A Survey of POMDP Applications

This paper was an introduction to a new stochastic method called POMDP which stands for “Partially observable Markov Decision Processes”. This paper presents the wide application domains of the method and its applications in various processes that we observe almost daily.

This model takes into account the uncertainty in any decision process and extends the already prevailing MDP (Markov Decision process).The POMDP model consists of the following:

a finite set of states -S
a finite set of actions -A
a finite set of observations- Z
a state transition function -τ
an observation function -o
an immediate reward function -r

The states represent all the possible underlying states in the process which may not be even directly visible. The state transition accounts for the uncertainty by providing probabilistic measure to for a certain transition. The action set is all the available control choices available at particular time, whereas the observation set is the set of all possible observations available at a particular state. The reward function gives immediate utility for performing an action in each of the underlining process states.

The main objective of the model is to derive the control policy that will utilize the minimum possible information to yield the optimum decision. This is important because in many cases, complete information about the process is either not available or is very expensive. Thus, this approach minimizes the cost associated with the decision and also the computational complexity of the decision. This objective has been demonstrated by the author by providing various examples of application of POMDP domains various domains like, structural inspection, elevate control policies, fishery industries, autonomous robots, network troubleshooting, behavioral ecology, machine vision, , distributed database queries, marketing, questionnaire design machine maintenance, weapon allocation, corporate policy, moving target search, search and rescue, target identification, education, medical diagnosis.

Since the model is very heavily dependent on some of the detailed partial process information, author has warned that the model may not be very useful if we do not have the partial information about the process. In other words, we require the model to provide us with complete information about every possible observation and immediate reward for each state, action and observation. This information may not be always available for every application domain. The other important problem highlighted by the author is the user interface issues which deals with how all information can be provided to a system. Apart from this, there are also issues like computational complexities.

Discussion:

This per provided a clear case motivation to the use of POMDPs in variety of domains some of which seemed interesting. I was impressed by the way he has highlighted all information available form a given process and framed them in form of requirements for the POMDP.As far as the implementation details are concerned, there was almost nothing in the paper which seemed bit disappointing, but keeping in mind many non engineer person dealing with uncertainty issues (like ecologist, fishery people), this is a nice over view. I believe, POMDP’s have good application in many search and limited information navigation systems as system is capable enough to decide based on the available information. Keeping in mind our goals, I believe the system could be useful if we have a limited posture sequence forming a distinct gesture and the library of such gestures is small. Otherwise, it is computationally very expensive and we may not be able to provide all information required to model a decision.