Gradient Based Policy Optimization of Constrained Markov Decision Processes

Details Gradient Based Policy Optimization of Constrained Markov Decision Processes Gradient Based Policy Optimization of Constrained Markov Decision Processes

2011-10-22

arxiv.org/abs/1110.4946v1

Mathematics

Optimization and Control

Scientific paper

We present stochastic approximation algorithms for computing the locally optimal policy of a constrained, average cost, finite state Markov Decision Process. The stochastic approximation algorithms require estimation of the gradient of the cost function with respect to the parameter that characterizes the randomized policy. We propose a spherical coordinate parameterization and implement a simulation based gradient estimation scheme using using potential realization factors. In this paper we analyse boundedness of moments of the estimators and provide a new analysis for the computational requirements of the estimators. We present a stochastic version of a primal dual (augmented) Lagrange multiplier method for the constrained algorithm. We show that the "large sample size" limit of the procedure is unbiased, but for small sample size (which is relevant for on-line real time operation) it is biased. We give an explicit expression of the asymptotic bias and discuss several possibilities for bias reduction. Numerical examples are given to illustrate the performance of the algorithms. In particular we present a case study in transmission control in wireless communications where the optimal policy has a simplified threshold structure and apply our stochastic approximation method to this example.

Affiliated with

Also associated with

No associations

Rating

Gradient Based Policy Optimization of Constrained Markov Decision Processes does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Gradient Based Policy Optimization of Constrained Markov Decision Processes, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Gradient Based Policy Optimization of Constrained Markov Decision Processes will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-564326