An Approximate Newton Method for Markov Decision Processes

Details An Approximate Newton Method for Markov Decision Processes An Approximate Newton Method for Markov Decision Processes

2012-04-05

arxiv.org/abs/1204.1227v2

Mathematics

Optimization and Control

Scientific paper

Gradient-based algorithms are one of the methods of choice for the optimisation of Markov Decision Processes. In this article we will present a novel approximate Newton algorithm for the optimisation of such models. The algorithm has various desirable properties over the naive application of Newton's method. Firstly the approximate Hessian is guaranteed to be negative-semidefinite over the entire parameter space in the case where the controller is log-concave in the control parameters. Additionally the inference required for our approximate Newton method is often the same as that required for first order methods, such as steepest gradient ascent. The approximate Hessian also has many nice sparsity properties that are not present in the Hessian and that make its inversion efficient in many situations of interest. We also provide an analysis that highlights a relationship between our approximate Newton method and both Expectation Maximisation and natural gradient ascent. Empirical results suggest that the algorithm has excellent convergence and robustness properties.

Affiliated with

Also associated with

No associations

Rating

An Approximate Newton Method for Markov Decision Processes does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with An Approximate Newton Method for Markov Decision Processes, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and An Approximate Newton Method for Markov Decision Processes will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-212866