Modeling transition dynamics in MDPs with RKHS embeddings of conditional distributions

Details Modeling transition dynamics in MDPs with RKHS embeddings of conditional distributions Modeling transition dynamics in MDPs with RKHS embeddings of conditional distributions

2011-12-20

arxiv.org/abs/1112.4722v1

Computer Science

Learning

Scientific paper

We propose a new, nonparametric approach to estimating the value function in reinforcement learning. This approach makes use of a recently developed representation of conditional distributions as functions in a reproducing kernel Hilbert space. Such representations bypass the need for estimating transition probabilities, and apply to any domain on which kernels can be defined. Our approach avoids the need to approximate intractable integrals since expectations are represented as RKHS inner products whose computation has linear complexity in the sample size. Thus, we can efficiently perform value function estimation in a wide variety of settings, including finite state spaces, continuous states spaces, and partially observable tasks where only sensor measurements are available. A second advantage of the approach is that we learn the conditional distribution representation from a training sample, and do not require an exhaustive exploration of the state space. We prove convergence of our approach either to the optimal policy, or to the closest projection of the optimal policy in our model class, under reasonable assumptions. In experiments, we demonstrate the performance of our algorithm on a learning task in a continuous state space (the under-actuated pendulum), and on a navigation problem where only images from a sensor are observed. We compare with least-squares policy iteration where a Gaussian process is used for value function estimation. Our algorithm achieves better performance in both tasks.

Affiliated with

Also associated with

No associations

Rating

Modeling transition dynamics in MDPs with RKHS embeddings of conditional distributions does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Modeling transition dynamics in MDPs with RKHS embeddings of conditional distributions, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Modeling transition dynamics in MDPs with RKHS embeddings of conditional distributions will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-58367