On the Use of Non-Stationary Policies for Infinite-Horizon Discounted Markov Decision Processes

Details On the Use of Non-Stationary Policies for Infinite-Horizon Discounted Markov Decision Processes On the Use of Non-Stationary Policies for Infinite-Horizon Discounted Markov Decision Processes

2012-03-25

arxiv.org/abs/1203.5532v2

Computer Science

Artificial Intelligence

Scientific paper

We consider infinite-horizon $\gamma$-discounted Markov Decision Processes, for which it is known that there exists a stationary optimal policy. We consider the algorithm Value Iteration and the sequence of policies $\pi_1,...,\pi_k$ it implicitely generates until some iteration $k$. We provide performance bounds for non-stationary policies involving the last $m$ generated policies that reduce the state-of-the-art bound for the last stationary policy $\pi_k$ by a factor $\frac{1-\gamma}{1-\gamma^m}$. In particular, the use of non-stationary policies allows to reduce the usual asymptotic performance bounds of Value Iteration with errors bounded by $\epsilon$ at each iteration from $\frac{\gamma}{(1-\gamma)^2}\epsilon$ to $\frac{\gamma}{1-\gamma}\epsilon$, which is significant in the usual situation when $\gamma$ is close to 1. Given Bellman operators that can only be computed with some error $\epsilon$, a surprising consequence of this result is that the problem of "computing an approximately optimal non-stationary policy" is much simpler than that of "computing an approximately optimal stationary policy", and even slightly simpler than that of "approximately computing the value of some fixed policy", since this last problem only has a guarantee of $\frac{1}{1-\gamma}\epsilon$.

Affiliated with

Also associated with

No associations

Rating

On the Use of Non-Stationary Policies for Infinite-Horizon Discounted Markov Decision Processes does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with On the Use of Non-Stationary Policies for Infinite-Horizon Discounted Markov Decision Processes, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and On the Use of Non-Stationary Policies for Infinite-Horizon Discounted Markov Decision Processes will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-75278