Multi-timescale Nexting in a Reinforcement Learning Robot

Details Multi-timescale Nexting in a Reinforcement Learning Robot Multi-timescale Nexting in a Reinforcement Learning Robot

2011-12-06

arxiv.org/abs/1112.1133v2

Computer Science

Artificial Intelligence

(13 pages)

Scientific paper

The term "nexting" has been used by psychologists to refer to the propensity of people and many other animals to continually predict what will happen next in an immediate, local, and personal sense. The ability to "next" constitutes a basic kind of awareness and knowledge of one's environment. In this paper we present results with a robot that learns to next in real time, predicting thousands of features of the world's state, including all sensory inputs, at timescales from 0.1 to 8 seconds. This was achieved by treating each state feature as a reward and applying temporal-difference methods to learn a corresponding value function with a discount rate corresponding to the timescale. That is, instead of predicting a single distinguished reward on a long timescale, as in conventional reinforcement learning, we predicted many state features at multiple short timescales. Although this approach is conceptually straightforward, there are many computational and performance challenges in implementing it in real time on a physical robot. We show that two thousand predictions, each dependent on six thousand state features, can be learned and updated online at better than 10Hz on a laptop computer, using the standard TD(\lambda) algorithm with linear function approximation. We show that this approach is efficient enough to be practical, with much of the learning complete within 30 minutes. We also show that a single tile-coded feature representation suffices to accurately predict many different signals at a significant range of timescales. Finally, we show that the accuracy of our learned predictions compares favorably with the optimal off-line solution, and with conventional auto-regressive prediction methods.

Affiliated with

Also associated with

No associations

Rating

Multi-timescale Nexting in a Reinforcement Learning Robot does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Multi-timescale Nexting in a Reinforcement Learning Robot, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Multi-timescale Nexting in a Reinforcement Learning Robot will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-380096