Hierarchical Reinforcement Learning using Spatio-Temporal Abstractions and Deep Neural Networks

Ramnandan Krishnamurthy, Aravind S. Lakshminarayanan, Peeyush Kumar, Balaraman Ravindran

Published 2016-05-17Version 1

This paper introduces an automated skill acquisition framework in reinforcement learning which involves identifying a hierarchical description of the given task in terms of abstract states and extended actions between abstract states. Identifying such structures present in the task provides ways to simplify and speed up reinforcement learning learning algorithms. These structures also help to generalize such algorithms over multiple tasks without relearning policies from scratch. We use ideas from dynamical systems to find metastable regions in the state space and associate them with abstract states. The spectral clustering algorithm PCCA+ is used to identify suitable abstractions aligned to the underlying structure. Skills are defined in terms of the transitions between such abstract states. The connectivity information from PCCA+ is used to generate these skills or options. The skills are independent of the learning task and can be efficiently reused across a variety of tasks defined over a common state space. Another major advantage of the approach is that it does not need a prior model of the MDP and can work well even when the MDPs are constructed from sampled trajectories. Finally, we present our attempts to extend the automated skills acquisition framework to complex tasks such as learning to play video games where we use deep learning techniques for representation learning to aid our spatio-temporal abstraction framework.