Fast reinforcement learning with generalized policy updates (Paper Explained)

#ai #research #reinforcementlearning Reinforcement Learning is a powerful tool, but it is also incredibly data-hungry. Given a new task, an RL agent has to learn a good policy entirely from scratch. This paper proposes a new framework that allows an agent to carry over knowledge from previous tasks into solving new tasks, even deriving zero-shot policies that perform well on completely new reward functions. OUTLINE: 0:00 - Intro & Overview 1:25 - Problem Statement 6:25 - Q-Learning Primer 11:40 - Multiple Rewards, Multiple Policies 14:25 - Example Environment 17:35 - Tasks as Linear Mixtures of Features 24:15 - Successor Features 28:00 - Zero-Shot Policy for New Tasks 35:30 - Results on New Task W3 37:00 - Inferring the Task via Regression 39:20 - The Influence of the Given Policies 48:40 - Learning the Feature Functions 50:30 - More Complicated Tasks 51:40 - Life-Long Learning, Comments & Conclusion Paper: https://www.pnas.org/content/early/2020/08/13/1907370117 My Video on Successor Features: https://youtu.be/KXEEqcwXn8w Abstract: The combination of reinforcement learning with deep learning is a promising approach to tackle important sequential decision-making problems that are currently intractable. One obstacle to overcome is the amount of data needed by learning systems of this type. In this article, we propose to address this issue through a divide-and-conquer approach. We argue that complex decision problems can be naturally decomposed into multiple tasks that unfold in sequence or in parallel. By associating each task with a reward function, this problem decomposition can be seamlessly accommodated within the standard reinforcement-learning formalism. The specific way we do so is through a generalization of two fundamental operations in reinforcement learning: policy improvement and policy evaluation. The generalized version of these operations allow one to leverage the solution of some tasks to speed up the solution of others. If the reward function of