Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning

July 2018

Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning

Authors:

Xinlei Pan, Eshed Ohn-Bar, Nicholas Rhinehart, Yan Xu, Yilin Shen, and Kris M. Kitani

Abstract:

Humans are able to understand and perform complex tasks by strategically structuring tasks into incremental steps or sub-goals. For a robot attempting to learn to perform a sequential task with critical subgoal states, these subgoal states can provide a natural opportunity for interaction with a human expert. This paper analyzes the benefit of incorporating a notion of subgoals into Inverse Reinforcement Learning (IRL) with a Human-In-The-Loop (HITL) framework. The learning process is interactive, with a human expert first providing input in the form of full demonstrations along with some subgoal states. These subgoal states defines a set of sub-tasks for the learning agent to complete in order to achieve the final goal. The learning agent queries for partial demonstrations corresponding to each sub-task as needed when the learning agent struggles with individual sub-task. The proposed Human Interactive IRL (HI-IRL) framework is evaluated on several discrete path-planning tasks. We demonstrate that subgoal-based interactive structuring of the learning task results in significantly more efficient learning, requiring only a fraction of the demonstration data needed for learning the underlying reward function with a baseline IRL model.
@conference{Pan-2018-109839,
author = {Xinlei Pan And Eshed Ohn-Bar And Nicholas Rhinehart And Yan Xu And Yilin Shen And Kris M. Kitani},
title = {Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning},
booktitle = {Proceedings of 17th International Conference on Autonomous Agents and MultiAgent Systems, (AAMAS '18)},
year = {2018},
month = {July},
pages = {1380 - 1387},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.