f-IRL: Inverse Reinforcement Learning via State Marginal Matching

November 2020

f-IRL: Inverse Reinforcement Learning via State Marginal Matching

Authors:

Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Benjamin Eysenbach
Conference Paper
Proceedings of (CoRL) Conference on Robot Learning

Abstract:

Imitation learning is well-suited for robotic tasks where it is difficult to directly program the behavior or specify a cost for optimal control. In this work, we propose a method for learning the reward function (and the corresponding policy) to match the expert state density. Our main result is the analytic gradient of any f-divergence between the agent and expert state distribution w.r.t. reward parameters. Based on the derived gradient, we present an algorithm, f-IRL, that recovers a stationary reward function from the expert density by gradient descent. We show that f-IRL can learn behaviors from a hand-designed target state density or implicitly through expert observations. Our method outperforms adversarial imitation learning methods in terms of sample efficiency and the required number of expert trajectories on IRL benchmarks. Moreover, we show that the recovered reward function can be used to quickly solve downstream tasks, and empirically demonstrate its utility on hard-to-explore tasks and for behavior transfer across changes in dynamics.
@conference{Ni-2020-125594,
author = {Tianwei Ni And Harshit Sikchi And Yufei Wang And Tejus Gupta And Lisa Lee And Benjamin Eysenbach},
title = {f-IRL: Inverse Reinforcement Learning via State Marginal Matching},
booktitle = {Proceedings of (CoRL) Conference on Robot Learning},
year = {2020},
month = {November},
keywords = {Imitation Learning, Inverse Reinforcement Learning, Machine Learning, Optimization},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.