Visual Semantic Planning using Deep Successor Representations

October 2017

Visual Semantic Planning using Deep Successor Representations

Authors:

Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi
Conference Paper
Proceedings of (ICCV) International Conference on Computer Vision

Abstract:

A crucial capability of real-world intelligent agents is their ability to plan a sequence of actions to achieve their goals in the visual world. In this work, we address the problem of visual semantic planning: the task of predicting a sequence of actions from visual observations that transform a dynamic environment from an initial state to a goal state. Doing so entails knowledge about objects and their affordances, as well as actions and their preconditions and effects. We propose learning these through interacting with a visual and dynamic environment. Our proposed solution involves bootstrapping reinforcement learning with imitation learning. To ensure cross task generalization, we develop a deep predictive model based on successor representations. Our experimental results show near optimal results across a wide range of tasks in the challenging THOR environment.
@conference{Zhu-2017-113307,
author = {Yuke Zhu And Daniel Gordon And Eric Kolve And Dieter Fox And Li Fei-Fei And Abhinav Gupta And Roozbeh Mottaghi And Ali Farhadi},
title = {Visual Semantic Planning using Deep Successor Representations},
booktitle = {Proceedings of (ICCV) International Conference on Computer Vision},
year = {2017},
month = {October},
pages = {483 - 492},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.