3D-Aware Scene Manipulation via Inverse Graphics

December 2018

3D-Aware Scene Manipulation via Inverse Graphics

Authors:

Shunyu Yao, Tzu Ming Hsu, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, Bill Freeman, and Josh Tenenbaum
Conference Paper
Proceedings of (NeurIPS) Neural Information Processing Systems

Abstract:

We aim to obtain an interpretable, expressive, and disentangled scene representation that contains comprehensive structural and textural information for each object. Previous scene representations learned by neural networks are often uninterpretable, limited to a single object, or lacking 3D knowledge. In this work, we propose 3D scene de-rendering networks (3D-SDN) to address the above issues by integrating disentangled representations for semantics, geometry, and appearance into a deep generative model. Our scene encoder performs inverse graphics, translating a scene into a structured object-wise representation. Our decoder has two components: a differentiable shape renderer and a neural texture generator. The disentanglement of semantics, geometry, and appearance supports 3D-aware scene manipulation, e.g., rotating and moving objects freely while keeping the consistent shape and texture, and changing the object appearance without affecting its shape. Experiments demonstrate that our editing scheme based on 3D-SDN is superior to its 2D counterpart.
@conference{Tao-2018-125684,
author = {Shunyu Yao And Tzu Ming Hsu And Jun-Yan Zhu And Jiajun Wu And Antonio Torralba And Bill Freeman And Josh Tenenbaum},
title = {3D-Aware Scene Manipulation via Inverse Graphics},
booktitle = {Proceedings of (NeurIPS) Neural Information Processing Systems},
year = {2018},
month = {December},
pages = {1891 - 1902},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.