Visual Object Networks: Image Generation with Disentangled 3D Representation

December 2018

Visual Object Networks: Image Generation with Disentangled 3D Representation

Authors:

Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenenbaum, and Bill Freeman
Conference Paper
Proceedings of (NeurIPS) Neural Information Processing Systems

Abstract:

Recent progress in deep generative models has led to tremendous breakthroughs in image generation. While being able to synthesize photorealistic images, existing models lack an understanding of our underlying 3D world. Different from previous works built on 2D datasets and models, we present a new generative model, Visual Object Networks (VON), synthesizing natural images of objects with a disentangled 3D representation. Inspired by classic graphics rendering pipelines, we unravel the image formation process into three conditionally independent factors—shape, viewpoint, and texture—and present an end-to-end adversarial learning framework that jointly models 3D shape and 2D texture. Our model first learns to synthesize 3D shapes that are indistinguishable from real shapes. It then renders the object's 2.5D sketches (i.e. silhouette and depth map) from its shape under a sampled viewpoint. Finally, it learns to add realistic textures to these 2.5D sketches to generate realistic images. The VON not only generates images that are more realistic than the state-of-the-art 2D image synthesis methods but also enables many 3D operations such as changing the viewpoint of a generated image, shape and texture editing, linear interpolation in texture and shape space, and transferring appearance across different objects and viewpoints.
@conference{Zhu-2018-125683,
author = {Jun-Yan Zhu And Zhoutong Zhang And Chengkai Zhang And Jiajun Wu And Antonio Torralba And Josh Tenenbaum And Bill Freeman},
title = {Visual Object Networks: Image Generation with Disentangled 3D Representation},
booktitle = {Proceedings of (NeurIPS) Neural Information Processing Systems},
year = {2018},
month = {December},
pages = {118 - 129},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.