Toward Multimodal Image-to-Image Translation

December 2017

Toward Multimodal Image-to-Image Translation

Authors:

Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli Shechtman
Conference Paper
Proceedings of (NeurIPS) Neural Information Processing Systems

Abstract:

Many image-to-image translation problems are ambiguous, as a single input image may correspond to multiple possible outputs. In this work, we aim to model a distribution of possible outputs in a conditional generative modeling setting. The ambiguity of the mapping is distilled in a low-dimensional latent vector, which can be randomly sampled at test time. A generator learns to map the given input, combined with this latent code, to the output. We explicitly encourage the connection between output and the latent code to be invertible. This helps prevent a many-to-one mapping from the latent code to the output during training, also known as the problem of mode collapse, and produces more diverse results. We explore several variants of this approach by employing different training objectives, network architectures, and methods of injecting the latent code. Our proposed method encourages bijective consistency between the latent encoding and output modes. We present a systematic comparison of our method and other variants on both perceptual realism and diversity.
@conference{Zhu-2017-125690,
author = {Jun-Yan Zhu And Richard Zhang And Deepak Pathak And Trevor Darrell And Alexei A. Efros And Oliver Wang And Eli Shechtman},
title = {Toward Multimodal Image-to-Image Translation},
booktitle = {Proceedings of (NeurIPS) Neural Information Processing Systems},
year = {2017},
month = {December},
pages = {465 - 476},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.