Generative Visual Manipulation on the Natural Image Manifold

October 2016

Generative Visual Manipulation on the Natural Image Manifold

Authors:

Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros
Conference Paper
Proceedings of (ECCV) European Conference on Computer Vision

Abstract:

Realistic image manipulation is challenging because it requires modifying the image appearance in a user-controlled way, while preserving the realism of the result. Unless the user has considerable artistic skill, it is easy to “fall off” the manifold of natural images while editing. In this paper, we propose to learn the natural image manifold directly from data using a generative adversarial neural network. We then define a class of image editing operations, and constrain their output to lie on that learned manifold at all times. The model automatically adjusts the output keeping all edits as realistic as possible. All our manipulations are expressed in terms of constrained optimization and are applied in near-real time. We evaluate our algorithm on the task of realistic photo manipulation of shape and color. The presented method can further be used for changing one image to look like the other, as well as generating novel imagery from scratch based on user’s scribbles.
@conference{Zhu-2016-125696,
author = {Jun-Yan Zhu And Philipp Krähenbühl And Eli Shechtman And Alexei A. Efros},
title = {Generative Visual Manipulation on the Natural Image Manifold},
booktitle = {Proceedings of (ECCV) European Conference on Computer Vision},
year = {2016},
month = {October},
pages = {597 - 613},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.