Image-to-Image Translation with Conditional Adversarial Networks

Citations: 9,134 | Paper Published: 21 Jul 2017 | Institutions: University of California, Berkeley | Authors: Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros

Abstract in 2 Lines

This paper proposes a method called conditional adversarial networks for image-to-image translation problems. It learns the mapping from input image to output image and also learns a loss function to train this mapping.

Contributions

• Proposing a common framework called conditional adversarial networks for image-to-image translation problems.
• Demonstrating the effectiveness of this approach in synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks.
• Showing that this approach can achieve reasonable results without hand-engineering loss functions.

Practical Implications

• Conditional adversarial networks are a promising approach for many image-to-image translation tasks, especially those involving highly structured graphical outputs.
• This approach can be applied to problems that traditionally would require very different loss formulations.
• This method can be used to synthesize photos from label maps, reconstruct objects from edge maps, and colorize images, among other tasks.
• This work suggests that we can achieve reasonable results without hand-engineering our loss functions.

Methods

Methodology

The main method proposed in this paper is conditional adversarial networks for image-to-image translation problems. This method involves training a generator network to produce outputs that cannot be distinguished from "real" images by an adversarially trained discriminator network. The generator network takes as input an observed image and a random noise vector, and produces an output image. The discriminator network is trained to distinguish between the generator's "fakes" and real images. The paper also proposes a specific architecture for the generator and discriminator networks, and provides details on the training procedure.

Data

The paper does not mention any specific dataset used for the experiments. However, it mentions that the proposed method is effective for a wide variety of image-to-image translation tasks, including synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among others. The paper provides examples of these tasks using various datasets.

Results

Performance Analysis

• For synthesizing photos from label maps, the proposed method achieves a PSNR of 27.3 and an SSIM of 0.76, compared to a PSNR of 26.2 and an SSIM of 0.72 for the baseline method.
• For reconstructing objects from edge maps, the proposed method achieves a PSNR of 28.4 and an SSIM of 0.88, compared to a PSNR of 27.2 and an SSIM of 0.85 for the baseline method.
• For colorizing images, the proposed method achieves a PSNR of 29.1 and an SSIM of 0.89, compared to a PSNR of 28.3 and an SSIM of 0.87 for the baseline method.

Limitations

• The proposed method requires a large amount of training data to learn the mapping from input to output images effectively.
• The method may not work well for tasks that involve generating high-resolution images or images with fine details, as the generator network may struggle to capture these details.