EnhanceNet
Single Image Super-Resolution
Through Automated Texture Synthesis

✉ Mehdi S. M. Sajjadi

Bernhard Schölkopf

Michael Hirsch
Max-Planck Instite for Intelligent Systems
Spemanstr. 34, 72076 Tübingen, Germany

Oral presentation at International Conference on Computer Vision (ICCV) 2017
ArXiv 1612.07919

input image highest PSNR our result

Abstract

Single image super-resolution is the task of inferring a high-resolution image from a single low-resolution input. Traditionally, the performance of algorithms for this task is measured using pixel-wise reconstruction measures such as peak signal-to-noise ratio (PSNR) which have been shown to correlate poorly with the human perception of image quality. As a result, algorithms minimizing these metrics tend to produce over-smoothed images that lack high-frequency textures and do not look natural despite yielding high PSNR values.

We propose a novel application of automated texture synthesis in combination with a perceptual loss focusing on creating realistic textures rather than optimizing for a pixel-accurate reproduction of ground truth images during training. By using feed-forward fully convolutional neural networks in an adversarial training setting, we achieve a significant boost in image quality at high magnification ratios. Extensive experiments on a number of datasets show the effectiveness of our approach, yielding state-of-the-art results in both quantitative and qualitative benchmarks.

Downloads

• Paper & Supplementary (pdf) • Pre-trained model (zip) • Poster (pdf) • Slides (pdf) •
• Set5 (3 MB) • Set14 (15 MB) • BSD100 (76 MB) • SunHays80 (258 MB) • Urban100 (381 MB) •


Results

bicubic ENet-E ENet-PAT ground truth
bicubic ENet-E ENet-PAT ground truth


Comparison with other methods

bicubic Glasner Kim SCSR SelfEx SRCNN
PSyCo VDSR DRCN ENet-E ENet-PAT ground truth


Comparison of previous state of the art at 2x (75% pixels missing)
with our method at 4x super-resolution (94% pixels missing)

2x downsampled input 2x downsampled input 4x downsampled input
2x VDSR 2x DRCN 4x with ENet-PAT


References

[DRCN] J. Kim, J. Kwon Lee, and K. Mu Lee. Deeply-recursive convolutional network for image super-resolution. In CVPR, 2016.
[Glasner] D. Glasner, S. Bagon, and M. Irani. Super-resolution from a single image. In ICCV, 2009.
[Kim] K. I. Kim and Y. Kwon. Single-image super-resolution using sparse regression and natural image prior. IEEE TPAMI, 2010.
[PSyCo] E. Perez-Pellitero, J. Salvador, J. Ruiz-Hidalgo, and B. Rosenhahn. PSyCo: Manifold span reduction for super resolution. In CVPR, 2016.
[SCSR] J. Yang, J. Wright, T. Huang, and Y. Ma. Image super-resolution as sparse representation of raw image patches. In CVPR, 2008.
[SelfEx] J.-B. Huang, A. Singh, and N. Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, 2015.
[SRCNN] C. Dong, C. C. Loy, K. He, and X. Tang. Learning a deep convolutional network for image super-resolution. In ECCV, 2014.
[VDSR] J. Kim, J. Kwon Lee, and K. Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016.