skip to main content
10.1145/3394171.3413585acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Light Field Super-resolution via Attention-Guided Fusion of Hybrid Lenses

Published: 12 October 2020 Publication History

Abstract

This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple low-resolution cameras. To tackle this challenge, we propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation; the other one constructs another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations via the learned attention maps, leading to the final high-resolution LF image. Extensive experiments demonstrate the significant superiority of our approach over state-of-the-art ones. That is, our method not only improves the PSNR by more than 2 dB, but also preserves the LF structure much better. To the best of our knowledge, this is the first end-to-end deep learning method for reconstructing a high-resolution LF image with a hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and also be beneficial to LF data storage and transmission. The code is available at https://github.com/jingjin25/LFhybridSR-Fusion.

Supplementary Material

MP4 File (3394171.3413585.mp4)
We explore the problem of reconstructing high-resolution light field images from hybrid lenses, including a high-resolution camera surrounded by multiple low-resolution cameras. To tackle this challenge, we propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation; the other one constructs another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations via the learned attention maps, leading to the final high-resolution LF image. To the best of our knowledge, this is the first end-to-end deep learning method for reconstructing a high-resolution LF image with a hybrid input.

References

[1]
[n.d.]. Stanford Lytro Light Field Archive. http://lightfields.stanford.edu/LF2016.html. [Online].
[2]
Vivek Boominathan, Kaushik Mitra, and Ashok Veeraraghavan. 2014. Improving resolution and depth-of-field of light field cameras using a hybrid imaging system. In IEEE International Conference on Computational Photography (ICCP). 1--10.
[3]
Hong Chang, Dit-Yan Yeung, and Yimin Xiong. 2004. Super-resolution through neighbor embedding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). I--I.
[4]
Jie Chen, Junhui Hou, and Lap-Pui Chau. 2018. Light field denoising via anisotropic parallax analysis in a cnn framework. IEEE Signal Processing Letters, Vol. 25, 9 (2018), 1403--1407.
[5]
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2016. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 38, 2 (2016), 295--307.
[6]
Reuben A Farrugia, Christian Galea, and Christine Guillemot. 2017. Super resolution of light field images using linear subspace projection of patch-volumes. IEEE Journal of Selected Topics in Signal Processing, Vol. 11, 7 (2017), 1058--1071.
[7]
Juliet Fiss, Brian Curless, and Richard Szeliski. 2014. Refocusing plenoptic images using depth-adaptive splatting. In IEEE International Conference on Computational Photography (ICCP). 1--9.
[8]
Mantang Guo, Junhui Hou, Jing Jin, Jie Chen, and Lap-Pui Chau. 2020. Deep Spatial-angular Regularization for Compressive Light Field Reconstruction over Coded Apertures. In European Conference on Computer Vision (ECCV).
[9]
Stefan Heber and Thomas Pock. 2014. Shape from light field meets robust PCA. In European Conference on Computer Vision (ECCV). 751--767.
[10]
Katrin Honauer, Ole Johannsen, Daniel Kondermann, and Bastian Goldluecke. 2016. A dataset and evaluation methodology for depth estimation on 4d light fields. In Asian Conference on Computer Vision (ACCV). 19--34.
[11]
Junhui Hou, Jie Chen, and Lap-Pui Chau. 2019. Light field image compression based on bi-level view compensation with rate-distortion optimization. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 29, 2 (2019), 517--530.
[12]
Fu-Chung Huang, Kevin Chen, and Gordon Wetzstein. 2015a. The light field stereoscope: immersive computer graphics via factored near-eye light field displays with focus cues. ACM Transactions on Graphics, Vol. 34, 4 (2015), 60.
[13]
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015b. Single image super-resolution from transformed self-exemplars. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5197--5206.
[14]
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. 2015. Spatial transformer networks. In Advances in Neural Information Processing Systems (NeurIPS). 2017--2025.
[15]
Jing Jin, Junhui Hou, Jie Chen, and Sam Kwong. 2020 a. Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2260--2269.
[16]
Jing Jin, Junhui Hou, Jie Chen, Wing Fung Henry Yeung, and Sam Kwong. 2018. Light Field Spatial Super-resolution via CNN Guided by A Single High-resolution RGB Image. In IEEE International Conference on Digital Signal Processing (DSP). 1--5.
[17]
Jing Jin, Junhui Hou, Hui Yuan, and Sam Kwong. 2020 b. Learning Light Field Angular Super-Resolution via a Geometry-Aware Network. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). 11141--11148.
[18]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision (ECCV). 694--711.
[19]
Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics, Vol. 35, 6 (2016), 193.
[20]
Changil Kim, Henning Zimmer, Yael Pritch, Alexander Sorkine-Hornung, and Markus H Gross. 2013. Scene reconstruction from high spatio-angular resolution light fields. ACM Transactions on Graphics, Vol. 32, 4 (2013), 73--1.
[21]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1646--1654.
[22]
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2017. Deep laplacian pyramid networks for fast and accurate super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 624--632.
[23]
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 105--114.
[24]
Nianyi Li, Jinwei Ye, Yu Ji, Haibin Ling, and Jingyi Yu. 2014. Saliency detection on light field. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2806--2813.
[25]
Chien-Hung Lu, Stefan Muenzel, and Jason W. Fleischer. 2013. High-Resolution Light-Field Microscopy. In Imaging and Applied Optics.
[26]
Michael Mathieu, Camille Couprie, and Yann LeCun. 2016. Deep multi-scale video prediction beyond mean square error. International Conference on Learning Representations (ICLR) (2016), 1--11.
[27]
Kaushik Mitra and Ashok Veeraraghavan. 2012. Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior. In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 22--28.
[28]
Mattia Rossi and Pascal Frossard. 2018. Geometry-Consistent Light Field Super-Resolution via Graph-Based Regularization. IEEE Transactions on Image Processing, Vol. 27, 9 (2018), 4207--4218.
[29]
Pratul P Srinivasan, Ren Ng, and Ravi Ramamoorthi. 2017. Light field blind motion deblurring. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2354--2362.
[30]
Jing Tian and Kai-Kuang Ma. 2011. A survey on super-resolution imaging. Signal, Image and Video Processing, Vol. 5, 3 (2011), 329--342.
[31]
Radu Timofte, Vincent De Smet, and Luc Van Gool. 2014. A+: Adjusted anchored neighborhood regression for fast super-resolution. In IEEE Asian Conference on Computer Vision (ACCV). 111--126.
[32]
Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A Efros, and Ravi Ramamoorthi. 2016b. A 4D light-field dataset and CNN architectures for material recognition. In European Conference on Computer Vision (ECCV). 121--138.
[33]
Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A Efros, and Ravi Ramamoorthi. 2017b. Light field video capture using a learning-based hybrid imaging system. ACM Transactions on Graphics, Vol. 36, 4 (2017), 133.
[34]
Xiang Wang, Lin Li, and GuangQi Hou. 2016a. High-resolution light field reconstruction using a hybrid imaging system. Applied optics, Vol. 55, 10 (2016), 2580--2593.
[35]
Yunlong Wang, Fei Liu, Kunbo Zhang, Guangqi Hou, Zhenan Sun, and Tieniu Tan. 2018. LFNet: A Novel Bidirectional Recurrent Convolutional Neural Network for Light-Field Image Super-Resolution. IEEE Transactions on Image Processing, Vol. 27, 9 (2018), 4274--4286.
[36]
Yuwang Wang, Yebin Liu, Wolfgang Heidrich, and Qionghai Dai. 2017a. The light field attachment: Turning a dslr into a light field camera using a low budget camera ring. IEEE Transactions on Visualization and Computer Graphics, Vol. 23, 10 (2017), 2357--2364.
[37]
Zhihao Wang, Jian Chen, and Steven CH Hoi. 2019. Deep Learning for Image Super-resolution: A Survey. arXiv preprint arXiv:1902.06068 (2019).
[38]
Sven Wanner and Bastian Goldluecke. 2014. Variational light field analysis for disparity estimation and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 36, 3 (2014), 606--619.
[39]
Wing Fung Henry Yeung, Junhui Hou, Jie Chen, Yuk Ying Chung, and Xiaoming Chen. 2018b. Fast Light Field Reconstruction With Deep Coarse-To-Fine Modeling of Spatial-Angular Clues. In European Conference on Computer Vision (ECCV). 137--152.
[40]
Wing Fung Henry Yeung, Junhui Hou, Xiaoming Chen, Jie Chen, Zhibo Chen, and Yuk Ying Chung. 2018a. Light Field Spatial Super-Resolution Using Deep Efficient Spatial-Angular Separable Convolution. IEEE Transactions on Image Processing, Vol. 28, 5 (2018), 2319--2330.
[41]
Youngjin Yoon, Hae-Gon Jeon, Donggeun Yoo, Joon-Young Lee, and In So Kweon. 2015. Learning a deep convolutional network for light-field image super-resolution. In IEEE International Conference on Computer Vision Workshops (ICCVW). 24--32.
[42]
Jingyi Yu. 2017. A light-field journey to virtual reality. IEEE MultiMedia, Vol. 24, 2 (2017), 104--112.
[43]
Yan Yuan, Ziqi Cao, and Lijuan Su. 2018. Light-Field Image Superresolution Using a Combined Deep CNN Based on EPI. IEEE Signal Processing Letters, Vol. 25, 9 (2018), 1359--1363.
[44]
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual dense network for image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2472--2481.
[45]
Mandan Zhao, Gaochang Wu, Yipeng Li, Xiangyang Hao, Fang Lu, and Yebin Liu. 2018. Cross-Scale Reference-Based Light Field Super-Resolution. IEEE Transactions on Computational Imaging, Vol. 4, 3 (2018), 406--418.
[46]
Haitian Zheng, Mengqi Ji, Haoqian Wang, Yebin Liu, and Lu Fang. 2018. CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping. In European Conference on Computer Vision (ECCV). 87--104.

Cited By

View all
  • (2024)基于深度学习的光场图像重建与增强综述(特邀)Laser & Optoelectronics Progress10.3788/LOP24140461:16(1611015)Online publication date: 2024
  • (2024)Learning-based light field imaging: an overviewEURASIP Journal on Image and Video Processing10.1186/s13640-024-00628-12024:1Online publication date: 30-May-2024
  • (2024)Neural light field reconstruction from hybrid camerasOptoelectronic Imaging and Multimedia Technology XI10.1117/12.3036229(4)Online publication date: 23-Nov-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

MM '20: Proceedings of the 28th ACM International Conference on Multimedia
October 2020
4889 pages
ISBN:9781450379885
DOI:10.1145/3394171
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. attention
  2. deep learning
  3. hybrid imaging system
  4. light field
  5. super-resolution

Qualifiers

  • Research-article

Funding Sources

Conference

MM '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)39
  • Downloads (Last 6 weeks)4
Reflects downloads up to 22 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)基于深度学习的光场图像重建与增强综述(特邀)Laser & Optoelectronics Progress10.3788/LOP24140461:16(1611015)Online publication date: 2024
  • (2024)Learning-based light field imaging: an overviewEURASIP Journal on Image and Video Processing10.1186/s13640-024-00628-12024:1Online publication date: 30-May-2024
  • (2024)Neural light field reconstruction from hybrid camerasOptoelectronic Imaging and Multimedia Technology XI10.1117/12.3036229(4)Online publication date: 23-Nov-2024
  • (2024)Hybrid Domain Learning towards Light Field Spatial Super-Resolution using Heterogeneous ImagingICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10446031(2400-2404)Online publication date: 14-Apr-2024
  • (2024)A survey for light field super-resolutionHigh-Confidence Computing10.1016/j.hcc.2024.100206(100206)Online publication date: Jan-2024
  • (2024)Light field image super-resolution using a Content-Aware Spatial–Angular Interaction networkDisplays10.1016/j.displa.2024.10278284(102782)Online publication date: Sep-2024
  • (2023)LFACon: Introducing Anglewise Attention to No-Reference Quality Assessment in Light Field SpaceIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324706929:5(2239-2248)Online publication date: 1-May-2023
  • (2023)Deep Light Field Spatial Super-Resolution Using Heterogeneous ImagingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.318404729:10(4183-4197)Online publication date: 1-Oct-2023
  • (2023)Light Field Reconstruction Via Deep Adaptive Fusion of Hybrid LensesIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.3287603(1-18)Online publication date: 2023
  • (2023)Joint Upsampling for Refocusing Light Fields Derived With Hybrid LensesIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2023.325388072(1-12)Online publication date: 2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy