Abstract
Developing and integrating advanced image sensors with novel algorithms in camera systems is prevalent with the increasing demand for computational photography and imaging on mobile platforms. However, the lack of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photography and imaging (MIPI). To bridge the gap, we introduce the first MIPI challenge including five tracks focusing on novel image sensors and imaging algorithms. In this paper, RGBW Joint Fusion and Denoise, one of the five tracks, working on the fusion of binning-mode RGBW to Bayer at half resolution is introduced. The participants were provided with a new dataset including 70 (training) and 15 (validation) scenes of high-quality RGBW and Bayer pair. In addition, for each scene, RGBW of 24 dB and 42 dB are provided. All the data were captured using a RGBW sensor in both outdoor and indoor conditions. The final results are evaluated using objective metrics including PSNR, SSIM [11], LPIPS [15] and KLD. A detailed description of all models developed in this challenge is provided in this paper. More details of this challenge and the link to the dataset can be found in https://github.com/mipi-challenge/MIPI2022
Q. Yang, J. Jiang, C. Li, S. Zhou, R. Feng, W. Sun, Q. Zhu, C. C. Loy, J. Gu, are the MIPI 2022 challenge organizers. The other authors participated in the challenge. Please refer to Appendix A for details.
MIPI 2022 challenge website: http://mipi-challenge.org/.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Camon 19 pro. https://www.tecno-mobile.com/phones/product-detail/product/camon-19-pro-5g
Oppo unveils multiple innovative imaging technologies. https://www.oppo.com/en/newsroom/press/oppo-future-imaging-technology-launch/
vivo x80 is the only vivo smartphone with a sony imx866 sensor: the world’s first RGBW bottom sensors. https://www.vivoglobal.ph/vivo-X80-is-the-only-vivo-smartphone-with-a-Sony-IMX866-Sensor-The-Worlds-First-RGBW-Bottom-Sensors/
Chen, L., Chu, X., Zhang, X., Sun, J.: Simple baselines for image restoration. arXiv preprint arXiv:2204.04676 (2022)
Liu, J., et al.: Learning raw image denoising with bayer pattern unification and bayer preserving augmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2019)
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)
Sun, B., Zhang, Y., Jiang, S., Fu, Y.: Hybrid pixel-unshuffled network for lightweight image super-resolution. arXiv preprint arXiv:2203.08921 (2022)
Wang, X., et al.: Esrgan: enhanced super-resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2018)
Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., Li, H.: Uformer: a general u-shaped transformer for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17683–17693 (2022)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Xu, S., Zhang, J., Zhao, Z., Sun, K., Liu, J., Zhang, C.: Deep gradient projection networks for pan-sharpening. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1366–1375 (2021)
Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023–6032 (2019)
Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5728–5739 (2022)
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Acknowledgements
We thank Shanghai Artificial Intelligence Laboratory, Sony, and Nanyang Technological University to sponsor this MIPI 2022 challenge. We thank all the organizers and participants for their great work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
A Teams and Affiliations
A Teams and Affiliations
BITSpectral
Title: Fusion Cross-Patch Attention Network for RGBW Joint Fusion and Denoise
Members: Zhen Wang (wzhstruggle@163.com), Daoyu Li, Yuzhe Zhang, Lintao Peng, Xuyang Chang, Yinuo Zhang, Liheng Bian
Affiliations: Beijing Institute of Technology
BIVLab
Title: Self-Guided Spatial-Frequency Complement Network for RGBW Joint Fusion and Denoise
Members: Bing Li (frigid@mail.ustc.edu.cn), Jie Huang, Mingde Yao, Ruikang Xu, Feng Zhao
Affiliations: University of Science and Technology of China
HIT-IIL
Title: NAFNet for RGBW Image Fusion
Members: Xiaohui Liu (xh720199@gmail.com), Xiaohui Liu, Rongjian Xu, Zhilu Zhang, Xiaohe Wu, Ruohao Wang, Junyi Li, Wangmeng Zuo
Affiliations: Harbin Institute of Technology
jzsherlock
Title: Dual Branch Network for Bayer Image Denoising Using White Pixel Guidance
Members: Zhuang Jia (jiazhuang@xiaomi.com)
Affiliations: Xiaomi
LLCKP
Title: Synthetic RGBW image and noise
Members: DongJae Lee (jhtwosun@kaist.ac.kr)
Affiliations: KAIST
MegNR
Title: HAUformer: Hybrid Attention-guided U-shaped Transformer for RGBW Fusion Image Restoration
Members: Ting Jiang (jiangting@megvii.com), Qi Wu, Chengzhi Jiang, Mingyan Han, Xinpeng Li, Wenjie Lin, Youwei Li, Haoqiang Fan, Shuaicheng Liu
Affiliations: Megvii Technology
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, Q. et al. (2023). MIPI 2022 Challenge on RGBW Sensor Fusion: Dataset and Report. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13805. Springer, Cham. https://doi.org/10.1007/978-3-031-25072-9_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-25072-9_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25071-2
Online ISBN: 978-3-031-25072-9
eBook Packages: Computer ScienceComputer Science (R0)