Skip to main content
Springer Nature Link
Log in

Dual Convolutional Neural Networks for Low-Level Vision

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

We propose a general dual convolutional neural network (DualCNN) for low-level vision problems, e.g., super-resolution, edge-preserving filtering, deraining, and dehazing. These problems usually involve estimating two components of the target signals: structures and details. Motivated by this, we design the proposed DualCNN to have two parallel branches, which respectively recovers the structures and details in an end-to-end manner. The recovered structures and details can generate desired signals according to the formation model for each particular application. The DualCNN is a flexible framework for low-level vision tasks and can be easily incorporated into existing CNNs. Experimental results show that the DualCNN can be effectively applied to numerous low-level vision tasks with favorable performance against the state-of-the-art methods that have been specially designed for each individual task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. As an extension of the details and structures learning, we do not assume that \(\phi (\cdot )\) is independent of \(\varphi (\cdot )\).

References

  • Berman, D., Treibitz, T., & Avidan, S. (2016). Non-local image dehazing. In CVPR (pp. 1674–1682).

  • Bulat, A., Yang, J., & Tzimiropoulos, G. (2018). To learn image super-resolution, use a GAN to learn how to do image degradation first. In ECCV (pp. 187–202).

  • Burger, H. C., Schuler, C. J., & Harmeling, S. (2012). Image denoising: Can plain neural networks compete with bm3d? In CVPR (pp. 2392–2399).

  • Burger, H., Schuler, C., & Harmeling, S. (2012). Image denosing: Can plain neural networks compete with BM3D. In CVPR.

  • Cai, B., Xu, X., Jia, K., Qing, C., & Tao, D. (2016). Dehazenet: An end-to-end system for single image haze removal. IEEE TIP, 25(11), 5187–5198.

    MathSciNet  MATH  Google Scholar 

  • Chen, D., & Davies, M. E. (2020). Deep decomposition learning for inverse imaging problems. In ECCV (pp. 510–526).

  • Chen, Y. L., & Hsu, C. T. (2013). A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In ICCV (pp. 1968–1975).

  • Chen, Q., Xu, J., & Koltun, V. (2017). Fast image processing with fully-convolutional networks. In ICCV (pp. 2516–2525).

  • Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. O. (2007). Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE TIP, 16(8), 2080–2095.

    MathSciNet  Google Scholar 

  • Dong, C., Deng, Y., Loy, C. C., & Tang, X. (2015). Compression artifacts reduction by a deep convolutional network. In ICCV (pp. 576–584).

  • Dong, C., Loy, C. C., & Tang, X. (2016). Accelerating the super-resolution convolutional neural network. In ECCV (pp. 391–407).

  • Dong, C., Loy, C. C., He, K., & Tang, X. (2014). Learning a deep convolutional network for image super-resolution. In ECCV (pp. 184–199).

  • Dong, C., Loy, C. C., He, K., & Tang, X. (2016). Image super-resolution using deep convolutional networks. IEEE TPAMI, 38(2), 295–307.

    Article  Google Scholar 

  • Dong, J., Roth, S., & Schiele, B. (2021). DWDN: Deep wiener deconvolution network for non-blind image deblurring. IEEE TPAMI, 52, 1. https://doi.org/10.1109/TPAMI.2021.3138787.

    Article  Google Scholar 

  • Eigen, D., Krishnan, D., & Fergus, R. (2013). Restoring an image taken through a window covered with dirt or rain. In ICCV (pp. 633–640).

  • Fan, Q., Chen, D., Yuan, L., Hua, G., Yu, N., & Chen, B. (2018a). Decouple learning for parameterized image operators. In ECCV (pp. 455–471).

  • Fan, Q., Yang, J., Wipf, D. P., Chen, B., & Tong, X. (2018b). Image smoothing via unsupervised learning. ACM TOG, 37(6), 259:1-259:14.

  • Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., & Paisley, J. (2017). Removing rain from single images via a deep detail network. In CVPR (pp. 3855–3863).

  • Fu, X., Huang, J., Ding, X., Liao, Y., & Paisley, J. (2017). Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing, 26(6), 2944–2956.

    Article  MathSciNet  MATH  Google Scholar 

  • Girshick, R. B. (2015). Fast R-CNN. In ICCV (pp. 1440–1448).

  • Guo, T., Li, X., Cherukuri, V., & Monga, V. (2019). Dense scene information estimation network for dehazing. In CVPR workshops (pp. 2122–2130).

  • Haris, M., Shakhnarovich, G., & Ukita, N. (2018). Deep back-projection networks for super-resolution. In CVPR (pp. 1664–1673).

  • He, K., Sun, J., & Tang, X. (2009). Single image haze removal using dark channel prior. In CVPR (pp. 1956–1963).

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In CVPR (pp. 770–778).

  • Huang, J. B., Singh, A., & Ahuja, N. (2015). Single image super-resolution from transformed self-exemplars. In CVPR (pp. 5197–5206).

  • Isobe, T., Jia, X., Gu, S., Li, S., Wang, S., & Tian, Q. (2020). Video super-resolution with recurrent structure-detail network. In ECCV (pp. 645–660).

  • Jain, V., & Seung, H. S. (2008). Natural image denoising with convolutional networks. In NIPS (pp. 769–776).

  • Kang, L. W., Lin, C. W., & Fu, Y. H. (2012). Automatic single-image-based rain streaks removal via image decomposition. IEEE TIP, 21(4), 1742–1755.

    MathSciNet  MATH  Google Scholar 

  • Kim, J., Lee, J. K., & Lee, K. M. (2016). Accurate image super-resolution using very deep convolutional networks. In CVPR (pp. 1646–1654).

  • Kim, J., Lee, J. K., & Lee, K. M. (2016). Deeply-recursive convolutional network for image super-resolution. In CVPR (pp. 1637–1645).

  • Krishnan, D., & Fergus, R. (2009). Fast image deconvolution using hyper-Laplacian priors. In NIPS (pp. 1033–1041).

  • Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In NIPS (pp. 1106–1114).

  • Lai, W. S., Huang, J. B., Ahuja, N., & Yang, M. H. (2019). Fast and accurate image super-resolution with deep Laplacian pyramid networks. IEEE TPAMI, 41(11), 2599–2613.

    Article  Google Scholar 

  • Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., & Shi, W. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In CVPR (pp. 4681–4690).

  • Levin, A., Weiss, Y., Durand, F., & Freeman, W. T. (2009). Understanding and evaluating blind deconvolution algorithms. In CVPR (pp. 1964–1971).

  • Levin, A., Fergus, R., Durand, F., & Freeman, W. T. (2007). Image and depth from a conventional camera with a coded aperture. ACM TOG, 26(3), 70.

    Article  Google Scholar 

  • Li, R., Pan, J., Li, Z., & Tang, J. (2018). Single image dehazing via conditional generative adversarial network. In CVPR (pp. 8202–8211).

  • Li, B., Peng, X., Wang, Z., Xu, J., & Feng, D. (2017). Aod-net: all-in-one dehazing network. In ICCV (pp. 4780–4788).

  • Li, Y., Tan, R.T., Guo, X., Lu, J., & Brown, M. S. (2016). Rain streak removal using layer priors. In CVPR (pp. 2736–2744).

  • Liao, R., Tao, X., Li, R., Ma, Z., & Jia, J. (2015). Video super-resolution via deep draft-ensemble learning. In ICCV (pp. 531–539).

  • Lim, B., Son, S., Kim, H., Nah, S., & Lee, K. M. (2017). Enhanced deep residual networks for single image super-resolution. In CVPR workshop (pp. 1132–1140).

  • Lin, T. Y., Roy Chowdhury, A., & Maji, S. (2015). Bilinear CNN models for fine-grained visual recognition. In ICCV (pp. 1449–1457).

  • Li, S., Ren, W., Zhang, J., Yu, J., & Guo, X. (2019). Single image rain removal via a deep decomposition-composition network. Computer Vision Image Understanding, 186, 48–57.

    Article  Google Scholar 

  • Liu, S., Pan, J., & Yang, M. H. (2016). Learning recursive filters for low-level vision via a hybrid neural network. In ECCV (pp. 560–576).

  • Martin, D. R., Fowlkes, C. C., Tal, D., & Malik, J. (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV (pp. 416–425).

  • Meng, G., Wang, Y., Duan, J., Xiang, S., & Pan, C. (2013). Efficient image dehazing with boundary constraint and contextual regularization. In ICCV (pp. 617–624).

  • Pan, J., Liu, S., Sun, D., Zhang, J., Liu, Y., Ren, J., Li, Z., Tang, J., Lu, H., Tai, Y. W., & Yang, M. H. (2018). Learning dual convolutional neural networks for low-level vision. In CVPR (pp. 3070–3079).

  • Pan, J., Dong, J., Liu, Y., Zhang, J., Ren, J. S. J., Tang, J., et al. (2021). Physics-based generative adversarial models for image restoration and beyond. IEEE TPAMI, 43(7), 2449–2462.

    Article  Google Scholar 

  • Pan, J., Sun, D., Pfister, H., & Yang, M. (2018). Deblurring images via dark channel prior. IEEE TPAMI, 40(10), 2315–2328.

    Article  Google Scholar 

  • Qian, R., Tan, R. T., Yang, W., Su, J., & Liu, J. (2018). Attentive generative adversarial network for raindrop removal from a single image. In CVPR (pp. 2482–2491).

  • Ren, W., Liu, S., Zhang, H., Pan, J., Cao, X., & Yang, M. H. (2016). Single image dehazing via multi-scale convolutional neural networks. In ECCV (pp. 154–169).

  • Ren, J. S. J., Xu, L., Yan, Q., & Sun, W. (2015). Shepard convolutional neural networks. In NIPS (pp. 901–909).

  • Saxena, A., Sun, M., & Ng, A. Y. (2009). Make3d: Learning 3d scene structure from a single still image. IEEE TPAMI, 31(5), 824–840.

    Article  Google Scholar 

  • Schmidt, U., & Roth, S. (2014). Shrinkage fields for effective image restoration. In CVPR (pp. 2774–2781).

  • Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., & Wang, Z. (2016). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR (pp. 1874–1883).

  • Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012). Indoor segmentation and support inference from RGBD images. In ECCV (pp. 746–760).

  • Singh, V., Ramnath, K., & Mittal, A. (2020). Refining high-frequencies for sharper super-resolution and deblurring. Computer Vision Image Understanding, 199, 103034.

    Article  Google Scholar 

  • Sun, Y., Chen, Y., Wang, X., & Tang, X. (2014). Deep learning face representation by joint identification-verification. In NIPS (pp. 1988–1996).

  • Tarel, J., Hautière, N., Caraffa, L., Cord, A., Halmaoui, H., & Gruyer, D. (2012). Vision enhancement in homogeneous and heterogeneous fog. IEEE Intelligent Transportation Systems Magazine, 4(2), 6–20.

    Article  Google Scholar 

  • Tian, C., Xu, Y., Zuo, W., Du, B., Lin, C. W., & Zhang, D. (2020). Designing and training of A dual CNN for image denoising. CoRR arXiv:2007.03951

  • Timofte, R., Smet, V. D., & Gool, L. J. V. (2014). A+: Adjusted anchored neighborhood regression for fast super-resolution. In ACCV (pp. 111–126).

  • Xie, J., Xu, L., & Chen, E. (2012). Image denoising and inpainting with deep neural networks. In NIPS (pp. 350–358).

  • Xu, L., Ren, J. S. J., Liu, C., & Jia, J. (2014). Deep convolutional neural network for image deconvolution. In NIPS (pp. 1790–1798).

  • Xu, L., Ren, J.S.J., Yan, Q., Liao, R., & Jia, J. (2015). Deep edge-aware filters. In ICML (pp. 1669–1678).

  • Xu, L., Lu, C., Xu, Y., & Jia, J. (2011). Image smoothing via \(L _{{0}}\) gradient minimization. ACM TOG, 30(6), 174:1-174:12.

    Google Scholar 

  • Xu, L., Yan, Q., Xia, Y., & Jia, J. (2012). Structure extraction from texture via relative total variation. ACM TOG, 31(6), 139:1-139:10.

    Google Scholar 

  • Yang, H., Pan, J., Yan, Q., Sun, W., Ren, J. S. J., & Tai, Y. W. (2017). Image dehazing using bilinear composition loss function. CoRR arXiv:1710.00279

  • Yang, A., Wang, H., Ji, Z., Pang, Y., & Shao, L. (2019). Dual-path in dual-path network for single image dehazing. In IJCAI (pp. 4627–4634).

  • Zhang, H., & Patel, V. M. (2018). Densely connected pyramid dehazing network. In CVPR (pp. 3194–3203).

  • Zhang, H., & Patel, V. M. (2018). Density-aware single image de-raining using a multi-stream dense network. In CVPR (pp. 695–704).

  • Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., & Fu, Y. (2018). Image super-resolution using very deep residual channel attention networks. In ECCV (pp. 294–310).

  • Zhang, J., Pan, J., Lai, W. S., Lau, R. W. H., & Yang, M. H. (2017). Learning fully convolutional networks for iterative non-blind deconvolution. In CVPR (pp. 6969–6977).

  • Zhang, Y., Tian, Y., Kong, Y., Zhong, B., & Fu, Y. (2018). Residual dense network for image super-resolution. In CVPR (pp. 2472–2481).

  • Zhang, Q., Xu, L., & Jia, J. (2014). 100+ times faster weighted median filter (WMF). In CVPR (pp. 2830–2837).

  • Zhang, K., Zuo, W., Gu, S., & Zhang, L. (2017). Learning deep CNN denoiser prior for image restoration. In CVPR (pp. 2808–2817).

  • Zhang, H., Sindagi, V., & Patel, V. M. (2020). Image de-raining using a conditional generative adversarial network. IEEE TCSVT, 30(11), 3943–3956.

    Google Scholar 

  • Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017). Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE TIP, 26(7), 3142–3155.

    MathSciNet  MATH  Google Scholar 

  • Zhu, H., Peng, X., Chandrasekhar, V., Li, L., & Lim, J. H. (2018). Dehazegan: When image dehazing meets differential programming. In IJCAI (pp. 1234–1240).

  • Zhu, H., Cheng, Y., Peng, X., Zhou, J. T., Kang, Z., Lu, S., et al. (2021). Single-image dehazing via compositional adversarial network. IEEE Transactions on Cybernetics, 51(2), 829–838.

    Article  Google Scholar 

  • Zoran, D., & Weiss, Y. (2011). From learning models of natural image patches to whole image restoration. In ICCV (pp. 479–486).

Download references

Acknowledgements

This work is supported in part by the National Key Research and Development Program of China under Grant 2018AAA0102001, the National Natural Science Foundation of China under Grants 61872421, 61922043, and 61925204, the Fundamental Research Funds for the Central Universities under Grant 30920041109, and NSF CAREER under Grant 1149783.

Author information

Authors and Affiliations

  1. Nanjing University of Science and Technology, Nanjing, 210094, China

    Jinshan Pan, Jinhui Tang & Jian Yang

  2. Google, New York, USA

    Deqing Sun

  3. SenseTime Research, Shenzhen, 518000, China

    Jiawei Zhang

  4. Kuaishou Technology, Shenzhen, 518000, China

    Yu-Wing Tai

  5. University of California at Merced, Merced, USA

    Ming-Hsuan Yang

  6. Yonsei University, Seoul, South Korea

    Ming-Hsuan Yang

  7. Google, Mountain View, USA

    Ming-Hsuan Yang

Authors
  1. Jinshan Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Deqing Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiawei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jinhui Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jian Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yu-Wing Tai
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ming-Hsuan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jinhui Tang or Ming-Hsuan Yang.

Additional information

Communicated by Subhransu Maji.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 27160 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pan, J., Sun, D., Zhang, J. et al. Dual Convolutional Neural Networks for Low-Level Vision. Int J Comput Vis 130, 1440–1458 (2022). https://doi.org/10.1007/s11263-022-01583-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-022-01583-y

Keywords

Search

Navigation

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy