Skip to main content
Springer Nature Link
Log in

Multi-branch feature fusion and refinement network for salient object detection

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

With the development of convolutional neural networks (CNNs), salient object detection methods have made great progress in performance. Most methods are designed with complex structures to aggregate the multi-level feature maps, to reach the goal of filtering noise and obtaining rich information. However, there is no differentiation when dealing with the multi-level features, and only a uniform treatment is used in general. Based on the above considerations, in this paper, we propose a multi-branch feature fusion and refinement network (MFFRNet), which is a framework for treating low-level features and high-level features differently, and effectively fuses the information of multi-level features to make the results more accurate. We propose a detail optimization module (DOM) designed for the rich detail information in low-level features and a pyramid feature extraction module (PFEM) designed for the rich semantic information in high-level features, as well as a feature optimization module (FOM) for refining the fused feature of multiple levels. Extensive experiments are conducted on six benchmark datasets, and the results show that our approach outperforms the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availibility Statement

The data that support the findings of this study are available from the author, Jinyu Yang, upon reasonable request.

References

  1. Collier, M., Mustafa, B., Kokiopoulou, E., Jenatton, R., Berent, J.: Correlated input-dependent label noise in large-scale image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1551–1560 (2021)

  2. Li, B., Li, Y., Eliceiri, K.W.: Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14318–14328 (2021)

  3. Wang, T.-C., Mallya, A., Liu, M.-Y.: One-shot free-view neural talking-head synthesis for video conferencing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10039–10049 (2021)

  4. Zhang, P., Zhuo, T., Huang, W., Chen, K., Kankanhalli, M.: Online object tracking based on cnn with spatial-temporal saliency guided sampling. Neurocomputing 257, 115–127 (2017)

    Article  Google Scholar 

  5. Zhang, P., Liu, W., Wang, D., Lei, Y., Wang, H., Lu, H.: Non-rigid object tracking via deep multi-scale spatial-temporal discriminative saliency maps. Pattern Recognit. 100, 107130 (2020)

    Article  Google Scholar 

  6. Zhang, L., Dai, J., Lu, H., He, Y., Wang, G.: A bi-directional message passing model for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1741–1750 (2018)

  7. Zhang, X., Wang, T., Qi, J., Lu, H., Wang, G.: Progressive attention guided recurrent network for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 714–722 (2018)

  8. Qin, X., Zhang, Z., Huang, C., Gao, C., Dehghan, M., Jagersand, M.: Basnet: boundary-aware salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7479–7489 (2019)

  9. Zhang, J., Shi, Y., Zhang, Q., Cui, L., Chen, Y., Yi, Y.: Attention guided contextual feature fusion network for salient object detection. Image Vis. Comput. 117, 104337 (2022)

    Article  Google Scholar 

  10. Zhao, T., Wu, X.: Pyramid feature attention network for saliency detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3085–3094 (2019)

  11. Wei, J., Wang, S., Huang, Q.: F\(^3\)net: fusion, feedback and focus for salient object detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 12321–12328 (2020)

  12. Zhang, M., Liu, T., Piao, Y., Yao, S., Lu, H.: Auto-msfnet: search multi-scale fusion network for salient object detection. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 667–676 (2021)

  13. Pang, Y., Zhao, X., Zhang, L., Lu, H.: Multi-scale interactive network for salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9413–9422 (2020)

  14. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)

    Article  Google Scholar 

  15. Zhang, J., Ehinger, K.A., Ding, J., Yang, J.: A prior-based graph for salient object detection. In: 2014 IEEE International Conference on Image Processing (ICIP). IEEE, pp. 1175–1178 (2014)

  16. Cheng, M.-M., Mitra, N.J., Huang, X., Torr, P.H., Hu, S.-M.: Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 569–582 (2014)

    Article  Google Scholar 

  17. Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1155–1162 (2013)

  18. Perazzi, F., Krähenbühl, P., Pritch, Y., Hornung, A.: Saliency filters: contrast based filtering for salient region detection. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp. 733–740 (2012)

  19. Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.-H.: Saliency detection via graph-based manifold ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3166–3173 (2013)

  20. Jiang, H., Wang, J., Yuan, Z., Wu, Y., Zheng, N., Li, S.: Salient object detection: a discriminative regional feature integration approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2083–2090 (2013)

  21. Hong, C., Yu, J., Zhang, J., Jin, X., Lee, K.-H.: Multimodal face-pose estimation with multitask manifold deep learning. IEEE Trans. Ind. Inform. 15(7), 3952–3961 (2018)

    Article  Google Scholar 

  22. Hong, C., Yu, J., Wan, J., Tao, D., Wang, M.: Multimodal deep autoencoder for human pose recovery. IEEE Trans. Image Process. 24(12), 5659–5670 (2015)

    Article  MathSciNet  Google Scholar 

  23. Li, K., Lu, J., Zuo, H., Zhang, G.: Dynamic classifier alignment for unsupervised multi-source domain adaptation. IEEE Trans. Knowl. Data Eng. 35(5), 4727–4740 (2022)

    Google Scholar 

  24. Li, K., Lu, J., Zuo, H., Zhang, G.: Multidomain adaptation with sample and source distillation. IEEE Trans. Cybern. (2023)

  25. Yu, J., Tan, M., Zhang, H., Rui, Y., Tao, D.: Hierarchical deep click feature prediction for fine-grained image recognition. IEEE Trans. Pattern Anal. Mach. Intell. 44(2), 563–578 (2019)

    Article  Google Scholar 

  26. Hou, Q., Cheng, M.-M., Hu, X., Borji, A., Tu, Z., Torr, P.H.: Deeply supervised salient object detection with short connections. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3203–3212 (2017)

  27. Liu, J.-J., Hou, Q., Cheng, M.-M., Feng, J., Jiang, J.: A simple pooling-based design for real-time salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3917–3926 (2019)

  28. Wu, Z., Su, L., Huang, Q.: Cascaded partial decoder for fast and accurate salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3907–3916 (2019)

  29. Wang, W., Shen, J., Cheng, M.-M., Shao, L.: An iterative and cooperative top-down and bottom-up inference network for salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5968–5977 (2019)

  30. Wei, J., Wang, S., Wu, Z., Su, C., Huang, Q., Tian, Q.: Label decoupling framework for salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13025–13034 (2020)

  31. Zhang, J., Liang, Q., Guo, Q., Yang, J., Zhang, Q., Shi, Y.: R2net: residual refinement network for salient object detection. Image Vis. Comput. 120, 104423 (2022)

    Article  Google Scholar 

  32. Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881–2890 (2017)

  33. Wu, Y.-H., Liu, Y., Zhang, L., Cheng, M.-M., Ren, B.: Edn: salient object detection via extremely-downsampled network. IEEE Trans. Image Process. 31, 3125–3136 (2022)

    Article  Google Scholar 

  34. Xia, C., Sun, Y., Fang, X., Ge, B., Gao, X., Li, K.-C.: Imsfnet: integrated multi-source feature network for salient object detection. Appl. Intell. 1–21 (2023)

  35. Ding, X., Guo, Y., Ding, G., Han, J.: Acnet: strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1911–1920 (2019)

  36. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv:1511.07122 (2015)

  37. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

  38. Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: Cbam: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  39. Liu, J.-J., Hou, Q., Liu, Z.-A., Cheng, M.-M.: Poolnet+: exploring the potential of pooling for salient object detection. IEEE (2022)

  40. Yu, J., Jiang, Y., Wang, Z., Cao, Z., Huang, T.: Unitbox: an advanced object detection network. In: Proceedings of the 24th ACM International Conference on Multimedia, pp. 516–520 (2016)

  41. Bokhovkin, A., Burnaev, E.: Boundary loss for remote sensing imagery semantic segmentation. In: International Symposium on Neural Networks. Springer, pp. 388–401 (2019)

  42. Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1155–1162 (2013)

  43. Li, G., Yu, Y.: Visual saliency based on multiscale deep features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5455–5463 (2015)

  44. Li, Y., Hou, X., Koch, C., Rehg, J.M., Yuille, A.L.: The secrets of salient object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 280–287 (2014)

  45. Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.-H.: Saliency detection via graph-based manifold ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3166–3173 (2013)

  46. Wang, L., Lu, H., Wang, Y., Feng, M., Wang, D., Yin, B., Ruan, X.: Learning to detect salient objects with image-level supervision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 136–145 (2017)

  47. Fan, D.-P., Cheng, M.-M., Liu, J.-J., Gao, S.-H., Hou, Q., Borji, A.: Salient objects in clutter: bringing salient object detection to the foreground. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 186–202 (2018)

  48. Perazzi, F., Krähenbühl, P., Pritch, Y., Hornung, A.: Saliency filters: contrast based filtering for salient region detection. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp. 733–740 (2012)

  49. Achanta, R., Hemami, S., Estrada, F., Susstrunk, S.: Frequency-tuned salient region detection. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp. 1597–1604 (2009)

  50. Margolin, R., Zelnik-Manor, L., Tal, A.: How to evaluate foreground maps? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2014)

  51. Fan, D.-P., Gong, C., Cao, Y., Ren, B., Cheng, M.-M., Borji, A.: Enhanced-alignment measure for binary foreground map evaluation. arXiv:1805.10421 (2018)

  52. Soleymani, R., Granger, E., Fumera, G.: F-measure curves: a tool to visualize classifier performance under imbalance. Pattern Recognit. 100, 107146 (2020)

    Article  Google Scholar 

  53. Liu, J.-J., Hou, Q., Cheng, M.-M.: Dynamic feature integration for simultaneous detection of salient object, edge, and skeleton. IEEE Trans. Image Process. 29, 8652–8667 (2020)

    Article  Google Scholar 

  54. Feng, M., Lu, H., Ding, E.: Attentive feedback network for boundary-aware salient object detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

  55. Deng, Z., Hu, X., Zhu, L., Xu, X., Qin, J., Han, G., Heng, P.-A.: R3net: recurrent residual refinement network for saliency detection. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence. AAAI Press, Menlo Park, pp. 684–690 (2018)

  56. Fan, D.-P., Cheng, M.-M., Liu, Y., Li, T., Borji, A.: Structure-measure: a new way to evaluate foreground maps. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4548–4557 (2017)

  57. Pang, Y., Zhao, X., Zhang, L., Lu, H.: Multi-scale interactive network for salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9413–9422 (2020)

  58. Ma, M., Xia, C., Li, J.: Pyramidal feature shrinking for salient object detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 2311–2318 (2021)

  59. Chen, Z., Xu, Q., Cong, R., Huang, Q.: Global context-aware progressive aggregation network for salient object detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 10599–10606 (2020)

  60. Lee, M.S., Shin, W., Han, S.W.: Tracer: extreme attention guided salient object tracing network. arXiv:2112.07380 (2021)

  61. Yu, S., Zhang, B., Xiao, J., Lim, E.G.: Structure-consistent weakly supervised salient object detection with local saliency coherence. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 3234–3242 (2021)

  62. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. Ieee, pp. 248–255 (2009)

  63. Wu, Z., Su, L., Huang, Q.: Cascaded partial decoder for fast and accurate salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3907–3916 (2019)

  64. Wu, Z., Su, L., Huang, Q.: Stacked cross refinement network for edge-aware salient object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7264–7273 (2019)

  65. Zhao, X., Pang, Y., Zhang, L., Lu, H., Zhang, L.: Suppress and balance: a simple gated network for salient object detection. In: European Conference on Computer Vision. Springer, pp. 35–51 (2020)

  66. Zhou, H., Xie, X., Lai, J.-H., Chen, Z., Yang, L.: Interactive two-stream decoder for accurate and fast saliency detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9141–9150 (2020)

  67. Zhao, Z., Xia, C., Xie, C., Li, J.: Complementary trilateral decoder for fast and accurate salient object detection. In: Proceedings of the 29th Acm International Conference on Multimedia, pp. 4967–4975 (2021)

  68. Li, J., Su, J., Xia, C., Ma, M., Tian, Y.: Salient object detection with purificatory mechanism and structural similarity loss. IEEE Trans. Image Process. 30, 6855–6868 (2021)

    Article  Google Scholar 

  69. Zhuge, M., Fan, D.-P., Liu, N., Zhang, D., Xu, D., Shao, L.: Salient object detection via integrity learning. IEEE Trans. Pattern Anal. Mach. Intell. (2022)

Download references

Acknowledgements

This work is supported in part by grants from the National Natural Science Foundation of China (No. 61806126, 61903256, 61976140, 61973307, 62062040), the Natural Science Foundation of Shanghai (19ZR1455300, 21ZR1462600), the Shanghai Science and Technology Innovation Action Plan (No. 22S31903900), the Outstanding Youth Project of Jiangxi Natural Science Foundation (No. 20212ACB212003), the Jiangxi Province Key Subject Academic and Technical Leader Funding Project (No. 20212BCJ23017) and Science and Technology Development Foundation of the Shanghai Institute of Technology (No. ZQ2023-15).

Author information

Authors and Affiliations

  1. School of Computer Science and Information Engineering, Shanghai Institute of Technology, Shanghai, 201418, People’s Republic of China

    Jinyu Yang, Yanjiao Shi, Jin Zhang, Qianqian Guo, Qing Zhang & Liu Cui

Authors
  1. Jinyu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yanjiao Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qianqian Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Liu Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JY wrote the main manuscript text, ZJ drew Figs. 1, 2, 3, 4, 5, 6, 7, 8 and 9, and QG prepared Tables 1, 2 and 3. All authors reviewed the manuscript.

Corresponding author

Correspondence to Yanjiao Shi.

Ethics declarations

Conflict of interest

The manuscript is approved by all co-authors and there are no Conflict of interest to report.

Additional information

Communicated by J. Gao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, J., Shi, Y., Zhang, J. et al. Multi-branch feature fusion and refinement network for salient object detection. Multimedia Systems 30, 190 (2024). https://doi.org/10.1007/s00530-024-01356-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00530-024-01356-2

Keywords

Search

Navigation

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy