Abstract
Automatic and accurate pavement crack detection is essential for cost-effective road maintenance. Deep convolutional neural networks (DCNNs) are widely used in recent methods for pavement crack segmentation. Although DCNNs can segment pavement cracks with great accuracy, the requirement for huge pixel-level labels is demanding. In this article, we propose a novel weakly supervised framework for pavement crack segmentation based on multi-scale object localization and incremental annotation refinement. A trained pavement crack classification network is used to produce initial annotations using multi-scale class activation mapping strategy. Then, a new segmentation network (U2-Net) with triplet attention (TA) module and multiple loss functions is trained using initial annotations. The TA module is developed to emphasize important features and ignore unimportant features, whereas multiple loss functions are employed to assist crack segmentation for a clean and full mask. Moreover, incremental annotation refinement (IAR) is proposed for iteratively optimizing the segmentation network and refining segmentation masks. Comparative experiments on DeepCrack and Crack500 datasets demonstrate that the proposed framework bridges the performance gap between weakly and fully supervised pavement crack segmentation methods, outperforms existing weakly supervised pavement crack segmentation methods, and achieves state-of-the-art performance while reducing human labeling efforts.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Zhong Q u, Cao C, Liu L, Zhou Dong-Yang (2021) A deeply supervised convolutional neural network for pavement crack detection with multiscale feature fusion. IEEE Trans Neural Netw Learning Syst:1–10
Protopapadakis E, Voulodimos A, Doulamis A, Doulamis N, Stathaki T (2019) Automatic crack detection for tunnel inspection using deep learning and heuristic image post-processing. Appl Intell 49 (7):2793–2806
Liu C, Zhu C, Xia X, Zhao J, Haihui Long. (2022) Ffedn: feature fusion encoder decoder network for crack detection
Dai Z, Yi J, Zhang Y, Bo Z, He L (2020) Fast and accurate cable detection using cnn. Appl Intell 50(12):4688–4707
Daipeng Y, Peng B, Al-Huda Z, Malik A, Zhai D (2022) An overview of edge and object contour detection. Neurocomputing
Xia H, Ma M, Li H, Song S (2022) Mc-net: multi-scale context-attention network for medical ct image segmentation. Appl Intell 52(2):1508–1519
Zhang J, Liu Y, Guo C, Zhan J (2022) Optimized segmentation with image inpainting for semantic mapping in dynamic scenes. Appl Intell:1–16
Liu M, Yan X, Wang C, Wang K (2021) Segmentation mask-guided person image generation. Appl Intell 51(2):1161–1176
Ma M, Xia H, Tan Y, Li H, Song S (2022) Ht-net: hierarchical context-attention transformer network for medical ct image segmentation. Appl Intell:1–14
Li J, Mei X, Prokhorov D, Tao D (2017) Deep neural network for structural prediction and lane detection in traffic scene. IEEE Transa Neural Netw Learning Syst 28(3):690–703
Zhong Q, Chen W, Wang S-Y, Yi T-M, Liu L (2021) A crack detection algorithm for concrete pavement based on attention mechanism and multi-features fusion. IEEE Trans Intell Transp Syst:1–10
Guo J-M, Markoni H, Lee J-D (2021) Barnet: boundary aware refinement network for crack detection. IEEE Trans Intell Transp Syst:1–16
Cheng JCP, Wang M (2018) Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques. Autom Constr 95:155–171
Yang X u, Wei S, Bao Y, Li H (2019) Automatic seismic damage identification of reinforced concrete columns from images by a region-based deep convolutional neural network. Struct Control Health Monit 26(3):e2313
Tang W, Huang S, Zhao Q, Li R, Huangfu L (2021) An iteratively optimized patch label inference network for automatic pavement distress detection. IEEE Trans Intell Transp Syst:1–10
Gopalakrishnan K, Khaitan SK, Choudhary A, Agrawal A (2017) Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection. Construct Build Mater 157:322–330
Shi Y, Cui L, Qi Z, Meng F, Chen Z (2016) Automatic road crack detection using random structured forests. IEEE Trans Intell Transp Syst 17(12):3434–3445
Yang F, Zhang L, Sijia Y u, Prokhorov D, Mei X, Ling H (2020) Feature pyramid and hierarchical boosting network for pavement crack detection. IEEE Trans Intell Transp Syst 21(4):1525–1535
Li H, Song D, Liu Y u, Li B (2019) Automatic pavement crack detection by multi-scale image fusion. IEEE Trans Intell Transp Syst 20(6):2025–2036
Bo P, Al-Huda Z, Xie Z, Xi W (2020) Multi-scale region composition of hierarchical image segmentation. Multimed Tools Appl:1–23
Dai J, He K, Sun J (2015) Boxsup: exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In: Proceedings of the IEEE international conference on computer vision, pp 1635–1643
Di L, Dai J, Jia J, He K, Jian Sun. (2016) Scribblesup: scribble-supervised convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3159–3167
Al-Huda Z, Zhai D, Yang Y, Algburi RNA (2021) Optimal scale of hierarchical image segmentation with scribbles guidance for weakly supervised semantic segmentation. Int J Pattern Recognit Artif Intell 35 (10):2154026
Kolesnikov A, Lampert CH (2016) Seed, expand and constrain: three principles for weakly-supervised image segmentation. In: European conference on computer vision. Springer, pp 695–711
Al-Huda Z, Bo P, Yang Y, Algburi RNA (2020) Object scale selection of hierarchical image segmentation with deep seeds. IET Image Process, (8)
Huang Z, Wang X, Wang J, Liu W, Wang J (2018) Weakly-supervised semantic segmentation network with deep seeded region growing. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7014–7023
Al-Huda Z, Bo P, Yang Y, Muqeet A (2019) Object scale selection of hierarchical image segmentation using reliable regions. In: 2019 IEEE 14th international conference on intelligent systems and knowledge engineering (ISKE). IEEE, pp 1081–1088
Al-Huda Z, Bo P, Yang Y, Algburi RNA, Ahmad M, Khurshid F, Moghalles K (2021) Weakly supervised semantic segmentation by iteratively refining optimal segmentation with deep cues guidance. Neural Comput Applic:1–26
Dong Z, Wang J, Bo C, Wang D, Wang X (2020) Patch-based weakly supervised semantic segmentation network for crack detection. Construct Build Mater 258:120291
Qin X, Zhang Z, Huang C, Dehghan M, Zaiane OR, Jagersand M (2020) U2-net: going deeper with nested u-structure for salient object detection. Pattern Recogn 106:107404
Misra D, Nalamada T, Arasanipalai AU, Hou Q (2021) Rotate to attend: convolutional triplet attention module. In: 2021 IEEE winter conference on applications of computer vision (WACV), pp 3138–3147
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444
Deepcrack (2019) A deep hierarchical feature learning architecture for crack segmentation. Neurocomputing 338:139–153
Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2018) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell 40(4):834–848
Liu Z, Cao Y, Wang Y, Wang W (2019) Computer vision-based concrete crack detection using u-net fully convolutional networks. Autom Constr 104:129–139
Wang M, Cheng JCP (2020) A unified convolutional neural network integrated with conditional random field for pipe defect segmentation. Comput-Aided Civil Infrastruc Eng 35(2):162–177
Li D, Cong A, Guo S (2019) Sewer damage detection from imbalanced cctv inspection data using deep convolutional neural networks with hierarchical classification. Autom Constr 101:199–208
Chen Liang-Chieh, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV), pp 801–818
Oliveira H, Correia PL (2009) Automatic road crack segmentation using entropy and image dynamic thresholding. In: 2009 17th European signal processing conference. IEEE, pp 622–626
Inoue Y, Nagayoshi H (2021) Crack detection as a weakly-supervised problem: towards achieving less annotation-intensive crack detectors. In: 2020 25th international conference on pattern recognition (ICPR). IEEE, pp 65–72
Griffiths D, Boehm J (2018) Rapid object detection systems, utilising deep learning and unmanned aerial systems (uas) for civil engineering applications. Int Archives Photogrammetry, Remote Sensing Spatial Inf Sci-ISPRS Archives 42:391–398. International society for photogrammetry and remote sensing (ISPRS)
Pizer SM, Johnston RE, Ericksen JP, Yankaskas BC, Muller KE (1990) Contrast-limited adaptive histogram equalization: speed and effectiveness. In: 1990 proceedings of the first conference on visualization in biomedical computing, pp 337–345
Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2921–2929
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
Bo W, Yuan C, Li B, Ding X, Li Z, Ying W, Weiming H (2021) Multi-scale low-discriminative feature reactivation for weakly supervised object localization. IEEE Trans Image Process 30:6050–6065
Jie H, Li S, Albanie S, Sun G, Enhua W (2020) Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell 42(8):2011–2023
Pereira S, Pinto A, Amorim J, Ribeiro A, Alves V, Silva CA (2019) Adaptive feature recombination and recalibration for semantic segmentation with fully convolutional networks. IEEE Trans Med Imaging 38(12):2914–2925
Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. Thrity-Seventh Asilomar Conf Signals Syst Comput, 2003 2:1398–1402. https://doi.org/10.1109/ACSSC.2003.1292216
Aggarwal G, Jain S (2019) Road crack detection and segmentation for autonomous driving. In: 2019 international conference on communication and electronics systems (ICCES), pp 198–202
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF (eds) Medical image computing and computer-assisted intervention – MICCAI 2015. Springer International Publishing, pp 234–241, Cham
Song W, Jia G, Jia D, Zhu H (2019) Automatic pavement crack detection and classification using multiscale feature attention network. IEEE Access 7:171001–171012
Song W, Jia G, Zhu H, Di J, Gao L (2020) Automated pavement crack damage detection using deep multiscale convolutional features. J Adv Transp:2020
Zou Q, Zhang Z, Li Q, Qi X, Wang Q, Wang S (2019) Deepcrack: learning hierarchical convolutional features for crack detection. IEEE Trans Image Process 28(3):1498–1512
Kolesnikov A, Lampert CH (2016) Seed, expand and constrain: three principles for weakly-supervised image segmentation. In: Leibe B, Matas J, Sebe N, Welling M (eds) Computer vision – ECCV 2016. Springer International Publishing
Huang Z, Wang X, Wang J, Liu W, Wang J (2018) Weakly-supervised semantic segmentation network with deep seeded region growing. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 7014–7023
Ahn J, Kwak S (2018) Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 4981–4990
Acknowledgements
This work was supported by the Natural Science Foundation of Sichuan, China (No. 2022NSFSC0502), the National Science Foundation of China (No. 61772435, 42075142) and Fundamental Research Funds for the Central Universities (No. 2682021ZTPY069).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
No conflict of interest exits in this manuscript
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Al-Huda, Z., Peng, B., Algburi, R.N.A. et al. Weakly supervised pavement crack semantic segmentation based on multi-scale object localization and incremental annotation refinement. Appl Intell 53, 14527–14546 (2023). https://doi.org/10.1007/s10489-022-04212-w
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-022-04212-w