Abstract
DNN testing is a critical way to guarantee the quality of DNNs. To obtain test oracle information, DNN testing requires a huge cost to label test inputs, which greatly affects the efficiency of DNN testing. To alleviate the labeling cost problem, the paper applies the idea of the spectrum-based fault location technique to DNN testing and proposes a novel test input prioritization approach for DNNs based on fault localization (called TPFL). TPFL first performs dynamic spectrum analysis on each neuron in the DNN. TPFL then proposes a suspiciousness measure that uses the neuron spectrum to identify suspicious neurons that cause the DNN to make wrong decisions. Finally, TPFL is based on the following key insight: a test input makes the suspicious neurons fully active, it indicates that this may be a bug-revealing input, so the input should have a higher priority. To evaluate, we conduct an empirical study on 3 widely used datasets and corresponding 8 DNN models. The experimental results show that TPFL performs well in both classification and regression models and overall outperforms most existing test input prioritization techniques.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abreu, R., Zoeteweij, P., Golsteijn, R.: A practical evaluation of spectrum-based fault localization. J. Syst. Softw. 82(11), 1780–1792 (2009)
Aggarwal, A., Lohia, P., Nagar, S.: Black box fairness testing of machine learning models. In: Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2019, Tallinn, Estonia, 26–30 August 2019, pp. 625–635. ACM (2019)
Bengio, Y., Mesnil, G., Dauphin, Y.N.: Better mixing via deep representations. In: Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16–21 June 2013. JMLR Workshop and Conference Proceedings, vol. 28, pp. 552–560. JMLR.org (2013)
Byun, T., Sharma, V., Vijayakumar, A.: Input prioritization for testing neural networks. In: IEEE International Conference On Artificial Intelligence Testing, AITest 2019, Newark, CA, USA, 4–9 April 2019, pp. 63–70. IEEE (2019)
Carlini, N., Wagner, D.A.: Towards evaluating the robustness of neural networks. In: 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, 22–26 May 2017, pp. 39–57. IEEE Computer Society (2017)
Chen, Z., Huang, X.: End-to-end learning for lane keeping of self-driving cars. In: IEEE Intelligent Vehicles Symposium, IV 2017, Los Angeles, CA, USA, 11–14 June 2017, pp. 1856–1860. IEEE (2017)
Cheng, D., Cao, C., Xu, C.: Manifesting bugs in machine learning code: an explorative study with mutation testing. In: 2018 IEEE International Conference on Software Quality, Reliability and Security, QRS 2018, Lisbon, Portugal, 16–20 July 2018, pp. 313–324. IEEE (2018)
Eniser, H.F., Gerasimou, S., Sen, A.: DeepFault: fault localization for deep neural networks. In: Hähnle, R., van der Aalst, W. (eds.) FASE 2019. LNCS, vol. 11424, pp. 171–191. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-16722-6_10
Feng, Y., Shi, Q., Gao, X.: DeepGini: prioritizing massive tests to enhance the robustness of deep neural networks. In: ISSTA 2020: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, 18–22 July 2020, pp. 177–188. ACM (2020)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015)
Guo, J., Jiang, Y., Zhao, Y.: DLFuzz: differential fuzzing testing of deep learning systems. In: Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, 04–09 November 2018, pp. 739–743. ACM (2018)
He, K., Zhang, X., Ren, S.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016, pp. 770–778. IEEE Computer Society (2016)
Khope, S.R., Elias, S.: Critical correlation of predictors for an efficient risk prediction framework of ICU patient using correlation and transformation of MIMIC-III dataset. Data Sci. Eng. 7(1), 71–86 (2022)
Kim, J., Feldt, R., Yoo, S.: Guiding deep learning system testing using surprise adequacy. In: Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, 25–31 May 2019, pp. 1039–1049. IEEE/ACM (2019)
Kurakin, A., Goodfellow, I.J.: Adversarial examples in the physical world. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 24–26 April 2017, Workshop Track Proceedings. OpenReview.net (2017)
Lee, S., Cha, S., Lee, D.: Effective white-box testing of deep neural networks with adaptive neuron-selection strategy. In: ISSTA 2020: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, 18–22 July 2020, pp. 165–176. ACM (2020)
Liu, W., Wang, Z., Liu, X.: A survey of deep neural network architectures and their applications. Neurocomputing 234, 11–26 (2017)
Ma, L., Juefei-Xu, F., Zhang, F.: DeepQauge: multi-granularity testing criteria for deep learning systems. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, 3–7 September 2018, pp. 120–131. ACM (2018)
Papernot, N., McDaniel, P.D., Jha, S.: The limitations of deep learning in adversarial settings. In: IEEE European Symposium on Security and Privacy, EuroS &P 2016, Saarbrücken, Germany, 21–24 March 2016, pp. 372–387. IEEE (2016)
Pearson, S., Campos, J., Just, R.: Evaluating and improving fault localization. In: Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, 20–28 May 2017, pp. 609–620. IEEE/ACM (2017)
Pei, K., Cao, Y., Yang, J.: DeepXplore: automated whitebox testing of deep learning systems. In: Proceedings of the 26th Symposium on Operating Systems Principles, Shanghai, China, 28–31 October 2017, pp. 1–18. ACM (2017)
Rothermel, G., Untch, R.H., Chu, C.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27(10), 929–948 (2001)
Sun, Y., Huang, X., Kroening, D.: Testing deep neural networks. CoRR abs/1803.04792 (2018)
Sun, Y., Wu, M.: Concolic testing for deep neural networks. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, 3–7 September 2018, pp. 109–119. ACM (2018)
Tian, Y., Pei, K., Jana, S.: DeepTest: automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27–June 03 2018, pp. 303–314. ACM (2018)
Wang, Z., You, H., Chen, J.: Prioritizing test inputs for deep neural networks via mutation analysis. In: 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22–30 May 2021, pp. 397–409. IEEE (2021)
Wong, W.E., Gao, R., Li, Y.: A survey on software fault localization. IEEE Trans. Softw. Eng. 42(8), 707–740 (2016)
Xie, X., Ma, L., Juefei-Xu, F.: DeepHunter: a coverage-guided fuzz testing framework for deep neural networks. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Beijing, China, 15–19 July 2019, pp. 146–157. ACM (2019)
Yan, R., Chen, Y., Gao, H.: Test case prioritization with neuron valuation based pattern. Sci. Comput. Program. 215, 102761 (2022)
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53
Zhang, J.M., Harman, M., Ma, L.: Machine learning testing: survey, landscapes and horizons. IEEE Trans. Softw. Eng. 48(2), 1–36 (2022)
Zhang, K., Zhang, Y., Zhang, L.: Neuron activation frequency based test case prioritization. In: International Symposium on Theoretical Aspects of Software Engineering, TASE 2020, Hangzhou, China, 11–13 December 2020, pp. 81–88. IEEE (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tao, Y., Tao, C., Guo, H., Li, B. (2022). TPFL: Test Input Prioritization for Deep Neural Networks Based on Fault Localization. In: Chen, W., Yao, L., Cai, T., Pan, S., Shen, T., Li, X. (eds) Advanced Data Mining and Applications. ADMA 2022. Lecture Notes in Computer Science(), vol 13725. Springer, Cham. https://doi.org/10.1007/978-3-031-22064-7_27
Download citation
DOI: https://doi.org/10.1007/978-3-031-22064-7_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-22063-0
Online ISBN: 978-3-031-22064-7
eBook Packages: Computer ScienceComputer Science (R0)