Skip to main content
Springer Nature Link
Log in

An extensive replication study of the ABLoTS approach for bug localization

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Bug localization is the task of recommending source code locations (typically files) that contain the cause of a bug and hence need to be changed to fix the bug. Along these lines, information retrieval-based bug localization (IRBL) approaches have been adopted, which identify the most bug-prone files from the source code space. In current practice, a series of state-of-the-art IRBL techniques leverage the combination of different components (e.g., similar reports, version history, and code structure) to achieve better performance. ABLoTS is a recently proposed approach with the core component, TraceScore, that utilizes requirements and traceability information between different issue reports (i.e., feature requests and bug reports) to identify buggy source code snippets with promising results. To evaluate the accuracy of these results and obtain additional insights into the practical applicability of ABLoTS, we conducted a replication study of this approach with the original dataset and also on two extended datasets (i.e., additional Java dataset and Python dataset). The original dataset consists of 11 open source Java projects with 8,494 bug reports. The extended Java dataset includes 16 more projects comprising 25,893 bug reports and corresponding source code commits. The extended Python dataset consists of 12 projects with 1,289 bug reports. While we find that the TraceScore component, which is the core of ABLoTS, produces comparable or even better results with the extended datasets, we also find that we cannot reproduce the ABLoTS results, as reported in its original paper, due to an overlooked side effect of incorrectly choosing a cut-off date that led to test data leaking into training data with significant effects on performance. Additionally, we conduct experiments to assess the performance of various composers that aggregate scores from different components, revealing that Logistic Regression, fixed weight, and CombSUM outperform the other composers across all three datasets, while decision tree and random forest exhibited subpar performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability

Artifacts of the experiment are available in an online replication package: https://github.com/feifeiniu-se/Replication2.

Notes

  1. https://github.com/feifeiniu-se/Replication2

  2. The other four projects from the original dataset were excluded due to some missing commits on GitHub.

  3. https://sites.google.com/view/mambalab/projects/amalgam

  4. https://github.com/exatoa/bench4bl

  5. https://sourceforge.net/projects/lemur/files/lemur/

  6. Since the main improvement of TraceScore over SimiScore lies in the leveraging of requirements and traceability information, but there is no such information in Extended-Python dataset, we do not compare the improvement over the Extended-Python dataset

References

  • Akbar SA, Kak AC (2020) A large-scale comparative evaluation of ir-based tools for bug localization. In: 17th International Conference on Mining Software Repositories, pp. 21–31

  • Al-Aidaroos AS, Bamzahem SM (2023) The impact of glove and word2vec word-embedding technologies on bug localization with convolutional neural network. International Journal of Science and Engineering Applications, 108–111

  • Anh BTM, Luyen NV (2021) An imbalanced deep learning model for bug localization. In: 2021 28th Asia-Pacific Software Engineering Conference Workshops (APSEC Workshops), pp. 32–40. IEEE

  • Anvik J, Hiew L, Murphy GC (2005) Coping with an open bug repository. In: 2005 OOPSLA Workshop on Eclipse Technology eXchange, pp. 35–39

  • Anvik J, Hiew L, Murphy GC (2006) Who should fix this bug? In: 28th International Conference on Software Engineering, pp. 361–370

  • Aslam JA, Montague MH (2001) Models for metasearch. In: Croft, W.B., Harper, D.J., Kraft, D.H., Zobel, J. (eds.) SIGIR 2001: 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, September 9-13, 2001, New Orleans, Louisiana, USA, pp. 275–284. ACM. https://doi.org/10.1145/383952.384007

  • Aslan Ö, Samet R (2017) Mitigating cyber security attacks by being aware of vulnerabilities and bugs. In: 2017 International Conference on Cyberworlds (cw), pp. 222–225. IEEE

  • Bachmann A, Bernstein A (2009) Software process data quality and characteristics: a historical view on open and closed source projects. In: Joint International and Annual ERCIM Workshops on Principles of Software Evolution (IWPSE) and Software Evolution (Evol) Workshops, pp. 119–128

  • Baeza-Yates R, Ribeiro-Neto B et al (1999) Modern Information Retrieval vol. 463. ACM press New York

  • Benton S, Ghanbari A, Zhang L (2019) Defexts: A curated dataset of reproducible real-world bugs for modern jvm languages. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 47–50. IEEE

  • Bettenburg N, Premraj R, Zimmermann T, Kim S (2008) Duplicate bug reports considered harmful... really? In: 2008 IEEE International Conference on Software Maintenance, pp. 337–345. IEEE

  • Cao J, Yang S, Jiang W, Zeng H, Shen B, Zhong H (2020) Bugpecker: Locating faulty methods with deep learning on revision graphs. In: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pp. 1214–1218

  • Carver JC (2010) Towards reporting guidelines for experimental replications: A proposal. In: 1st International Workshop on Replication in Empirical Software Engineering, vol. 1, pp. 1–4

  • Chris Lewis RO. Bug Prediction at Google. [EB/OL]. http://google-engtools.blogspot.com/2011/12/bug-prediction-at-google.html Accessed 12, 2011

  • Ciborowska A, Damevski K (2022) Fast changeset-based bug localization with bert. In: 44th International Conference on Software Engineering, pp. 946–957

  • Ciborowska A, Damevski K (2022) Fast changeset-based bug localization with bert. In: Proceedings of the 44th International Conference on Software Engineering, pp. 946–957

  • Da Silva FQ, Suassuna M, França ACC, Grubb AM, Gouveia TB, Monteiro CV, Santos IE (2014) Replication of empirical studies in software engineering research: a systematic mapping study. Empir Softw Eng 19(3):501–557

    Google Scholar 

  • Fox EA, Koushik MP, Shaw J, Modlin R, Rao D et al (1993) Combining evidence from multiple searches. In: The First Text Retrieval Conference (TREC-1), pp. 319–328

  • Fox E, Shaw J (1994) Combination of multiple searches. NIST special publication SP, 243–243

  • Garnier M, Garcia A (2016) On the evaluation of structured information retrieval-based bug localization on 20 c# projects. In: XXX Brazilian Symposium on Software Engineering, pp. 123–132

  • Gómez OS, Juristo N, Vegas S (2014) Understanding replication of experiments in software engineering: A classification. Inf Softw Technol 56(8):1033–1048

    Article  Google Scholar 

  • Haben G, Habchi S, Papadakis M, Cordy M, Le Traon Y (2021) A replication study on the usability of code vocabulary in predicting flaky tests. In: 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pp. 219–229. IEEE

  • Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. ACM SIGKDD Explorations Newsl 11(1):10–18

    Article  Google Scholar 

  • Han J, Huang C, Sun S, Liu Z, Liu J (2023) bjxnet: an improved bug localization model based on code property graph and attention mechanism. Autom Softw Eng 30(1):12

    Article  Google Scholar 

  • Huo X, Thung F, Li M, Lo D, Shi S-T (2019) Deep transfer bug localization. IEEE Trans Software Eng 47(7):1368–1380

    Article  Google Scholar 

  • Imblearn (2022) Imblearn. https://imbalanced-learn.org/stable/

  • Jeffrey D, Feng M, Gupta N, Gupta R (2009) Bugfix: A learning-based tool to assist developers in fixing bugs. In: 2009 IEEE 17th International Conference on Program Comprehension, pp. 70–79. IEEE

  • JIRA (2018) Jira Issue Tracking Software. https://www.jira.com

  • Jones KS (1972) A statistical interpretation of term specificity and its application in retrieval. Journal of documentation

  • Kim S, Zimmermann T, Whitehead Jr EJ, Zeller A (2007) Predicting faults from cached history. In: 29th International Conference on Software Engineering (ICSE’07), pp. 489–498. IEEE

  • Lam AN, Nguyen AT, Nguyen HA, Nguyen TN (2015) Combining deep learning with information retrieval to localize buggy files for bug reports (n). In: 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 476–481. IEEE

  • Lam AN, Nguyen AT, Nguyen HA, Nguyen TN (2017) Bug localization with combination of deep learning and information retrieval. In: 2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC), pp. 218–229 . IEEE

  • B. Le TD, Lo D, Le Goues C, Grunske L (2016) A learning-to-rank based fault localization approach using likely invariants. In: 25th International Symposium on Software Testing and Analysis, pp. 177–188

  • Lee J, Kim D, Bissyandé TF, Jung W, Traon YL (2018) Bench4bl: Reproducibility study of the performance of ir-based bug localization. In: 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. ISSTA 2018, pp. 1–12. https://doi.org/10.1145/3213846.3213856

  • Lewis C, Lin Z, Sadowski C, Zhu X, Ou R, Whitehead EJ (2013) Does bug prediction support human developers? findings from a google case study. In: 2013 35th International Conference on Software Engineering (ICSE), pp. 372–381. IEEE

  • Li W, Li Q, Ming Y, Dai W, Ying S, Yuan M (2022) An empirical study of the effectiveness of ir-based bug localization for large-scale industrial projects. Empir Softw Eng 27(2):1–31

    Article  Google Scholar 

  • Li G, Liu H, Chen X, Gunawi HS, Lu S (2019) Dfix: automatically fixing timing bugs in distributed systems. In: 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 994–1009

  • Loyola P, Gajananan K, Satoh F (2018) Bug localization by learning to rank and represent bug inducing changes. In: 27th ACM International Conference on Information and Knowledge Management, pp. 657–665

  • Lucia, Lo D, Xia X (2014) Fusion fault localizers. In: 29th ACM/IEEE International Conference on Automated Software Engineering, pp. 127–138

  • Lukins SK, Kraft NA, Etzkorn LH (2010) Bug localization using latent dirichlet allocation. Inf Softw Technol 52(9):972–990

    Article  Google Scholar 

  • Manning CD, Raghavan P, Schütze H (2008) Introduction to Information Retrieval. Cambridge university press

  • Massey FJ Jr (1951) The kolmogorov-smirnov test for goodness of fit. J Am Stat Assoc 46(253):68–78

    Article  Google Scholar 

  • McMillan C, Grechanik M, Poshyvanyk D, Fu C, Xie Q (2011) Exemplar: A source code search engine for finding highly relevant applications. IEEE Trans Software Eng 38(5):1069–1087

    Article  Google Scholar 

  • Muvva S, Rao AE, Chimalakonda S (2020) Bugl–a cross-language dataset for bug localization. arXiv:2004.08846

  • Niu F, Assunçao WKG, Huang L, Mayr-Dorn C, Ge J, Luo B, Egyed A (2023) Rat: A refactoring-aware traceability model for bug localization. In: 2023 45th International Conference on Software Engineering (ICSE). IEEE. accepted

  • Niu F, Mayr-Dorn C, Assunção WK, Huang L, Ge J, Luo B, Egyed A (2023) The ablots approach for bug localization: is it replicable and generalizable? In: 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pp. 576–587. IEEE

  • NLTK (2022) NLTK library. http://www.nltk.org

  • Piessens F (2002) A taxonomy of causes of software vulnerabilities in internet software. In: Supplementary 13th International Symposium on Software Reliability Engineering, pp. 47–52. Citeseer

  • Qi B, Sun H, Yuan W, Zhang H, Meng X (2021) Dreamloc: A deep relevance matching-based framework for bug localization. IEEE Trans Reliab 71(1):235–249

    Article  Google Scholar 

  • Rahman F, Posnett D, Hindle A, Barr E, Devanbu P (2011) Bugcache for inspections: hit or miss? In: 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, pp. 322–331

  • Rath M, Mäder P (2019) The seoss 33 dataset–requirements, bug reports, code history, and trace links for entire projects. Data Brief 25:104005

    Article  Google Scholar 

  • Rath M, Lo D, Mäder P (2018) Analyzing requirements and traceability information to improve bug localization. In: 15th International Conference on Mining Software Repositories, pp. 442–453

  • Rath M, Lo D, Mäder P (2018) Replication Data for: Analyzing Requirements and Traceability Information to Improve Bug Localization

  • Saha RK, Lease M, Khurshid S, Perry DE (2013) Improving bug localization using structured information retrieval. In: 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 345–355. IEEE

  • Sangle S, Muvva S, Chimalakonda S, Ponnalagu K, Venkoparao VG (2020) Drast–a deep learning and ast based approach for bug localization. arXiv:2011.03449

  • Schütze H, Manning CD, Raghavan P (2008) Introduction to Information Retrieval vol. 39. Cambridge University Press Cambridge

  • scikit-learn (2022) scikit-learn. https://scikit-learn.org/stable/

  • SCM G (2018) Git SCM. https://www.git-scm.com

  • Shepperd M, Ajienka N, Counsell S (2018) The role and value of replication in empirical software engineering results. Inf Softw Technol 99:120–132

    Article  Google Scholar 

  • Shi X, Ju X, Chen X, Lu G, Xu M (2022) Semirfl: Boosting fault localization via combining semantic information and information retrieval. In: 2022 IEEE 22nd International Conference on Software Quality, Reliability, and Security Companion (QRS-C), pp. 324–332. IEEE

  • Sisman B, Kak AC (2012) Incorporating version histories in information retrieval based bug localization. In: 2012 9th IEEE Working Conference on Mining Software Repositories (MSR), pp. 50–59. IEEE

  • Strohman T, Metzler D, Turtle H, Croft WB (2005) Indri: A language model-based search engine for complex queries. In: International Conference on Intelligent Analysis, vol. 2, pp. 2–6. Citeseer

  • Tan L, Liu C, Li Z, Wang X, Zhou Y, Zhai C (2014) Bug characteristics in open source software. Empir Softw Eng 19(6):1665–1705

    Article  Google Scholar 

  • Voorhees EM et al (1999) The trec-8 question answering track report. Trec 99:77–82

    Google Scholar 

  • Wang S, Lo D (2016) Amalgam+: Composing rich information sources for accurate bug localization. Journal of Software: Evolution and Process. 28(10):921–942

    Google Scholar 

  • Wang B, Xu L, Yan M, Liu C, Liu L (2020) Multi-dimension convolutional neural network for bug localization. IEEE Trans Serv Comput 15(3):1649–1663

    Article  Google Scholar 

  • Wang S, Lo D (2014) Version history, similar report, and structure: Putting them together for improved bug localization. In: 22nd International Conference on Program Comprehension, pp. 53–63

  • Wen M, Wu R, Cheung S-C (2016) Locus: Locating bugs from software changes. In: 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 262–273. IEEE

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in Software Engineering: An Introduction. Kluwer Academic Publishers, USA. https://doi.org/10.1007/978-1-4615-4625-2

    Article  Google Scholar 

  • Wong C-P, Xiong Y, Zhang H, Hao D, Zhang L, Mei H (2014) Boosting bug-report-oriented fault localization with segmentation and stack-trace analysis. In: 2014 IEEE International Conference on Software Maintenance and Evolution, pp. 181–190. IEEE

  • Wu S (2012) Data Fusion in Information Retrieval. Adaptation, Learning, and Optimization, vol. 13. Springer. https://doi.org/10.1007/978-3-642-28866-1

  • Xiao Y, Keung J, Bennin KE, Mi Q (2019) Improving bug localization with word embedding and enhanced convolutional neural networks. Inf Softw Technol 105:17–29

    Article  Google Scholar 

  • Xiao X, Xiao R, Li Q, Lv J, Cui S, Liu Q (2023) Bugradar: Bug localization by knowledge graph link prediction. Information and Software Technology, 107274

  • Xu G, Wang X, Wei D, Shao Y, Chen B (2023) Bug localization with features crossing and structured semantic information matching. International Journal of Software Engineering and Knowledge Engineering

  • Yang S, Cao,J, Zeng H, Shen B, Zhong H (2021) Locating faulty methods with a mixed rnn and attention model. In: 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC), pp. 207–218. IEEE

  • Ye X, Bunescu R, Liu C (2014) Learning to rank relevant files for bug reports using domain knowledge. In: 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 689–699

  • Yong J, Zhu Z, Li Y (2023) Decomposing source codes by program slicing for bug localization. In: 2023 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE

  • Youm KC, Ahn J, Kim J, Lee E (2015) Bug localization based on code change histories and bug reports. In: 2015 Asia-Pacific Software Engineering Conference (APSEC), pp. 190–197. IEEE

  • Zhang H (2009) An investigation of the relationships between lines of code and defects. In: 2009 IEEE International Conference on Software Maintenance, pp. 274–283. IEEE

  • Zhou J, Zhang H, Lo D (2012) Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports. In: 2012 34th International Conference on Software Engineering (ICSE), pp. 14–24 . IEEE

Download references

Acknowledgements

This work is supported by Natural Science Foundation of Jiangsu Province, China (BK20201250), Cooperation Fund of Huawei-NJU Creative Laboratory for the Next Programming, and also supported in part by NSF Grant 2034508 (USA), by a Sam Taylor Fellowship Award, the Austrian Science Fund (FWF) grant P31989-N31 and P34805-N as well as the LIT Secure and Correct System Lab sponsored by the province of Upper Austria.

Author information

Authors and Affiliations

  1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China

    Feifei Niu, Enshuo Zhang, Jidong Ge & Bin Luo

  2. Institute for Software Systems Engineering, Johannes Kepler University, Linz, Austria

    Christoph Mayr-Dorn & Alexander Egyed

  3. Department of Computer Science, North Carolina State University, North Carolina, USA

    Wesley Klewerton Guez Assunção

  4. Department of Computer Science, Southern Methodist University, Dallas, Texas, USA

    Liguo Huang

Authors
  1. Feifei Niu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Enshuo Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Christoph Mayr-Dorn
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Wesley Klewerton Guez Assunção
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Liguo Huang
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Jidong Ge
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Bin Luo
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Alexander Egyed
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Feifei Niu.

Ethics declarations

Conflict of interest

The authors declared that they have no conflict of interest.

Additional information

Communicated by: Patanamon Thongtanunam,Bogdan Vasilescu, Emad Shihab

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Mining Software Repositories (MSR).

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Niu, F., Zhang, E., Mayr-Dorn, C. et al. An extensive replication study of the ABLoTS approach for bug localization. Empir Software Eng 29, 143 (2024). https://doi.org/10.1007/s10664-024-10537-6

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-024-10537-6

Keywords

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy