See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/224503795

Improving Bug Tracking Systems

Conference Paper in Proceedings - International Conference on Software Engineering · June 2009

DOI: 10.1109/ICSE-COMPANION.2009.5070993 · Source: IEEE Xplore

CITATIONS READS
57 7,918

4 authors, including:

R. Premraj Jonathan Sillito

Universität des Saarlandes The University of Calgary
38 PUBLICATIONS 3,289 CITATIONS 34 PUBLICATIONS 1,853 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by R. Premraj on 03 June 2014.

The user has requested enhancement of the downloaded file.

 Improving Bug Tracking Systems

Thomas Zimmermann1,2 Rahul Premraj3 Jonathan Sillito2 Silvia Breu4

tz@acm.org rpremraj@cs.vu.nl sillito@ucalgary.ca silvia.breu@cl.cam.ac.uk
1 Microsoft Research, Redmond, USA
2 Department of Computer Science, University of Calgary, Canada
3 Vrije Universiteit, Amsterdam, The Netherlands
4 Computer Laboratory, University of Cambridge, UK

Abstract resolve bugs [4]. Items such as stack traces, steps to re-
produce, observed and expected behavior, test cases, and
It is important that information provided in bug reports is screenshots ranked high on the list of preferred information
relevant and complete in order to help resolve bugs quickly. by developers.
However, often such information trickles to developers af- Previous research has shown that reporters often omit
ter several iterations of communication between develop- these important items [4, 6]. Developers are then forced to
ers and reporters. Poorly designed bug tracking systems actively solicit information from reporters and, depending
are partly to blame for this exchange of information being on their responsiveness, this may stall development. The
stretched over time. Our paper addresses the concerns of effect of this delay is that bugs take longer to be fixed and
bug tracking systems by proposing four broad directions for more and more unresolved bugs accumulate in the project’s
enhancements. As a proof-of-concept, we also demonstrate bug tracking system. We believe that one reason for this
a prototype interactive bug tracking system that gathers rel- problem is that current bug tracking systems are merely in-
evant information from the user and identifies files that need terfaces to relational databases that store the reported bugs.
to be fixed to resolve the bug. They provide little or no support to reporters to help them
provide the information that developers need.
Our on-going work is directed towards rectifying the sit-
uation by proposing ideas that can fundamentally transform
1. Introduction
bug tracking systems to enhance their usability such that
relevant information is easier to gather and report by a ma-
Alice, a real person: My ECLIPSE crashed. jority of users (Section 2). Further, we present a simulation
Bob, a bug-tracking system: What did you do?
of an interactive bug tracking system, using a decision tree,
Alice: I clicked on File → New Project and OK.
which extracts relevant information from the user to iden-
Bob: Did you choose a Java project?
Alice: No. tify the file that contains the bug, as in the above discourse,
as a means to help developers. (Section 3). The paper closes
. . . (a few questions later)
with a look at related work (Section 4) followed by conclu-
Bob: Thanks Alice. The bug is likely in ProjectCreator.java sions (Section 5).
and we will fix it soon.

The use of bug tracking systems as a tool to organize main- 2. Better bug tracking systems
tenance activities is widespread. The systems serve as a
central repository for monitoring the progress of bug re- Having complete information in the initial bug report (or as
ports, requesting additional information from reporters, and soon as possible) helps developers to quickly resolve the
discussing potential solutions for fixing the bug. Develop- bug. The focus of our work is on improving bug track-
ers use the information provided in bug reports to identify ing systems with the goal of increasing the completeness
the cause of the defect, and narrow down plausible files of bug reports. Specifically, we are working on improving
that need fixing. A survey conducted amongst developers bug tracking systems in four ways (see Figure 1):
from the APACHE, ECLIPSE, and MOZILLA projects found
out which information items are considered useful to help 1 Many thanks to Sascha Just for creating the template for Figure 1.
 User-centric. This includes both reporters and developers.
Reporters can be educated on what information to pro-
vide and how to collect it. Developers too can benefit
Tool Information from similar training on what information to expect in
Centric Centric
bug reports and how this information can be used to
resolve bugs.
Improving
bug tracking The following section presents a prototype developed
around the information-centric direction that gathers infor-
systems mation from reporters to identify candidate files to be fixed.

User Process 3. Asking the right questions in bug reports

Centric Centric

When a user submits a bug report she is asked many ques-

tions: What is the name of the product? In which plug-
in/component? What is the Build ID? What is the bug
Figure 1. Improving bug tracking systems.1 about? What are the steps to reproduce the bug? Any ad-
ditional information? However, the initial information pro-
vided in a bug report is often incomplete and developers of-
ten have follow-up questions: Do you have flash installed?
Tool-centric. Tool-centric enhancements are made to the
Can you provide a screenshot? Getting replies by users
features provided by bug tracking systems. They can
takes time (often weeks) and slows down the progress on
help to reduce the burden of information collection and
open bugs. Furthermore, research has shown that develop-
provision. For example, bug tracking systems can be
ers get responses to only two out of three questions [6].
configured to automatically locate the relevant stack
Ideally, follow-up questions would be asked immediately
trace and add it to a bug report. Also, providing steps
after a user has submitted a bug report and is still within
to reproduce can be automated by using capture/replay
reach. We therefore propose to build expert systems that
tools or macro-recorders; observed behavior can be
automatically ask relevant questions and gather all required
easily shown by simplifying screen capture; test cases
information once an initial failure description has been pro-
can be automatically generated [3]. All of the above
vided. The selection and order of questions should not be
examples aim to help with the collection of informa-
static (as in current bug tracking systems), but rather de-
tion needed by developers to fix bugs.
pend on previous responses. We believe that such an expert
Information-centric. These improvements focus directly system can provide better bug descriptions and also narrow
on the information being provided by the reporter. Bug down the location of the defect.2
tracking systems can be embedded with tools such as To create such a system the following data is needed:
CUEZILLA [4] that provide real-time feedback on the
1. Information that developers use to find the location of
quality of the information provided and what can be
a defect. For each kind of information one can then
added to increase value. With helpful reminders from
formulate a question.3 Some might be general such
the system, reporters may be motivated to go the ex-
as asking for screenshots, build identifiers, and stack
tra mile and collect more information. Systems can
traces, but other questions might be more specific, for
be further modified to perform validity checks such as
example asking about certain menus or dialogs.
verifying whether the reported stack trace is consistent
and complete, submitted patches are valid, and the like. 2. The defect location and question/answer-pairs for a
large number of fixed bugs. One can then use these
Process-centric. Process-centric improvements to bug
bugs to build machine learning models (e.g., decision
tracking systems focus on administration of activities
trees, neural networks) which select questions and pre-
related to bug fixing. For example, bug triaging, i.e.,
dict the location of the defect based on the responses.
determining which developer should resolve the bug,
can be automated to speed up the task [1, 5]. Other ex- 2 One can think of this approach to bug reporting as a computerized ver-

amples are better awareness of the progress made on sion of the twenty questions game. The bug tracking system asks the user
several questions (one question at a time) and the user provides responses.
bug reports (so that reporters are aware of the actions After this process is finished, the developer would have a description of the
taken as a response for their efforts) or to provide users failure and ideally the bug tracking system could make a good guess at the
with an early estimate for when their bug will be fixed. location of the defect.
In this paper, we show an early proof-of-concept study that In Figure 2, the path that leads to JaveEditor.java would
uses data that is readily available in bug reports (such as ask only about the component (Text). In contrast the path
component, version, reporter). We are currently working to ASTConverter15Test.java would ask about component
on compiling a catalog of information frequently needed by (Core, UI), version (3.1 or higher), and the reporter (R4).
developers. Once we have this catalog we will implement a This justifies the need for interactive bug tracking systems
tool and conduct a fully-fledged evaluation. (rather than static web-pages) because for certain responses
fewer questions are needed. With more questions and more
3.1. A first experiment files we expect the decision tree to become more complex.
We also expect that the order of questions will change for
To check whether question/answer-pairs can predict the lo- different paths in the tree.
cation where a bug has been fixed, we conducted the fol-
lowing experiment. For the twenty most frequently fixed 3.2. Next steps and challenges
files in ECLIPSEJDK we selected all related bug reports. We
then used the resulting 2,875 cases to train a decision tree We showed that for some of the questions asked in bug re-
to predict the location of the fix (name of the file).4 ports (component, version), the responses can potentially
As input features, we used the following questions predict the location of the defect. Our study is very prelim-
for which we could extract the answers easily from the inary, and our next steps will be the following:
ECLIPSE bug database.
• Identify information needs in a large sample of bug
• How severe is the bug? (bug severity)
reports through manual inspection. This will help to
• On what operating system does it occur? (op sys) compile a catalog of questions that can be used for the
• What is the affected component? (component id) expert system.
• How important is the bug? (priority) • Using this catalog, collect answers and defect locations
• Which version of ECLIPSE is affected? (version) for another large sample of bug reports. This dataset
will be used to automatically learn a prediction model.
• What is your name?5 (reporter)
• Evaluate the predictions and conduct usability studies.
• What platforms are affected? (rep platform)
The resulting decision tree is in Figure 2. The root node One of the challenges for this project will be to find a
shows the defect distribution for all twenty selected files, representative catalog of questions that is able to predict
e.g., 11% of bugs have been fixed in GenericTypeTest.java. defects. In addition, scalability will become an issue once
For all other nodes, we report only the three files with the more questions with many unique values are used.
highest likelihood.
Out of the seven input features, only three influenced the 4. Related work
location of a defect. The feature with the most influence is
component id, which comes as no surprise because compo-
Many researchers have investigated what factors aid or
nents almost directly map to source code. The next most
indicate the quick resolution of bugs. Hooimeijer and
influential feature is version, which indicates that defect-
Weimer [8] observed that bug reports with more comments
prone hot-spots in components change over time.
get fixed sooner. They also noted that bug reports that are
The reporter of a bug also predicts the location of the
easier to read also have shorter life times (also confirmed by
defect, likely because reporters use ECLIPSE differently and
our previous work [4]). More recently, Aranda and Venolia
thus trigger bugs in different parts. For example in Figure 2,
have examined communication that takes place between de-
the reporter set R1 mostly reveals bugs related to Java for-
velopers outside the bug tracking system [2]. In our earlier
matting, while R2 reveals bugs related to AST conversions
work, we discussed shortcomings of existing bug tracking
and R3 reveals bugs related to the ECLIPSE Java model.6
systems [9], which led to the four areas of improvements
3 The user can be an end-user or a tester and both would require different presented in this paper. Asking the right questions, is a cru-
sets of questions. For example, an end-user will not be familiar with the cial part of debugging and several techniques such as al-
internal structure of a system, while a tester might be.
4 We decided to use a decision tree for illustrative purposes. It is possi- gorithmic debugging [7] and the WhyLine tool [10] support
ble and likely that neural networks or other machine learning models will developers in doing so. In contrast, we propose to move this
yield better results. process from the developer side to the user side. Instead of
5 Decision trees are computationally expensive when an input feature
recorded feedback by many users as it happens in Collab-
can take many values, which is the case for reporter. We considered there-
fore only the twenty most active reporters and modeled less active reporters
orative Bug Isolation [11], our proposed approach requires
as “other”. explicit feedback by only a single user.
6 For privacy reasons, we omit the reporter names in Figure 2.
 ASTConverter15Test.java (0.056) GenericTypeTest.java (0.11) JDIThread.java (0.043)
ASTConverter.java (0.069) JavaCore.java (0.044) MatchLocator.java (0.033)
ASTConverterTest2.java (0.036) JavaEditor.java (0.043) MethodVerifyTest.java (0.035)
CodeFormatterVisitor.java (0.049) JavaModelManager.java (0.034) Parser.java (0.055)
CompletionEngine.java (0.048) JavaProject.java (0.036) ProblemReporter.java (0.056)
DefaultBindingResolver.java (0.038) JavaRuntime.java (0.043) Scope.java (0.068)
FormatterRegressionTests.java (0.055) JDIModelPresentation.java (0.05)

component_id=Text

component_id=Core,UI component_id=Debug

ASTConverter.java (0.084) CompletionEngine.java (0.0095) JavaRuntime.java (0.31)

GenericTypeTest.java (0.13) JavaCore.java (0.0095) JDIModelPresentation.java (0.37)
Scope.java (0.082) JavaEditor.java (0.98) JDIThread.java (0.31)

version=1.0,2.0,2.1.1 version=3.1 or higher

version=2.0.2,2.1,2.1.1,3.2
version=2.0,2.1.2,3.0,3.1
version=2.0.1,2.0.2,2.1,2.1.2,3.0,3.0.1,3.0.2

ASTConverter.java (0.31) ASTConverter.java (0.082) ASTConverter15Test.java (0.097) JavaRuntime.java (0.47) JavaRuntime.java (0.26)
DefaultBindingResolver.java (0.25) FormatterRegressionTests.java (0.12) GenericTypeTest.java (0.23) JDIModelPresentation.java (0.18) JDIModelPresentation.java (0.43)
Parser.java (0.13) Parser.java (0.08) Scope.java (0.12) JDIThread.java (0.35) JDIThread.java (0.3)

reporter=R1 reporter=R2 reporter=R3 reporter=R4 reporter=R5

FormatterRegressionTests.java (0.2) ASTConverter15Test.java (0.14) JavaCore.java (0.093) ASTConverter15Test.java (0.17) GenericTypeTest.java (0.32)
JavaCore.java (0.078) ASTConverter.java (0.16) JavaModelManager.java (0.14) GenericTypeTest.java (0.097) MethodVerifyTest.java (0.11)
Parser.java (0.076) ASTConverterTest2.java (0.11) JavaProject.java (0.16) ProblemReporter.java (0.089) Scope.java (0.17)

Figure 2. The decision tree illustrates how questions could narrow down the location of a defect.
Each path corresponds to a question/answer series. The file in bold is the likely defect location.

5. Conclusions and consequences [2] J. Aranda and G. Venolia. The secret life of bugs: Going past the
errors and omissions in software repositories. In ICSE’09: Proceed-
ings of the 31st International Conference on Software Engineering
Current bug tracking systems do not effectively elicit all of (to appear), 2009.
the information needed by developers. Without this infor- [3] S. Artzi, S. Kim, and M. D. Ernst. Recrash: Making software fail-
mation developers cannot resolve bugs in a timely fashion ures reproducible by preserving object states. In ECOOP’08: Pro-
ceedings of the 22nd European Object-Oriented Programming Con-
and so we believe that improvement to the way bug tracking ference, pages 542–565, 2008.
systems collect information are needed. [4] N. Bettenburg, S. Just, A. Schröter, C. Weiss, R. Premraj, and
This paper proposes four broad areas for improvements. T. Zimmermann. What makes a good bug report? In FSE’08:
Proceedings of the 16th International Symposium on Foundations
While implementing a range of improvements from these of Software Engineering, pages 308–318, November 2008.
areas may be ideal, bug tracking systems may instead pre- [5] N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim. Duplicate
fer to specialize, thus providing a rich set of choices. This bug reports considered harmful ... really? In ICSM’08: Proceed-
ings of the 24th IEEE International Conference on Software Main-
would be a healthy change to the current situation where tenance, pages 337–345, 2008.
they all provide identical functionality. [6] S. Breu, J. Sillito, R. Premraj, and T. Zimmermann. Frequently
As an example of the kind of improvements we envi- asked questions in bug reports. Technical report, University of Cal-
sion, we have described an interactive system for collecting gary, March 2009.
[7] P. Fritzson, T. Gyimothy, M. Kamkar, and N. Shahmehri. General-
information from reporters and leveraging that information ized algorithmic debugging and testing. In PLDI’91: Proceedings
to locate the source of the bug. To demonstrate the potential of the ACM SIGPLAN Conference on Programming Language De-
of this idea we have conducted an initial study in which we sign and Implementation, pages 317–326, 1991.
[8] P. Hooimeijer and W. Weimer. Modeling bug report quality. In
simulated an interactive bug tracking system. The system ASE’07: Proceedings of the 22nd International Conference on Au-
asks the user context-sensitive questions to extract relevant tomated Software Engineering, pages 34–43, 2007.
information about the bug early on to suggest candidate files [9] S. Just, R. Premraj, and T. Zimmermann. Towards the next gener-
ation of bug tracking systems. In VL/HCC’08: Proceedings of the
that will need to be fixed. This is likely to speed up the pro- 2008 IEEE Symposium on Visual Languages and Human-Centric
cess of resolving bugs. In the future, we will move from the Computing, pages 82–85, September 2008.
current prototype of the interactive system to a full-scale [10] A. J. Ko and B. A. Myers. Debugging reinvented: asking and an-
swering why and why not questions about program behavior. In
system that can deal with a variety of information to gather, ICSE’08: Proceedings of the International Conference on Software
as commonly observed in the real world. Engineering, pages 301–310, 2008.
[11] B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan. Scal-
able statistical bug isolation. In PLDI’05: Proceedings of the 2005
References ACM SIGPLAN Conference on Programming Language Design and
Implementation, pages 15–26, 2005.

[1] J. Anvik, L. Hiew, and G. C. Murphy. Who should fix this bug?
In ICSE’06: Proceedings of the 28th International Conference on
To learn more about our work on bug tracking systems, we
Software engineering, pages 361–370, 2006. invite you to visit http://www.softevo.org/

View publication stats