CSE 403: Software Engineering, Spring 2015

courses.cs.washington.edu/courses/cse403/15sp/

Unit Testing

Emina Torlak
emina@cs.washington.edu
Outline

• Software quality control

• Effective unit testing
• Coverage and regression testing

2
intro
basics of software quality control
Errors and faults

Ariane 5: 37 sec after

launch. Cost: $1 billion.

4
Errors and faults

• Error: incorrect software behavior

• Example: Software controller for the Ariane 5 rocket
crashed (and so did the rocket).

Ariane 5: 37 sec after

launch. Cost: $1 billion.

4
Errors and faults

• Error: incorrect software behavior

• Example: Software controller for the Ariane 5 rocket
crashed (and so did the rocket).

• Fault: mechanical or algorithmic cause of error (bug)

• Example: Conversion from 64-bit floating point to 16-bit
signed integer value caused an exception.
• Requirements specify desired behavior; if the system
deviates from that, it has a fault.

Ariane 5: 37 sec after

launch. Cost: $1 billion.

4
Software quality control techniques

• Fault avoidance: prevents errors before the

system is released.
• reviews, inspections, walkthroughs, development
methodologies, testing, verification

• Fault tolerance: enables the system to

recover from (some classes of) errors by itself.
• rollbacks, redundancy, mirroring

5
Showing the presence and absence of bugs …

testing verification

6
Showing the presence and absence of bugs …

Detects the presence of bugs

by running the code on a few
carefully chosen inputs.

testing verification

6
Showing the presence and absence of bugs …

Detects the presence of bugs

Shows the absence of bugs on
by running the code on a few
all possible inputs.
carefully chosen inputs.

testing verification

6
Common kinds of testing

• Unit testing: tests the behavior of an individual

module (method, class, interface)
• Black-box testing
• White-box testing

• System testing: tests the behavior of the system as

a whole, with respect to scenarios and requirements
• Functional testing, integration testing
• Performance, load, stress testing
• Acceptance, usability, installation, beta testing

7
unit
effective unit testing
Two rules of unit testing

• Do it early and do it often

• Catch bugs quickly, before they have a chance to hide
• Automate the process if you can

• Be systematic
• If you thrash about arbitrarily, the bugs will hide in
the corner until you're gone

9
Four basic steps of a test

10
Four basic steps of a test

1. Choose input data

• without looking at the implementation: black box
• with knowledge of the implementation: white box

10
Four basic steps of a test

1. Choose input data

• without looking at the implementation: black box
• with knowledge of the implementation: white box

2. Define the expected outcome

10
Four basic steps of a test

1. Choose input data

• without looking at the implementation: black box
• with knowledge of the implementation: white box

2. Define the expected outcome

3. Run on the input to get the actual outcome

10
Four basic steps of a test

1. Choose input data

• without looking at the implementation: black box
• with knowledge of the implementation: white box

2. Define the expected outcome

3. Run on the input to get the actual outcome
4. Compare the actual and expected outcomes

10
Four basic steps of a test

This is hard! Need a set

1. Choose input data of test cases that is small
• without looking at the implementation: black box enough to run quickly, yet
• with knowledge of the implementation: white box large enough to cover [all]
interesting program
2. Define the expected outcome behaviors.
3. Run on the input to get the actual outcome
4. Compare the actual and expected outcomes

10
Choosing inputs: two key ideas

11
Choosing inputs: two key ideas

• Partition the input space

• Identify subdomains with the same behavior
• Pick one input from each subdomain

11
Choosing inputs: two key ideas

• Partition the input space

• Identify subdomains with the same behavior
• Pick one input from each subdomain

• Boundary values
• Pick inputs at the edges of the subdomains.
• Effective at finding corner case bugs:
• off-by-one, overflow, aliasing, empty container

11
Partitioning the input space

// returns the maximum of a, b

public static int max(int a, int b) { … }

• Partition into
• a < b, a = b, a > b

• Pick an input from each class

• (1, 2), (0, 0), (2, 1)

12
Partitioning the input space

// returns the maximum of a, b

public static int max(int a, int b) { … }

• Partition into
• a < b, a = b, a > b
How would you partition the
input space for
• Pick an input from each class • BigInteger multiplication?
• (1, 2), (0, 0), (2, 1) • Set intersection?

12
Choosing boundary values

// returns|x|
public static int abs(int a) { … }

• Partition into
• a < 0, a > 0, a = 0 (boundary)

• Other boundary values

• Integer.MAX_VALUE
• Integer.MIN_VALUE

13
Choosing boundary values

// returns|x|
public static int abs(int a) { … }

• Partition into
• a < 0, a > 0, a = 0 (boundary)
What are good boundary
• Other boundary values values for objects?
• Integer.MAX_VALUE
• Integer.MIN_VALUE

13
Black box testing

• Explores alternate paths through the specification.

• Module under test is a black box: interface visible,
internals hidden.

// If a >= b, returns a. Otherwise returns b.

public static int max(int a, int b) { … }

• 3 paths, so 3 subdomains
• (1, 2) => 2
• (2, 1) => 2
• (0, 0) => 0

14
Advantages of black box testing

• Process is not influenced by component being tested

• Assumptions embodied in code not propagated to test data.

• Robust with respect to changes in implementation

• Test data need not be changed when code is changed

• Allows for independent testers

• Testers need not be familiar with code

15
Disadvantage of black box testing

• It will miss bugs in the implementation that are not

covered by the specification
• Control-flow details
• Performance optimizations
• Alternate algorithms for different cases

16
White box testing

• Explores alternate paths through the implementation

• Module under test is a clear box: internals visible.

boolean[] primeTable = new boolean[CACHE_SIZE];

boolean isPrime(int x) {
if (x>CACHE_SIZE) {
for (int i=2; i<x/2; i++) {
if (x%i==0) return false;
}
return true;
} else {
return primeTable[x];
}
}

• Important transition at around x = CACHE_SIZE

17
(Dis)advantages of white box testing

18
(Dis)advantages of white box testing

• Advantages
• Finds an important class of boundaries.
• Yields useful test cases.
• In isPrime example, need to check numbers on each
side of CACHE_SIZE
• CACHE_SIZE-1, CACHE_SIZE, CACHE_SIZE+1

18
(Dis)advantages of white box testing

• Advantages
• Finds an important class of boundaries.
• Yields useful test cases.
• In isPrime example, need to check numbers on each
side of CACHE_SIZE
• CACHE_SIZE-1, CACHE_SIZE, CACHE_SIZE+1

• Disadvantages
• Tests may have the same bugs as implementation!

18
Properties of good and bad unit tests

• Tests should be self-contained and not depend

on each other implicitly or explicitly.
• "Smells" (bad things to avoid) in tests:
• Constrained test order
• Test A must run before Test B.
• Tests call each other
• Test A calls Test B.
• Mutable shared state
• Tests A/B both use a shared object.

19
cover
coverage and regression testing
Measuring test suite quality with coverage

21
Measuring test suite quality with coverage

• Various kinds of coverage

• Statement: is every statement run by some test case?
• Branch: is every direction of an if or while statement
(true or false) taken by some test case?
• Path: is every path through the program taken by
some test case?

21
Measuring test suite quality with coverage

• Various kinds of coverage

• Statement: is every statement run by some test case?
• Branch: is every direction of an if or while statement
(true or false) taken by some test case?
• Path: is every path through the program taken by
some test case?

• Limitations of coverage
• Coverage is just a heuristic.
• 100% coverage may not be achievable.
• High-cost to approach the limit.

21
Coverage measuring tools: EclEmma

22
Regression testing

23
Regression testing

• Whenever you find a bug

• Store the input that elicited that bug, plus the correct output
• Add these to the test suite
• Check that the test suite fails
• Fix the bug and verify the fix

23
Regression testing

• Whenever you find a bug

• Store the input that elicited that bug, plus the correct output
• Add these to the test suite
• Check that the test suite fails
• Fix the bug and verify the fix

• Why is this a good idea?

• Ensures that your fix solves the problem.
• Helps to populate test suite with good tests.
• Protects against reversions that reintroduce bug:
• It happened at least once, and it might happen again

23
Summary

• Unit testing helps

• convince others that a module works;
• catch problems earlier.

• Choose test data to cover

• specification (black box testing)
• code (white box testing)

• Testing can’t generally prove the absence

of bugs, but it can increase quality and
confidence in the implementation.

24