SOFTWARE TESTING

CHAPTER 5:
CRITERIA-BASED TEST
DESIGN

Updated October 2023

 Old View : Black and White Boxes
 Black-box testing : Derive tests from external
descriptions of the software, including specifications,
requirements, and design
 White-box testing : Derive tests from the source
code internals of the software, specifically including
branches, individual conditions, and statements
 Model-based testing : Derive tests from a model of
the software (such as a UML diagram)

MDTD makes these distinctions less important.

The more general question is:
from what abstraction level do we derive tests?
2
 Changing Notions of Testing
 Old view focused on testing at each software
development phase as being very different from
other phases
◦ Unit, module, integration, system …

 New view is in terms of structures and criteria

◦ input space, graphs, logical expressions, syntax
 Test design is largely the same at each phase
◦ Creating the model is different
◦ Choosing values and automating the tests is
different
3
 Model-Driven Test Design
refined
model / test
requirements /
structure requirements
test specs

DESIGN
Criteria give us ABSTRACTION
test requirements LEVEL

software IMPLEMENTATION
artifact ABSTRACTION input
LEVEL values

pass / test test test

fail results scripts cases
4
New : Test Coverage Criteria
A tester’s job is simple : Define a model of the
software, then find ways
to cover it
▪ Test Requirements : A specific element of a software
artifact that a test case must satisfy or cover

▪ Coverage Criterion : A rule or collection of rules that

impose test requirements on a test set

Testing researchers have defined dozens of criteria, but they

are all really just a few criteria on four types of structures …
5
 Source of Structures
 These structures can be extracted from lots of
software artifacts
◦ Graphs can be extracted from UML use cases, finite state
machines, source code, …
◦ Logical expressions can be extracted from decisions in
program source, guards on transitions, conditionals in use
cases, …
 This is not the same as “model-based testing,”
which derives tests from a model that describes
some aspects of the system under test
◦ The model usually describes part of the behavior
◦ The source is explicitly not considered a model
6
 Criteria Based on Structures
Structures : Four ways to model software
A: {0, 1, >1}
1. Input Domain B: {600, 700, 800}
Characterization (sets) C: {swe, cs, isa, infs}

2. Graphs

3. Logical Expressions (not X or not Y) and A and B

if (x > y)
z = x - y;
4. Syntactic Structures
else
(grammars) z = 2 * x;
7
 1. Graph Coverage – Structural
2 6

1 5 7

Node
Edge (Statement)
3 Path(Branch)
Cover
Cover every node
Cover everyedge
every path
• 12567
• 12567
• 12567
4 • 1343567
This graph may represent • 1343567
• 1257
• statements & branches • 1357
• 13567
• methods & calls
• 1357
• components & signals
• 1343567
• states and transitions
• 134357 …
8
1. Graph Coverage – Data Flow
def = {m}
def = {a , m}
use = {y}
2 6
def = {x, y} use = {x} use = {a} use = {m}

1 5 use = {a} 7
use = {x} def = {a}
Defs
All & Uses Pairs
Defs
3 All Uses
• (x, 1, (1,2)), (x, 1, (1,3))
Every
Everydefdefused once
“reaches”
• (y, 1, 4), (y, 1, 6)
• every
1, 2, 5,use
6, 7
4 • (a, 2, (5,6)), (a, 2, (5,7)),
• •1,
(a, 3,1,
2,2,5,5,76,
(5,6)), (a,73, (5,7)),
This graph contains:
def = {m}
• defs: nodes & edges • •1,1,2,
• (m, 3,2,4,5,3,
7), 75,4,7 7), (m, 6,
(m,
use = {y}
where variables get values 7)
• 1, 3, 5, 6, 7
• uses: nodes & edges • 1, 3, 5, 7
where values are accessed
• 1, 3, 4, 3, 5,7 9
 1. Graph - FSM Example
Memory Seats in a Lexus ES 300
Guard (safety constraint) Trigger (input)

[Ignition = off] | Button2

Driver 1 Driver 2
Configuration Configuration
[Ignition = off] | Button1
[Ignition = on] | seatBack ()
[Ignition = on] | seatBottom ()
New Ignition = off
[Ignition = on] | lumbar ()
(to Modified) Configuration
[Ignition = on] | sideMirrors ()
Driver 2
Ignition = off
[Ignition = on] | Reset AND Button2
New
Modified
Configuration
Configuration
Driver 1
[Ignition = on] | Reset AND Button1

10
2. Logical Expressions
( (a > b) or G ) and (x < y)

Transitions
Logical
Program Decision Statements
Expressions
Software Specifications

11
2. Logical Expressions
( (a > b) or G ) and (x < y)
 Predicate Coverage : Each predicate must be true and
false
◦ ( (a>b) or G ) and (x < y) = True, False
 Clause Coverage : Each clause must be true and false
◦ (a > b) = True, False
◦ G = True, False
◦ (x < y) = True, False
 Combinatorial Coverage : Various combinations of clauses
◦ Active Clause Coverage: Each clause must determine
the predicate’s result
12
2. Logic – Active Clause Coverage
( (a > b) or G ) and (x < y)
1 T F T
With these
values for G and
2 F F T
(x<y), (a>b)
3 F T T duplicate
determines the
value of the
predicate 4 F F T
5 T T T
6 T T F
13
3. Input Domain Characterization
 Describe the input domain of the software
◦ Identify inputs, parameters, or other categorization
◦ Partition each input into finite sets of representative
values
◦ Choose combinations of values
 System level
◦ Number of students { 0, 1, >1 }
◦ Level of course { 600, 700, 800 }
◦ Major { swe, cs, isa, infs }
 Unit level
◦ Parameters F (int X, int Y)
◦ Possible values X: { <0, 0, 1, 2, >2 }, Y : { 10, 20, 30 }
◦ Tests F (-5, 10), F (0, 20), F (1, 30), F (2, 10), F (5, 20)
14
4. Syntactic Structures
 Based on a grammar, or other syntactic definition
 Primary example is mutation testing
1. Induce small changes to the program: mutants
2. Find tests that cause mutant programs to fail: killing mutants
3. Failure is defined as different output from the original program
4. Check the output of useful tests on the original program
 Example program and mutants
if (x > y)
if (x >= y)
if (x > y) z = x - y;
z = x - y;  z = x + y;
else  z = x – m;
z = 2 * x; else
z = 2 * x;
15
 Example : Jelly Bean Coverage
Flavors : Colors :
1. Lemon 1. Yellow (Lemon, Apricot)
2. Pistachio 2. Green (Pistachio)
3. Cantaloupe 3. Orange (Cantaloupe,
4. Pear Tangerine)
5. Tangerine 4. White (Pear)
6. Apricot
▪ Possible coverage criteria :
1. Taste one jelly bean of each flavor
• Deciding if yellow jelly bean is Lemon or Apricot is a
controllability problem
2. Taste one jelly bean of each color
16
 Coverage
Given a set of test requirements TR for coverage
criterion C, a test set T satisfies C coverage if and
only if for every test requirement tr in TR, there is
at least one test t in T such that t satisfies tr

 Infeasible test requirements : test requirements

that cannot be satisfied
◦ No test case values exist that meet the test requirements
◦ Example: Dead code
◦ Detection of infeasible test requirements is formally undecidable for
most test criteria

 Thus, 100% coverage is impossible in practice

17
More Jelly Beans
T1 = { three Lemons, one Pistachio, two Cantaloupes, one
Pear, one Tangerine, four Apricots }

 Does test set T1 satisfy the flavor criterion ?

T2 = { One Lemon, two Pistachios, one Pear, three

Tangerines }

 Does test set T2 satisfy the flavor criterion ?

 Does test set T2 satisfy the color criterion ?

18
 Coverage Level
The ratio of the number of test requirements satisfied by
T to the size of TR

 T2 on the previous slide satisfies 4 of 6 test

requirements

19
 Two Ways to Use Test Criteria
1. Directly generate test values to satisfy the criterion
◦ Often assumed by the research community
◦ Most obvious way to use criteria
◦ Very hard without automated tools

2. Generate test values externally and measure against

the criterion
◦ Usually favored by industry
◦ Sometimes misleading
◦ If tests do not reach 100% coverage, what does that
mean?

Test criteria are sometimes called metrics

20
 Generators and Recognizers
 Generator : A procedure that automatically
generates values to satisfy a criterion
 Recognizer : A procedure that decides whether a
given set of test values satisfies a criterion

 Both problems are provably undecidable for most

criteria
 It is possible to recognize whether test cases
satisfy a criterion far more often than it is possible
to generate tests that satisfy the criterion

 Coverage analysis tools are quite plentiful

21
Comparing Criteria with Subsumption (5.2)

 Criteria Subsumption : A test criterion C1

subsumes C2 if and only if every set of test cases
that satisfies criterion C1 also satisfies C2
 Must be true for every set of test cases
 Examples:
◦ The flavor criterion on jelly beans subsumes the color
criterion … if we taste every flavor we taste one of every
color
◦ If a test set has covered every branch in a program
(satisfied the branch criterion), then the test set is
guaranteed to also have covered every statement

22
Advantages of Criteria-Based Test Design (5.3)

 Criteria maximize the “bang for the buck”

◦ Fewer tests that are more effective at finding faults
 Comprehensive test set with minimal overlap
 Traceability from software artifacts to tests
◦ The “why” for each test is answered
◦ Built-in support for regression testing
 A “stopping rule” for testing—advance knowledge
of how many tests are needed
 Natural to automate

23
Characteristics of a Good Coverage Criterion

1. It should be fairly easy to compute test

requirements automatically
2. It should be efficient to generate test values
3. The resulting tests should reveal as many
faults as possible

• Subsumption is only a rough approximation of fault

revealing capability
• Researchers still need to gives us more data on
how to compare coverage criteria

24
 Test Coverage Criteria
 Traditional software testing is expensive and labor-
intensive
 Formal coverage criteria are used to decide which
test inputs to use
 More likely that the tester will find problems
 Greater assurance that the software is of high
quality and reliability
 A goal or stopping rule for testing
 Criteria makes testing more efficient and effective

How do we start applying these ideas in practice?

25
 How to Improve Testing ?
 Testers need more and better software tools
 Testers need to adopt practices and techniques
that lead to more efficient and effective testing
◦ More education
◦ Different management organizational strategies
 Testing & QA teams need more technical expertise
◦ Developer expertise has been increasing dramatically
 Testing & QA teams need to specialize more
◦ This same trend happened for development in the 1990s

26
 Four Roadblocks to Adoption
1. Lack of test education
Microsoft and Google say half their engineers are testers, programmers test half the time
Number of UG CS programs in US that require testing ? 0
Number of MS CS programs in US that require testing ?
0
Number of UG testing classes in the US ? ~50
2. Necessity to change process
Adoption of many test techniques and tools require changes in development process
This is expensive for most software companies
3. Usability of tools
Many testing tools require the user to know the underlying theory to use them
Do we need to know how an internal combustion engine works to drive ?
Do we need to understand parsing and code generation to use a compiler ?

4. Weak and ineffective tools

Most test tools don’t do much – but most users do not realize they could be better
Few tools solve the key technical problem – generating test values automatically
27
 Needs From Researchers
1. Isolate : Invent processes and techniques that
isolate the theory from most test practitioners
2. Disguise : Discover engineering techniques,
standards and frameworks that disguise the
theory
3. Embed : Theoretical ideas in tools
4. Experiment : Demonstrate economic value of
criteria-based testing and ATDG (ROI)
◦ Which criteria should be used and when ?
◦ When does the extra effort pay off ?
5. Integrate high-end testing with development
28
 Needs From Educators
1. Disguise theory from engineers in classes

2. Omit theory when it is not needed

3. Restructure curricula to teach more than test

design and theory
◦ Test automation
◦ Test evaluation
◦ Human-based testing
◦ Test-driven development

29
 Changes in Practice
1. Reorganize test and QA teams to make effective
use of individual abilities
◦ One math-head can support many testers
2. Retrain test and QA teams
◦ Use a process like MDTD
◦ Learn more testing concepts
3. Encourage researchers to embed and isolate
◦ We are very responsive to research grants
4. Get involved in curricular design efforts through
industrial advisory boards

30
Criteria Summary
• Many companies still use “monkey testing”
• A human sits at the keyboard, wiggles the mouse
and bangs the keyboard
• No automation
• Minimal training required
• Some companies automate human-designed tests
• But companies that use both automation and criteria-
based testing
Save money
Find more faults
Build better software
31
31
Structures for Criteria-Based Testing
Four Structures for
Modeling Software

Input Space Graphs Logic Syntax

Applied Applied to
to
Applied
Source FSMs
to
Specs DNF

Source Specs Source Models

Design Use cases Integ Input

32
 Summary of Part 1’s New Ideas
1. Why do we test – to reduce the risk of using software
◦ Faults, failures, the RIPR model
◦ Test process maturity levels – level 4 is a mental discipline that
improves the quality of the software
2. Model-Driven Test Design
◦ Four types of test activities – test design, automation, execution and
evaluation
3. Test Automation
◦ Testability, observability and controllability, test automation
frameworks
4. Test Driven Development
5. Criteria-based test design
◦ Four structures – test requirements and criteria

Earlier and better testing empowers test managers

33