Introduction to Software

Testing
(2nd edition)
Chapter 4

Putting Testing First

Paul Ammann & Jeff Offutt

http://www.cs.gmu.edu/~offutt/software
test/

August 2014
 The Increased Emphasis on Testing
 Philosophy of traditional software development
methods
– Upfront analysis
– Extensive modeling
– Reveal problems as early as possible
More work must be
revised
Root problem is harder to
find
Cost
Delta

Time
Original Revision
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 2
 Traditional Assumptions
1. Modeling and analysis can identify potential
problems early in development
2. Savings implied by the cost-of-change curve
justify the cost of modeling and analysis over
the life
 These areoftrue
the ifproject
requirements are always
complete and current
 But those annoying customers keep changing
their minds!
– Humans are naturally good at approximating
– But pretty bad at perfecting
 These two assumptions have made software
engineering frustrating and difficult for
Thus, agile
decades
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 3
 Why Be Agile ?
 Agile methods start by recognizing that neither
assumption is valid for many current software
projects
– Software engineers are not good at developing
requirements
– We do not anticipate many changes
– Many of the changes we do anticipate are not
needed
 Requirements (and other “non-executable
artifacts”) tend to go out of date very quickly
– We seldom take time to update them
– Many current software projects change continuously
 Agile methods expect software to start small
and evolve over time
–
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 4
 Supporting Evolutionary Design
Traditional design advice says to anticipate
changes
Designers often anticipate changes that don’t
happen Anticipat
ed

Anticipated Change
change that
Evolving
doesn’t Design
happen
Unanticipa
ted
Change

Both anticipated and unanticipated changes affect

design
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 5
 The Test Harness as Guardian (4.2)
What is Correctness ?
Traditional Correctness Agile Correctness
(Universal) (Existential)
V x,y, x ≥ y

10
{ (1, 1)  T
(1, 0)  T
5
(0, 1)  F
Y (10, 5)  T
1 (10, 12)  F }
1 X 5 10

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 6

 A Limited View of Correctness
 In traditional methods, we try to define all
correct behavior completely, at the beginning
– What is correctness?
– Does “correctness” mean anything in large
engineering products?
– People are VERY BAD at completely defining
correctness
 In agile methods, we redefine correctness to be
relative to a specific set of tests
– If the software behaves correctly on the tests, it is
“correct”
– Instead of defining all behaviors, we demonstrate
some behaviors
But software engineers ain’t
– Mathematicians may be disappointed at the lack of
completenessmathematicians!
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 7
Test Harnesses Verify Correctness
A test harness runs all automated tests
efficiently and reports results to the
developers
 Tests must be automated
– Test automation is a prerequisite to test driven
development
 Every test must include a test oracle that can
evaluate whether that test executed correctly
 The tests replace the requirements
 Tests must be high quality and must run
quickly
 We run tests every time we make a change to
the software
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 8
 Continuous Integration
 Agile methods work best when the current
version of the software can be run against all
tests at any time
A continuous integration server rebuilds
the system, returns, and reverifies tests
whenever any update is checked into the
 Mistakes are caughtrepository
earlier
 Other developers are aware of changes early
 The rebuild and reverify must happen as soon
as possible
– Thus, tests need to execute quickly
A continuous integration server doesn’t
just run tests, it decides if a modified system
is still correct
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 9
 Continuous Integration Reduces
Risk

Inventory of non-integrated work

Integrated Functional-
ity

Non-integrated functionality is
dangerous!
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 10
 System Tests in Agile Methods

Traditional testers often

design system tests from
requirements
Requiremen
tsRequiremen Syste
tsRequiremen ? m
ts tests

But … what if there are

no traditional
requirements
documents ?
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 11
 User Stories
A user story is a few sentences that
captures what a user will do with the
software
Agent sees a list of
Withdraw money
today’s interview
from checking
applicants
account
Support technician
sees customer’s
history on demand

– In the language of the end user

– Usually small in scale with few details
– Not archived
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 12
 Acceptance Tests in Agile
Methods
Acceptan
User TDD
ce Test
Story Test 1
(Failing)
Change
software
Tests &
Acceptan archive
Refactor
ce Test d
Continue adding
(Passing) TDD tests until TDD
Change
acceptance test Test 2
passes software
& Refactoring
Refactor avoids
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 13
 Adding Tests to Existing Systems
 Most of today’s software is legacy
– No legacy tests
– Legacy requirements hopelessly outdated
– Designs, if they were ever written down, lost
 Companies sometimes choose not to change
software out of fear of failure

How to apply TDD to legacy

software with no tests?
 Create an entire new test set? — too
expensive!
 Give up? — a mixed project is unmanageable
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 14
 Incremental TDD
 When a change is made, add TDD tests for just
that change
– Refactor
 As the project proceeds, the collection of TDD
tests continues to grow
 Eventually the software will have strong TDD
tests

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 15

 The Testing Shortfall
 Do TDD tests (acceptance or otherwise) test
the software well?
– Do the tests achieve good coverage on the code?
– Do the tests find most of the faults?
– If the software passes, should management feel
confident the software is reliable?

NO!

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 16

 Why Not?
 Most agile tests focus on “happy paths”
– What should happen under normal use
 They often miss things like
– Confused-user paths
– Creative-user paths
– Malicious-user paths

The agile methods

literature does not give
much guidance

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 17

 What Should Testers Do?

Ummm ... Excuse me,

Professor ...

What do I DO?

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 18

 Design Good
Use a human-based
Tests
1. approach
• Create additional user stories
that describe non-happy paths
• How do you know when you’re
finished?
• Some people are very good at
this, some are bad, and it’s hard
to teach Use modeling and criteria
• 2. Model the input domain to design
tests
Part 2 of • Model software behavior with
book … graphs, logic, or grammars
• A built-in sense of completion
• Much easier to teach—
engineering
• Requires discrete math knowledge
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 19
 Summary
 More companies are putting testing first
 This can dramatically decrease cost and
increase quality
 A different view of “correctness”
– Restricted but practical
 Embraces evolutionary design
 TDD is definitely not test automation
– Test automation is a prerequisite to TDD
 Agile tests aren’t enough

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 20