Life Cycle-Based Testing

Levels and Life Cycle Models

• Levels of testing depend primarily on the software
life cycle used.
• BUT, most forms of testing levels are derived from
the V-Model version of the good, old Waterfall
Model.
• Iterative models introduce the need for regression
testing.
• System testing is greatly enhanced when an
executable specification is used.
 The Waterfall Lifecycle
Requirements
Specification what

how
Preliminary
Design what

how Detailed
Design what

how
Coding

Unit
Testing

Integration
Testing

System
Testing
 High Level
Design Requirements
specification

DetailedDesign
Coding

Unit, Integration,
and System Testing

Maintenance
 The Waterfall Lifecycle Model
• Earliest lifecycle model
– NATO conference in 1968
– probably “in use” before that
• Very sequential—output of one phase expresses
“what” must be done in the next phase
• Strong emphasis/importance of design
• What-How cycles are natural points for software
technical inspections
• Basis for unit, integration, and system levels of
testing (see V-Model)
• Basis for many spinoff lifecycle models
 The V-Model
Requirements System
Specification Testing

Preliminary Integration
Design Testing

Detailed Unit
Design Testing

Coding
 Evaluation of the Waterfall Model
• Advantages
– hierarchical structure maps nicely into large projects
– phases have well-defined end products
• (see IBM’s entry and exit criteria)
– Unit level work can be done in parallel, reducing overall
project interval
• Disadvantages
– Extremely long feedback cycle for customer
– Very late synthesis (begins at integration testing)
– Staff limitations may not support the advantage of massive
parallel development at the unit level
– Requires “perfect foresight”, otherwise early faults propagate
 Spin-off Models
• Practitioner responses to waterfall limitations
• Iterative Development
• The Spiral Model
• Rapid Prototyping
• Executable Specification
• Agile models
– Scrum
– eXtreme Programming (XP)
– Test-Driven Development
• Two promising hybrids
– Agile Model-Driven Development (AMDD)
– Model-Driven Agile Develoipment (MDAD)
 Iterative Development
Build Requirements
Definition what Specification what

how Detailed how Preliminary

Design what Design what

how how Build

Coding Sequence

Unit
Testing

Integration
Testing

Regression
Testing

Progression
Testing
 Iterative Development
• Preserves a single high level design phase
– amortizing design across increments is risky. Early design
decisions may eliminate later design choices
– defines the sequence and content of “builds” (or increments)
• Builds create the need for regression testing
• Preserves the advantages of Waterfall, AND
• Responds to Waterfall defects
– staffing limitations
– late synthesis
– long feedback cycle with customer
 The Spiral Model
• Proposed by Barry Boehm in 1988
• Very similar to the Iterative Model
– builds are selected based on risk and feasibility
• Pictured as an expanding spiral superimposed on the
x-y plane (see internet for copyrighted images)
• “Quadrants” correspond to a sequence of build
activities
– determining objectives
– risk analysis
– development and test
– planning next increment
• Single high level design phase is lost (which might be
an inherent risk)
 “Perfect Foresight?”
• Waterfall and the iterative variations have no
answer for the customer who does not have a clear,
complete idea of what is needed.
• “Requirements Elicitation” is the process of helping
customers and developers reach a common
understanding of a proposed system.
• Three lifecycle responses...
– Rapid Prototyping
– Executable Specification
– the Agile methods
 Rapid Prototyping
Prototype Cycle
what
Prototype Build
Objectives Prototype
Preliminary how
Design what
Exercise
how Detailed Prototype
Design what

how
Coding

Unit
Testing

Integration
Testing

Regression
Testing

Progression
Testing
 Rapid Prototyping
• Helps customer identify needs and defects
– “I’ll know what I want when I see it.”
– provides the “does view” that customers appreciate
– ideal to give the “look and feel” of menu-driven systems
– modify prototype per customer feedback
– Sometimes done for feasibility
• Advantages:
– improved and early feedback with customer
– better basis for design
• Keep or dispose?
– once it has served its purpose, the prototype can be
archived.
– possible to use to identify test scenarios
Executable Specification
what

Preliminary how Executable Specification Cycle

Design what
Define/Revise
how Model
Detailed
Design what
Execute
how Model
Coding

Unit
Testing

Integration
Testing

Regression
Testing

Progression
Testing
 Executable Specification
• Very similar to Rapid Prototyping
– early feedback
– “look and feel”
• Best for event-driven systems
• Executable model is the specification
– finite state machine
– StateChart
– some form of Petri Net
• Need an “engine”
– model is executed by the engine
– usually interactively with customer
Evaluation of Executable
Specification
• Intended for “reactive systems” (event-driven)
• Advantages
– early feedback
– automatic generation of system test cases
– can be used for operator training
– support for early analysis
• Disadvantages
– modeling can be difficult
– training may be necessary
– engine can be expensive
Generic Agile Lifecycle
Customer
Expectations

Iteration
Plan

User Story

Integration
Testing
Design
Code
Test
 Agile Development

• Best response to the customer who does not know

what is needed, i.e., “perfect foresight”
• (see the Agile Manifesto)
• Customer-driven, hence excellent customer
feedback
• Short increments (early synthesis)
• We look at
– eXtreme Programming (XP)
– Test-Driven Development (TDD)
– Scrum
eXtreme Programming
User
Stories

Iteration
Iteration
Plan

Release Pair Acceptance Small

Plan Coding Test Release

Unit Test
 eXtreme Programming
• Kent Beck, 1996
• Distinguishing characteristic: pair programming
– one person has the detailed view (and the keyboard)
– partner has the overall view, and acts as a constant
reviewer
– roles can change
• Bottom-up development precludes a single, high
level design phase
– (but that might not be possible with an uncertain
customer anyway)
Test-Driven Development (TDD)
User
Story

No

Story Story Test

Tasks Cases

Run Tests Pass Refactor?

Fail Yes

“Just Enough” Refactor

Code Existing code
 Test-Driven Development (TDD)
• Extreme case of agile development
• Bottom-up development based on test cases
– derived from customer-provided user stories
– very quick feedback
• Very small increments
– early synthesis
– excellent fault isolation
– refactoring results in clean code
• BUT, no opportunity for a comprehensive design
 TDD Example: a Boolean Function
to Determine Leap Years
• Definition: Leap year divisible by 4, Except for
years which are both divisible by 100 and not
divisible by 400. The second part of the rule
effects century years. For example;
the century years 1600 and 2000 are leap
years, but the century years 1700, 1800, and
1900 are not.
• Test-Driven Development would break this into
small, individual user stories (also called tasks).

• “Coded” here in a pseudo-code (a lingua franca)

that resembles Visual Basic.
User Story 1:A year divisible
by 4 is a leap
year
Test Case 1
Input: 2004
Expected Output: True

(existing) Pseudo-Code in normal font

Function isLeap(year) As Boolean
End isLeap

Running Test Case 1 on this code fails.

Add just enough code to make the test pass.
 User Story 1: A year
divisible by 4 is a
leap year
Test Case 1
Input: 2004
Expected Output: True

(updated) Pseudo-Code in bold face font

Function isLeap(year) As Boolean

dim year AS Integer
'MOD is the modulo arithmetic built-in operator in most languages
If (( year MOD 4) = 0) Then
IsLeap = True
EndIf
End isLeap

Test Case 1 passes. Now do User Story 2.

User Story 2: A year not
divisible by 4 is a
common year
Test Case 1 Input: 2004
Expected Output: True
Test Case 2 Input: 2007
Expected Output: False

(existing) Pseudo-Code in normal font

Function isLeap(year) As Boolean

dim year AS Integer
If (( year MOD 4) = 0) Then
IsLeap = True
EndIf
End isLeap

Test Case 1 passes. Test Case 2 fails. Now add just enough code so
that Test Case 2 passes.
User Story 2: A year not
divisible by 4 is a
common year
Test Case 1 Input: 2004
Expected Output: True
Test Case 2 Input: 2007
Expected Output: False

(updated) Pseudo-Code In bold face font

Function isLeap(year) As Boolean

dim year AS Integer
isLeap = False
If (( year MOD 4) = 0) Then
IsLeap = True
EndIf
End isLeap

Test Cases 1 and 2 pass. Now do User Story 3

User Story 3: A century year
not divisible by 400 is a
common year
Test Case 1 Input: 2004, Expected Output: True
Test Case 2 Input: 2007, Expected Output: False
Test Case 3 Input: 1900, Expected Output: False

(existing) Pseudo-Code In normal font

Function isLeap(year) As Boolean

dim year AS Integer
isLeap = False
If (( year MOD 4) = 0) Then
IsLeap = True
EndIf
End isLeap

Test Cases 1 and 2 pass. Test Case 3 fails. Now add just enough code
so that Test Case 3 passes.
User Story 3: A century year
not divisible by 400 is a
common year
Test Case 1 Input: 2004, Expected Output: True
Test Case 2 Input: 2007, Expected Output: False
Test Case 3 Input: 1900, Expected Output: False

(updated) Pseudo-Code In bold face font

Function isLeap(year) As Boolean

dim year AS Integer
isLeap = False
If ((( year MOD 4) = 0) AND NOT((year MOD 100) = 0))) Then
IsLeap = True
EndIf
End isLeap

Test Cases 1, 2 and 3 pass. Now do User Story 4

 User Story 4: A century
year divisible by 400
is a leap year
Test Case 1 Input: 2004, Expected Output: True
Test Case 2 Input: 2007, Expected Output: False
Test Case 3 Input: 1900, Expected Output: False
Test Case 4 Input: 2000, Expected Output: True

(existing) Pseudo-Code In normal font

Function isLeap(year) As Boolean

dim year AS Integer
isLeap = False
If ((( year MOD 4) = 0) AND NOT((year MOD 100) = 0))) Then
IsLeap = True
EndIf
End isLeap

Test Cases 1, 2 and 3 pass. Test case 4 fails. Now add just enough
code so that test case 4 passes.
User Story 4: A century
year divisible by 400
is a leap year
Test Case 1 Input: 2004, Expected Output: True
Test Case 2 Input: 2007, Expected Output: False
Test Case 3 Input: 1900, Expected Output: False
Test Case 4 Input: 2000, Expected Output: True

(updated) Pseudo-Code In bold face font

Function isLeap(year) As Boolean

dim year AS Integer
isLeap = False
If ((( year MOD 4) = 0) AND NOT((year MOD 100) = 0))) OR
((year MOD 400 = 0)) Then
IsLeap = True
EndIf
End isLeap

Test Cases 1, 2, 3 and 4 pass. Done with function isLeap.

Advantages of Test Driven
Development
• In this example, the steps are deliberately small.

• Customer and developer can (should!) jointly

determine granularity of user stories.

• Fault isolation is greatly simplified (in fact, trivial).

If a test case fails, the fault must be in the most
recently added code.

• Once a new test case passes, a working (subset)

of the desired software can always be delivered.

• Something always works!

Disadvantages of Test Driven
Development
• Useful granularity is an issue.

• There is no guarantee that user stories “arrive”

in a sensible order.

• There is no guarantee that user stories are the

“same size” (or require similar effort)

• Bottom-up coding often results in poorly

structured code, making refactoring necessary.

• Tool support (e.g. jUnit) is essential.

SCRUM
Scrum—New Terms for Existing
Ideas
• Created in 1993 by Jeff Sutherland
• Formalized in 1995 by Ken Schwaber
• Very popular today, both in US and Europe
• Named for the importance of teamwork in rugby
• n.b.: material in the series of slides on Scrum is
taken from http://www.scrumalliance.org/
learn_about_scrum)
• Emphasis on Team work and Team member
• New name for old ideas
Scrum—New Terms for Existing
Ideas
• Three roles(Participant)
– Product owner(customer)
– Scrum Master(Supervisor)
– Self-organizing team(development
team)
• Three ceremonies(Meeting)
– Sprint planning meeting
– Daily scrum meeting
– Sprint review meeting
• Three artifacts(work Product)
– Product backlog,
– Sprint backlog
– Burndown chart
 Product Owner Responsibilities
• Define the features of the product;
• Decide on release date and content;
• Be responsible for the profitability of the product
(ROI);
• Prioritize features according to market value;
• Adjust features and priority every 30 days, as
needed; and
• Accept or reject work results.

• Question: Product Owner = Customer?`

 Scrum Master Responsibilities
• Ensure that the team is fully functional and
productive
• Enable close cooperation across all roles and
functions
• Remove barriers
• Shield the team from external interferences
• Ensure that the process is followed (sprint
planning, daily meeting, sprint review)

• Question: Scrum Master = Supervisor?

 Scrum Team Responsibilities
• Cross-functional with 7 +/- 2 members
• Selects the Sprint goal and specifies work results
• Has the right to do everything within the boundaries of
the project guidelines to reach the Sprint goal
• Organizes itself and its work
• Demonstrates work results to Product Owner.

• Question: Scrum Team = Development Team?

Sprint planning meeting
Daily scrum meeting
Sprint Review Meeting
Scrum artifacts
 Product backlog
 Sprint backlog
 Burn-down chart
Product Backlog
Sprint Backlog
Sprint Backlog
Burn-down chart
Burn down chart
 Scrum Lifecycle
Daily Activities

Standup
Product
Meeting
Backlog

Design
Sprint Sprint
Backlog Definition

Coding

Test

Small Sprint
Release Test
Scrum Team
 Scrum team to focus on what happened the
preceding day and what needs to be done in the
new day.
 Testing in Scrum occurs in two levels:
 Unit Level-at each day

 Integration Level-small release at the end of

spirit.
 Sprint Backlog:
 Selection of the Sprint backlog from the
product backlog is done by the product owner
(the customer),

 Sprint Definition:
 It looks a lot like preliminary design because
this is the point where the Scrum team
identifies the sequence and contents of
individual sprints
Comparison of Model Driven
Development (MDD) and Test
Driven Development (TDD)
• First American’s view of Eagles and Mice
– Eagles have the “big picture”
– Mice focus on the details
– (both views are important!)

• MDD is a rigorous, top-down approach.

• TDD is a bottom-up approach.

TDD isLeap in Visual Basic
(refactored)
Public Function isLeap(year) As Boolean
Dim year As Integer 1

Dim c1, c2, c3 As Boolean

2

 1. c1 = (year Mod 4 = 0)
3
 2. c2 = (year Mod 100 = 0)
 3. c3 = (year Mod 400 = 0) 4

 4, isLeap = False
5

 5. If ( (c1 AND NOT(c2)) OR (c3)) Then

6 7

 6. IsLeap = True
 7. Else 8

 8. IsLeap = False 9

 9. EndIf
 End Function
 Decision Table Model of isLeap
Conditions r1 r2 r3 r4 r5 r6 r7 r8

C1. year is a multiple of 4 T T T T F F F F

C2. year is a century year T T F F T T F F

C3. year is a multiple of 400 T F T F T F T F

Actions

(logically impossible) X X X X

A1. year is a common year X x

A2. year is a leap year X X

test case: year = 2000 1900 2012 2011

 MDD isLeap in Visual Basic 1

Public Function isLeap(year) As Boolean

Dim year As Integer 2

Dim c1, c2, c3 As Boolean

1. c1 = (year Mod 4 = 0) 3

2. c2 = (year Mod 100 = 0)

3. c3 = (year Mod 400 = 0) 4

4, isLeap = False
5. If c1 Then
5
6. If c2 Then
7. If c3 Then 6
8. isLeap = True ‘rule r1
9. Else 7

10. isLeap = False ‘rule r2

11. End If 8 9 12 15

12. Else
13. isLeap = True ‘rule r4 10 13 16

14. End If
15. Else 11

16. isLeap = False ‘rule r8 14

17. End If
End Function 17
 Observations

• The TDD version is less complex (really?)

– Why?

• The TDD version gradually built up to a

compound condition (that might be hard to
understand, and to modify).
• The decision table model assures
– What?

• Both versions require 4 test cases

 Open Questions
• Does Agile (bottom up) Development actually
result in better designs?

• Can any form of Agile Testing reveal “deep

faults”?
– e.g.: faults revealed only by data flow (define/use)
testing
– computational faults
– time-dependent faults

• What about maintenance? Agile code has

– well-named variables and components
– (hopefully) been refactored carefully
– no comments
– test cases are the specification
 A Compromise?
• How might we capture/combine the advantages of
various lifecycles to avoid the known deficiencies?
• Lessons from Georg (German Ph.D.
mathematician) and Go player
• A successful Go player must have both good
strategy and good tactics
• Georg’s contention...
– Design = strategy(overall design)
– TDD = tactics(unit level development)
Agile Model-Driven
Development
• Scott Ambler
• Model just enough for the present user
story
• Design is necessary!
• BUT the modeling is still not in one
phase (possibly unavoidable with the
uncertain customer)
Agile Model-Driven
Development
Project Iteration
Inception Iteration
Modeling

Iteration
Plan Model
Storming

Test-Driven
Development
 Model-Driven Agile
Development
• Re-arrangement of AMDD
• Early emphasis on design as one step (thanks, Georg)
• Implementation uses Test-Driven Development
Model-Driven Agile Development
Requirements
Specification

Project
Modeling Iteration
Iteration
“Final” Modeling
Series of
System
Iterations
Testing
Test-Driven
Development

Iteration
Integration