Chapter 2

Software Testing
ISTQB / ISEB Foundation Exam Practice

Testing in the Lifecycle

1 Principles 2 Lifecycle 3 Static testing

4 Dynamic test
5 Management 6 Tools
techniques

1
 Lifecycle

1 2 3 ISTQB / ISEB Foundation Exam Practice

4 5 6

Contents
Models for testing, economics of testing
High level test planning
Component Testing
Integration testing in the small
System testing (non-functional and functional)
Integration testing in the large
Acceptance testing
Maintenance testing
2
V-Model: test levels
Business Acceptance
Requirements Testing

Project Integration Testing

Specification in the Large

System System
Specification Testing

Design Integration Testing

Specification in the Small

Component
Code
Testing
3
V-Model: late test design
Tests
Business Acceptance
Requirements Testing
Tests
Project “We don’t have Integration Testing
Specification time to design in the Large
tests early”Tests
System System
Specification Testing
Tests
Design Integration Testing
Specification in the Small
Tests
Component
Code
Testing
Design
Tests?
4
 V-Model: early test design
Tests Tests
Business Acceptance
Requirements Testing
Tests Tests
Project Integration Testing
Specification in the Large
Tests Tests
System System
Specification Testing
Tests Tests
Design Integration Testing
Specification in the Small
Tests Tests
Component
Design Code
Testing
Run
Tests Tests
5
• In traditional models, defects were found too late in the life
cycle, as testing was not involved until the end of the project.
Testing also added lead time due to its late involvement.
• The V-model, on the other hand, shows that testing is
not only an execution-based activity.
• There are a variety of activities that need to be
performed before the end of the coding phase.
• These activities should be carried out in parallel with
development activities, and testers need to work with
developers and business analysts so they can
perform these activities and tasks and produce a set
of test deliverables.
• The work products produced by the developers and
business analysts during development are the basis
6
of testing in one or more levels.
Early test design

 test design finds faults

 faults found early are cheaper to fix
 most significant faults found first
 faults prevented, not built in

Early test design helps to build quality,

stops fault multiplication 7
VV&T
 Verification – checks that the work-product meets the requirements set
out for it.
- An example of this would be to ensure that a website being built follows the
guidelines for making websites usable by as many people as possible.
- Verification helps to ensure that we are building the product in the right way.

 Validation – changes the focus of work-product evaluation to evaluation

against user needs. This means ensuring that the behaviour of the work-
product matches the customer needs as defined for the project.
- For example, for the same website above, the guidelines may have been written
with people familiar with websites in mind. It may be that this website is also
intended for novice users. Validation would include these users checking that
they too can use the website easily.
- Validation helps to ensure that we are building the right product as far as the
users are concerned.

 Testing – the process of exercising software to verify that it satisfies

specified requirements and to detect faults

8
Verification, Validation and Testing

Validation

Testing
Any

Verification

9
Another model--incremental development
models

10
 Rapid Application Development
Components/functions are developed in parallel as if they were
mini projects, the developments are time-boxed, delivered, and
then assembled into a working prototype.

This methodology
allows early validation
of technology risks and
a rapid response to
changing customer
requirements.

11
Agile development

Extreme Programming (XP) is one of the most well-known agile development life
cycle models.
Some characteristics of XP are:
• It promotes the generation of business stories to define the functionality.
• It demands an on-site customer for continual feedback and to define and carry
out functional acceptance testing.
• It promotes pair programming and shared code ownership amongst the
developers.
• It states that component test scripts shall be written before the code is written and
that those tests should be automated.
• It states that integration and testing of the code shall happen several times a day.
• It states that we always implement the simplest solution to meet today's
problems.

12
How would you test this spec?

 A computer program plays chess with one

user. It displays the board and the pieces on
the screen. Moves are made by dragging
pieces.

13
“Testing is expensive”

 Compared to what?
 What is the cost of NOT testing, or of faults
missed that should have been found in test?
- Cost to fix faults escalates the later the fault is found
- Poor quality software costs more to use
• users take more time to understand what to do
• users make more mistakes in using it
• morale suffers
• => lower productivity
 Do you know what it costs your organisation?14
What do software faults cost?

 Have you ever accidentally destroyed a PC?

- knocked it off your desk?
- poured coffee into the hard disc drive?
- dropped it out of a 2nd storey window?
 How would you feel?
 How much would it cost?

15
 Hypothetical Cost - 1

(Loaded Salary cost: £50/hr)

Fault Cost Developer User
- detect ( .5 hr) £25
- report ( .5 hr) £25
- receive & process (1 hr) £50
- assign & bkgnd (4 hrs) £200
- debug ( .5 hr) £25
- test fault fix ( .5 hr) £25
- regression test (8 hrs) £400

£700 £50
16
 Cost of fixing faults

1000

100

10

Req Des Test Use

17
 Lifecycle

1 2 3 ISTQB / ISEB Foundation Exam Practice

4 5 6

Contents
Models for testing, economics of testing
omponent Testing
Integration testing in the small
System testing (non-functional and functional)
Integration testing in the large
Acceptance testing
Maintenance testing

18
Component testing

 Component testing, also known as unit, module and program

testing, searches for defects in, and verifies the functioning of
software (e.g. modules, programs, objects, classes, etc.) that are
separately testable.
 lowest level
 tested in isolation
 most thorough look at detail
- error handling
- interfaces
 usually done by programmer
19
Component test process
BEGIN

Component
Test Planning

Component
Test Specification

Component
Test Execution

Component
Test Recording

Checking for
Component END
Test Completion 20
Component test process
BEGIN Component test planning
- how the test strategy and
Component
Test Planning
project test plan apply to
the component under test
Component
Test Specification -any exceptions to the strategy
Component
Test Execution
- all parts the component
will interact with (e.g. stubs
Component and drivers
Test Recording

Checking for
Component END
Test Completion 21
Component test process
BEGIN

Component Component test specification

Test Planning
- test cases are designed
Component using the test case design
Test Specification techniques specified in the
test plan
Component -A Test case include:
Test Execution
- objective
Component • initial state of component
Test Recording • input
• expected outcome
Checking for - test cases should be
Component END
Test Completion repeatable 22
Component test process
BEGIN

Component
Test Planning

Component
Test Specification
Component test execution
Component - Each test case is executed
Test Execution - Either executed manually
or using an automated
Component
Test Recording tool

Checking for
Component END
Test Completion 23
Component test process
Component test recording
BEGIN using a testing management tool
- identities & versions of
Component
Test Planning
component, test specification
- actual outcome recorded &
Component compared to expected outcome
Test Specification - discrepancies logged
- repeat test activities to establish
Component removal of the discrepancy
Test Execution (fault in test or verify fix)
- record coverage levels achieved
Component for test completion criteria
Test Recording
specified in test plan
Checking for
Component Sufficient
END to show test
Test Completion activities carried out 24
Component test process
BEGIN

Component Checking for component

Test Planning
test completion
Component - check test records against
Test Specification specified test completion
criteria (e.g. coverage)
Component - if not met, repeat test
Test Execution
activities
Component - may need to repeat test
Test Recording specification to design test
cases to meet completion
Checking for criteria (e.g. white box)
Component END
Test Completion 25
 Also a measurement
Test design techniques technique? = Yes
= No

 “Black box”  “White box”

- Equivalence partitioning - Statement testing
- Boundary value analysis - Branch / Decision testing
- State transition testing - Data flow testing
- Cause-effect graphing - Branch condition testing
- Syntax testing - Branch condition
- Random testing combination testing
 How to specify other - Modified condition
techniques decision testing
- LCSAJ testing
- Linear Code Sequence and Jump
26
 Lifecycle

1 2 3 ISTQB / ISEB Foundation Exam Practice

4 5 6

Contents
Models for testing, economics of testing
High level test planning
Component Testing
Integration testing in the small
System testing (non-functional and functional)
Integration testing in the large
Acceptance testing
Maintenance testing
27
Integration testing in the small
 more than one (tested) component
 communication between components
 what the set can perform that is not possible
individually
 non-functional aspects if possible
 integration strategy:
- big-bang vs incremental (top-down, bottom-up,
functional)
 done by designers, analysts, or
independent testers
28
Integration Testing Example

 For example you have to test the keyboard of a

computer then it is a unit testing but when you
have to combine the keyboard and mouse of a
computer together to see its working or not then it
is the integration testing. So it is prerequisite
that for performing integration testing a system
must be unit tested before.

https://youtu.be/QYCaaNz8emY
http://www.guru99.com/integration-testing.html

29
Example of Integration Test
Cases
Test Case ID Test Description Input Expected Result

Test integration between the Payment is processed

User books a service
TS-1 payment gateway and the correctly and the booking is
and makes a payment
booking system confirmed

The message is delivered

Test the integration between the User sends a message
correctly and the user
TS-2 messaging system and the and receives a
receives the expected
notification system notification
notification

Test the integration between the System generates a The bill is correctly
TS-3 billing system and the accounting bill and sends it to the processed and recorded in
system accounting system the accounting system

The search results and

Test the integration between the User performs a
recommendations are
TS-4 search engine and the search and receives
accurate and relevant to the
recommendation engine recommendations
users search query 30
Big-Bang Integration

 In theory:
- if we have already tested components why not just
combine them all at once? Wouldn’t this save time?
- (based on false assumption of no faults)
 In practice:
- takes longer to locate and fix faults
- re-testing after fixes more extensive
- end result? takes more time

31
Incremental Integration

 Baseline 0: tested component

 Baseline 1: two components
 Baseline 2: three components, etc.
 Advantages:
- easier fault location and fix
- easier recovery from disaster / problems
- interfaces should have been tested in component
tests,

32
Top-Down Integration
 Baselines:
- baseline 0: component a
- baseline 1: a + b a
- baseline 2: a + b + c
- baseline 3: a + b + c + d b c
- etc.
d e f g
 Need to call to lower
level components not
h i j k l m
yet integrated
 Stubs: simulate missing
n o
components
33
Stubs
 Stub replaces a called component for
integration testing
 Keep it Simple
 Examples
- print/display name (I have been called)
- reply to calling module (single value)
- computed reply (variety of values)
- prompt for reply from tester
- search list of replies
- provide timing delay
34
Pros & cons of top-down approach
 Advantages:
- critical control structure tested first and most often
- can demonstrate system early (show working
menus)
 Disadvantages:
- needs stubs
- may be difficult to "see" detailed output (but should
have been tested in component test)

35
Bottom-up Integration

 Baselines: a
- baseline 0: component n
b c
- baseline 1: n + i
- baseline 2: n + i + o
d e f g
- baseline 3: n + i + o + d
- etc.
h i j k l m
 Needs drivers to call
the baseline configuration
n o
 Also needs stubs
for some baselines
36
Drivers

 Driver: test harness: scaffolding

 specially written or general purpose
(commercial tools)
- invoke baseline
- send any data baseline expects
- receive any data baseline produces (print)

37
Pros & cons of bottom-up approach
 Advantages:
- lowest levels tested first and most thoroughly (but
should have been tested in unit testing)
- good for testing interfaces to external environment
(hardware, network)
 Disadvantages
- no working system until last baseline
- needs both drivers and stubs
- major control problems found last
38
Minimum Capability Integration
(also called Functional)
 Baselines: a
- baseline 0: component a
b c
- baseline 1: a + b
- baseline 2: a + b + d
d e f g
- baseline 3: a + b + d + i
- etc.
h i j k l m
 Needs stubs
 Shouldn't need drivers
n o
(if top-down)

39
Thread Integration
(also called functional)
 order of processing some event a
determines integration order
 interrupt, user transaction b c

d e f g
 advantages:
- critical processing first h i j k l m

 disadvantages: n o
- may need complex drivers and stubs
40
Inegration Example

 Stubs and Drivers Example

 https://w3.cs.jmu.edu/bernstdh/Web/common/
help/stubs-and-drivers.php

41
 Example:
 For Example we have 3 modules Sign-in, home-page, and
user-database module. Sign-in module is ready and need to
test it, but we call functions from home-page and user-
database (which is not ready). To test at a selective module
we write a short dummy piece of a code which simulates
home-page and user-database, which will return values for
Sign-in, this piece of dummy code is always called Stubs
and it is used in a top down integration.
 Considering the same Example above: If we have home-
page and user-database modules get ready and Sign-in
module is not ready, and we need to test home-page and
user-database modules Which return values from Sign-in
module, So to extract the values from Sign-in module We
write a Short Piece of Dummy code for Sign-in which
returns value for home-page and user-database, So these
pieces of code is always called Drivers and it is used in
Bottom Up Integration.

42
 Lifecycle

1 2 3 ISTQB / ISEB Foundation Exam Practice

4 5 6

Contents
Models for testing, economics of testing
High level test planning
Component Testing
Integration testing in the small
System testing (non-functional and functional)
Integration testing in the large
Acceptance testing
Maintenance testing
43
System testing

 last integration step to test

- functional
• functional requirements and requirements-based
testing
•business process-based testing
- non-functional
• as important as functional requirements
• often poorly specified
• must be tested
 often done by independent test group
44
Functional system testing

 Functional requirements
- a requirement that specifies a function that a system
or system component must perform (ANSI/IEEE
Std 729-1983, Software Engineering Terminology)

45
Non-functional system testing

 different types of non-functional system tests:

- usability - configuration / installation
- security - reliability / qualities
- documentation - back-up / recovery
- storage - performance, load, stress
- volume

46
Performance Tests
 Timing Tests
- response and service times
- database back-up times
 Capacity & Volume Tests
- maximum amount or processing rate
- number of records on the system
- graceful degradation
 Endurance Tests (24-hr operation?)
- robustness of the system
- memory allocation 47
Multi-User Tests
 Concurrency Tests
- small numbers, large benefits
- detect record locking problems
 Stress Tests
- go beyond limits for the system - know what will
happen
- particular relevance for e-commerce

48
Source: Sue Atkins, Magic Performance Management
Load testing
Load testing in a nutshell consists of:
- Conducting the anticipated load in terms of virtual users or requests per
second.
- Comparing actual metrics (response time, throughput, percentage of
errors, resource usage, etc.) with the expected metrics.
- Deciding whether the current build is ready to go or not.
 tools
- JMeter: http://jakarta.apache.org/jmeter/
16 Top Load Testing Software Tools for 2024 | Test Guild
Front-end performance testing measures how fast your website loads and
displays content for users. In contrast, back-end performance testing
involves sending multiple requests to your servers to see if they can
handle simultaneous requests.

49
Stress Tests
 The goal of stress testing is to evaluate and determine the
behavior of a software component while the offered load
is in excess of its designed capacity
 The system is deliberately stressed by pushing it to and
beyond its specified limits
- It ensures that the system can perform acceptably under worst-
case conditions, under an expected peak load. If the limit is
exceeded and the system does fail, then the recovery mechanism
should be invoked
 Stress tests are targeted to bring out the problems
associated with one or more of the following:
- Memory leak
- Buffer allocation and memory carving
50
Usability Tests
 messages tailored and meaningful to (real) users?
 coherent and consistent interface?
 sufficient redundancy of critical information?
 within the "human envelope"? (7±2 choices)
 feedback (wait messages)?
 clear mappings (how to escape)?
 http://www.usefulusability.com/24-usability-testing-
tools/
 http://www.feng-gui.com/
 https://www.usertesting.com/be-a-user-tester
Who should design / perform these tests?
51
 Security Tests
- Security tests are designed to verify that the system
meets the security requirements
• Confidentiality
– It is the requirement that data and the processes be protected
from unauthorized disclosure
• Integrity
– It is the requirement that data and process be protected from
unauthorized modification
• Availability
– It is the requirement that data and processes be protected from
the denial of service to authorized users
- Security test scenarios should include negative
scenarios such as misuse and abuse of the software
52 system
 Security Tests (cont’d) :
 useful types of security tests includes the following:
1. Verify that only authorized accesses to the system are permitted
2. Verify the correctness of both encryption and decryption
algorithms for systems where data/messages are encoded.
3. Verify that illegal reading of files, to which the perpetrator is not
authorized, is not allowed
4. Ensure that virus checkers prevent or limit entry of viruses into
the system
5. Ensure that the system is available to authorized users when a
zero-day attack occurs
6. Try to identify any “backdoors” in the system usually left open
by the software developers

53
Configuration and Installation

 Configuration Tests
- different hardware or software environment
- configuration of the system itself
 Installation Tests
- distribution (CD, network, etc.) and timings
- physical aspects: electromagnetic fields, heat,
humidity, motion, chemicals, power supplies
- uninstall (removing installation)

54
Reliability Tests

 Reliability tests are designed to measure the

ability of the system to remain operational
for long periods of time.
 The reliability of a system is typically
expressed in terms of mean time to failure
(MTTF)
 The average of all the time intervals between
successive failures is called the MTTF
Back-up and Recovery

 Back-ups
- computer functions
- manual procedures (where are tapes stored)
 Recovery
- real test of back-up
- manual procedures unfamiliar
- should be regularly rehearsed
- documentation should be detailed, clear and
thorough
56
Documentation Testing

 Documentation review
- check for accuracy against other documents
- gain consensus about content
- documentation exists, in right format
 Documentation tests
- is it usable? does it work?
- user manual
- maintenance documentation

57
Tools used for System Testing
 JMeter
 Gallen Framework
 HP Quality Center/ALM
 IBM Rational Quality Manager
 Microsoft Test Manager
 Selenium
 Appium
 LoadRunner
 Gatling
 Apache JServ
 SoapUI
 Top 20 Performance Testing Tools in 2024 | BrowserStack
58
 Lifecycle

1 2 3 ISTQB / ISEB Foundation Exam Practice

4 5 6

Contents
Models for testing, economics of testing
High level test planning
Component Testing
Integration testing in the small
System testing (non-functional and functional)
Integration testing in the large
Acceptance testing
Maintenance testing
59
Integration testing in the large

 Tests the completed system working in

conjunction with other systems, e.g.
- LAN / WAN, communications middleware
- other internal systems (billing, stock, personnel,
overnight batch, branch offices, other countries)
- external systems (stock exchange, news, suppliers)
- intranet, internet / www
- 3rd party packages
- electronic data interchange (EDI)
60
 Lifecycle

1 2 3 ISTQB / ISEB Foundation Exam Practice

4 5 6

Contents
Models for testing, economics of testing
High level test planning
Component Testing
Integration testing in the small
System testing (non-functional and functional)
Integration testing in the large
Acceptance testing
Maintenance testing
61
User acceptance testing

 Final stage of validation

- customer (user) should perform or be closely
involved
- customer can perform any test they wish, usually
based on their business processes
- final user sign-off

- 18 Best User Acceptance Testing (UAT) Tools in

2024
62
Why customer / user involvement

 Users know:
- what really happens in business situations
- complexity of business relationships
- how users would do their work using the system
- variants to standard tasks (e.g. country-specific)
- examples of real cases
- how to identify sensible work-arounds

Benefit: detailed understanding of the new system

63
Contract acceptance testing
 Based on the results, the customer considers
whether the software system is free of (major)
deficiencies and whether the service defined by
the development contract has been accomplished
and is acceptable.
 The test criteria are the acceptance criteria
determined in the development contract.
Therefore, these criteria must be stated as
unambiguously as possible.
 Additionally, conformance to any governmental,
legal, or safety regulations must be addressed
here. 64
Alpha and Beta tests: similarities
 Testing by [potential] customers or
representatives of your market
- not suitable for bespoke software
 When software is stable
 Use the product in a realistic way in its
operational environment
 Give comments back on the product
- faults found
- how the product meets their expectations
- improvement / enhancement suggestions?
65
Alpha and Beta tests: differences
 Alpha tests are carried out at the producer’s
location, while beta tests are carried out at the
customer’s site.
 The idea is that “if you make dogfood, try it
yourself first.”
 Large suppliers of software like Microsoft and
Google advocate this approach before beta
testing.

66
Acceptance testing motto

If you don't have patience to test the system

the system will surely test your patience

67
 Lifecycle

1 2 3 ISTQB / ISEB Foundation Exam Practice

4 5 6

Contents
Models for testing, economics of testing
High level test planning
Component Testing
Integration testing in the small
System testing (non-functional and functional)
Integration testing in the large
Acceptance testing
Maintenance testing
68
 Maintenance testing
Testing New Product Versions
 Testing to preserve quality after software
maintenance
 when a product is adapted to new operational
conditions (adaptive maintenance, updates of
operating systems, databases, middleware) or
 when defects are corrected (corrective
maintenance).
 Testing changes made during maintenance can
be difficult because the system’s specifications
are often out of date or missing.
69
What to test in maintenance testing
 Test any new or changed code
 Impact analysis
- what could this change have an impact on?
- how important is a fault in the impacted area?
- test what has been affected, but how much?
• most important affected areas?
• areas most likely to be affected?
• whole system?
 The answer: “It depends”

70
Regression testing
 regression testing involves executing test
cases that have been executed before.
 Regression tests are executed whenever
- the software changes, either as a result of fixes or
new or changed functionality.
- It is also a good idea to execute them when some
aspect of the environment changes, for example
when a new version of a database management
system is introduced or a new version of a source
code compiler is used.
71
Criteria Re-testing Regression Testing
The process of re-executing selected test
The process of executing test cases that
cases to ensure that the software still
Definition failed in previous test cycles to verify if
functions correctly after any new changes
the issues have been fixed.
have been made.
To ensure that the software still works
Focuses on executing test cases related
Purpose correctly and have not been negatively
to fixed defects or failures.
impacted by the changes.
Uses an environment that closely
Uses the same environment as the resembles the production environment,
Test Type
previous test cycle. including hardware, software, and network
configurations.

It may require a range of test data to cover

It may require specific test data related
Test Data different scenarios and ensure coverage of
to the previously failed test cases.
the impacted functionalities.

Test The test cases related to the changed or

The failed test cases are given priority
Execution impacted areas are given priority, followed
and executed first.
Order by other relevant test cases.
Typically performed immediately after Performed after changes have been made
Timeframe
the fixes have been implemented. to the software.
Focuses on the specific areas where Focuses on a wider range of test cases to
Coverage
defects or failures were reported. cover the impacted functionalities.
It makes sure that the original fault is
Bug detection It checks for unexpected side effects.
corrected. 72
 19 Best Regression Testing Tools Reviewed
For 2024 - The CTO Club
 Top 20 Regression Testing Tools in 2024 |
BrowserStack
 https://testsigma.com/regression-
testing#best-regression-testing-tools

73
 Lifecycle

1 2 3 ISTQB / ISEB Foundation Exam Practice

4 5 6

Summary: Key Points

V-model shows test levels, early test design
High level test planning
Component testing using the standard
Integration testing in the small: strategies
System testing (non-functional and functional)
Integration testing in the large
Acceptance testing: user responsibility
Maintenance testing to preserve quality
74