UNIT – I

1. SOFTWARE DEVELOPMENT LIFE CYCLE MODELS

1.1 PHASES OF SOFTWARE PROJECT

A software project is made up of a series of phases. Broadly, most software projects
comprise the following phases.
 Requirements gathering and analysis
 Planning
 Design
 Development or coding
 Testing
 Deployment and maintenance

 Requirements Gathering and Analysis

During requirement gathering, the specific requirements of the software to be built are
gathered and documented. The requirements get documented in the form of a system
requirements specification (SRS) document.

 Planning
The purpose of the planning is to come up with a schedule, the scope, and resource
requirements for a release. A plan explains how the requirements will be met and by which
time. The planning phase is applicable for both development and testing activities. At the end
of this phase, both project plan and test plan documents are delivered.

 Design
The purpose of the design phase is to figure out how to satisfy the requirements
enumerated in the system Requirements Specification document. The design phase produces a
representation that will be used by the following phase, the development phase. This
 Unit – I Software Testing

representation should serve two purposes. First, from this representation, it should be possible
to verify that all the requirements are satisfied.

 Development or coding
This development or coding phase comprises coding the programs in the chosen
programming language.

 Testing

Testing is the process of exercising the software product in pre – defined ways to check
if the behavior is the same as expected behavior. By testing the product, an organization
identifies and removes as many defects as possible before shipping it out.

 Deployment and maintenance

Once a product is tested, it is given to the customers who deploy it in their environments.
As the users start using the product in their environments, they may observe discrepancies
between the actual behavior of the product and what they were given to expect (either
discrepancies could end up as product defects, which need to be corrected). Maintenance is
made up of corrective maintenance (for example, fixing customer – reported problems),
adaptive maintenance (for example, making the software run on a new version of an operating
system or database), and preventive maintenance (for example, changing the application
program code to avoid a potential security hole in an operating system code).

1.2 QUALITY, QUALITY ASSURANCE, AND QUALITY CONTROL:

A software product is designed to satisfy certain requirements of a given customer (or
set of customers). For each such feature, the expected behavior is characterized by set test
cases. Each test case is further characterized by

Page 2 of 86
 Unit – I Software Testing

 The environment under which the test case is to be executed:

 Inputs that should be provided for that test case;
 How these inputs should get processed;
 What changes should be produced in the internal state or environment; and
 What out puts should be produced.
The actual behavior of a given test case, under a given set of inputs ,in a given internal state is
characterized by
 How these inputs actually get processed.
 What changes are actually produced in the internal state or environment and
 What outputs are actually produced

1.3 TESTING, VERIFICATION AND VALIDATION

The narrow definition of the term “testing” is the phase that follows coding and
precedes deployment. Testing is traditionally used to mean testing of the program code.
Timely testing increases the chances of a product or service meeting the customer’s
requirements. The purpose of testing is to uncover defects in the system,(and to have someone
fix the defects). Testing is done by a set of people with in a software product organization
whose goal and charter is to uncover the defects in the product before it reaches the customer.
Testing is not meant to replace other ways of ensuring quality (like reviews).Verification takes
care of activities to focus on the question “are we building the product right” and validation
takes care of a set of activities to address the question” are we building the right product”

1.4 PROCESSMODEL TO REPRESENT DIFFERENT PHASES

A process model is a way to represent any given phase of software development that
effectively builds in the concepts of validation and verification to prevent and minimize the
delay between defect injection and defect detection. In this model each phase of a software
project is characterized by the following:
 Entry criteria, which specify when that phase can be started. Also included are
the inputs for the phase.

Page 3 of 86
 Unit – I Software Testing

 Tasks or steps that need to be carried out in that phase along with
measurements that characterize the tasks.
 Verification, which specifies methods of checking that the tasks have been
carried out correctly.
 Exit criteria, which stipulate the conditions under which one can consider the
phase as done. Also included are the outputs for only the phase.

1.5 LIFE CYCLE MODELS

The ETVX model characterizes a phase of a project. A Life Cycle model describes how the
phases combine together to form a complete project or life cycle. The activities performed In
any given software project, apart from the most common activities or phases-requirements
gathering, design, development, testing & maintenance- there could be other activities as well.
The deliverables from each activity, Each activity produces a set of deliverables, which
are the end products of that activity. Methods of validation of the deliverables. The outputs
produced by a given activity represent the goal to be satisfied by that activity. The sequence of
activities. The different activities work together in unison in a certain sequence of steps to
achieve overall project goals.
Methods of verification of each activity, including the mechanism of communication
amongst the activities. The different activities interact with one another by means of
communication methods. We will now look at some of the common life cycle models that are
used in software projects. For each model, we will look at:
 A brief description of the model;
 The relationship of the model to verification and validation activities; and
 Typical scenarios where that life cycle model is useful.

1. 5.1 Waterfall model

In the Waterfall model, a project is divided into a set of phases (or activities). Each phase is
distinct, that is, there are clear lines of separation between the phases, with very clear
demarcation of the functions of each of the phases.

Page 4 of 86
 Unit – I Software Testing

one example of a project in the Waterfall model in Figure .

Waterfall Model

Overall business
requirements.

Software
requirements

Planning

High-level design

Low-level design

coding

Testing

The project goes through a phase of requirements gathering. At the end of

requirements gathering, a System Requirements Specification document is produced. This
becomes the input to the design phase. During the design phase, a detailed design is
produced in the form of a System Design Description. Suppose a design was created for a
given set of requirements and the project passed on to the programming/ development
phase. At this point of time, it was found that it was not possible to develop the programs
because of some limitations.

Page 5 of 86
 Unit – I Software Testing

1.5.2 Prototyping and rapid application development models

 Early and frequent user feedback will increase the chances of a software project
meeting the customers’ requirements.
 Changes are unavoidable and the software development process must be able to
adapt itself to rapid changes.
The Prototyping model comprises the following activities.
 The software development organization interacts with customers to understand their
requirements.
 The software development organization produces a prototype to show how the
eventual software system would look like.
 The customer and the development organization review the prototype frequently so
that customer’s feedback is taken very early in the cycle (that is, during the
requirements Specification document.)
 Once the SRS document is produced, the prototype can be discarded.
Computer Aided Software Engineering (CASE) tool is used throughout the life cycle, right from
requirements gathering. Such CASE tools have
 Methodologies to elicit requirements;
 Repositories to store the gathered requirements and all downstream entities such as
design objects; and
 Mechanisms to automatically translate the requirements stored in the repositories to
design and generate the code in the chosen programming environment.

1.5.3 Spiral or iterative model

The spiral or interactive model follows a process in which the requirements gathering
design, coding, and testing are performed iteratively till all requirements are met.

Page 6 of 86
 Unit – I Software Testing

This model enables incremental development which the product evolves, with
requirements getting added to it dynamically.

1.5.4 The v model

The waterfall model viewed testing as a post development activity. The spiral model
took this one step further and tried to break up the product into increments each of which can
be tested separately. The v model starts off being similar to the waterfall model in that it
envisages product development to be made up of a number of phases or levels.
Phases of testing for different development Phases.

Page 7 of 86
 Unit – I Software Testing

Overall business Acceptance testing

requirements

Software System testing

requirements

High-level design Integration testing

Low-level design Component testing

coding Unit testing

Testing

Page 8 of 86
 Unit – I Software Testing

This is the basis for v model, which presents excellent advantages for verification and
validation. As shown in above figure. Along with the actual activities and retain the test
execution downstream, after the product is built.

Page 9 of 86
 Unit – I Software Testing

1.5. 5. Modified v model

The V model split the design and execution portion of the various types of tests and
attached the test design portion to the corresponding earlier phases of the software life cycle.
The V model does not explicitly assure this natural parallelism commonly found in product
development. Just as the V model introduced various types of testing.
The modified V model introduces various phases of testing. It moved to integration-
testing phase and so on. These are actually two sides of the same coin and thus provide
complimentary views. The main advantage the modified V model brings to the table is the
recognition of the parallelism present in different parts of testing the is possible. It is
determined by a set of entry criteria.
Design Unit test Components(1,2….) Components IT System
complete complete complete complete Complete complete
Unit testing

Component
testing
Integration
testing Entry
System criteria
testing
Acceptance
testing
Exit criteria

They have been introduced just to denote that integration testing can start after two
components have been completed, and when all components are integrated and tested, the
next phase of testing that is, system testing can start.

Page 10 of 86
 Unit – I Software Testing

WHITE BOX TESTING:

WHAT IS WHITE BOX TESTING?
White box testing id a way of testing the external functionality of the code by
examining and testing the program code that realizes the external functionality. This is also
known as clear box, or glass box or open box testing. A number of defects come about because
of incorrect translation of requirements and design into program code. Some other defects are
created by programming errors and programming language idiosyncrasies. The different
methods of white box testing discussed in this chapter can help reduce the delay between the
injection of a defect in the program code and its detection. White box testing is classified into
“static” and “structural” testing.

Page 11 of 86
 Unit – I Software Testing

1. STATIC TESTING
Static testing is a type which requires only the source code of the product, not the binaries
or executable.
 The code works according to the functional requirement;
 The code has been written in accordance with the design developed earlier in the
project life cycle;
 The code for any functionality has been missed out;
 The code handles error properly.
Static testing can be done by humans or with the help of specialized tools.

 Static testing by humans

This process has several advantages.
1. Sometimes humans can find errors that computers cannot.
2. By making multiple humans read and evaluate the program, we can get multiple
perspectives and therefore have more problems identified upfront than a computer
could.
3. A human evaluation if the code can compare it against the specifications or design and
thus ensure that it does what is intended to do. This may not always be possible when a
computer runs a test.
4. A human evaluation can detect many problems it one go and can even try to identify the
root causes of the problems.
5. By making humans test the code, before execution, computer resources can be saved.
Of course, this comes at the expense of human resources.
6. A proactive method of testing like static testing minimizes the delay on identification of
the problems.
7. From a psychological point of view, finder defects later in the cycle (for example, after
the code is compiled and system is being put together) creates immense pressure on
programmers.

Page 12 of 86
 Unit – I Software Testing

There are multiple method of static testing by humans.

1. Desk checking if the code.
2. Code walkthrough.
3. Code review.
4. Code inspection.

 Desk checking
Normally done manually by the author, of the code, desk checking is a method to verify the
portions of the code for correctness. Such ,verification is done by comparing the code with the
design or specifications to make sure that the code does what it is supposed to do and
effectively. Whenever errors are found, the author applies the corrections for errors pm the
spot.
1. A developer is not the person to detect problems in his or her own code.
2. Developers generally prefer to write new code rather than do any, form of testing1!
3. This method is essentially person-dependent and thus may not work consistently across
all developers.

 Code walkthrough
This methods and formal inspection(described in the nest section) are group-oriented
methods. In walkthrough, set of people look at the program code, and raise the questions for
the author.

 Formal inspection
Code inspection-also called Fagan inspection (named after the original formulator)-is a
method, normally with high degree of formalism. The focus of this method is to detect all
faults, violations and other side-effects.
1. Demanding thorough preparation before an inspection/review;
2. Enlisting multiple diverse views;
3. Assigning specific roles to the multiple participants; and
4. Going sequentially through the code in a structured manner.

Page 13 of 86
 Unit – I Software Testing

 Static analysis tools

The review and inspection mechanisms described above involve significant amount of
manual work.
1. Whether there are unreachable codes (usage of GOTO statements sometimes
creates this situation; there could be other reasons too)
2. Variables declared but not used
3. Mismatch in definition and assignment of values to variables
4. Illegal or error prone typecasting of variables
5. Use of non-portable or architecture-dependent programming constructs
6. Memory allocated but not having corresponding statements for freeing them up
memory
7. Calculation of cyclamate complexity (covered in the sectio0n3.3)

2. STRUCTURAL TESTING:
Structural testing takes into account the code, code structure, internal design, and
how they are coded. The fundamental difference between structural testing and static testing Is
that in structural testing tests are actually run by the computer on built product, whereas in
static testing, the product is tested by humans using the source code and not the executables
or binaries.
Structural testing can be further classified into unit/code functional testing, code
coverage, and code complexity testing.

 Unit/code functional testing:

This initial part of structural testing corresponds to some quick checks that a developer
performs before the code to more extensive code coverage testing or code complexity testing. This can
happen by several methods:

Page 14 of 86
 Unit – I Software Testing

1. Initially, the developer can perform certain obvious tests, knowing the input
variables and the corresponding expected output variables. By repeating these tests
for multiple values of input variables, the confidence level of the developer to go to
the next level increases.
2. For modules with complex logic or conditions, the developer can build a “debug
version” of the product.
3. Another approach to do the initial test is to run the product under a debugger or an
Integrated Development Environment (IDE).

 Code coverage testing:

Code coverage testing involves designing and executing test cases and finding out
the percentage of code that is covered by testing. The percentage of code coverage by a test is
found by adopting a technique called instrumentation of code.

Code coverage testing is made up of the following type of coverage:

1. Statement coverage.
2. Path coverage.
3. Condition coverage.
4. Function coverage.

 Statement coverage:
Program constructs in the conventional programming language can be
classified as

1. sequential control flow

2. Two-way decision statement like if then else
3. Multi-way decision statement like switch
4. Loops like while do, repeat until and for
Statement coverage refers to writing test cases that execute each of the program statement.

Page 15 of 86
 Unit – I Software Testing

 Path coverage
In path coverage we split a program into a number of distinct paths. A program can start
from the beginning and take any of the paths to completion.
 A
 B-D-G
 B-C-E-H
 B-C-F-G
 B-C-F-H

 Condition coverage
In the above example even if we have covered all the paths possible it would not mean
that the program is fully tested.

 Function coverage
This is a new addition to structural testing to identify how many program functions are
covered by test cases.
1. Functions are easier to identify in a program and hence it is easier to write test cases
to provide function coverage.
2. It function to achieve 100% function coverage than 100% coverage in any of the
earlier methods
3. Functions have a more logical mapping to requirements
4. Since functions are a means of realizing the importance of functions can be prioritized
based on the importance of the requirements they realize.
5. Functions coverage provides a natural transition to black box testing
Thus function coverage can help in improving the performance as well as quality of the product.

Code complexity testing

Cyclomatic complexity is a metric that quantifies the complexity of a program and thus
provides answers to the above questions.

Page 16 of 86
 Unit – I Software Testing

In previous sections we saw the different types of coverage that can be provided to test a
program.
1. Identify the predicates or decision points. (Boolean conditions and conditional statements)
2. Ensure that the predicates are simple
3. Combine all sequential statements into a single node
4. A simple predicate as simplified in (2) above edges emanating from this one node
5. The edges terminate at some node.

Challenges in white box testing:

White box testing requires a sound knowledge of the program code and the programming
language. Human tendency of a developer being unable to find the defects in his or her code as
we saw earlier most of us have blind spots in detecting errors in our own products since white
box testing involves programming who write the code it is quite possible
That they May not be most effective in detecting defects in their own work products. An
independent perspective could certainly help.

-- x - -

Page 17 of 86
 Unit – II Software Testing

UNIT – II

2. BLACK-BOX TESTING
2.1 WHAT IS BLACK-BOX TESTING?
 Black Box testing is done from customer’s view point.
 The text engineer gives a set of inputs and expects outputs and is unaware of how those
inputs are transformed into outputs by the software.
 Black box testing convenient to administer because they use the complete finished
product and do not require any knowledge of its construction.
 Independent test laboratories can administer black box tests to ensure functionality and
compatibility.
 These testing require a functional knowledge of the product to be tested.

2.2 WHY BLACK-BOX TESTING?

Black box testing helps in the overall functional verification of system under test.
 Black box testing is done on requirements: It was identify any incomplete, inconsistent
requirement, when system is complete entity after it is tested.
 Black box testing addresses a stated requirements as well as implied requirements: No
all the requirements are stated explicitly, but they are deemed implicit.
 Black box testing encompasses end user perspectives: We want to test a product from
external perspective, end-user perspective are an integral part of black box testing.
 Black box testing was handle valid and invalid inputs: It is natural for users to make
errors while using a product. It is not sufficient for black box testing to simply have valid
inputs. It ensure a product behaves as expect in valid and does not hang an invalid
input.
The tester may not know the technology or the internal logic of the product.

2.3 WHEN TO DO BLACK-BOX TESTING?

 Testers get an involved right from the requirements gathering and analysis phase for
system under test.

Page 18 of 86
 Unit – II Software Testing

 Test scenarios and data are prepare the test construction phase of the test life cycle, the
software is design phase.
 Once the code is ready for delivered for testing, test execute can be done.
 All the test scenarios develop the construction phase are executed

2.4 HOW TO DO BLACK-BOX TESTING?

Black box testing deals with a various techniques to execute test scenarios.
The various techniques are
1. Requirements based testing
2. Positive and negative testing
3. Boundary value analysis
4. Decision tables
5. Equivalence partitioning
6. State based testing
7. Compatibility testing
8. User documentation testing
9. Domain testing

2.4.1. Requirements based testing:

 Requirement based testing deals with the validating the requirements given in the
software requirements specification (SRS).

 All explicit and implicit requirements are collected as ”test requirements specification”
(TRS)

Page 19 of 86
 Unit – II Software Testing

Sample requirement specification

Requirement Priority
S.No Description
identifier (High,med,low)
Inserting the key numbered 123-456 and
1 BR-01 turning it clockwise should facilitate H
locking
Inserting the key numbered 123-456 and
2 BR-02 turning it anticlockwise should facilitate H
unlocking
Only key number 123-456 can be used to
3 BR-03 H
lock and unlock
4 BR-04 No other can be used to lock M
5 BR-05 No other can be used to unlock
The lock must not open even when it is
6 BR-06 M
hit with a heavy object
The lock and key must be made of metal
7 BR-07 and must weight approximately 150 L
grams.
L
Lock and unlock directions should be
8 BR-08
changeable for usability of left-handers

 Requirements are tracked by a requirements traceability matrix (RTM)

Page 20 of 86
 Unit – II Software Testing

Sample Requirement Traceability Matrix

Test
Priority Test Phase of
Req.ID Description case
(H,M,L) conditions testing
IDs
Inserting the key numbered
123-456 and turning it Use key Lock- Unit,
BR-01 H
clockwise should facilitate 123-456 001 Component
locking

Inserting the key numbered

H
123-456 and turning it Use key Lock- Unit,
BR-02
anticlockwise should 123-456 002 Component
facilitate unlocking

Use key
123-456 to
Only key number 123-456
Lock Lock-
BR-03 can be used to lock and H Component
Use key 003
unlock
123-456 to
Unlock

 The test conditions column lists the different ways of testing the requirements.
 The test cases IDs column can used to complete the mapping between test cases and
requirements.
 The following combinations are possible between the requirements and test cases.
-One to one
-One to many
-Many to one
-Many to many
-One to none

Page 21 of 86
 Unit – II Software Testing

The requirement traceability matrix provides a various test metrics. They are
 Requirements addressed priority wise
 Number of test cases requirements.
 Total number of test cases prepared
Once the test cases are executed, the results in metrics are
 Total number of test cases passed.
 Total number of test cases failed.
 Total number of defects in requirements.
 Number of requirements completed.
 Number of requirements pending

2.4.2. Positive and negative testing:

Positive testing is done to verify the known test conditions and negative testing is done
to break the product with unknowns.
Positive testing tries to prove that a given product does what t is supposed to do. When
a test case verifies the requirements of the product with a set of expected output, it is called
positive testing. The purpose of positive testing is to prove that the product works as per
specifications and expectations. A product delivering an error when it is expected to give an
error is also a part of positive testing.
Negative testing is done to show that the product does not fail when an unexpected
input is given. The purpose of negative testing is to try and break the system. Negative testing
covers scenarios for which the product is not designed and coded.
The difference between positive and negative testing is in their coverage .For positive
testing if all documented requirements and test conditions are covered , then coverage can be
considered to be 100 percentage. Negative testing requires a high degree of creativity among
the testers to cover as many “unknown’ as possible to avoid failure at a customer site.
2.4.3. Boundary value analysis:
Most of the defects in software products hover around conditions
and boundaries. By boundary means “limits” of values of the various variables.
Boundary value analysis is useful to generate test cases when the input (or

Page 22 of 86
 Unit – II Software Testing

output) data is made up of clearly identifiable boundaries or ranges. t is a method

for arriving at tests that are effective in catching defects.
To summarize boundary value testing:
 Look for any kind of gradation in data value which affects computation. The
discontinuities are the boundary values, which require testing.
 Look for any internal limits such as limits of resources. The behavior of the
product at these limits should also be the subject of boundary value testing.
 In the list of boundary values, documented limits on hardware resources.
For examble , if it is documented that a product will run with minimum 4
MB of RAM , make sure you include test cases for the minimum RAM(4MB
in this case.

Case: 1 all buffers free for use

Case: 2 after inserting two buffers and still having free buffers

Case: 3 after inserting the last available buffer. No free buffers

Case: 4 no free buffers and new buffer coming in. first needs freeing

Page 23 of 86
 Unit – II Software Testing

Internal data structures like arrays, stacks and queues need to be checked for boundary
or limit conditions; when they are linked lists used as internal structures, the behavior of the list
at the beginning and end have to be tested thoroughly.
Boundary values and decision tables help identify the test cases that are most likely to
uncover defects. A generalization of both these concepts is the concept of equivalence classes.

2.4.4 Decision tables

A decision table lists the various decision variables, the conditions assumed by each of decision
variables. Let us take a simple example of calculation of standard deduction on taxable income.
The example is meant to illustrate the use of decision tables and not to be construed as tax
advice or a realistic tax scenario in any specific country.
1. The first factor that determines the standard deduction in the filing status. The basic
standard deduction for the various filling status are

Single $4,750
Married, filling a joint return $9,500
Married, filling a separate return $7,000

2. If a married couple is filling separate returns and one spouse is not taking standard
deduction, the other spouse also is not eligible for standard deduction.
3. An additional $1000 is allowed as standard deduction. If either the filer is 65 years or
older or the spouse is 65 years or older cable when the filling status is “Married” and
filling “joint”.
4. An additional $1000 is allowed as standard deduction. If either the filer is blind or
the spouse is blind.
From the above description, it is clear that the calculation of a standard deduction depends
on the following factors.

Page 24 of 86
 Unit – II Software Testing

1. Status of filling of the filer

2. Age of the filer
3. Whether the filer is blind or not.
In addition, some factors also come into play in calculating standard deduction.
1. Whether spouse has claimed standard deduction.
2. Whether spouse is blind
3. Whether spouse is more than 65 years old.
The steps in decision table are
 Identify the decision variables.
 Identify the possible values of each of the decision variables.
 Enumerate the combinations of the allowed values of each of the variables.
 Identify the cases when values assumed by a variable are immaterial for a given
combination of other input variables. Represent such variables by the don’t care
symbol.
 For each combination of values of decision variables list out the action or expected
result.
 Form a table. Listing in each but the last column a decision variable. In the last column,
list the action item for the combination of variables in that row.
Once a decision table is useful when input and output data can be expressed as Boolean
conditions (TRUE, FALSE and DON’T CARE). Once a decision table if formed, each row of the
tables acts as the specification for one test case.

2.4.5 Equivalence partitioning:

It is a software testing technique that involves identifying a small set of representative input
values that produce as many different output conditions as possible. This reduces the number
of permutations and combinations of input, output values used for testing, thereby increasing
the coverage and reducing the effort involved in testing.

Page 25 of 86
 Unit – II Software Testing

The set of input value generate one single expected output is called a partition. When
the behavior of the software is the same for set values, then the set is termed as an
equivalence class or a partition.

EXAMPLE : LIFE INSURANCE PREMIUM RATES

A Life insurance company has basic premium of $0.50 for all ages. Based on the age group
, an additional monthly premium has to be paid.
AGE GROUP ADDITIONAL PREMIUM
Under 35 $1.65
35-59 $2.87
60+ 0$6.00
Based on the equivalence partitioning technique, the equivalence partitions that based on age
are given below.
 Below 35 years of age. (valid input)
 Between 35 and 59 years of age(valid input)
 Above 60 years of age(valid input)
 Negative age. (invalid)
 Age as any three-digit number. (valid)
 Age as 0. (invalid)
2.4.6 State based testing:

State based testing is very useful in situations where

 The product under test is a language processor.
 Workflow modelling depends upon current state.
 Dataflow modelling the system is modelled as a set of dataflow, lead from one state to
another.
Consider the application was have a validate of number according to simple rules
 The number can start with an optional sign.

Page 26 of 86
 Unit – II Software Testing

 The optional sign can be followed by digits.

 The digits can be optionally followed by decimal points.
 If there is a decimal point, then there should be two digits after the decimal.

+ or
1 2 3 4 5

Digit digit
Decimal point
blank blank
digit 6

2.4.7 Compatibility testing

Common techniques that are used for performing compatibility, using a compatibility table are
 Horizontal combination: All values of parameters that can coexist with the product for
executing the set test cases are grouped together as a row in the compatibility matrix.
 Intelligent sampling: To solve this problem, combinations of infrastructure parameter
are combined with the set of features intelligently and tested.
Compatibility testing of a product involving parts of it can be divide two types.
 Backward compatibility testing.
 Forward compatibility testing

2.4.8 User documentation testing:

 It covers all manuals, user guides, installation guides, setup guides, read me file and
online help provide with a software help.
User documentation testing has two objectives.
 To check if what is stated in document available in the product.
 To check if what there in the product is explain correctly in the document.
Some benefits of user documentation testing are,

Page 27 of 86
 Unit – II Software Testing

 It is highlighting problems.
 Ensures consistency of documentation and product.
 Results in less difficult support calls.
 New programmers and testers documentation to learn external functionality.
 Customers need less training.
2.4.9 Domain testing:
 Black box testing perform test without looking at the program code but looking at the
specification.
 Domain testing can be consider as the next level of testing.
 Thus the domain testing can be consider as an extension of black box testing.
 The test engineers performing this type of testing are selected because have a in-depth
knowledge of the business domain.

WHITE-BOX
TESTING

BLACK-BOX
TESTING

DOMAIN
TESTING

Page 28 of 86
 Unit – II Software Testing

Domain testing is ability to design and execute test cases relate to people who will buy
and use the software. It helps in understanding the problems they are trying to solve and the
ways in which they are using the software to solve them. If a tester does not understand the
system or the business processes, it would be very difficult for him or her to use, let alone test,
the application without the aid of test scripts and cases. Domain testing exploits the tester’s
domain knowledge to test the suitability of the product to what the users do on a typical day.

2. INTEGRATION TESTING
 Testing that interact between module and other system.
 It defines as the set of integrations among components.
 This test is a both type of testing and phase of testing.

2.1 INTEGRATION TESTING AS A TYPE OF TESTING

 It was test an interface.
 It contains both internal and external interface.
 Internal are those provide communication of two modules within project.
 Integration Testing is a black box testing approach.
 It is also called as a “grey box testing”.
A set of modules and interfaces

Component 1

Component 2 Component 3 Component 4

Component 5 Component 6 Component 7 Component 8

Page 29 of 86
 Unit – II Software Testing

The integration testing divided into four types. They are

 Top-down integration
 Bottom-up integration
 Bi-directional integration
 System integration

2.2.1 Top-down integration:

 It involves testing the topmost component interface with other components in same
order as navigate from top to bottom, till cover all the components.

Example of top down integration

Component 1

Component 2 Component 3 Component 4

Component 5 Component 6 Component 7 Component 8

Order of testing interfaces for the example

Step Interfaces tested
1 1-2
2 1-3
3 1-4

Page 30 of 86
 Unit – II Software Testing

4 1-2-5
5 1-3-6
6 1-3-6-(3-7)
7 (1-2-5)-(1-3-6-(3-7))
8 1-4-8
9 (1-2-5)-(1-3-6-(3-7))-(1-4-8)

They are two approaches used in top down integration testing.

*Breadth first approach
*Depth first approach
2.2.2 Bottom-up integration:
 It was opposite of top-down integration.
 When the components for a new product development becomes available in reverse
order.
Example of Bottom-up integration

Component 8

Component 5 Component 6 Component 7

Component 1 Component 2 Component 3 Component 4

Page 31 of 86
 Unit – II Software Testing

Order of interfaces tested using bottom up integration

Step Interfaces tested

1 1-5
2 2-6,3-6
3 2-6-(3-6)
4 4-7
5 1-5-8
6 2-6-(3-6)-8
7 4-7-8
8 (1-5-8)-(2-6-(3-6)-8)-(4-7-8)

One difference between top down and bottom up integration is the arrows are two-way
arrows. The arrows from top to bottom indicate interaction or control flow. The arrows from bottom to
top indicate integration approach or integration path.

2.2.3 Bi-directional integration:

 It is a combination of top-down and bottom-up integration approaches used together to
derive integration steps.

Page 32 of 86
 Unit – II Software Testing

Bi-directional integration

Component 1

Component 2 Component 3 Component 4

Component 5 Component 6 Component 7 Component 8

 The connectivity is stubs and drivers. Drivers are used to provide upstream connectivity,
Stubs provide downstream connectivity.
 This approach is also called as “sandwich integration”.

Steps for integration using sandwich testing

Step Interfaces tested

1 6-2
2 7-3-4
3 8-5
4 (1-6-2)-(1-7-3-4)-(1-8-5)

An area where this approach comes in handy is when migrating from a two-tier to a
three-tier environment.

Page 33 of 86
 Unit – II Software Testing

2.2.4 System integration:

 It means all components of system integrated and tested as a single unit.
 It is divide into two types;
 Component or sub-system integration
 Final integration or system integration
 This approach is called as “big-bang” integration.
 This approach saves time and effort.
This approach also has some disadvantages. They are
 The ownership for connecting the root cause of defect may be a difficult pinpoint.
 A certain component may take an excessive amount of time to be ready.
 When integration testing happens in the end, the pressure from the approach release
date is very high.
 When a failure or defect is encountered during system integration, it is very difficult to
locate the problem.

3 Choosing Integration Method

S.no Factors Suggested integration

method
1 Clear requirements and design Top-down
2 Dynamically changing requirements, design Bottom-up
,architecture
3 Changing architecture, stable design Bi-directional
4 Limited changes to existing architecture with less Big-bang
impact
5 Combination of above Select one of the above after
careful analysis

Page 34 of 86
 Unit – II Software Testing

2.4 INTEGRATION TESTING AS A PHASE OF TESTING

 The focus was not only in function of the components work.
 It is used to find defect, the defects are not arise of interface.
 It should have ensured completeness and coverage of testing.
 It was termed as “ad hoc testing”.
 There are different technologies of ad hoc, such as exploratory, monkey, out of the
testing.
 It has a sequence of related operations.
 The combine of test cases are called scenario test cases.

2.5 SCENARIO TESTING

 It is a set of realistic user activities that are used for evaluating the product.
 It has two types, they are
 System scenarios
 Use-case scenarios

2.5.1 System scenarios:

System scenario is a method of set of activities used for scenario testing covers several
components of system.

 Story line: It develops a story line combine of various activities of product by an end
user.
 Life cycle: Consider an object of different transition that happen to the object.
 Deployment stories from customer: Develop a scenario from customer deployment
details and create a set of activities by various users in implementation.
 Business verticals: It likes a pharmaceuticals, software houses and government
organization.
 Battle ground: Effective scenarios will have a combination of current customer
implementation.

Page 35 of 86
 Unit – II Software Testing

2.5.2 Use-case scenarios:

 It is a stepwise procedure with different user and associated parameters. A use case
scenario can include stories, pictures and deployment details.
 A use case can involve several roles perform different activities based on the role.
 Product should do for a particular activity is a system behavior.
 Users with specific role to interact between the actors and the system are called agents
CHEQUE

QUERY

CASH RESPONSE

2.6 DEFECT BASH

 Defect bash is a ad hoc testing
 It was perform in different roles in an organization test the product. It is popular among
application development companies.
 The test by all the participants during defect bash is not based on written test cases.
 Defect bash brings together plenty of good practices that are popular in testing industry.
They are as follows.
 Enabling people.
 Bring in people who have different levels of product.

Page 36 of 86
 Unit – II Software Testing

 Bring fresh pairs of eyes to uncover new defects.

 Enable people to say “system works” as well enable them to “break the system”
All activities in the defect bash are planned activities, except for what to be tested.
It involves several steps. They are
Choosing the frequency and duration of defect bash:
 Optimizing the small duration is a big saving as a large number of people involved.
 It will in occur low return on investment, too few defect bashes.

Selecting the right product build:

 Defect bashes involve large number of people, effort and plan a good quality.
 Large number of people are prepare a confidence of good product.

Communicating the objective of defect bash:

 Defect bash is an ad hoc activity, its purpose and objectives have to be clear.
 The objective was find a large number of uncovered defects.

Setting up and monitoring the lab:

 The majority of defect bash fail due to inadequate hardware, wrong software
configuration.
 There are two type of defect are functional and non-functional defects.

Taking actions and fixing issues:

 Many defects are duplicate defects.
 The defects need to be classified in to issue at a higher level.
Optimizing the effort involved in defect bash:
 There are several ways to optimize effort involved in a defect bash if a record of
objectives.
 Hence the defect bash further can classified into,
 Feature defect bash
 Integration defect bash
 Product defect bash

-- x - -

Page 37 of 86
 Unit – III Software Testing

UNIT – III
SYSTEM AND ACCEPTANCE TESTING

3.1 SYSTEM TESTING OVERVIEW:

The testing conducted on the complete integrated products and solutions to evaluate
system compliance with specified requirements on functional and non-functional aspect is
called system testing.
System testing is defined as a testing phase conducted on the complete integrated
system, to evaluate the system compliance with its specified requirements it is done after unit,
component, and integrating testing phases.
A system can also be defined as a set of hardware, software and other parts that
together provide product features and solutions. System testing helps in uncovering the defects
that may not be directly attributable to a module or an interface. System testing brings out
issues that are fundamental to design, architecture, and code of the whole product.
On the non functional side, system brings in different testing types.
1. Performance/ load testing: To evaluate the time taken or response time of the system
performs it is required function in composition with different versions of same product.

2. Scalability testing :That requires enormous amount of resource of to find out the
maximum capability of system parameters.

3. Reliability testing: To evaluate ability of the system or an independent component of

the system to perform its required functions repeatedly for specified performed of
times.

4. Stress testing: Evaluate a system beyond the limits of the specified requirements or
system resources (such as disk space, memory processor, utilization) to ensure the
system does not break down unexpectedly.

Page 38 of 86
 Unit – III Software Testing

5. Interoperability testing: This testing is done to ensure that to or more products can
exchange information, use the information and work closely.

6. Localization testing: Testing conducted to verify that the localized product works in
different language.

The definition of system testing keep changing covering wider and more high-level aspects
depending on the context. Global perspective
Test phase

Component System testing Sub-System testing

supplier

Product System testing System testing

organization

Solution System testing Solution testing

integration

System testing is performed on the basis of the return test cases according to
information collected from details architectural/design documents, module specifications, and
system requirements, specifications, system test cases are created after looking at component
and integration test cases and are also at the same time designed to include the functionality
that tests the system together.

3.2 WHY IS SYSTEM TESTING DONE?

An independent test team normally does system testing. It is conducted with an
objective to find product level defects and building the confidence before the product is
released the customer.
System testing helps in identifying as many defects as possible before the customer
finds them in the deployment. System testing helps in reducing the risk of releasing of product.

Page 39 of 86
 Unit – III Software Testing

The system testing phase helps in switching focus of the product development team
towards customers and their use of the product. System testing is done for the following
reasons.
1. Provide independent perspective in testing
2. Bring in customer perspective in testing
3. Provide a “fresh” pair of eyes” to discover defect not found earlier by testing
4. Test product behavior in a holistic, complete and realistic environment
5. Test both functional and non-functional, aspects of the product
6. Build confidence in the product
7. Analysed and reduce the risk of releasing of the product
8. Ensure all requirements are met and ready the product for acceptance testing

3.3 FUNCTIONAL VERSUS NON-FUNCTIONAL TESTING

Functional testing involves testing products functionality and features. Non-functional
testing involves testing the products quality factors. System testing combines both Functional
and non-functional test verification.
Functional testing helps in verifying what the system is supposed to do. It has only two
results as far as requirements fulfilment is concerned-met or not met. It requires in-depth
customer and product knowledge as well as domain knowledge so as to develop different test
cases and find critical defects as the focus of the testing is to find defects. Functional testing is
performed in all phases of testing such as unit testing, component testing, integration testing
and system testing.
Non-functional testing is performed to verify the quality factors. These quality factors
are also called non-functional requirements. Non-functional testing requires large amount to
resources and the results are different for different configurations and resources.

Page 40 of 86
 Unit – III Software Testing

Functional versus non-functional testing

Non-functional testing
Testing aspects Functional testing

Product features and

Involves Quality factors
functionality
Tests Product behavior Behavior and experience
Simple steps written to check Huge data collected and
Result conclusion
expected results analysed
Product implementation ,
Result varies due to Product implementation
resources and configurations
Testing focus Defect detection Qualification of product
Product, domain, design,
Knowledge required Product and domain
architecture, statistical skills
Failures normally due to Code Architecture, design and code
Unit, Component, integration
Testing phase System
system
Repeated only in case of
Test case repeatability Repeated many times failures and for different
configurations
One-time setup for a set of test Configuration changes for each
Configuration
cases test case

3.4 FUNCTIONAL SYSTEM TESTING

Functional testing is performed at various testing phases, there are to obvious
problems.
*Duplication
*Gray area
Duplication refers to the same tests being performed multiple times and gray area
refers to certain tests being missed out in all the phases
Gray areas in testing happen due to lack of product know ledge, lack of languages of
customer usages and lack of co-ordination across test teams.

Page 41 of 86
 Unit – III Software Testing

There are multiple ways system functional testing is performed. There are also many
ways product level test cases derived for functional testing. Some of the common techniques
are given below
1. Design/architecture verification.
2. Business vertical testing
3. Development testing
4. Beta testing
5. Certification, standards, and testing for compliance

3.4.1. Design/architecture verification

The test case are developed and checked against the design and architecture to see
whether they are actual product-level test case. Whereas the functional system test focuses on
the behavior of the complete product.
Some of the guidelines used to reject test cases for system functional testing include as
following
1. Is this focusing on code logic, data structures and unit of the product?
2. Is this specified in the functional specification of any component?
3. Is this specified in design and architecture specification for integration testing?
4. Is it focusing on product implementation but not visible to customers?
5. Is it the right mix of customer usage and product implementation?

3.4.2. Business vertical testing

Workflow automation system can be used by different businesses and services. Using
and testing the product for different business verticals such as insurance, banking, asset
management, and verifying the business operations and usage is called “Business Vertical
Testing”. It is important that product understands the business processes and includes
customer as a feature so that different business verticals can use the product. Some aspects
involves in business vertical testing. They are follows.
*Terminology
*Syndication
*Customization

Page 42 of 86
 Unit – III Software Testing

Business vertical testing can be done in two ways-simulation and replication. In

simulation of a vertical test, the customer or the tester assumes requirements and the business
flow tested. In replication, customer data and process is obtained and the product is completely
customized, tested.

3.4.3. Deployment testing

Deployment testing divided into two types.
*Onsite deployment
* Offsite deployment
Deployment testing that happens in a product development company to ensure that
customer deployment requirements are met is called offsite deployment. Deployment testing is
also conducted after the release of the product is called as onsite deployment.

3.4.4. Beta testing

Developing a product involves a significant amount of effort and time delays in product
releases and the product not meeting the customer requirement are common. A product
rejected by the customer after delivery means a huge loss to the organization. There are many
reasons for a product not meeting the customer requirements. They are as follows.
1. There are implicit and explicit requirements for the product.
2. Since product development involves a good amount of times, some of the requirements
given at the project would have changed by the time the product is delivered
3. The requirements are high-level statements with a high degree of ambiguity.
4. The understanding of the requirement may be correct but their implementation could
be wrong.
5. Lack of usability and documentation makes it difficult for the customer to use the
product and may also result in rejection of the product.
One of the mechanisms used is sending the product that is under test to the customers and
receiving the feedback. This type program is called beta program.
1. Collecting the list of customer and their beta testing requirements along with their
expectations on the product

Page 43 of 86
 Unit – III Software Testing

2. Not all the customer in the list need to agree to the start date and end date of the beta
program,
3. Sending some documents for reading in advance and training the customer on product
usage
4. “beta testing entry criteria” the customer and the product development groups of the
vendor together prepare seta of entry/exit criteria for beta testing
5. Sending the beta product to the customer and enable them to carry out their own
testing
6. Collecting the feedback periodically from the customers and prioritizing the defects for
fixing
7. Responding to customer feedback with product fixes or documentation changers and
closing
8. Analyzing and concluding whether the beta program met the exit criteria
9. Communicate the progress and action items to customer and formally closing the beta
program
10. Incorporating the appropriate changes in the product

3.4.5. Certification, standards, and testing for compliance

A product needs to be certified to be certified with the popular hardware, operating
system, database, and other infrastructure pieces. This is called certification testing.
Testing the product to ensure that these standards are properly implemented is called
testing for standards.
The following are some examples of compliance testing.
 Compliance to FDA this act by the food and drug administrations requires that
adequate testing be done for products such as cosmetics, drugs, and medical sciences.
 508 accessibility guidelines this accessibility set of guidelines require the product to
meet some requirements for its physically challenged user.
 SOX (Sarbanes- Oxley’s act) this act requires that products and services be audited to
prevent financial fraud in the organization

Page 44 of 86
 Unit – III Software Testing

 OFAC and patriot act this act requires the transaction of the banking applications be
audited for misuse of funds for terrorism

3.5 NON-FUNCTIONAL SYSTEM TESTING

The process for non- functional system testing has to be more robust stronger than
functional testing to minimize the need for better planning, and by setting up the configuration
with data population in advance for test execution.

3.5.1 Setting up the configuration

The biggest challenge, setting up the configuration, is common to all types of for non-
functional testing. There are two ways the setup is done-simulates environment and real-life
customer environment
Setting up a configuration is a challenge for the following reasons.
1. The cost involved in setting up such environment is quite high.
2. Some of the components of the environment could be from competing companies
products and it may not be easy to get these
3. The people may not have the skills to set up the environment
4. It is difficult to predict the exact type and nature of data that the customer may use.

3.5.2 Coming up with entry/exit criteria

Coming up with entry and exit criteria is another critical factor in for non- functional
testing?
Meeting the entry criteria is the responsibility of the previous test phase. Before accepting the
product for system testing.

3.5.3 Balancing key resources

The non- functional testing with respect to four key resources CPU, disk, memory, and
network. These four resources in a computer require equal attention as they need to be
judiciously balanced to enhance the quality factors of the product

Page 45 of 86
 Unit – III Software Testing

The following are some examples of basic assumptions that can be made about resources
and configuration.
1. The CPU can be fully utilized as long as it can freed
2. The available memory can be completely used by product as long as the memory
relinquished
3. The cost adding cup or memory is not that expensive as it was earlier
4. More disk space or the complete i/o band width can be used for the product as long as
they are available
5. The customer gets the maximum return on investment(ROI)only if the resources such as
cpu , disk, memory and network are optimally used

3.5.4 Scalability testing

The objective of scalability testing is to find out the maximum capability of the product
parameters. At the beginning of the scalability exercise their many not be an obvious clue about
the maximum capability
 Scalability should increase 50% when the number of CPUs is doubled from minimum
requirement and 40% thereafter, till a given number of CPUs are added. Such attest will be
applicable if the product is CPU intensive.
 Scalability should increase 40% when memory is doubled from the minimum requirement
and 30% thereafter. T his will be applicable if the product is memory intensive.
 Scalability should increase by at least 30% when the number of NIC cards or network band
width is doubled. This aspect has to be tested if the product is network intensive.
 Scalability should increase by at least 50% when the I/O band width is doubled. This aspect
has to be tested if the product is I/O intensive.

3.5.5 Reliability Testing

The reliability a product should not be confused with reliability testing. Producing a
reliable product requires sound techniques, good discipline, and strong management.

Page 46 of 86
 Unit – III Software Testing

1. Defined engineering process

Software reliability can be achieved by following clearly defined processes. The team is
mandated to understand requirements for reliability right from beginning and focuses on
creating a reliable design upfront.
2. Review of work products at each stage
At the end of each stage of the product development lifecycle, the work products
produced are reviewed.
3. Change management procedures
Many errors percolate to the product due to improper impact analysis of changes made
to the product.
4. Review of testing coverage
Allocating time for the different phases and types of testing can help in catching errors
as and when the product is developed.
5. Ongoing monitoring of the product
Once the product has been delivered, it is analyzed proactively for any possibly missed
errors.
(defects)

Time/ iterative tests Time/ iterative tests

To summarize reliability tested product will have the following characteristics:
 No errors or very few errors from repeated transactions.
 Zero downtime.
 Optimum utilization of resources.
 Consistent performance and response time of the product for repeated transactions for
a specified time duration.
 No side-effects after the repeated transactions are executed.

Page 47 of 86
 Unit – III Software Testing

3.5.6 Stress testing:

Stress testing is done to evaluate a system beyond the limits of specified requirements
or resources, to ensure that system does not break. Stress testing helps in understanding how
the system can behave under extreme and realistic situations.
Sometimes the product may not recover immediately when the load is decreased. There are
several reasons for this.
 Some transactions may be in the wait queue , delaying the recovery.
 Some rejected transactions many need to be purged, delaying the recovery.
 Due to failures, some clean-up operations may be needed by the product, delaying the
recovery.
 Certain data structures may have got corrupted and may permanently prevent recovery
from the stress point.

Failure rate
Stress point

Increasing load decreasing load

\The following guidelines can be used to select the tests for stress testing.
1. Repetitive tests
Executing repeated tests ensures that all times the code works as expected.
There are some operations that are repeatedly executed by the customer.
2. Concurrency
Concurrent tests ensure that the code is exercised in multiple paths and
simultaneously. The operations that are used by multiple users are selected and
performed concurrently for stress testing.
3. Magnitude
This refers to the amount to lead to be applied to the product to stress the
system. It can be a single operation being executed for a large volume of users or mix of
operations distributed over different users.

Page 48 of 86
 Unit – III Software Testing

4. Random variation
As explained earlier, stress testing depends on increase/decrease variable load.
Tests that stress the system with random inputs at random instances and random
magnitude are selected and executed as part of stress testing.

3.5.7 Interoperability testing

Interoperability testing is done to ensure the two or more products can exchange
information, use information, and work properly together. The two-way interoperability is
represented by exchange of information between email management and Microsoft word,
where information can be cut and pasted on both directions
1. Consistency of information flow across system
2. Changes to data representation as per the system requirement
3. Correlated interchange of message and receiving appropriate
4. Communication and messages
5. Meeting quality factors

3.6 ACCEPTANCE TESTING

Acceptance testing is a phase after system testing that is normally done by the
customers or representatives of the customer the customer defines a set of test cases that will
be executed to qualify and accept the product.
In cases where the acceptance tests are performed by the product organization alone,
acceptance tests are executed to verify if the product meets the acceptance criteria defined
during the requirements definition phase of the project
Acceptance test cases failing in a customer site may cause the product to be rejected
and may mean financial loss or may mean rework of product involving effort and time

3.6.1 ACCEPTANCE CRITERIA

Acceptance criteria-product acceptance
During the requirements phase each requirements is associated with acceptance
criteria is possible that one or more requirements may be mapped to form acceptance criteria.

Page 49 of 86
 Unit – III Software Testing

Acceptance criteria-procedure acceptance

Acceptance criteria can be defined based on the procedures followed for delivery.
1. User, administration and troubleshooting documentation should be part of the release.
2. Along with binary code, the source code of the product with builds scripts to be
delivered in a CD.
3. A minimum of 20 employees are trained on the product usage prior to deployment.
Acceptance criteria-service level agreements
Service level agreements are generally part a contract signed by the customer and product
organization
 All major defects that come up during first three months of deployment need to be fixed
free of cost;
 Downtime of the implemented system should be less than 0.1%
 All major defects are to be fixed within 48 hours of reporting

3.6.2 Selecting Test Cases for Acceptance Testing

The test cases for acceptance testing are selected from the existing set of test cases
from different phases of testing. Some guidelines on what test cases can be included for
acceptance testing.
1.End-to-end functionality verification’
2. Domain tests
3. User scenario tests
4. Basic sanity test
5. New functionality
6. A few non-functional tests
7. Tests pertaining to legal obligations and service level agreements
8. Acceptance test data
3.6.3 Executing Acceptance Tests
An acceptance test team usually comprises members who are involved in the day-to-day
activities of the product usage or familiar with such scenarios.
Defects reported during acceptance tests could be of different priorities. Test teams help
acceptance test team report defects.
-- x - -

Page 50 of 86
 Unit – IV Software Testing

UNIT-IV
PERFORMANCE TESTING
4.1 INTRODUCTION
The testing performed to evaluate the response time, throughput, and utilization of the
system, to execute its required functions in compared with different versions of the same
product or a different competitive product is called performance testing.

4.2 FACTORS GOVERNING PERFORMANCE TESTING

Some factors governing for performance testing. The factors are
*Throughput
*Response time
*Latency
*Tuning
*Benchmarking
*Capacity planning

throughput

Saturation point

user load

Page 51 of 86
 Unit – IV Software Testing

Client WEB SERVER DATABASE

INTERNET
SERVER
N1 N1
A1
N3
A2
N4

A3

The capability of the system or the product in handling multiple transactions is

determined by a factor called throughput. Throughput represents the number of
request/business transactions processed by the product in specified time duration. The load to
the product can be increasing the number of concurrent operations of the product. This is the
ideal situation for any product and indicates that the product is capable of delivering more
when there are more users trying to use the product. In the second part of the graph, beyond
certain user load conditions, it can be noticed that the throughput comes down.
The next factor is Latency. Latency is a delay caused by the application operating system
and by the environment that are calculated separately.
The next factor that governs the performance testing is tuning. Tuning is a procedure by
which the product performance is enhanced by setting different values to the parameters of
the product performance without having to touch the source code of the product.
Another important factor is Benchmarking. Benchmarking is to compare the throughput
and response time of the product with those of the competitive products.
The next one is capacity planning. A right kind of configuration (both hardware and
software) is needed to derive the best results from performance testing and for deployments.
The exercise to find out what resource and configurations are needed is called capacity
planning.

Page 52 of 86
 Unit – IV Software Testing

4.3 METHDOLOGY FOR PERFORMANCE TESTING

Performance testing is complex and expensive due to large resource requirements and the
time it takes. Hence it requires careful planning and a robust methodology. A good number of
defects that get uncovered during performance testing may require design and architecture
change. A methodology for performance testing involves the following steps,
1.) Collecting requirements
2.) Writing test cases
3.) Automating performance test cases
4.) Executing performance testing cases
5.) Analysing results
6.) Performance test tuning
7.) performance test benchmarking
8.) Recommending right configuration for the customers.

4.3.1 COLLECTING REQUIREMENTS

Collecting requirements is the first step in planning Performance testing. Performance
testing generally needs elaborate documentation and environment setup and the expected
results may not well know in advance. Firstly, Performance testing requirement should be
testable. Secondly, a Performance testing requirements needs to clearly stat what factors needs
to be measured and improved. Lastly, Performance testing requirements needs to be
associated with the actual number or percentage of improvement that is desired.
There are several sources for deriving p requirements. Some of them are as
follows.
1) Performance compared to the previous release of the same product. A
performance requirement can be something like “an ATM withdrawal
transaction will be faster than the previous release by 10%”
2) Performance compared to the competitive product. A performance
requirement can be documented as “ATM withdrawal will be as fast as or
faster than competitive product XYZ.”

Page 53 of 86
 Unit – IV Software Testing

3) Performance compared to absolute numbers derived from actual need. A

requirement can be documented such as” ATM machine should be capable
handling 1000 transactions not taking more than a minute.”
4) Performance numbers derived from architecture and design. The architect
or designers of a product would normally be in a much better position than
anyone else to say what the performance is expected out of the product.
There are two types of requirements performance testing focuses on generic requirements
and specific requirements. Generic requirements are those that are common across all
products in the product domain area. Specific requirements are those that depend on
implementation for a particular product and differ from one product to another in a given
domain.

4.3.2 WRITING TEST CASES:

The next step involved in performance testing is writing test cases. The following details
defined,
1.) List of operations and transactions to be tested.
2.) Steps for executing those operations/transactions.
3.) List of product, OS parameters that impact the performance testing & their values
4.) Loading pattern
5.) Resource and their configuration
6.) The expected results
7.) The product versions/competitive products to be compared with and related
information such as their corresponding fields.
Performance test cases are repetitive in nature. These test cases are normally executed
repeatedly for different values of parameters, different load conditions, and different
configurations.

4.3.3 AUTOMATING PERFORMANCE TEST CASES

Automation is an important step in the methodology for performance testing.
1.) Performance testing repetitive
2.) Performance test cases cannot be effective without automation

Page 54 of 86
 Unit – IV Software Testing

3.) The result of performance testing need to be accurate and manually

calculating the response time, throughput, and so on can introduce
inaccurate
4.) Performance testing takes into account several factors.
5.) The analysis of performance results and failures needs to take into
account related information such as resource utilization, log files, trace
files
End-to-end automation is required for performance testing. Not only the steps of the test cases
but also the setup require for the test cases setting different values to parameters, creating
different load condition.

4.3.4 EXECUTING PERFORMANCE TEST CASES

Performance testing generally involves less effort for execution but more effort for
planning, data collection, and analysis. Data corresponding to the following points needs to
be collected while executing performance tests,

1.) Start and end time of the test case execution

2.) Log and trace audit files of the product and operating system
3.) Utilization of resources on a periodic basis
4.) The response time, throughput, latency and so on as specified in the test
case documentation at regular intervals

4.3.5 ANALYSING THE PERFORMANCE TEST RESULTS

This is the most complex of part of performance testing where product knowledge,
analytical thinking, and statistical background are all absolutely essential.
Before analysing the data some calculations of the data are required. The following
come under this category.
1.) Calculating the mean of the performance test result data
2.) Calculating the standard deviation

Page 55 of 86
 Unit – IV Software Testing

3.) Removing the noise and re-plotting and re-calculating the mean and standard
division.
4.) In terms of catching and other technologies implemented in the product, the
data coming from the cache need to be differentiated from the data that get
processed by the product and presented.
5.) Differentiating the performance data when the resource is available completely
as against when some background activities were going on.
The process of removing some unwanted values in a set is called noise removal. The
majority of the sever-client, internet and database application store the data in a local high
speed buffer. Once the data sets organized the analysis performance data is carried out to
conclude the following,
1.) Whether performance of the product is consist when tests are executed multiple
times.
2.) What performance can be expected for what type of configuration resource
3.) What parameters impact performance and how they can be used to derive
better performance
4.) What is the effect scenarios involving several mix of operations for the
performance factors
5.) What is the effect of product technologies such as caching on performance
improvement
6.) Up to what load are the performance numbers acceptable and whether
performance of the product meets the criteria of “graceful degradation.”
7.) What is the optimum throughput/response time of the product for a set of
factors such as load, resource, and parameters
8.) What performance requirement are met and how the performance looks when
compared to the previous version or the expectation set earlier or the
competition
9.) Sometime high-end configuration may not be available for performance testing.

Page 56 of 86
 Unit – IV Software Testing

4.3.6 PERFORMANCE TUNING

Analysing performance data helps in narrowing down the list of parameter that really
impact the performance results and improving product performance. There are two steps
involved in getting the optimum mileage from performance tuning. They are
1.) Tuning the product parameters and
2.) Tuning the operating system and parameters
The product parameters in isolation as well as in combination have an impact on product
performance. Hence it is important to
1.) Repeat the performance tests for different values of each parameter that
impact performance
2.) Sometimes when a particular parameter value is changed it needs
changes in other parameters. Repeat the performance tests for a group
of parameters and their different values
3.) Repeat the performance tests for default values of all parameters
4.) Repeat the performance tests foe low and high values of each parameter
and combination
These parameter in the operating system are grouped under different categories to
explain their impact as given below

1.) File system reflected parameters

2.) Disk management parameter
3.) Memory management parameters
4.) Processor management parameters
5.) Network parameters

4.3.7 PERFORMANCE BENCHMARKING

Performance benchmarking is about comparing the performance of product
transactions with that of the competitors. No two products can have the same architecture,
design, functionality, and code.

Page 57 of 86
 Unit – IV Software Testing

The steps involved in performance benchmarking are the following:

1. Identifying the transactions/scenarios and the test configuration
2. Comparing the performance of different of different products
3. Tuning the parameters of the products being compared fairly to deliver the best
performance
4. Publishing the results of performance benchmarking

4.3.8 CAPACITY PLANNING

Capacity planning corresponding to short, medium and long _term requirements are called
1. Minimum required configuration;
2. Typical configuration; and
3. Special configuration.

Minimum required configuration denotes that with anything less than this configuration
,the product may not even work .a typical configuration denotes the under that configuration
the product will work fine for meeting the performance require ments .a special configuration
denotes that capacity planning was done considering all future requirements.
There are two techniques that play a major role in capacity planning .they are load
balancing and high availability .load balancing ensures that the multiple machines available are
used equally to service the transactions this ensures that by adding more machines ,more load
can be handled by the product.

4.4 TOOLS FOR PERFORMANCE TESTING

There are two types of tools that can be used for performance testing functional
performance tools and load tools. Functional performance tools help in recording and playing
back the transactions and obtaining performance numbers .this test generally involves very few
machines. Load testing tools simulate the load condition for performance testing without
having to keep that many users or machines .the load testing tools simplify the complexities
involved in creating the load and without such load tools it may be impossible to perform these
kinds of tests.
The list below some popular performance tools

Page 58 of 86
 Unit – IV Software Testing

* Functional performance tools

1. Win runner from mercury
2. A partner from Compuware
3. Silk test from segue
* Load testing tools
1. Load runner from mercury
2. A load from Compuware
3. Silk performer from segue

4.5 PROCESS FOR PERFORMANCE TESTING

Performance testing allows the same process as any other testing type .the only
difference is in getting more details and analysis .as mentioned earlier, the effort involved in
performance testing is more and tests are generally repeated several times.
The next step in the performance testing process is to create a performance test plan.
1. Resource Requirements All Additional Resources That Are Specially Needed For
Performance Testing Need To Be Planned And Obtained.
2. Test Bed (Simulated And Real Life), Test _Lab Setup The Test Lab ,With All Required
Equipment And Software Configuration ,Has To Be Set Up Prior To Execution.
3. Responsibilities Performance Defects , As Explained Earlier ,May Cause Changes
To Architecture ,Design ,And Code.
4. Setting Up Product Traces ,Audits ,And Traces (External And Internal) Performance
Test Results Need To Be Associated With Traces And Audit Trails To Analyze The Results And
Defects.
5. Entry And Exit Criteria Performance Tests Require A Stable Product Due To Its
Complexity And The Accuracy That Is Needed .Change To The Product Affect Performance
Numbers And May Mean That The Tests Have To Be Repeated.

Page 59 of 86
 Unit – IV Software Testing

Evaluate exit criteria

Obtain measurable, testable
requiremwnts

Create a performance test Perform and analyse

plan performance test cases

Design test casues

Automate test cases Evaluate entry criteria

4.6 CHALLENGES
Performance testing is not a very well understood topic in the testing community. The

availability of skills is a major problem facing performance testing .as discussed at several places
of this chapter, product knowledge, knowledge of competition, tools usage, automation
process, knowledge on statics, and analytical skills are needed to do performance testing.
Performance testing requires a large number and amount of resources such as hardware,
software, effort, time, tools and people. Performance test results need to reflect real _life
environment and expectations. Selecting the right tool for the performance testing is another
challenge. Interfacing with different teams that include a set of customers is yet another
challenge in performance testing. Lack of seriousness on performance tests by the
management and development team is another challenge .once all functionalities are working
fine in a product, it is assumed that the product is ready to ship.

Page 60 of 86
 Unit – IV Software Testing

REGRESSING TESTING
4.1 WHAT IS REGRESSION TESTING
Regression testing is done to ensure that enhancements or defect fixes made to the
software works properly and does not affect the existing functionality.
Software undergoes constant changes. Such changes are necessitated because of
defects to be fixed, enhancements to existing functionality, or new functionality to be added.
Anytime such changes are made, it is important to ensure that.
1. The changes or additions work as designed
2. The changes or additions do not break something that is already working and should
continue to work.
Regression testing is designed to address the above two purpose.

4.2 TYPES OF REGRESSION TESTING

There are two types of regression testing.
 Regular regression testing
 Final regression testing
A “regular regression testing” is done between test cycles to ensure that the defect fixes
that are done and the functionality that were working with the earlier test cycles continue to
work.
A “final regression testing” is done to validate the final build before release. The CM
engineer delivers the final build with the media and other contents exactly as it would go to the
customer. The final regression test cycle is conducted for a specific period of duration, which is
mutually agreed upon between the development and testing teams. This is called the “cook
time” for regression testing. The final regression test cycle is more critical.

4.3 WHEN TO DO REGRESSION TESTING

Whenever changes happen to software, regression testing is done to ensure that these do
not adversely affect adversely the existing functionality. Regression testing is done between

Page 61 of 86
 Unit – IV Software Testing

test cycles to find out if the software delivered is as good as or better than the builds received
in the past. As testing involves large amount of resources (hardware, software, and people), a
quick testing is needed to assess the quality of build and changes to software initially a few test
engineers with very few machines do regression testing. Regression testing is done whenever
such a requirement arises
 A reasonable amount of initial testing is already carried out.
 A good number of defects have been fixed.
 Defect fixes that can produce side-effects are taken care of.
Regression testing may also be performed periodically, as a pro-active measure. Regression
testing can be performed irrespective of which test phase the product is in.

Types
Regression testing
Regular regression –final
regressi on

When?
What? Why?
Defects creep in due to When set of defect fixes
Select re-testing to
changes arrive after formal test is
ensure defect fixes
Defect fixes may cause complete
exist function to fail

4.4 HOW TO DO REGRESSION TESTING

A well-defined methodology for regression testing is very important. There are several
methodologies for regression testing that are used by different organizations. The
methodologies contains some following steps.
 Performing an initial “Smoke” or “Sanity” test
 Understanding the criteria for selecting the test cases
 Classifying the test cases into different priorities
 A methodology for selecting test cases

Page 62 of 86
 Unit – IV Software Testing

 Resetting the test cases for test execution

 Concluding the results of a regression cycle.

4.4.1 Performing an Initial “Smoke” or “Sanity” Test

Whenever changes are made to a product, it should first be made sure that nothing
basic breaks.
Smoke testing consists of
 Identifying the basic functionality that a product must satisfy;
 Designing test cases to ensure that these basic functionality work and packaging them
into a smoke test suite;
 Ensuring that every time a product is built, this suite is run successfully before anything
else is ran; and
 If this suite fails, escalating to the developers to identify the changes and perhaps
changes or roll back the changes to a state where the smoke test suits succeeds.

DOCUMENTATION PROCESS people

7% 10% 8%

CM
15%

OTHERS
CODING 5%
20%
REQUIREMENTS
15%

DESIGN
20%

Page 63 of 86
 Unit – IV Software Testing

Defects in the product can get introduced not only by the code, but also by the build scripts
that are used for compiling and linking the program. Smoke testing enables the uncovering of
such errors introduced by (also) the build procedures.

4.4.2 Understanding the Criteria for selecting the Test cases

There are two approaches to selecting the test cases for a regression run. First, an
organization can choose to have a constant set of regression tests that are run for every build
or change. In such a case, deciding what tests to run is simple. But this approach is likely to be
sub-optimal because
 In order to cover all fixes, the constant set of tests will encompass all features and tests
which are not required may be run every time; and
 A given set of defect fixes or changes may introduce problems for which there may not
be ready-made test cases in the constant set.
A second approach is to select the test cases, dynamically for each build by making judicious
choices of the test cases.
 The defect fixed and changes made in the current build;
 The ways to test the current change;
 The impact that the current changes may have on other parts of the system; and
 The ways of testing other impacted parts.
Some of the criteria to select test cases for regression testing are as follows.
 Include test cases that have produced the maximum defects in the past
 Include test cases for a functionality in which a change has been made
 Include test cases in which problems are reported
 Include test cases that the basic functionality or the core features of the product which
are mandatory requirements of the customer.
 Include test cases that the end-to-end behavior of the application or the product
 Include test cases to test the positive test conditions

 Includes the area which is highly visible to the users.

Page 64 of 86
 Unit – IV Software Testing

4.4.3 Classifying Test Cases

When the test cases have to be selected dynamically for each regression run .Choosing
the right tests for a regression run, the test cases can be classified into various priorities based
on importance and customer usage. The test cases divided into three categories.
*Priority-0 These test cases can be called sanity test cases which check basic
functionality and are run for accepting the build, for further testing. They are also run when a
product goes through a major change.
*Prioity-1 User the basic and normal setup and these test cases deliver high project
value to both development team and to customers.

Priority-0
10%

Priority-1
25%

Priority-2
65%

Prioity-2 These test cases deliver moderate project value. They are executed as part
of the testing cycle and selected for regression testing on a need basis.

Page 65 of 86
 Unit – IV Software Testing

4.4.4 Methodology for Selecting Test Cases

Case 1 If the criticality and impact of the defect fixes are low then it is enough that a test
engineer selects a few test cases from test case database (TCDB),(a repository that stores all the
test cases that can be used for testing a product. These test cases can fall under any priority (0,
1, or, 2).
Case 2 If the criticality and the impact of the defect fixes are medium, test all test cases for
priority-0 and priority-1 and few for priority-2.
Case 3 If the criticality and impact of the defect fixes are high, to execute all priority-0, priority-
1 and a carefully selected subset of priority-2 test cases.
The alternative methodologies given below can be considered.
 Regress all: For regression testing, all priority 0, 1 and 2 test cases are rerun. This
means all the test cases in the regression test bed/ suite are executed.
 Priority based regression: For regression testing based on this priority, all priority 0,1,
and 2 test cases are run in order, based on the availability of time. Deciding when to
stop the regression testing is based on the availability of time.
 Regress changes: For regression testing using this methodology code changes are
compared to the last cycle of testing and test cases are selected based on their impact
on the code (gray box testing)
 Random regression: Random test cases are selected and executed for this regression
methodology.
 Context based dynamic regression: A few priority-0 test cases are selected, and based
on the context created by the analysis of those test cases after the execution (for
example , find new defects, boundary value) and outcome, additional related cases are
selected for continuing the regression testing.

Page 66 of 86
 Unit – IV Software Testing

Legend
P0 priority = 0 Methodology for Selecting Test Cases

P1 priority = 1

P2 priority = 2

C Very few
All

C subset
Critically
impact

low MED HIGH

P0 c P0 P0

P1 P1
P1 c
P2 c P2 c
P2 c
4.4.5 Resetting the Test Cases for Regression Testing

Page 67 of 86
 Unit – IV Software Testing

Resetting test cases reduces the risk involved in testing defect fixes by making the testers go
through all the test cases and selecting appropriate test cases based on the impact of those
defect fixes. Resetting of test cases is not expected to be done often, and it needs to be done
with the following considerations in mind.
 When there is a major change in the product.
 When there is a change in the build procedure which affects the product.
 Large release cycle where some test cases were not executed for a long time.
 When the product is in the final regression test cycle with a few selected test cases.
 Where there is a situation, that the expected results of the test cases could be quite
different from the previous cycles.
 The test cases relating to defect fixes and production problems need to be evaluated
release after release. In case they are found to be working fine, they can be reset.
 Whenever existing application functionality is removed, the related test cases can be
reset.
 Test cases that consistently produce a positive result can be removed.
 Test cases relating to a few negative test conditions (not producing any defects) can be
removed.

4.4.6 Concluding the Results of Regression Testing

The test manager can compare test with the previous results of the test cases to conclude
whether regression was successful or not.
 If the result of a particular test case was a pass using the previous builds and a fail in the
current build, then regression has failed.
 If the result of a particular test case was a fail using the previous builds and a pass in the
current build, then it is safe to assume the defect fixes working.
 If the result of a particular test case was a fail using the previous builds and a fail in the
current build, and if there are on defect fixes for this particular test case.
 If the result of a particular test case is a fail using the previous builds but works with a
documented workaround and if you are satisfied with the workaround.

Page 68 of 86
 Unit – IV Software Testing

 If not satisfied with the workaround, then it should be considered as a fail system test
cycle but may be considered as a pass for regression test cycle.

4.5 BEST PRACTICES IN REGRESSION TESTING

Regression methodology can be applied when
 Need to assess the quality of product between test cycles (both planned and need
based);
 Doing a major release of a product, have executed all test cycles, and are planning a
regression test cycle for defect fixes; and
 Doing a minor release of a product (support packs, patches, and so on) having only
defect fixes, and we can plan for regression test cycles to test care of those defect fixes.

Practice 1: Regression can be used for all types of releases

Practice 2: Mapping defect identifiers with test cases improve regression quality.
Practice 3: Create and execute regression test bed daily
Practice 4: Ask your best test engineer to select the test code

-- x - -

Page 69 of 86
 Unit – V Software Testing

UNIT - V
TEST MANAGEMENT EXECUTION, AND REPORTING
5.1TEST PLANNING
5.1.1 Preparing a Test Plan
Testing- like any project- should be driven by a plan. The test plan acts as the anchor for
the execution, tracking and reporting of the entire testing project and covers.
 What needs to be tested
 How the testing is going to be performance.
 What resources needed for testing
 The time lines by which the testing activities performed
 Risk that may be faced in all of the above, with appropriate mitigation and contingency
plans.

5.1.2 Scope Management: Deciding Features to be Tested/not Tested

Scope management pertains to specifying scope of a project. For testing scope
management entails
 Understanding what constitutes a release of a product.
 Breaking down the release into features
 prioritizing the features for testing
 Deciding which features will be tested which all not be
 Gathering details to prepare estimation of resources for testing.

The following factor drives the choice and prioritization of features to be tested.
 Features that are new and critical for the release
 Features whose failures can be catastrophic
 Features that are expected to be complex to test

Page 70 of 86
 Unit – V Software Testing

 Features which are extensions of earlier features that have been defect
prone.


5.1.3 Deciding Test Approach/Strategy

What needs to be tested, to enable estimation of size, effort and schedule? This is
includes identifying
1. What type of testing would you use for testing functionality?
2. What are the configurations or scenarios for testing the features?
3. What integration testing would you do to ensure these features work together?
4. What localization validations would be needed?
5. What “non-functional” testes would you need to do?

5.1.4 Setting up Criteria for Testing

Resumption criteria specify when the suspended tests can be resumed. Some of the
typical suspension criteria include
 Encountering more than a certain number of defects causing frequent stoppage of
testing activity
 Hitting show stoppers that prevent further progress of testing
 Developers releasing a new version which they advice should be used in lieu of the
product under test

5.1.5 Identifying Responsibility, Staffing, and Training Needs

Scope management identifies what needs to be tested. The test strategy outlines how to
do it. The next aspect of planning is the who part of it. Identifying responsibilities, staffing and
training needs addresses this aspect.
The different role definitions should
 Ensure there is clear accountability for a given as task and so that each person
knows he or she has to do.

Page 71 of 86
 Unit – V Software Testing

 The clearly list he responsibilities for various functions to various to that every
one knows how his or her work fits into the entire project
 Complement each other, ensuring no one steps on an others toes, and
 supplement each other, so that no task is left unassigned

5.1.6 Identifying Resources Requirements

As a part of planning for a testing project, the project manager should provide estimates
for the various hardware and software resources required. Some of the following factors need
to be considered
 Machine configuration needed to run the product under test
 Overheats required by the test automation tool if any,
 Supporting tools such as compilers, test data generators configuration
management tools and so on.
 the different configuration test separating software that must be present
 special requirements for running machine- intensive test such as performance
 Appropriate numbered of licenses of all the software.

5.1.7 Identifying Test Deliverables

The test plan identifies the deliverables that should come out of the test cycle testing activity.
The deliverables includes the following all reviewed and appropriate by the people.
 Test plan itself master test plan and various other test plans for the project
 Test case design specifications
 Test cases including any automation that is specified in the plan
 Test logs produced by running the tests
 Test summary reports

5.1.8 Testing Tasks: Size and Effort Estimation

Estimation happens broadly in three phases.
 Size estimation
 Effort estimation
 Schedule estimation

Page 72 of 86
 Unit – V Software Testing

Size estimation quantifies the actual amount of testing that needs to be done. Several
factors contribute size estimate of a testing project.
Size of the product under test :
This obviously determines the amount of testing that needs to be done. Some of measure
of product under test is as follows
 Lines of code [LOC]
 A function point[FP]
Size estimate is expressed in terms of any of the following:
1. Number of test cases
2. Number of test scenarios
3. Number of configurations to be tested
The other factors that drive the effort estimate are as follows.
 Productivity data
 Reuse opportunities
 Robustness of process

5.1.9 Activity Breakdown and Scheduling

Activity breakdown and schedule estimation entail translating the effort required
into specific time frames. The following steps make up this translation.
 Identifying internal and external dependencies among the activities
 Sequencing the activities, based on the expected duration as well as on the
dependencies.
 Identifying the time required for each of the WBS activities, taking into account the
above two factors
 Monitoring the progress in terms of time and effort.
 Rebalancing schedules and resources as necessary.
Some of the common external dependencies are,
 Availability of the product from developers,

Page 73 of 86
 Unit – V Software Testing

 Hiring
 Training
 Acquisition of hardware/software required for training
 Availability of translated message files for testing.
Some of the internal dependencies could be,
 Completing the test specification
 Coding/scripting the tests
 Executing the tests.

5.1.10 Risk Management

Every project also faces risk. Risks are events that could potentially affect a project outcome.
These events are normally beyond the control of the project manager. Risk management
entails,
 Identifying the possible risks
 Quantifying the risks
 Planning how to mitigate the risk
 Responding to risk when they become a reality,

Risk Risk
identification qualification

Risk response Risk mitigation

planning

Page 74 of 86
 Unit – V Software Testing

Risk identification consists of identifying the possible risks that may hit a project.
Although there could potentially be many risks that hit a project, the risk identification step
should focus on those risks that are more likely to happen. The flowing are some of the
common ways to identify risks in testing.
 Use of checklist: Over time an organization may find new gleanings on testing that can
be captured in the form of a checklist.
 Use of organizational history and metrics: When an organizational collects and analyzes
the various metrics, the information can provide valuable insights into what possible
risks can hit a project
 Informal networks across the industry: The informal networking across the industry can
help identifying risks that other organizations.
Risk quantification deals with expressing the risk in numerical terms. There are two
components to the quantification of risk. One is probability of the risk happening and other is
the impact of the risk, if the risk happens.
The Risk mitigation deals with identifying alternative strategies to combat a risk event,
should that risk materialize. The following are some of the common risks encountered in testing
projects and their characteristics:
 Unclear requirements
 Schedule dependence
 Insufficient time for testing
 “Show stopper “defects
 Availability of skilled and motivated people for testing
 Inability to get test automation tool

5.2 TEST MANAGEMENT

5.2.1 Choice of standards
Standards are of two types – external standards and internal standards. External
standards are standards that a product should comply with are externally visible, and are

Page 75 of 86
 Unit – V Software Testing

usually stipulated by external consortia. Internal standards are standards formulated by a

testing organization to bring in consistency and predictability.
Some of the internals standards include:
1. Naming and storage
2. Document standards
3. Test coding standards
4. Test reporting standards

5.2.2 Test Infrastructure Management

Testing requires robust infrastructure to be planned upfront. This infrastructure is
made up of three essential elements.
1. A test case database(TCDB)
2. A defect repository
3. Configuration management repository and tool
Test case database captures all the relevant information about the test cases in an
organization. A defect repository captures all the relevant details of defects reported for a
product. A CM repository keeps track of change control and version control of all the files
/entities that make up a software product. A particular case of the files is test files. Change
control ensures that
1. Changes to test files are made in a controlled fashion and only with proper approvals.
2. Changes made by one test engineer are not accidentally lost or overwritten by other
changes.
3. Each change produces a distinct version of the file that is recreatable at any point of
time.
4. At any point of time, everyone gets access to only the most recent version of the test
files.

5.2.3 Test People Management

People management is an integral part of any project management. Often, it is difficult
chasm for engineers-turned-managers to cross. As an individual contributor, a person relies
only on his or her own skills to accomplish an assigned activity; the person is not necessarily

Page 76 of 86
 Unit – V Software Testing

trained on how to document what needs to be done so that it can be accomplished by

someone else. Furthermore, people management also requires the ability to hire, motivate and
retain right people.

5.2.4 Integrating with Product Release

Ultimate, the success of a product depends on the effectiveness of integration of the
development and testing activities. The following are some of the points to be decided for
this planning
1. Sync points between development and testing as to when different types of testing can
commence.
2. Service level agreement between development and testing as to how long it would take
for the testing.
3. Consistent definitions of the various priorities and severities of the defects. This will
bring in a shared vision between developments.
4. Communication mechanisms to the documentation group to ensure that the
documentation.

5.3 TEST PROCESS

5.3.1 Putting Together and Base lining a Test Plan
A test plan combines all points discussed above into a single document that acts as
an anchor point for entire testing project. An organization normally arrives at a template
that is to be used across the board. Each testing project puts together a test plan based on
the template.

5.3.2 Test Case Specification

Using the test plan as the basis, the testing team designs test case specification,
which then
becomes the basis for preparing individual test cases. Hence, a test case specification should
clearly identify

Page 77 of 86
 Unit – V Software Testing

1. The purpose of the test: This list what feature or part the test is intended for. The test
case should follow the naming conventions that are consistent with the feature/ module
being tested.
2. Items being tested, along with their version/release numbers as appropriate.
3. Environment that needs to be set up for running the test case.
4. Input data to be used for the test case.
5. Steps to be followed to execute the test.
6. The expected results that are considered to be “correct result”
7. A step to compare the actual results produced with the expected results
8. Any relationship between this test and other tests.

5.3.3 Update of traceability matrix

The traceability matrix is a tool to validate that every requirement is tested. The
traceability matrix is created during the requirements gathering phase itself by filling up the
unique identifier for each requirement

5.3.4 Identifying Possible Candidates for Automation

The test case designs forms the basis for writing the test cases. Before writing the test
cases, a decision should be taken as to which tests are to be automated and which should
be run manually. Some of the criteria that will be used in deciding which scripts to automate
include
1. Repetitive nature of the test.
2. Effort involved in automation
3. Amount of manual intervention required for the test; and
4. Cost of automation tool.

5.3.5 Developing and Base lining Test Cases

Based on the test case specification and the choice of candidates for automation, test
cases have to be developed. The development of test cases entails translating the test
specifications to a form from which the tests can be executed. If a test case is a candidate for
automation, then this step requires writing test scripts in the automation language. If a test

Page 78 of 86
 Unit – V Software Testing

case is a manual test case, then test case writing maps to writing detailed step-by-step
instructions for executing the test and validating the results. The test cases should also have
change history documentation, which specifies
1. What was the change;
2. Why the change was necessitated;
3. Who made the change;
4. When was the change;
5. A brief description of how the change has been implement;
6. Other files affected by the change.

5.3.6 Executing Test Cases and Keeping Traceability Matrix Current

As the test cases are executed during a test cycle, the defect repository is updated with
1. Defects from the earlier test cycles that are fixed in the current build
2. New defects that get uncovered in the current run of the tests.

5.3.7 Collecting and Analysing Metrics

When tests are executed, information about test execution gets collected in test logs
and other files. The basic measurements from running the tests are then converted to
meaningful metrics by the use of appropriate transformation and formulas.

5.3.8 Preparing test summary report

At the completion of a test cycle, a test summary report is produced. This report gives
insights to the senior management about the fitness of the product release.

5.3.9 Recommending Product Release Criteria

The job of the testing team is to articulate to the senior management and the product
release team
1. What defects the product has;
2. What is the impact / severity of each of the defects,
3. What would be the risks of releasing the product with the existing defects.

Page 79 of 86
 Unit – V Software Testing

5.4 TEST REPORTING

Test reporting divided into three types. They are
1. Test incident report
2. Test cycle report
3. Test summary report

1. Test incident report

A test incident report is a communication that happens through the testing cycle as and
when defects are encountered. A test incident report is nothing but an entry made in the defect
repository. Each defect has a unique ID and this is used to identify the incident. The high impact
test incidents are highlighted in the test summary report.

2. Test cycle report

A test cycle entails planning and running certain tests in cycles, each cycle using a
different build of the product. A test cycle report, at the end of each cycle, gives
1. A summary of the activities carried out during that cycle
2. Defects that were uncovered during that cycle based on their severity and impact
3. Progress from the previous cycle to the current cycle in terms of defects fixed
4. Outstanding defects that are yet to be fixed in this cycle
5. Any variations observed in effort or schedule

3. Test summary report

There are two types of test summary reports
1. Phase-wise test summary, which is produced at the end of every phase
2. Final test summary reports.
A summary report should present
A summary of the activities carried out during the test cycle
 Variance of the activities carried out from the activities planned.
This include

Page 80 of 86
 Unit – V Software Testing

 The tests that were planned to be run but could not be run
 Modifications to tests from what was in the original test specification
 Additional tests that were run
 Difference in effort and time taken between what was planned and what was
executed
 Any other deviations from plan
 Summary of results should include
 Tests that failed, with any root cause descriptions
 Severity of impact of the defects uncovered by the tests
 Comprehensive assessment and recommendation for release should include
 “fit for release” assessment
 Recommendation of release



5.4.1 Recommending Product Release
Based on the test summary report, an organization can take a decision on whether to
release the product or not.

TEST METRICS AND MEASUREMENTS

Metrics derive information from raw data with a view to help in decision making. Some of the
areas that such information would shed light on are
1. Relationship between the data points
2. Any cause and effect correlation between the observed data points
3. Any pointers to how the data can be used for future planning and continuous
Improvements
Metrics are thus derived from measurements using appropriate formula or calculations.
Obviously, the same set of measurements can help product different set of metrics, of interest
to different people.

Page 81 of 86
 Unit – V Software Testing

TEST METRICS AND MEASUREMENTS

identify ehat
to measure

refine transform
measurements measurements
and metrics to metrics

decide
take actions
operational
and follow up
requirements

perform
metrics
analysis

WHY METRICS IN TESTING

Metrics are needed to know test case execution productivity and to estimate test
completion date.

Page 82 of 86
 Unit – V Software Testing

TYPES OF METRICS
Metrics can be classified as product metrics and process metrics. Product metrics can be
further classified into
1. Project metrics
2. Progress metrics
3. Productivity metrics

Process Product
metrics metrics

Project metric Producitivity

Process metrics
metrics

Defect find rate Compenent- Defects per 100hrs of

wise defect testing
Defect fix rate
distribution Test cases executed per
Outstanding 100 hrs of testing
Defect
defects rate
density and Test cases developed per
Priority 100 hours
defect
outstanding
removal rate
Defects per 100 test cases
rate
Age analysis
Defects trend Defects per 100 failed test
of
cases
Defect outstanding
classification defects Test phases effectiveness

trend Introduced
Closed defects distribution
Weighted and reopened
defects trend defects rate

Page 83 of 86
 Unit – V Software Testing

Effort variance Defect cause

distribution
Schedule variance

Effort distribution

Testing defect
Development
metrics
defect metrics

PROJECT METRICS
A typical project starts with requirements gathering and ends with product release. All
phases that fall between these points need to be planned and tracked. In the planning cycle,
the scope of the project is finalized. The project scope gets translated to size estimates, which
specify the quantum to work to be done. This size estimate gets translated to effort estimate
for each phase and activities by using the available productivity data available. This initial effort
is called baselined effort.
The project progresses and if the scope of the project changes or if the available
productivity numbers are not correct, then the effort estimates are re-evaluated again and this
re-evaluated effort estimate is called revised effort.
The basic measurements that are very natural, simple to capture and form the inputs to
the metrics in this section are

1. The different activities and the initial baselined effort and schedule for each of the activities
2. The actual effort and time taken for the activities; this is entered as and when
The activities take place.

Page 84 of 86
 Unit – V Software Testing

3. The revised estimate of effort and schedule these are re-calculated at appropriate times in
the project life.

PROGRESS METRICS
Defects get detected by the testing team and get fixed by the development team. Defect
metrics are further classified into test defect metrics and development defect metrics .

Test Defect Metrics

The purpose of testing is to find defects early in the test cycle.
Defect find rate: The total number of defects found in the product at regular intervals(daily or
weekly) from beginning to end of a product development life cycle.
Defect fix rate: The number of defects is fixed from total number of defects.
Outstanding defects rate: The number of defects outstanding in the product is calculated by
subtracting the total defects fixed from the total defects found in the product.

Defect classification trend:

Defects are classified into
 Cosmetic
 Minor
 Important
 Critical
 Extreme

Development defect metrics

 Component-wise defect distribution
 Defect density and defect removal rate
 Age analysis of outstanding defects
 Introduced and reopened defect trend

Page 85 of 86
 Unit – V Software Testing

PRODUCTIVITY METRICS
Productivity metrics combine several measurements and parameters with effort spent on
the product. They help in finding out the capability of the team as well as for other purpose
such as

1. Estimating for the new release

2. Finding out how well the team is progressing, understanding the reasons for variations in
results.
3. Estimating the number of defects that can be found
4. Estimating release date and quality
5. Estimating the cost involved in the release.

RELEASE METRICS

Release metrics is used to decide the project is released or not.

Page 86 of 86