Software Testing

Lecture - 1

1
Course Outline

❏ Class Attendance: 10
❏ Assignment: 40
❏ Presentation: 40
❏ Project (Term Paper): 60

❏ Final Exam: 150

-----------------------------------------
❏ Total: 300

2
Referred Books

• The Art of Software Testing

▪ Glenford J. Myers

• Introduction To Software Testing

▪ Jeff Offutt

• Software Test Automation- Effective use of test

execution tools
▪ Mark Fewster & Dorothy Graham

3
Outline

• Psychology of Software Testing

• Software Testing Principles
• Inspections, Walkthroughs and Reviews
• General Testing Guidelines

4
Psychology of Software Testing

One of the primary causes of poor program testing is the fact that
most programmers begin with a false definition of the term.

They might say:

‘The purpose of testing is to show that a program performs its
intended functions correctly.’
‘Testing is the process of demonstrating that errors are not present.’
‘Testing is the process of establishing confidence that a program does
what it is supposed to do.’

“These definitions are upside-down.”

5
Adding Value Through Testing

● Adding value through testing means raising the quality or reliability

of the program.
● Raising the reliability of the program means finding and removing
errors.

● Therefore, don’t test a program to show that it works;

○ Rather, we should start with the assumption that the program contains
errors
○ and then test the program to find as many of the errors as possible.

● Thus a more appropriate definition is this:

○ Testing is the process of executing a program with the intent of finding
errors.
6
Why does this definition matter?

● Myers asserts that “establishing the proper goal has an important

psychological effect.”

● If our goal is to demonstrate that a program has no errors,

● We tend to select test data that have a low probability of causing the program to
fail.
● but if our goal is to demonstrate that a program has errors,
● then our test data will have a higher probability of finding errors.

7
What is a Successful Test?

● Two types of testing can be distinguished:

○ Testing for defect identification, a successful test is one which causes the
system to fail.
○ Testing to demonstrate that the software meets its specifications or other
desired properties, in which case testing is successful if no failures are
observed.

● In any event, a test case should only be run if it has a non-zero

chance of revealing a defect.
○ Otherwise it’s just waste of time with the test

8
More on the Psychology of Testing

● Myers asserts that “… people perform poorly when they set out on a task that
they know to be infeasible or impossible.”
● “Defining program testing as the process of uncovering errors in a program makes it a
feasible task.”
● “… the process of demonstrating that errors are not present” is “impossible to achieve
for virtually all programs”.

● If we define testing as “the process of demonstrating that a program does

what it is supposed to do”, then
● it can still have serious bugs (e.g. security faults) on inputs that cause it to do
something it is not intended to do.

9
Two Definitions from Myers?

✧ “Software testing is a process, or a series of processes,

designed to make sure computer code does what it is designed
to do and that it does not do anything unintended.”

✧ “Software testing is a process,

▪ Or a series of processes,
✧ “designed to make sure computer code
▪ Does what it is designed to do and
▪ That it does not do anything unintended.”
Software Testing Principles

11
Software Testing Principles

★ A necessary part of a test case is a definition of the expected

output or result.
★ A programmer should avoid attempting to test his or her own
program.
★ A programming organization should not test its own
programs.
★ Any testing process should include a thorough inspection of
the results of each test.
★ Test cases must be written for input conditions that are invalid
and unexpected, as well as for those that are valid and
expected.
Software Testing Principles

★ Examining a program to see if it does not do what it is

supposed to do is only half the battle; the other half is seeing
whether the program does what it is not supposed to do.
★ Avoid throwaway test cases unless the program is truly a
throw away program.
★ Do not plan a testing effort under the tacit assumption that no
errors will be found.
★ The probability of the existence of more errors in a section of
a program is proportional to the number of errors already
found in that section.
★ Testing is an extremely creative and intellectually challenging
task.
 Static Testing

Inspections, Walkthroughs and Reviews

14
Static Testing

Work Products that Can Be Examined by Static Testing:

● Specifications, including business requirements, functional
requirements, and security requirements
● User stories, and acceptance criteria
● Architecture and design specifications
● Code
● Testware, including test plans, test cases, test procedures, and
automated test scripts
● User guides
● Web pages
● Contracts, project plans, schedules, and budget planning
● Configuration set up and infrastructure set up
● Models which may be used for Model-Based testing
Benefits of Static Testing

● Detecting and correcting defects more efficiently, and prior to

dynamic test execution
● Defects found early are often much cheaper to remove
● Identifying defects which are not easily found by dynamic testing
● Preventing defects in design or coding by uncovering
inconsistencies, ambiguities, contradictions, omissions,
inaccuracies, and redundancies in requirements
● Increasing development productivity (e.g., due to improved
design, more maintainable code)
● Reducing development cost and time
● Reducing testing cost and time
● Reducing total cost of quality over the software’s lifetime, due to
fewer failures later in the lifecycle or after delivery into operation
● Improving communication between team members in the course of
participating in reviews
Differences between Static and Dynamic
Testing
❖ Static testing finds defects in work products directly rather than
identifying failures caused by defects when the software is run.
❖ Static testing can be used to improve internal quality of the software,
while dynamic testing focuses on externally visible behaviors.
Compared with dynamic testing, typical defects that are easier and cheaper
to find and fix through static testing include:
● Requirement defects
● Design defects
● Coding defects
● Deviations from standards
● Incorrect interface specifications
● Security vulnerabilities
● Gaps or inaccuracies in test basis traceability or coverage
Inspections

✧ Inspections involve a team of people reading or visually

inspecting a program

✧ These are so-called ‘static’ V & V technique

✧ Do not require execution of a system
✧ Focuses on static system representations such as requirements,
design model, source code etc. to discover defects
 Inspections (Cont.)
General Procedure:

✧ Inspection Team:
▪ Moderator
▪ Programer
▪ Program’s Designer
▪ Test Specialist
✧ Moderator’s Duties:
▪ Distributing materials for, and scheduling, the inspection session.
▪ Leading the session.
▪ Recording all errors found.
▪ Ensuring that the errors are subsequently corrected
 Inspections (Cont.)

✧ Inspection Agenda:
▪ The programmer narrates, statement by statement, the logic of the
program. And other participants raise questions
▪ The program is analyzed with respect to checklists of historically
common programming errors
✧ Human Agenda:
▪ The testing group must adopt an appropriate attitude to make the
inspection process effective
▪ The programmer must take the process in a positive and constructive
way
▪ If managers somehow make use of the inspection results, the purpose
of the process may be defeated.
 Inspections (Cont.)

✧ Side Benefits of the Inspection Process:

1. The programmer usually receives valuable feedback
concerning programming style, choice of algorithms, and
programming techniques
2. Other participants gain in a similar way
3. A way of identifying early the most error-prone sections
of the program
 Inspections (Cont.)

✧ An Error Checklist for Inspections:

Data Reference Errors
Data Declaration Errors
Computation Errors
Comparison Errors
Interface Errors
Input/Output Errors
Other Checks
 Walkthroughs

✧ The code walkthrough is a set of procedures and

error-detection techniques for group code reading
✧ It is an uninterrupted meeting of one to two hours in duration

✧ Team Members:
▪ Moderator
▪ Secretary (a person who records all errors found)
▪ Tester
▪ Programer
Walkthroughs (Cont.)

✧ Suggestions for the other participants include:

▪ A highly experienced programmer
▪ A programming-language expert
▪ A new programmer (to give a fresh, unbiased outlook)
▪ The person who will eventually maintain the program
▪ Someone from a different project
▪ Someone from the same programming team as the programmer
Walkthroughs (Cont.)

✧ Procedure:
▪ Rather than simply reading the program or using error
checklists, the participants ‘‘play computer”
▪ The tester prepares a small set of paper test cases
▪ During the meeting, each test case is mentally executed
 Peer Ratings

✧ Peer rating is a technique of evaluating anonymous programs

in terms of their overall quality, maintainability, extensibility,
usability, and clarity.
✧ The purpose of the process is to allow programmers to
self-assess their programming skills.
✧ The process appears to be useful in both industrial and
classroom environments.
Peer Ratings

Procedure:
○ A programmer is selected to serve as an administrator
○ The administrator selects approximately 6 to 20 participants
○ Each participant is asked to select two of his or her own programs
to be reviewed.
○ Then, they are randomly distributed to the participants.
○ Each participant spends 30 minutes reviewing each program and
then completes an evaluation form by rating the programs.
○ After the review, the participants are given the anonymous
evaluation forms for their two contributed programs.
Individual Review Techniques

❏ Ad hoc
❏ Reviewers are provided with little or no guidance on how this task
should be performed.
❏ Reviewers often read the work product sequentially, identifying and
documenting issues as they encounter them
❏ This technique is highly dependent on reviewer skills and may lead to
many duplicate issues

❏ Desk Checking
❏ A desk check can be viewed as a one-person inspection or
walkthrough: A person reads a program, checks it with respect to an
error list, and/or walks test data through it.
❏ Desk checking is relatively unproductive
Individual Review Techniques

❏ Checklist-based
❏ It is a systematic technique, whereby the reviewers detect issues
based on checklists that are distributed at review initiation
❏ A review checklist consists of a set of questions based on potential
defects, which may be derived from experience.
❏ Checklists should be specific to the type of work product
❏ The main advantage of the checklist-based technique is a systematic
coverage of typical defect types

❏ Scenarios and dry runs

❏ Reviewers are provided with structured guidelines on how to read
through the work product.
❏ These scenarios provide reviewers with better guidelines on how to
identify specific defect types than simple checklist entries
Individual Review Techniques

❏ Perspective-based
❏ Reviewers take on different stakeholder viewpoints
❏ Typical stakeholder viewpoints include end user, marketing, designer,
tester, or operations.
❏ It requires the reviewers to attempt to use the program under review
to generate the product they would derive from it

❏ Role-based
❏ A role-based review is a technique in which the reviewers evaluate
the work product from the perspective of individual stakeholder roles.
❏ Typical roles include specific end user types (experienced,
inexperienced, senior, child, etc.),
❏ Specific roles in the organization (user administrator, system
administrator, performance tester, etc.)
 Desk Checking

✧ A desk check can be viewed as a one-person inspection or

walkthrough: A person reads a program, checks it with respect
to an error list, and/or walks test data through it.
✧ Desk checking is relatively unproductive
▪ it is a completely undisciplined process
▪ it runs counter to testing principle 2
✧ Desk checking is best performed by a person other than the
author of the program
Inspection and Testing

32
A model of the software testing process

33
General Testing Guidelines

● Testing shows the presence of defects, not their absence

● Exhaustive testing is impossible
● Early testing saves time and money
● Defects cluster together
● Beware of the pesticide paradox
● Testing is context dependent
● Absence-of-errors is a fallacy

34
General Testing Guidelines

● Choose inputs that force the system to generate all error

messages
● Design inputs that cause input buffers to overflow
● Repeat the same input or series of inputs numerous times
● Force invalid outputs to be generated
● Force computation results to be too large or too small

● Testing shows the presence of defects, not their absence

● Exhaustive testing is impossible
● Early testing saves time and money
● Absence-of-errors is a fallacy

35