Software Testing

Lecture - 1

1
Course Outline

❏ Class Attendance: 10
❏ CT/Assignment: 40
❏ Mid/Presentation: 40
❏ Project (Term Paper): 60

❏ Final Exam: 150

-----------------------------------------
❏ Total: 300

2
Referred Books

• The Art of Software Testing

▪ Glenford J. Myers

• Introduction To Software Testing

▪ Jeff Offutt

• Software Test Automation- Effective use of test

execution tools
▪ Mark Fewster & Dorothy Graham

3
Outline

• Introduction to Software Testing

• Psychology of Software Testing
• Software Testing Principles
• Inspections, Walkthroughs and Reviews
• General Testing Guidelines

4
Introduction to Software Testing

★ Software Testing is not a stand-alone activity

★ It is a predefined process that should be followed to ensure that the

software works as expected

★ It involves two approaches: Static and Dynamic

Objectives of Software Testing

● Finding defects early in the testing phase

● Ensuring the quality of the software
● To determine that the software meets the requirements
● To ensure that the software is fit for use
Types of Software Testing
Can be divided in 2 broader categories:
★ Functional Testing
★ Non-Functional Testing

Functional Testing Non-Functional Testing

Checks functionalities or system Checks performance of the
requirements functionalities
Validates functional aspects of a Validates non-functional aspects of a
software software
Focuses on user requirements Focuses on user expectations
Can be done manually Difficult to done manually, needs to
be automated
Functional Testing

★ Example of Functional Testing

○ Unit Testing
○ Integration Testing
○ System Testing
○ Acceptance Testing
○ Regression Testing
○ Sanity Testing
○ Smoke Testing
Non-Functional Testing

★ Example of Non-Functional Testing:

○ Performance Testing
○ Load Testing
○ Stress Testing
○ Soak Testing
○ Volume Testing
○ Spike Testing
○ Compatibility Testing
○ Security Testing
What is Test Case

★ Test case is a set of steps that is used to verify the functionalities of an

application and to check whether it meets the specified requirements.

★ It mainly contains 3 parts:

1. Input/Test data
2. Test Steps
3. Expected result
Why test cases are important?

★ Why test cases are important?

a. Traceability
b. Reusability
c. Requirement Coverage
Psychology of Software Testing

12
Psychology of Software Testing

One of the primary causes of poor program testing is the fact that
most programmers begin with a false definition of the term.

They might say:

‘The purpose of testing is to show that a program performs its
intended functions correctly.’
‘Testing is the process of demonstrating that errors are not present.’
‘Testing is the process of establishing confidence that a program does
what it is supposed to do.’

“These definitions are upside-down.”

13
Adding Value Through Testing

● Adding value through testing means raising the quality or reliability

of the program.
● Raising the reliability of the program means finding and removing
errors.

● Therefore, don’t test a program to show that it works;

○ Rather, we should start with the assumption that the program contains
errors
○ and then test the program to find as many of the errors as possible.

● Thus a more appropriate definition is this:

○ Testing is the process of executing a program with the intent of finding
errors.
14
Why does this definition matter?

● Myers asserts that “establishing the proper goal has an important

psychological effect.”

● If our goal is to demonstrate that a program has no errors,

● We tend to select test data that have a low probability of causing the program to
fail.
● but if our goal is to demonstrate that a program has errors,
● then our test data will have a higher probability of finding errors.

15
More on the Psychology of Testing

● Myers asserts that “… people perform poorly when they set out on a task that
they know to be infeasible or impossible.”
● “Defining program testing as the process of uncovering errors in a program makes it a
feasible task.”
● “… the process of demonstrating that errors are not present” is “impossible to achieve
for virtually all programs”.

● If we define testing as “the process of demonstrating that a program does

what it is supposed to do”, then
● it can still have serious bugs (e.g. security faults) on inputs that cause it to do
something it is not intended to do.

16
Two Definitions from Myers?

✧ “Software testing is a process, or a series of processes,

designed to make sure computer code does what it is designed
to do and it does not do anything unintended.”

✧ “Software testing is a process,

▪ Or a series of processes,
✧ “designed to make sure computer code
▪ Does what it is designed to do and
▪ That it does not do anything unintended.”
What is a Successful Test?

● Two types of testing can be distinguished:

○ Testing for defect identification, a successful test is one which causes the
system to fail.
○ Testing to demonstrate that the software meets its specifications or other
desired properties, in which case testing is successful if no failures are
observed.

● In any event, a test case should only be run if it has a non-zero

chance of revealing a defect.
○ Otherwise it’s just waste of time with the test

18
Software Testing Principles

19
Software Testing Principles

★ A necessary part of a test case is a definition of the expected

output or result.
★ A programmer should avoid attempting to test his or her own
program.
★ A programming organization should not test its own
programs.
★ Any testing process should include a thorough inspection of
the results of each test.
★ Test cases must be written for input conditions that are invalid
and unexpected, as well as for those that are valid and
expected.
Software Testing Principles

★ Examining a program to see if it does not do what it is

supposed to do is only half the battle; the other half is seeing
whether the program does what it is not supposed to do.
★ Do not plan a testing effort under the tacit assumption that no
errors will be found.
★ The probability of the existence of more errors in a section of
a program is proportional to the number of errors already
found in that section.
★ Testing is an extremely creative and intellectually challenging
task.
★ Avoid throwaway test cases unless the program is truly a
throw away program.
 Static Testing

Inspections, Walkthroughs and Reviews

22
Static Testing

● Static testing is a process of checking the code and designing

documents and requirements before it's run to find errors.
● Static testing checks code, requirements, and design documents
for errors, while dynamic testing checks the functionality of a
software system
● The main goal is to find flaws in the early stages of development
because it is normally easier to find the sources of possible
failures this way.
Static Testing

Work Products that Can Be Examined by Static Testing:

● Specifications, including business requirements, functional
requirements, and security requirements
● User stories, and acceptance criteria
● Architecture and design specifications
● Code
● Testware, including test plans, test cases, test procedures, and
automated test scripts
● User guides
● Web pages
● Contracts, project plans, schedules, and budget planning
● Configuration set up and infrastructure set up
● Models which may be used for Model-Based testing
Benefits of Static Testing

● Detecting and correcting defects more efficiently, and prior to

dynamic test execution
● Defects found early are often much cheaper to remove
● Identifying defects which are not easily found by dynamic testing
● Preventing defects in design or coding by uncovering
inconsistencies, ambiguities, contradictions, omissions,
inaccuracies, and redundancies in requirements
● Increasing development productivity (e.g., due to improved
design, more maintainable code)
● Reducing development cost and time
● Reducing testing cost and time
● Reducing total cost of quality over the software’s lifetime, due to
fewer failures later in the lifecycle or after delivery into operation
● Improving communication between team members in the course of
participating in reviews
Differences between Static and Dynamic
Testing
❖ Static testing finds defects in work products directly rather than
identifying failures caused by defects when the software is run.
❖ Static testing can be used to improve internal quality of the software,
while dynamic testing focuses on externally visible behaviors.
Compared with dynamic testing, typical defects that are easier and cheaper
to find and fix through static testing include:
● Requirement defects
● Design defects
● Coding defects
● Deviations from standards
● Incorrect interface specifications
● Security vulnerabilities
● Gaps or inaccuracies in test basis traceability or coverage
Inspections

✧ Inspections involve a team of people reading or visually

inspecting a program

✧ These are so-called ‘static’ V & V technique

✧ Do not require execution of a system
✧ Focuses on static system representations such as requirements,
design model, source code etc. to discover defects
 Inspections (Cont.)
General Procedure:
✧ Inspection Team:
▪ Moderator
▪ Programer
▪ Program’s Designer
▪ Test Specialist
✧ Moderator’s Duties:
▪ Distributing materials for, and scheduling, the inspection session.
▪ Leading the session.
▪ Recording all errors found.
▪ Ensuring that the errors are subsequently corrected
Inspection and Testing

29
 Inspections (Cont.)

✧ Inspection Agenda:
▪ The programmer narrates, statement by statement, the logic of the
program. And other participants raise questions
▪ The program is analyzed with respect to checklists of historically
common programming errors
▪ The moderator is responsible for ensuring that the discussions proceed
along productive lines
▪ Upon the conclusion of the inspection session, the programmer is
given a list of the errors uncovered
Inspections (Cont.)

✧ Human Agenda:
▪ The testing group must adopt an appropriate attitude to make the
inspection process effective
▪ The programmer must take the process in a positive and
constructive way
▪ If managers somehow make use of the inspection results (to assume
or imply that the programmer is inefficient or incompetent), the
purpose of the process may be defeated.
 Inspections (Cont.)

✧ Side Benefits of the Inspection Process:

1. The programmer usually receives valuable feedback
concerning programming style, choice of algorithms, and
programming techniques
2. Other participants gain in a similar way
3. A way of identifying early the most error-prone sections
of the program
 Inspections (Cont.)

✧ An Error Checklist for Inspections:

Data Reference Errors
Data Declaration Errors
Computation Errors
Comparison Errors
Interface Errors
Input/Output Errors
Other Checks
 Walkthroughs

✧ The code walkthrough is a set of procedures and

error-detection techniques for group code reading
✧ It is an uninterrupted meeting of one to two hours in duration

✧ Team Members:
▪ Moderator
▪ Secretary (a person who records all errors found)
▪ Tester
▪ Programer
Walkthroughs (Cont.)

✧ Suggestions for the other participants include:

▪ A highly experienced programmer
▪ A programming-language expert
▪ A new programmer (to give a fresh, unbiased outlook)
▪ The person who will eventually maintain the program
▪ Someone from a different project
▪ Someone from the same programming team as the programmer
Walkthroughs (Cont.)

✧ Procedure:
▪ Rather than simply reading the program or using error
checklists, the participants ‘‘play computer”
▪ The tester prepares a small set of paper test cases
▪ During the meeting, each test case is mentally executed
▪ The test data are ‘‘walked through’’ the logic of the program
and the state of the program is monitored on paper or a
whiteboard.
 Peer Ratings

✧ Peer rating is a technique of evaluating anonymous programs

in terms of their overall quality, maintainability, extensibility,
usability, and clarity.
✧ The purpose of the process is to allow programmers to
self-assess their programming skills.
✧ The process appears to be useful in both industrial and
classroom environments.
Peer Ratings

Procedure:
○ A programmer is selected to serve as an administrator
○ The administrator selects approximately 6 to 20 participants
○ Each participant is asked to select two of his or her own programs to
be reviewed.
○ Then, they are randomly distributed to the participants.
○ Each participant spends 30 minutes reviewing each program and then
completes an evaluation form by rating the programs.
○ After the review, the participants are given the anonymous evaluation
forms for their two contributed programs.
○ They are also given a statistical summary showing the overall and
detailed ranking of their original programs across the entire set of
programs
Individual Review Techniques

Ad hoc
❏ Reviewers are provided with little or no guidance on how this
task should be performed.
❏ Reviewers often read the work product sequentially, identifying
and documenting issues as they encounter them
❏ This technique is highly dependent on reviewer skills and may
lead to many duplicate issues
 Individual Review Techniques

Desk Checking
✧ A desk check can be viewed as a one-person inspection or
walkthrough: A person reads a program, checks it with respect
to an error list, and/or walks test data through it.
✧ Desk checking is relatively unproductive
▪ it is a completely undisciplined process
▪ it runs counter to testing principle 2
✧ Desk checking is best performed by a person other than the
author of the program
Individual Review Techniques

Checklist-based
❏ It is a systematic technique, whereby the reviewers detect
issues based on checklists that are distributed at review
initiation
❏ A review checklist consists of a set of questions based on
potential defects, which may be derived from experience.
❏ Checklists should be specific to the type of work product
❏ The main advantage of the checklist-based technique is a
systematic coverage of typical defect types
Individual Review Techniques

Scenarios and dry runs

❏ Reviewers are provided with structured guidelines on how to
read through the work product.
❏ These scenarios provide reviewers with better guidelines on
how to identify specific defect types than simple checklist
entries
Individual Review Techniques

Perspective-based
❏ Reviewers take on different stakeholder viewpoints
❏ Typical stakeholder viewpoints include end user, marketing,
designer, tester, or operations.
❏ It requires the reviewers to attempt to use the program under
review to generate the product they would derive from it
Individual Review Techniques

Role-based
❏ A role-based review is a technique in which the reviewers evaluate
the work product from the perspective of individual stakeholder roles.
❏ Typical roles include specific end user types (experienced,
inexperienced, senior, child, etc.),
❏ Specific roles in the organization (user administrator, system
administrator, performance tester, etc.)
A model of the software testing process

45
General Testing Guidelines

● Testing shows the presence of defects, not their absence

● Exhaustive testing is impossible
● Early testing saves time and money
● Absence-of-errors is a fallacy
● Choose inputs that force the system to generate all error
messages
● Design inputs that cause input buffers to overflow
● Repeat the same input or series of inputs numerous times
● Force invalid outputs to be generated
● Force computation results to be too large or too small

46