9/13/2007

Cause effect graph Example of testing techniques

Functional testing: Cause effect graphs Syntactic testing Random Test Equivalence class analysis Boundary analysis
Proposed by G. J. Mayers
Establish a relation between the effect (an output) and its causes (inputs). 4 basic symbols express the interdependency between inputs and outputs:

Negation Identity

Logic And Logic OR

V

Cause effect graph, an example

File management example:

Cause effect graph: decision table

1 = true, 0 = false Test data Causes effect

The character of the first column should be A or B. The second column should be a number. In this case the file is considered up dated. If the first character is erroneous, we should print then the message X12. If in the second column we dont have a number, we print the message X13.

A1 A2 1 2 0 0 0 0 1 1 0 0 1 1 0 0

A3 0 1 0 1 0 1

M1 M2 M3 0 0 0 1 0 1 1 1 0 0 0 0 1 0 1 0 1 0

M2 A1 V A2 E M1

A1: first char is A A2:the first char is :B A3: the second is a number

3 4 5 6

M1: update the file M2: message X12 M3: the message X13

A3 M3

3 inputs, 8 possibilities But A1 = true and A2= true is an impossible combination

9/13/2007

Cause effect graph: optimization rules

A

Cause effect graph: methodology

Test technique based on cause effect graph consist in the following steps: 1- decompose of the unit to be tested, if it has many functionalities 2- identify the causes (input combinations or classes of input conditions) 3- identify the effects (output conditions, or system transformations) 4- establish the graph of relations between causes and effects g p f ff 5- complete the graph by adding the constraints between causes and effects 6- convert the graph to a decision table - assign true in sequence to all effects - create a line for each combination of causes that produces an effect - for each combination determine the state of all effects. 7- Produce a test per line after simplification of the table following the rules.

V
B

Disjunctive case R1: for X at 1 then dont take the case A= B=1 R2: for X at 0, enumerates all the situations where A=B=0

Until 6 variables the use of Boolean algebra based optimization techniques such as Karnaugh-Veitch map

Conjunctive case R3: for X at 1, enumerate all the situations Where A=B=C=1 R4: For X at 0 R4.1 include only the case A=B=C=0 R4.2 for all other cases( 001, 010, 100, 011,, 101, 110), include only one sample

Data validation & Syntax testing

Generators & Recognizers
The goal is to determine data tests based on formal description. The formal description is a BNF or an FSM. BNF= Backus- Naur Form ( grammar) FSM= Finite State Machine
The routine is a string recognizer The tester is string generator

Data validation & Syntax testing: example

Recognizer for copy aa to bb rename aa to bb copy a to b First step: define the grammar <command>::= < command type> < file name> to < file name> <command type>::= copy | rename <file name> ::= <letter> | <letter> <letter> <letter>::= a | b

What to capture? 3 possibilities:

- the recognizer does not recognize a good string - it accepts a bad string - it may accept or reject a good or bad string, but in doing so, it fails

9/13/2007

Data validation & Syntax testing

Second step: build a corresponding derivation tree
1 <command>

Data validation & Syntax testing

Third step: Produce data test based on the derivation tree cover valid commands, then invalid commands.
For the example, Valid commands Cover the non terminal nodes (1,2,3,5,8,9,10,11) to check if the 2 commands are recognized: copy a to b (covers the non terminals nodes 1,2,3,5,8,10 and terminal nodes ,6,12,15) 6 copy aa to bb (covers the non terminals nodes 1,2,3,5,9, 11 and terminal nodes ,6, 16, 18,21,23) Cover at least one time all terminal nodes (those not covered yet are: 7,13,14,17,19,20,22. We have to add then the following tests: rename b to a ( that covers 7,13,14) copy bb to aa ( 17, 19, 20, 22)

2 < command type>

6 copy 7 rename

3 <file name>

4 to

5 <file name>

or

9 8 or <letter> <letter> <letter>

11 10 or <letter> <letter> <letter> 14 15 a or b 22 23 a or b

12 13 a or b 16 17 a or b 18 19 a or b

20 21 a or b

Data validation & Syntax testing

Invalid commands It may exist an infinite number of invalid commands. There is need for sampling.

We distinguish 3 levels in the derivation tree Level 1: nodes 2,3,4,5 Level 2: nodes 8,9,10,11 Level 3: all other nodes

Level 2: copy a a to b copy a to b b rename a a to b etc. Level 3: copy c to b copy a to c copy aaa to b copy a to bbb copy a to + rename aaa to b cop a to b copy a t b renome a to b etc.

Produce various faults and simulate the addition or omission of nodes at each level

Level 1: a to b ( node 2 is omitted) copy to b (node 3 is omitted) copy a b (node is omitted) copy a to ( node 5 is omitted) rename a to rename to b etc

9/13/2007

Random testing
All test selection techniques seen before are deterministic. Random testing is based on a probabilistic approach. It uses a uniform sampling on definition domains of input variables. Random Number Generator (RNG), a function in your computer. See page 137 Kaner book. Statistical testing uses the structure to select equal number of values Over the domain definitions to solve the following example: Spec: x is an integer between 1 and 10 If x >8 the action1 else action2 the probability of executing the then is 2/10, the else is 8/10 for input selection based on random testing.

Black Box Software Testing

Presentation Outline
- Introduction to Black Box Software Testing? - Definition - Why Black Box Testing? - Testing Objectives and Focuses - An Example p - Graph-based Testing Methods - Equivalence Partitioning - Boundary Value Analysis - Requirements-Based Testing - Random Testing

Introduction to Black Box Testing

What is black box testing? - Black box testing also known as specification-based testing. - Black box testing refer to test activities using specification-based testing methods and criteria to discover program errors based on program requirements and product specifications. The major testing focuses: - specification-based function errors - specification-based component/system behavior errors - specification-based performance errors - user-oriented usage errors interface - black box interface errors input Black Box (Component or System) output operation

9/13/2007

Introduction to Black Box Testing

Testing a triangle analyzer: Under test units in black-box: What do you need? - For software components, you need component specification, user interface doc. -For a software subsystem or system, you need requirements specification, and product specification document. You also need: - Specification-based software testing methods - Specification-based software testing criteria Software components, subsystems, or systems Program specification:

An Example

Input: 3 numbers separated by commas or spaces Processing: Determine if three numbers make a valid triangle; if not, print message NOT A TRIANGLE. If it is a triangle, classify it according to the length of the sides as scalene (no sides equal), isosceles (two sides equal), or equilateral (all sides equal). If it is a triangle, classify it according to the largest angle as acute (less than 90 degree), obtuse (greater than 90 degree), or right (exactly 90 degree). Output: One line listing the three numbers provided as input and the classification or the not a triangle message.

- good understanding of software components (or system)

Example:

3,4,5 6,1,6 5,1,2

Scalene Right Isosceles Acute Not a triangle

An Example
Functional Test Cases: Acute Scalene: Isosceles: Equilateral: Functional Test Cases: F ti lT tC Input 4,4,4 1,2,8 6,5,3 5,6,10 3,4,5 6,1,6 7,4,4 1,1,2^(0.5) Expected Results Equilateral acute Not a triangle Scalene acute Scalene obtuse Scalene right Isosceles acute Isosceles obtuse Isosceles right 6,5,3 6,1,6 4,4,4 Obtuse 5,6,10 7,4,4 Not possible Right 3,4,5 1,2, 2^(0.5) Not possible

An Example
Test cases for special inputs and invalid formats: 3,4,5,6 646 3,,4,5 3 4,5 3.14.6,4,5 4,6 Four sides Three-digit single number Two commas Missing comma Two decimal points Two sides

5,5,A 55A 6,-4,6 -3,-3,-3

Character as a side Ch id Negative number as a side All negative numbers Empty input

9/13/2007

Software Testing Principles An Example

Boundary Test Cases: Davids [DAV95] suggests a set of testing principles: (1) Boundary conditions for legitimate triangles - All tests should be traceable to customer requirements. 1,1,2 Makes a straight line, not a triangle 0,0,0 Makes a point, not a triangle 4,0,3 A zero side, not a triangle 1,2,3.00001 Close to a triangle but still not a triangle 9170,9168,3 9170 9168 3 Very V small angle ll l Scalene, acute S l t .0001,.0001,.0001 Very small triangle Equilateral, acute 83127168,74326166,96652988 Very large triangle, scalene, obtuse Boundary conditions for sides classification: 3.0000001,3,3 Very close to equilateral, Isosceles, acute 2.999999,4,5 Very close to isosceles Scalene, acute Boundary conditions for angles classification: 3,4,5.000000001 Near right triangle 1,1,1.41141414141414 Near right triangle - Tests should be planned long before testing begins. - The Pareto principle applies to software testing. - 80% of all errors uncovered during testing will likely be traceable f ll dd i i ill lik l b bl to 20% of all program modules. - Testing should begin in the small and progress toward testing in the large. - Exhaustive testing is not possible. - To be most effective, testing should be conducted by an independent third party. Scalene, obtuse Isosceles, acute

Software Testability

Equivalence Partitioning

According to James Bach: Software testability is simply how easily a computer program can be tested. A set of program characteristics that lead to testable software: - Operability: the better it works, the more efficiently it can be tested. p y , ff y - Observability: What you see is what you test. - Controllability: The better we can control the software, the more the testing can be automated and optimized. - Decomposability: By controlling the scope of testing, we can more quickly isolate problems and perform smarter retesting. - Simplicity: The less there is to test, the more quickly we can test it. - Stability: The fewer the changes, the fewer the disruptions to testing. - Understandability:The more information we have, the smarter we will test. Test case design for equivalence partitioning is based on an evaluation of equivalence classes for an input domain. f p An equivalence class represents a set of valid or invalid states for input condition. An input condition is: - a specific numeric value, a range of values - a set of related values, or a Boolean condition Equivalence partitioning is a black-box testing method - divide the input domain of a program into classes of data - derive test cases based on these partitions.

9/13/2007

Equivalence Partitioning
Equivalence partitioning is a black-box testing method - divide the input domain of a program into classes of data - derive test cases based on these partitions. Test case design for equivalence partitioning is based on an evaluation of equivalence classes for an input domain. An equivalence class represents a set of valid or invalid states for input condition. An input condition i A i t diti is: - a specific numeric value, a range of values - a set of related values, or a Boolean condition
Valid inputs

Equivalence Classes
Equivalence classes can be defined using the following guidelines: - If an input condition specifies a range, one valid and two invalid equivalence class are defined. - If an input condition requires a specific value, one valid and two invalid equivalence classes are defined. - If an input condition specifies a member of a set, one valid and one invalid equivalence classes are defined. - If an input condition is Boolean, one valid and one invalid classes are defined. Examples: area code: input condition, Boolean - the area code may or may not be present. input condition, range - value defined between 200 and 900

system invalid inputs partition outputs All Rights Reserved

password: input condition, Boolean - a password nay or may not be present. input condition, value - six character string. command: input condition, set - containing commands noted before.

Boundary Value Analysis

Boundary Value Analysis

Boundary value analysis(BVA) - a test case design technique - complements to equivalence partition Objective: Boundary value analysis leads to a selection of test cases that exercise bounding values. Guidelines: - If an input condition specifies a range bounded by values a and b, test cases should be designed with value a and b, just above and below a and b. Example: Integer D with input condition [-3, 10], test values: -3, 10, 11, -2, 0 - If an input condition specifies a number values, test cases should be developed to exercise the minimum and maximum numbers. Values just above and below minimum and maximum are also tested. Example: Enumerate data E with input condition: {3, 5, 100, 102} test values: 3, 102, -1, 200, 5

- Guidelines 1 and 2 are applied to output condition. - If internal program data structures have prescribed boundaries, be certain to design a test case to exercise the data structure at its boundary Such as data structures: - array input condition: empty, single element, full element, out-of-boundary search for element: - element is inside array or the element is not inside array You can think about other data structures: - list, set, stack, queue, and tree

9/13/2007

One-Dimensional Domain Bugs in Open Boundaries

One-Dimensional Domain Bugs in Closed Boundaries

B A Closed Domain (A): B Closure Bug: B Boundary Shifted Right: B Boundary Shifted Left: A A A A

B An Open Domain (A): B Closure Bug: B Boundary Shifted Right: B Boundary Shifted Left: B Missing Boundary:

A A A A

A Missing Boundary:

B Extra Boundary:

A Extra Boundary:

If the domain boundary is open, an off point is a point near the boundary but in the domain being tested.

If the domain boundary is closed, an off point is a point near the boundary but in the adjacent domain.

An Example
Generic Domain Bugs in One-Dimensional Domains
Correct: Incorrect: Shifted Boundaries:

Given a function module which implements function Z= F(X, Y), which defined as follows: Z = F(X,Y), where X and Y are integer parameters for F. The detailed definition is given below. -| X+Y when 10<=X<=20, and 12<=Y<=30 Z= | X-Y when 0<=X<10, and 0<=Y<12 | 0 under other conditions -Please answer the following questions: (4%) Identify the equivalence classes in [X, Y]. (hints: Considering Z as a function F with X and Y input variables. ) (4%) List your test cases in [X, Y] based on the equivalence classes. (4%) Perform boundary value analysis, and list all boundary conditions for X and Y. (3%) List your test cases in [X, Y] based on boundary value analysis. X

Tilted Boundaries:

Open/Close Error:

Extra Boundary:

Missing Boundary:

9/13/2007

Domain Boundary

Testing Two-Dimensional Domains

- Closure bug. For example, using a wrong operator (for example, x >= k when x > k is intended or vise versa). This bug could be detected due to the testing of different boundaries or trying interior and off points. - Shifted boundary: For example, a boundary is shifted due to the use of an incorrect constant in a predicate, such as x+y>=17 when x + y >=7 was intended. The off point catches this bug. - Titled boundary: A tilted boundary occurs when coefficients in the boundary inequality are wrong. For example, 3x+7y > 17 when 7x + 3y > 17 was intended. Testing different domain points can detect the bug. - Extra boundary: An extra boundary is created by an extra predicate. Try different boundary points can detect this bug.

Extreme point

Interior point

Boundary point Off point

Two-dimension Domain Boundary

- Missing boundary: A missing boundary is created by leaving a boundary predicate out.