Experimental Design

Dr. Mussa I. Mgwatu

 Course Contents
1. Introduction to Experimental Design
2. Factorial Designs
3. Response Surface Designs
4. Taguchi Designs
 Introduction
Benefits of experimental design:
• Design a proper set of experiments for measurement
or simulation.
• Develop a model that best describes the data obtained.
• Estimate the contribution of each alternative to the
performance.
• Isolate the measurement errors.
• Estimate confidence intervals for model parameters.
• Check if the alternatives are significantly different.
• Check if the model is adequate.
 Introduction (Cont.)
Terminology:
• Response Variable: Outcome.
E.g., throughput, response time

• Factors: Variables that affect the response variable.

E.g., CPU type, memory size, number of disk drives,
workload used, and user's educational level.
Also called predictor variables or predictors.
 Introduction (Cont.)
• Levels: The values that a factor can assume, E.g., the
CPU type has three levels: 68000, 8080, or Z80.
# of disk drives has four levels.
Also called treatment.

• Primary Factors: The factors whose effects need to be

quantified. E.g., CPU type, memory size only, and
number of disk drives.
 Introduction (Cont.)
• Secondary Factors: Factors whose impact need not be
quantified. E.g., the workloads.

• Replication: Repetition of all or some experiments.

• Design: The number of experiments, the factor level

and number of replications for each experiment. E.g.,
Full Factorial Design with 5 replications: 3×3×4×3×3
or 324 experiments, each repeated five times.
 Introduction (Cont.)
• Experimental Unit: Any entity that is used for
experiments. E.g., users. Generally, no interest in
comparing the units.

• Goal - minimize the impact of variation among the

units.

• Interaction ⇒ Effect of one factor depends upon the

level of the other.
 Introduction (Cont.)
Common mistakes in experimentation:
• The variation due to experimental error is ignored.
• Important parameters are not controlled.
• Effects of different factors are not isolated
• Simple one-factor-at-a-time designs are used
• Interactions are ignored
• Too many experiments are conducted.
 Introduction (Cont.)
Principles of experimentation:
• Three principles of experimental design such as
randomization, replication and blocking can be utilized
in experiments to improve the efficiency of
experimentation.

• They are applied to reduce or even remove

experimental bias. Large experimental bias could result
in wrong interpretation on the effect of the really
significant factors.
 Introduction (Cont.)
Randomization:
• While designing industrial experiments, there are
factors, such as power surges, operator errors,
fluctuations in ambient temperature and humidity, raw
material variations, etc. which may influence the process
output performance because they are often expensive
or difficult to control

• Such factors can adversely affect the experimental

results and therefore must be either minimized or
removed from the experiment.
 Introduction (Cont.)
• Randomization is one of the methods used to reduce
the effect of experimental bias by averaging out the
effects of noise factors that may be present in the
process.

• Randomization can ensure that all levels of a factor

have an equal chance of being affected by noise factors.
 Introduction (Cont.)
Replication:
• Replication means repetitions of an entire experiment
or a portion of it, under more than one condition.

• Replication has two important properties. The first

property is that it allows to obtain an estimate of the
experimental error. The second property is that it
permits to obtain a more precise estimate of the
factor/interaction effect.
 Introduction (Cont.)
• Sufficient number of replicates would make satisfactory
inferences about the effect of factors/interactions.

• However, replication can result in a substantial increase

in the time to conduct an experiment, and if the
material is expensive, replication may lead to exorbitant
material costs. Therefore, the use of replication in real
life must be justified in terms of time and cost.
 Introduction (Cont.)
Blocking:
• Blocking is a method of eliminating the effects of
unrelated variation due to noise factors and thereby
improving the efficiency of experimental design.

• The main objective is to eliminate unwanted sources of

variability such as batch-to- batch, day-to-day, shift-to-
shift, etc. , and arrange similar experimental runs into
blocks (or groups). Generally, a block is a set of
relatively homogeneous experimental conditions.
 Introduction (Cont.)
• The blocks can be batches of raw materials, different
operators, different vendors, etc. Observations collected
under the same experimental conditions (i.e. same day,
same shift, etc.) are said to be in the same block.

• Variability between blocks must be eliminated from the

experimental error, which leads to an increase in the
precision of the experiment
 Factorial Designs
Full factorial design:
• Two or more independent variables are manipulated in
a single experiment
• They are referred to as factors
• The major purpose of the research is to explore their
effects jointly
• Factorial design produce efficient experiments, each
observation supplies information about all of the
factors - All possible combinations
 2 2 Factorial Design
• Two factors, each at two levels Cache Memory size
• Example: workstation design size 4M byte 8M byte
– Factor 1: memory size 1K 15 45
– Factor 2: cache size 2K 25 75
– Dependent variable:
performance
22 Factorial Design (Cont.)

Interaction in a 22 experiment
 2K Factorial Design
• K factors, each at two level
Factor Level -1 Level 1
• 2K experiments
Memory size 4Mbytes 16Mbytes
• 23 design example
Catch size 1Kbytes 2Kbytes
– In designing a personal
workstation, the three Number of 1 2
factors needed to be processors
studied are: cache size,
memory size and
number of processors
2 3 Factorial Design

Interaction in a 23 experiment
2K Factorial Design with Replication
• r replications of 2K experiments
– 2Kr observations
– Allows estimation of experimental errors
– 223 design example
• The memory-cache experiments were repeated
three times each. The result is shown below

Cache size Memory size

4M 8M
1K 15, 18, 12 45, 48,51
2K 25, 28, 19 75,75,81
 General Full Factorial with Replication

• In the general full factorial design with replication

– Measure system response with all possible input
combinations
– Replicate each measurement n times to determine
effect of measurement error
• m factors, v levels, n replications
→ n vm experiments
• m = 5 input factors, v = 4 levels, n = 3
– → 3(45) = 3,072 experiments!
 Special Case of General Full Factorial

• Special case of generalized m-factor experiments

• Restrict each factor to two possible values
– High, low
– On, off
– n2m experiments

• Find factors that have largest impact

 Interaction Effect
• An interaction effect exists when differences on one
factor depend on the level of another factor
• How do we know if there is an interaction in a factorial
design?
– Statistical analysis will report all main effects and
interactions.
– If you can not talk about effect on one factor
without mentioning the other factor
– Spot an interaction in the graphs – whenever there
are lines that are not parallel there is an interaction
present!
Interaction Effect (Cont.)

Factorial Experiment, no interaction

Interaction Effect (Cont.)

Factorial Experiment, with interaction

 Fractional Factorial Design
• Full factorial design
– Study all combinations
– Can find effect of all factors
– May try 2K factorial design first

• Fractional (incomplete) factorial design

– Leave some treatment groups empty
– Less information
– May not get all interactions
– No problem if interaction is negligible
 k-p
2 Fractional Factorial Design
• Large number of factors
– Large number of experiments
– Full factorial design too expensive
– Use a fractional factorial design

• 2k-p design allows analyzing k factors with only 2k-p

experiments.
– 2k-1 design requires only half as many experiments
– 2k-2 design requires only one quarter of the
experiments
 27-4 Design
Exp No. A B C C E F G
1 -1 -1 -1 1 1 1 -1
2 1 -1 -1 -1 -1 1 1
3 -1 1 -1 -1 1 -1 1
4 1 1 -1 1 -1 -1 -1
5 -1 -1 1 1 -1 -1 1
6 1 -1 1 -1 1 -1 -1
7 -1 1 1 -1 -1 1 -1
8 1 1 1 1 1 1 1

• Study 7 factors with only 8 experiments

• When quantify the effects, just calculate the main effects
• Will be able to eliminate some factors in further study.
 One-factor ANOVA
• Each individual yij  y..  i  eij
measurement is y..  overall mean
composition of
i  effect due to A
– Overall mean eij  measuremen t error
– Effect of
alternatives
– Measurement
errors
 One-factor ANOVA

Variation Alternatives Error Total

Sum of squares SSA SSE SST
Deg freedom k 1 k (n  1) kn  1
M ean square sa2  SSA (k  1) se2  SSE [k (n  1)]
Computed F sa2 se2
Tabulated F F[1 ;( k 1), k ( n 1)]
 Two-factor ANOVA
• Each individual yijk  y...   i   j   ij  eijk
measurement is y...  overall mean
composition of  i  effect due to A
– Overall mean  j  effect due to B
– Effects  ij  effect due to interaction of A and B
– Interactions eijk  measuremen t error

– Measurement
errors
 Two-factor ANOVA
A B AB Error
Sum of squares SSA SSB SSAB SSE
Deg freedom a 1 b 1 (a  1)(b  1) ab(n  1)
M ean square sa2  SSA (a  1) sb2  SSB (b  1) sab 2
 SSAB [(a  1)(b  1)] se2  SSE [ab(n  1)]
Computed F Fa  sa2 se2 Fb  sb2 se2 Fab  sab 2
se2
Tabulated F F[1 ;( a 1), ab( n 1)] F[1 ;(b 1), ab( n 1)] F[1 ;( a 1)( b 1), ab( n 1)]
 Response Surface Methodology and Designs

• Response surface methodology, or RSM , is a

collection of mathematical and statistical techniques
that are useful for modeling and analysis in applications
where a response of interest is influenced by several
variables and the objective is to optimize this response.

• For example: Find the levels of temperature (x1) and

pressure (x2) to maximize the yield (y) of a process.
y  f ( x1 , x2 )  
 RSM and Designs (Cont.)

• Approximate the true relationship between y and the

independent variables by the lower-order polynomial
model.
y  0  1x1    k x k   The first-order model
k k
y  0   i xi   ii xi2  ij xi x j  
i 1 i 1 i j
The second-order model

• For the second order model, we determine whether the

stationary point is a point of maximum or minimum
response or a saddle point.
 RSM and Designs (Cont.)

A three-dimensional response surface showing the

expected yield as a function of temperature and feed
concentration.
RSM and Designs (Cont.)

Contour plot of yield response surface

 RSM and Designs (Cont.)

• The following designs are widely used for fitting a

quadratic model:
– Central Composite Design (CCD)(uniform
precision of effect estimates)
– Box-Behnken Design (almost uniform precision of
effect estimates, but usually fewer runs required than
CCD)
 RSM and Designs (Cont.)

• Central Composite Design is always executed by adding

points to an already performed 2k design
• Box-Behnken design consists of combinations from 2k
design
 Taguchi Designs
• Introduced by Dr. Genichi Taguchi (1980)
– Comparable in importance to Statistical Process
Control (SPC), the Deming approach and the
Japanese concept of TQC
• Unique aspects of the Taguchi method
– The Taguchi definition of quality
– The Taguchi Quality Loss Function (QLF)
– The concept of Robust Design
 Taguchi Designs (Cont.)
• The Taguchi definition of quality
– Ideal quality refers to a target value for determining
the quality level
– Ideal quality is delivered if a product or service
tangible performs its intended function throughout
its projected life under reasonable operating
conditions without harmful side effects
– Ideal quality is a function of customer perception
and satisfaction
– Service quality is measured in terms of loss to
society
 Taguchi Designs (Cont.)
• The concern is deviation of a characteristic from its
nominal value. Uncontrollable factors (noise) are often
responsible for this deviation (loss of quality).

• Therefore, Taguchi’s approach to experimental design

has as its goal the design of products/process that are
robust to these noise factors.
 Taguchi Designs (Cont.)
• Taguchi designed a number of orthogonal arrays to aid
in the development of experiments

• These orthogonal arrays are balanced fractional

orthogonal designs and can be used to estimate main
effects using only a few experimental runs.

• Two of the Taguchi’s simpler Orthogonal arrays are:

L4(23) and the L8(27)
 Taguchi’s Two Level Designs

L 4(23) Orthogonal Array The L8(27) Orthogonal Array

Factors Factors
Run 1 2 3 Run 1 2 3 4 5 6 7
1 1 1 1 1 1 1 1 1 1 1 1
2 1 2 2 2 1 1 1 2 2 2 2
3 1 2 2 1 1 2 2
3 2 1 2
4 1 2 2 2 2 1 1
4 2 2 1 5 2 1 2 1 2 1 2
6 2 1 2 2 1 2 1
7 2 2 1 1 2 2 1
8 2 2 1 2 1 1 2
Taguchi’s Three Level Designs
L9(34) orthogonal arrays

Factors
Run 1 2 3 4
1 1 1 1 1
2 1 2 2 2
3 1 3 3 3
4 2 1 2 3
5 2 2 3 1
6 2 3 1 2
7 3 1 3 2
8 3 2 1 3
9 3 3 2 1
 Signal to Noise (SN) Ratios
• To determine the effect each variable has on the output,
the signal-to-noise ratio, or the SN number, needs to be
calculated for each experiment conducted.

• The signal to noise ratios are derived from the Taguchi

loss function.

• The following three signal to noise ratios are widely

used:
 SN Ratios (cont.)
 2
y 
Nominal is Best: SN N 10log  2 
s 
 

 n 

  1
2
yi

Larger is Better: SN L   10log 

i 1
n


 
 
 
 n 

  yi2 

Smaller is Better: SN S   10log 

i 1
n


 
 
 
– yi is the mean value - the performance characteristic for a
given experiment and s is the variance