EGR 601

Advanced Engineering Experimentation

Uncertainty Analysis

Dr. Ernur Karadoğan

3
4 http://berkeleysciencereview.com/wp-content/uploads/2014/04/to_err_is_human_by_velica-d4i9wjr.jpg
Uncertainty Analysis
 Estimating the quality of a test result
 Answer the question: “± 𝒘𝒉𝒂𝒕?”

 Since we don’t know true value of a measurement, we

don’t know the error exactly

 We can only estimate the (range of) error based on what

is available to us: “our measurements”.

 The estimate of the range of the probable error is

assigned a value called uncertainty.
5
Why uncertainty is important?

 To assess the reliability of a result

 To know the confidence that can be placed in any

decisions based on its use (medical diagnosis etc.)

 To compare measurement results

6
Uncertainty Analysis
 Error is an inherent property of the measurement.
 Error is the difference between the measurement and the true
value.

 Uncertainty is an inherent property of the result.

 Outcome of a measurement is a result, and the uncertainty
quantifies the quality of that result.

 Errors are effects (that cause a measured value to differ

from the true value).

 Uncertainties are numbers (that quantify the probable

ranges of the errors).

7
Uncertainty Analysis
 We will discuss a systematic approach for identifying,
quantifying, and combining the estimates of the errors in a
measurement.

 The quality of an uncertainty analysis depends on the:

 engineer’s knowledge of the test (What variables, factors come
into play during testing and measurement?)
 measured variables (each measured variable will have an
uncertainty associated with the measurement)
 Equipment (each equipment has an uncertainty level associated
with it)
 measurement procedures
8
Uncertainty Analysis
 Two accepted professional documents

 “The American National Standards Institute/American

Society of Mechanical Engineers (ANSI/ASME) Power Test
Codes (PTC) 19.1 Test Uncertainty” is the United
States engineering test standard,

 The International Organization on Standardization’s

‘‘Guide to the Expression of Uncertainty in
Measurement’’ (ISO GUM) is an international metrology
standard.*

*9 Available on Bb.
Measurement Errors
 Errors are grouped into two categories:
 systematic error and random error

 Best estimate of the true value still holds:

x  x  u x
 Previously, we dealt with random uncertainty (based
on the statistics from a sample measurement)

 Now, we will include all known errors

**Measurement blunders are not considered!

10
Error types
 Random error- from variation found during repeated
measurements (precision error) =𝑥𝑖 − 𝑥ҧ
 Bias error- from offset in measurement system
(systematic error) =𝑥ҧ − 𝑡𝑟𝑢𝑒 𝑣𝑎𝑙𝑢𝑒

11
Assumptions of Uncertainty
Analysis
 Test objective is known and measurement process is well
defined

 Any known correction for systematic error have been applied

(calibration of the equipment is done!)

 Normal distribution of errors

 Errors are independent of each other (we will deal with

correlated errors as well)

 The engineer has some prior knowledge of what to expect

 Has an idea about the uncertainty outcome based on prior
experience and/or manufacturer’s performance specifications

12
Uncertainty Analysis
 Three measurement situations will be considered:

 Design stage
 where tests are planned but information is limited
 Considers only the resolution and calibration errors

 Advanced stage or single measurement

 where additional information about process control can be used to
improve a design-stage uncertainty estimate

 Multiple measurements
 where all available test information is combined to assess the
uncertainty in a test result
13
 Design stage uncertainty
 Performed initially when tests are planned when
information is somewhat limited,

 Quick estimate of the minimum uncertainty to be

expected (based on the instruments and method chosen)

 If too large, seek alternative ways (equipment,

measurement system etc.)

 Never used for reporting results…

14
 Zero-order & Instrument Uncertainty
 Even when all errors are zero, the value of the measurand
must be affected by the ability to resolve the information
provided by the instrument. This is called zero-order
uncertainty (Interpolation Error). At zero-order, we
assume that the variation expected in the measurand will be
less than that caused by the instrument resolution. And that
all other aspects of the measurement are perfectly
controlled (ideal conditions)

1
u0   resolution = 1 LSD (Least significant digit)
2
 Instrument uncertainty, uc, estimates systematic
uncertainty of the instrument (instrument calibration errors).
May be composed of several elemental errors.
15
More on Zero-order uncertainty
Analog: ½ cm

Digital: 0.1/2=0.05 units

Rounding is at most half of the least significant digit displayed/read.

16
 Combining Elemental Errors:
RSS Method
 Uncertainty Propagation: Can combine individual/elemental
errors by RSS (root-sum-squares) method to estimate the total
instrument uncertainty:

 Each individual error, 𝑢𝑘 , is called elemental error.

 The RSS method of combining uncertainties is based on the

assumption that the square of an uncertainty is a measure of the
variance (i.e., 𝜎 2 ) assigned to an error, and the propagation of these
variances yields a probable estimate of the total uncertainty.
17
 elemental
numbers associated with
errors
elemental error (i.e., uncertainties)

 P% = 95% probability level is common in test engineering

(assuming that the probability covered by 2𝜎)

 “Standard” uncertainty assumes that

the probability covered by 1𝜎 (68%
probability level)
18
Design stage uncertainty
(Minimum value for Uncertainty)

 Same units, and confidence level for each ui

 Generally use 95% confidence interval or ± 2𝜎
 The design stage uncertainty is

ud  u0  uc
2 2 2

19
Design stage uncertainty

Example:
 Consider a temperature probe with the following specifications:

hysteresis : ±0.1°C
linearization error : ±0.2% of reading
resolution : 0.05°C
zero offset error : ±0.03°C

Determine the types of error (random or systematic)

and the design stage uncertainty at a reading of 120°C

20
Design stage uncertainty

Solution:
 Identify the sources of error:
hysteresis : ±0.1°C Systematic
linearization error : ±0.2% of reading Systematic
resolution : 0.05°C Random
zero offset error : ±0.03°C Systematic

 Assume same confidence level for each uncertainty

u d  u0  uc
2 2 2

ud  0.0252  [0.12  (0.002 *120) 2  0.032 ]  0.26 C

2

21
Example

*
*

22
* “over” the range, not “of FSO”
Another Example

*
*

23
* Here “mV/psi” implies “of the reading” over the entire range
Another Example

Pressure
Voltmeter
transducer

Design-stage uncertainty
analysis shows us that a
better transducer, not a better
voltmeter, is needed if we
must improve the uncertainty
in this measurement!
24
Sources of error
 Coincide with three distinct stages of the measurement
process:
 Calibration errors
Measurement
 Data acquisition errors process
 Data reduction errors

• Within each group, list all elemental errors and assign

uncertainty values to each error

• For bookkeeping purposes; not mandatory

25
Calibration errors
 Occur during the calibration of the measuring system

26
Data acquisition errors
 Occur during actual measurement(s)

27
Data reduction errors
 Errors introduced in how data are handled following
collection
 Example: Curve fit error, truncation error, interpolation
error, modeling error etc.
 Systematic and random errors in each element

(standard error of the fit in regression)

(errors from assumed models)

28
Systematic uncertainty
 Systematic (bias) error is constant in repeated
measurements
 Systematic “standard” uncertainty
 ±b (confidence level of one 𝜎; 68% probability level for normal
distribution)

 Systematic uncertainty
 ±2b (confidence level of 2𝜎; 95% probability level for normal
distribution)

 At any probability level (assume large dof for t)

(95% probability level)

29
t-table
Systematic uncertainty

 Recognizing systematic error

 Calibration
 Concomitant measurement
 Interlaboratory comparisons (common procedure in
metrology)
 Judgement (based on experience)

30
Random uncertainty
 Manifested as scatter of the measured data (easily
observed)

 Random “standard” uncertainty

 The random uncertainty at a desired confidence level

31
Uncertainty Analysis: Propagation of Error

 Consider the experiment to measure the volumetric flow

rate through a pipe
 We measure time, t, and bucket volume, V
𝑉
 𝑄 = 𝑓(𝑡, 𝑉) =
𝑡

 How do uncertainties in either measured quantity

propagate to the uncertainty in flow rate?

 Is Q more sensitive to V or t?

33
 Propagation of error
Known:
• 𝑦 = 𝑓(𝑥)
• We measure x a number of times at some operating condition to establish its
mean value 𝑥ҧ and uncertainty 𝑢𝑥 .
• True value of 𝑥 lies somewhere within the interval 𝑥ҧ ±𝑢𝑥

y   y  f ( x  ux )

34
Taylor Series expansion

y   y  f ( x  ux )

35
Propagation of error
y   y  f ( x  ux )
Taylor Series expansion:
 dy 1 d2y 
f ( x  u x )  f ( x )  ( ) x  x u x  ( 2 ) x  x u x 2  ...
 dx 2 dx 

 dy 1 d2y 
y   y  f ( x )  ( ) x  x u x  ( 2 ) x  x u x  ...
2

 dx 2 dx 

By inspection: linear approximation for  y (neglect H.O.T.):

dy
y  f (x)  y  ( ) x x ux
dx
Uncertainty in y due to uncertainty in x
36
 Multi-variable Error Propagation
Errors in a measured variable are propagated through to a resultant variable
in a predictable way.
y  f (x)
Sensitivity of y to changes in x
dy
y ( ) x x ux
dx

Uncertainty in y due to uncertainty in x

uy   y

dy
u y  ( ) x x ux
dx
37
 Multi-variable Error Propagation
 Can generalize previous idea by extending to 𝐿 independent variables
Single variable (x) Multiple variables (xi, i=1, 2, …, L)

Functional relationship y  f ( x) R  f x1 , x2 , xL 

True value estimation 𝑥 ′ = 𝑥ҧ ±𝑢𝑥 R  R  uR
Sample mean y  f (x) R  f x1 , x2 , xL 
dy
y ( ) x x ux
Uncertainty dx
dy

uR  f u x1 , u x2 , u xL 
u y  ( ) x x ux
dx R
i  Sensitivity index
Develop a general sensitivity index: xi xx
38
Propagation of error
 Use sum-of-squares to determine uncertainty in R:

 L
2
1/2
R
uR     (i u xi )  where i 
 i 1  xi xx

each 𝑢𝑥ҧ𝑖 ,𝑖 = 1, 2, . . . , 𝐿 represents the uncertainty associated

with the best estimate of 𝑥1 and so forth through 𝑥𝐿.

 Same confidence level for each 𝑢𝑥ҧ𝑖

 Each variable must be independent

39
 1/2
 L

uR     (i u xi )2 
Example  i 1 

R
i 
xi xx

40
Approximating a Sensitivity Index:
Dithering*
 Apply a small perturbation to an input variable
 Useful when:
 The input-output relationship is from an experimental or numerical
model
 Analytical differentiation is complicated

 𝛿𝑥 is a small perturbation value (the change it causes in the

output provides an estimate of the sensitivity)
𝜕𝑦
 Reduce 𝛿𝑥 until the value of no longer changes (within a
𝜕𝑥
tolerance)

*41to dither: to be indecisive.

 Example

Analytical solution

Dithering (using the point x = 4 for illustration)

133.704 − 80.776
2(0.4)

In reality, y+ and y- values would be obtained by means of experimentation, i.e.

42
no analytical relationship between x and y needed a priori.
Only Numerical Data (tabulated or graphed):
Dithering
When there is no functional relationship readily available…

Central difference formula (can use for

2( )

43
Propagation of Uncertainty (Numerical Approach):
Sequential Perturbation
Uses the finite difference to approximate the derivatives (sensitivity index)

Establish the
operating point

Based on a single variable

44
Propagation of Uncertainty (Numerical Approach):
Sequential Perturbation

Based on a single variable

𝑅ത + 𝛿𝑦

45
Propagation of Uncertainty (Numerical Approach):
Sequential Perturbation

Based on a single variable

𝑅ത − 𝛿𝑦

46
Propagation of Uncertainty (Numerical Approach):
Sequential Perturbation

Based on a single variable

47
Propagation of Uncertainty (Numerical Approach):
Sequential Perturbation

1/2
 L 
uR     (i u xi )2 
 i 1 

R  R  uR

48
Example

1.

2.

3.

4.

Operating point

49 Operating
Point
Advanced-stage uncertainty analysis
(Single measurement uncertainty analysis)

 Design stage only considered errors due to instrumentation

resolution and calibration

 Takes design-stage analysis further by incorporating effects of

procedural and test control errors

 Single measurement uncertainty analysis used

 To estimate uncertainty beyond design stage uncertainty
 To report results of a pilot test where one or more variables were
perturbed, but no/few repeated measurements performed

51
Multiple measurement uncertainty analysis
 Assesses uncertainty of a variable based on a set of
measurements

 Major steps:
 Identify elemental errors (error sources)
 Estimate systematic and random error in each
 Estimate the uncertainty of the result

 Consider measuring of variable x subject to:

 Standard random uncertainty, (𝑠𝑥ҧ )k
 Systematic uncertainty, (𝑏𝑥ҧ )k
 k refers to each element of error

57
Multiple measurement uncertainty analysis

 Propagate random and systematic uncertainties by the RSS

method
1/2
s x   s x 1   s x 2  ...   s x  K 
2 2 2
random
 
1/2 systematic
bx   bx 1   bx 2  ...   bx  K 
2 2 2
 
 Uncertainty in x, ux, is a combination of systematic uncertainty
and standard random uncertainty
2 1/2
u x   bx    s x  
2
( P %) combined “standard” uncertainty in x
  (68%)
2 1/2
u x  t , P  bx    s x  
2
( P %) expanded uncertainty in x
  (95%)
58
Multiple measurement uncertainty analysis
 Degrees of freedom, , must be determined from total degrees
of freedom in random uncertainties, i.e. different degrees of
freedom may exist in different elements of 𝑠𝑥ҧ

 Use Welch-Satterthwaite formulation:

total degrees of freedom in only “random uncertainties”

 K: total number of elemental errors,

59
Propagation of Elemental Errors
coverage factor

expanded uncertainty in x at P%

Standard
Uncertainties
(68%)

60
Example

The uncertainty due to data scatter here could be classified as being due to a
temporal variation error
61
Example
𝐵1 = ±2𝑏1 = ±0.20 𝑁
force measuring instrument

Estimate the systematic uncertainty in the measurement instrument.

𝐵2 = ±2𝑏2 = ±0.30 𝑁

The systematic standard uncertainties due to each error source

The systematic standard uncertainty in the transducer

The expanded systematic uncertainty

62 Systematic
Uncertainty
Example

63
Example

64
65
 Propagation of uncertainty to a “result”
 Consider R = f(xn), n=1 to L , L is # of independent variables

Best estimate of true value of result: R  R  uR

Mean value determined at: R  f1 ( x1 , x2 ,..., x L )

uR  f 2 (bx1 , bx 2 ,..., bxL , s x1 , s x 2 ,..., s xL )
 L 
Propagation of systematic uncertainty: bR 2
   (i bxi )2 
 i 1 
 L 
Propagation of random uncertainty: sR 2
   (i s xi )2 
 i 1 
1/2
uR  t , P  bR   ( sR ) 
2 2
( P %)
Uncertainty in result:  
66
expanded uncertainty in the result
Propagation of uncertainty to a “result”
 When N is large, ≥ 30, can assume t,95 = 2
 Degrees of freedom, , must be determined from
total degrees of freedom in random uncertainties, i.e.
different degrees of freedom may exist in different
elements of R, particularly if n < 20-30
 Use Welch-Satterthwaite formulation* to calculate the
degree of freedom of the “result”:

Usually 𝜈𝑏𝑖 is very large;

Second term in the denominator is zero

No θ
67 * When the degrees of freedom in each of the variables, xi, is not the same.
Uncertainty Analysis Procedure
 Define the measurement process

 List all elemental error sources

 Estimate the elemental errors

 Calculate the systematic and random uncertainty for each

measured variable

 Propagate the systematic uncertainties and standard deviations all

the way to the result(s)

 Calculate the total uncertainties of the results

ASME
68 (1998) Test Uncertainty Part 1, ASME Power Test Code, 19.1-1998
 psfa: pounds per square foot absolute
Example

Assume a zero (negligible) systematic error in the gas constant.

70 101.3 kPa = 14.7 psia = 2116 psfa; 1 °R = 0.555556 K

[PRESSURE]

[PRESSURE] The instrument error is assigned a systematic uncertainty based on

the manufacturer’s statement, which is assumed to be stated at 95% confidence:

2253.91 11.27 psfa

[PRESSURE] The temporal variation causes a random uncertainty in

establishing the mean value of pressure and is calculated as
Temporal
variation

no systematic uncertainty to this error gives

71
Example
[TEMPERATURE] data-acquisition source error in temperature.

[TEMPERATURE ] The temporal variation causes a random uncertainty in

establishing the mean value of temperature

Temporal
variation

72
Example
 Combine
(11.27) 11.27 psfa
Higher contribution by pressure
Highest contribution overall

Lower contribution by temperature

The degrees of freedom in the systematic standard uncertainties,

73
Example
Propagation to the result

𝜕𝜌 𝑝 1
=−
𝜕𝑇 𝑅 𝑇2
𝜕𝜌 1
=
𝜕𝑝 𝑅𝑇

74
~0 (Assume large degrees of freedom) 𝜈
 (3.54%)
relative uncertainty (of the reading)

COMMENTS (1) We did not consider the uncertainty associated with our assumption of
exact ideal gas behavior, a potential modeling error. (2) Note how pressure contributes
more to either standard uncertainty than does temperature and that the systematic
uncertainty is small compared to the random uncertainty. The uncertainty in density is best
reduced by actions to reduce the effects of the random errors on the pressure
measurements.

75
 Researcher mistakes are not included.
Example
The mean temperature in an oven is to be estimated by using
the information obtained from a temperature probe. The
manufacturer states an uncertainty of ∓0.6°C (95%) for this
probe. Determine the oven temperature.
A measurement procedure
Sources of Error:

Temperature measuring device

Spatial variation (where in the oven

we are measuring the temperature)

Temporal variation (just sit back and see

the measurements scatter)

76
=?
Measurements
10 measurements are
made at each position.

77
 Mean oven temperature (pooled)

[Error source I] The temperature probe system (instrument

error)
Manufacturer statement of ∓0.6°C is considered a
systematic uncertainty at 95% confidence level.

𝐵1 = ±2𝑏1 = ± 0.6°C

78
[Error source II] Spatial variation (where in the oven we are
measuring the temperature), i.e., the spatial error contribution
to the estimate of the mean temperature 𝑇. ത

spatial nonuniformity in the oven temperature.

mean temperatures
at each measured
location

79
Assessing the spatial variation as a systematic uncertainty wouldn’t change the
overall uncertainty. It is more important to recognize and include the source of
error than to classify it.

[Error source II] Spatial variation (where in the oven we are

measuring the temperature), i.e., the spatial error contribution
to the estimate of the mean temperature 𝑇. ത
standard deviation

spatial nonuniformity in the oven temperature.

random standard uncertainty

(standard error)

No systematic uncertainty
80 associated with spatial variation
[Error source III] Temporal variation (just sit back and see the
temperature measurements scatter)
Time variations in probe output during each of the 10 measurements at each
location cause data scatter, as evidenced by the respective 𝑠𝑇𝑚 values.

The pooled standard deviation*

random standard uncertainty

(standard error)
81 Pooled
* Individual measurements are not given; therefore use the pooled formulas statistics
 Combine Uncertainties
The measurement systematic standard uncertainty

The measurement random standard uncertainty

The combined standard uncertainty

82
 The combined standard uncertainty

The combined expanded uncertainty

Welch-Satterthwaite formula

0.452 + 0.122 + 0.32 2

𝜈= = 16.07
0ൗ + 0.454ൗ + 0.124ൗ 0.3 4 0 0
ൗ𝜈1 + ൗ7 + ൗ72
𝜈1 7 72 +
83 Due to instrument error; assume very large DOF
Correlated Errors
 If two errors are not independent, they are ‘‘correlated.’’
 Multiple instruments calibrated against the same standard
 Consider a voltmeter used in two different measurement
systems (e.g. pressure, and temperature in a tank)
 Each system has a combined uncertainty that will include the
voltmeter’s uncertainty (random and systematic)
 Propagating the voltmeter’s uncertainty to the result
will include dependent errors (correlated)

 Numerical effect of the correlated errors depend on the

functional relationship

84
 Systematic (Correlated) Error

H of these K elemental errors are correlated between variables while the rest (K – H) are
uncorrelated.

the systematic standard uncertainty in a result is estimated by

𝛿𝑖𝑗 = 1 correlated Number of correlated errors

Covariance coefficient
𝛿𝑖𝑗 = 0 no correlation

85
 Random (Correlated) Error

H of these K elemental errors are correlated between variables while the rest (K – H) are
uncorrelated.

the random standard uncertainty in a result is estimated by

Number of correlated errors

r
Example

These cover all possible correlations

𝛿12 = 0 𝛿13 = 0 𝛿23 = 1

88
Example

𝛿12 = 0 𝛿13 = 0

# of correlated elemental errors, H

# of independent variables, L=3

# of correlated elemental errors, H = 2

89
 Example

𝜃𝑋 = 𝜃𝑌 = 1
systematic standard uncertainty is estimated as

# of correlated elemental errors

# of independent variables

90
229.4% of uncorrelated
 Effect of correlated errors depends on the
functional relationship
In this case, the correlated systematic errors have a large impact on the systematic
uncertainty. This is not always the case as it depends on the functional relationship
itself. If R = X/Y, then the covariance term in this problem would have little impact on
the systematic standard uncertainty in the result.

𝑋 1 1 𝑋ത 10.1
𝑅= 𝜃𝑋ത = = = 0.082 𝜃𝑌ത = − 2 = − = −0.068
𝑌 𝑌ത 12.2 𝑌ത 12.22

−0.068 0.082 0.0358

0.082
−0.068
0.11 34% of uncorrelated
91
Example:
Uncorrelated errors

Two 2000 Ω resistors

Two 500 Ω resistors

92
Example:
Uncorrelated errors

93
Example:
Uncorrelated errors

94
 Example:
Uncorrelated errors

Case 2 provides the smaller uncertainty at the design-stage. We should proceed using this design.

95
Example:
Correlated errors
 Suppose the resistors are certified by the manufacturer
to have specifications based on a common calibration.
 Assume the errors are correlated systematic errors

96
 Example:
Correlated errors

Assume uncertainty values given are at two standard deviations

The propagation of uncertainty

The expanded uncertainty Uncorr

97
Example:
Correlated errors

Assume uncertainty values given are at two standard deviations

The propagation of uncertainty

The expanded uncertainty Uncorr

98
Correlated Errors- Summary
 If you are aware of correlations incorporate them into
your uncertainty analysis
 The correlation may decrease or increase you uncertainty
(depending on the functional relationship, which affects the
sensitivity indices, 𝜃𝑖 )

 You may be able to decrease uncertainty by creating

correlations (again, based on the functional relationship)
 Use the same voltage measuring device in two measurement
systems (pressure, temperature, LVDT for distance etc.)

99
Monte Carlo simulation for Uncertainty
 Same procedure as finding the PDF of the result of a
functional relationship

 Standard uncertainty in the result is the standard

deviation, 𝑠𝑥 (not the standard error, 𝑠𝑥ҧ )

100
Example

Let’s estimate uncertainty this time…

101
 Sample Results (100K iterations)

Standard (68%)
𝐸 = 100.003 ∓ 10.411 V

Expanded (95%)
( 𝑡𝜈,95 = 1.96)

𝐸 = 100.003 ∓ 20.406 V

102