2/24/2020

Chapter 4

Roundoff and
Truncation Errors

Chapter Objectives
• Understanding the distinction between accuracy
and precision.
• Learning how to quantify error.
• Learning how error estimates can be used to
decide when to terminate an iterative calculation.
• Understanding how roundoff errors occur because
digital computers have a limited ability to represent
numbers.
• Understanding why floating-point numbers have
limits on their range and precision.

1
 2/24/2020

Objectives (cont)
• Recognizing that truncation errors occur when
exact mathematical formulations are represented
by approximations.
• Knowing how to use the Taylor series to estimate
truncation errors.
• Understanding how to write forward, backward, and
centered finite-difference approximations of the first
and second derivatives.
• Recognizing that efforts to minimize truncation
errors can sometimes increase roundoff errors.

Accuracy and Precision

• Accuracy refers to how closely a computed or measured
value agrees with the true value, while precision refers to
how closely individual computed or measured values
agree with each other.
a) inaccurate and imprecise
b) accurate and imprecise
c) inaccurate and precise
d) accurate and precise

2
 2/24/2020

Error Definitions
• True error (Et): the difference between the
true value and the approximation.
• Absolute error (|Et|): the absolute difference
between the true value and the
approximation.
• True fractional relative error: the true error
divided by the true value.
• Relative error (t): the true fractional relative
error expressed as a percentage.

Error Definitions
𝐸𝑡 = true value − approximation

• Et is also called the absolute error and

expressed as an absolute value

• A shortcoming of this definition is that it takes

no account of the order of magnitude of the
value under examination

3
 2/24/2020

• Suppose that you are asked to measure the

volume of a storage drum and a beaker
• The volumes of the storage drum and the
beaker are measured as 49,995 and 245 mL,
respectively
• If the true values are 50,000 and 250 mL,
respectively, the error in both cases is 5 mL

• True fractional relative error

true value − approximation
True fractional relative error =
true value

• True percent relative error

true value − approximation
ε𝑡 = × 100%
true value

4
 2/24/2020

• Going back to the storage drum and beaker

example,

5
• For the storage drum, 𝜀𝑡 = 50000 × 100 = 0.01%

5
• For the beaker, 𝜀𝑡 = 250 × 100 = 2.0%

Error Definitions (cont)

• The previous definitions of error relied on knowing
a true value. If that is not the case, approximations
can be made to the error.
• The approximate percent relative error can be
given as the approximate error divided by the
approximation, expressed as a percentage - though
this presents the challenge of finding the
approximate error!
• For iterative processes, the error can be
approximated as the difference in values between
sucessive iterations.

10

5
 2/24/2020

approximate error
ε𝑎 = × 100%
approximation

• For iterative procedures

present approximation − previous approximation

ε𝑎 = × 100%
present approximation

• t and a maybe +ve or –ve but are more

often represented as absolute values

11

Using Error Estimates

• Often, when performing calculations, we may
not be concerned with the sign of the error
but are interested in whether the absolute
value of the percent relative error is lower
than a pre-specified tolerance s.
• For such cases, the computation is repeated
until |a| < s
• This relationship is referred to as a stopping
criterion.

12

6
 2/24/2020

Using Error Estimates

• It is also convenient to relate these errors to
the number of significant figures in the
approximation.
• It can be shown (Scarborough, 1966) that if
the following criterion is met, we can be
assured that the result is correct to at least 𝑛
significant figures.

𝜀𝑠 = 0.5 × 102−𝑛 %

13

Error Estimates for Iterative

Methods
• Example 4.1: In mathematics, functions can often
be represented by infinite series, e.g., the
exponential function can be computed using

𝑥
𝑥2 𝑥3 𝑥𝑛
𝑒 = 1 +𝑥 + + +⋯+
2 3! 𝑛!

• Thus, as more terms are added in sequence, the

approximation becomes a better and better
estimate of the true value of 𝑒𝑥.

14

7
 2/24/2020

• Starting with the simplest version, 𝑒𝑥 = 1, add

.
terms one at a time in order to estimate 𝑒0 5.
After each new term is added, compute the
true and approximate percent relative errors.
.
• Note that the true value is 𝑒0 5 = 1.648721.
• Add terms until the absolute value of the
approximate error estimate a falls below a
pre-specified error criterion s conforming to
three significant figures.

15

• s = (0.5 × 102−3)% = 0.05%

Terms Result t % a %
1 1 39.3 -
2 1.5 9.02 33.3
3 1.625 1.44 7.69
4 1.645833333 0.175 1.23
5 1.648437500 0.0172 0.158
6 1.648697917 0.00142 0.0158

16

8
 2/24/2020

Computer Algorithm for Iterative

Calculations
function [fx,ea,iter] = IterMeth(x,es,maxit)
% Maclaurin series of exponential function
% [fx,ea,iter] = IterMeth(x,es,maxit)
% input: x = value at which series evaluated
% es = stopping criterion (default = 0.0001)
% maxit = maximum iterations (default = 50)
% output: fx = estimated value
% ea = approximate relative error (%) 𝑛
%
% defaults:
iter = number of iterations
𝑥
𝑥𝑛
if nargin<2|isempty(es),es=0.0001;end 𝑒 ≅෍
if nargin<3|isempty(maxit),maxit=50;end 𝑛!
% initialization 𝑖=0
iter = 1; sol = 1; ea = 100;
% iterative calculation
while (1)
solold = sol;
sol = sol + x ^ iter / factorial(iter);
iter = iter + 1;
if sol~=0
ea=abs((sol - solold)/sol)*100;
end
if ea<=es | iter>=maxit,break,end
end
fx = sol;
end

17

Roundoff Errors
• Roundoff errors arise because digital
computers cannot represent some quantities
exactly.
• They are important to engineering and
scientific problem solving because they can
lead to erroneous results.
• In certain cases, they can actually lead to a
calculation going unstable and yielding
obviously erroneous results.

18

9
 2/24/2020

Roundoff Errors
• There are two major aspects of roundoff
errors involved in numerical calculations:

– Digital computers have size and precision limits

on their ability to represent numbers.
– Certain numerical manipulations are highly
sensitive to roundoff errors.

19

Computer Number Representation

• Roundoff errors are directly related to the manner
in which numbers are stored in a computer
• The fundamental unit whereby information is
represented is called a word.
• This is an entity that consists of a string of binary
digits, or bits.
• Numbers are typically stored in one or more words.

20

10
 2/24/2020

Base 2 system
• The signed magnitude method, employs the
first bit of a word to indicate the sign, with a 0
for positive and a 1 for negative.
• The remaining bits are used to store the
number.
• For example, the integer value of 173 is
represented in binary as 10101101.

10101101 2 = 27 + 25 + 23 + 22 + 20 = 128 + 32 + 8 + 4 + 1 = 173 10

21

Base 2 system
• If such a scheme is employed, there clearly
is a limited range of integers that can be
represented.
• Again assuming a 16-bit word size, if one bit
is used for the sign, the 15 remaining bits
can represent binary integers from 0 to
111111111111111. The upper limit can be
converted to a decimal integer, as in:
1 × 214 + 1 × 213 + ⋯ + 1 × 21 + 1 × 20 = 32,767

22

11
 2/24/2020

Base 2 system

• a 16-bit computer word can store decimal integers

ranging from −32,767 to 32,767.
• because zero is already defined as
0000000000000000, it is redundant to use the
number 1000000000000000 to define a “minus zero”
• Therefore, it is conventionally employed to represent
an additional negative number: −32,768, and the
range is from −32,768 to 32,767.

23

Base 2 system

• For an n-bit word, the range would be

from −2𝑛−1 to (2𝑛−1 − 1).

• Thus, 32-bit integers would range from

−2,147,483,648 to +2,147,483,647.

24

12
 2/24/2020

Implications of Floating-Point
Representation
• Example 4.2: Suppose that we had a
hypothetical base-10 computer with a 5-digit
word size. Assume that one digit is used for
the sign, two for the exponent, and two for
the mantissa. For simplicity, assume that one
of the exponent digits is used for its sign,
leaving single digit for its magnitude.

25

• A general representation of the number

following normalization would be:

𝑠1 . 𝑑1 . 𝑑2 × 10𝑠0 𝑑0

• Largest value = +9.9 x 10+9

• Smallest value = +1.0 x 10-9

26

13
 2/24/2020

• Large positive and negative numbers that fall

outside the range would cause an overflow error.
• In a similar sense, for very small quantities there is
a “hole” at zero, and very small quantities would
usually be converted to zero.
• Recognize that the exponent overwhelmingly
determines these range limitations.
• Whereas the significand plays a minor role in
defining the range, it has a profound effect on
specifying the precision

27

28

14
 2/24/2020

Computer Number Representation

• By default, MATLAB has adopted the IEEE double-
precision format in which eight bytes (64 bits) are
used to represent floating-point numbers:
𝑛 = ± 1 + 𝑓 × 2𝑒
• The sign is determined by a sign bit
• The mantissa 𝑓 is determined by a 52-bit binary
number
• The exponent 𝑒 is determined by an 11-bit binary
number, from which 1023 is subtracted to get 𝑒

29

Floating Point Ranges

• Values of -1023 and +1024 for 𝑒 are reserved for
special meanings, so the exponent range is -1022
to 1023.
• The largest possible number MATLAB can store has
▪ f of all 1’s, giving a significand of 2 - 2-52, or approximately 2
▪ e of 111111111102, giving an exponent of 2046 - 1023 = 1023
▪ This yields approximately 21024 = 1.799710308
• The smallest possible number MATLAB can store with full
precision has
▪ f of all 0’s, giving a significand of 1
▪ e of 000000000012, giving an exponent of 1-1023 = -1022
▪ This yields 2-1022 = 2.225110-308

30

15
 2/24/2020

Floating Point Precision

• The 52 bits for the mantissa 𝑓 correspond to
about 15 to 16 base-10 digits.
• The machine epsilon - the maximum relative
error between a number and MATLAB’s
representation of that number, is thus
2-52 = 2.220410-16

31

Roundoff Errors with

Arithmetic Manipulations
• Roundoff error can happen in several
circumstances other than just storing numbers - for
example:
– Large computations - if a process performs a large
number of computations, roundoff errors may build up to
become significant
– Adding a Large and a Small Number - Since the small
number’s mantissa is shifted to the right to be the same
scale as the large number, digits are lost
– Smearing - Smearing occurs whenever the individual
terms in a summation are larger than the summation
itself.
• (𝑥 + 10-20) - 𝑥 = 10-20 mathematically, but
𝑥 = 1; (𝑥 + 10-20) - x gives a 0 in MATLAB!

32

16
 2/24/2020

Roundoff Errors with

Arithmetic Manipulations
• Subtracting 26.86 from 36.41 • Adding 4000 and 0.0010

0.3641 × 102 0.4000 × 104

−0.2686 × 102 +0.0000001 × 104
0.0955 × 102 0.4000001 × 104
• Since we have a leading • If we are using a
hypothetical 4-digit mantissa
zero in the result, the
and 1-digit exponent, this is
decimal point has to be chopped as:
shifted to the right to give:
0.4000 × 104
0.9550 × 101
• Hence, the result is as if we
• Hence, a non-significant have not even performed an
zero has been added to the addition operation
result

33

Truncation Errors
• Truncation errors are those that result from
using an approximation in place of an exact
mathematical procedure.
• Example 1: approximation to a derivative
using a finite-difference equation:
𝑑𝑣 ∆𝑣 𝑣 𝑡𝑖+1 − 𝑣 𝑡𝑖
≅ =
𝑑𝑡 ∆𝑡 𝑡𝑖+1 − 𝑡𝑖

• Example 2: The Taylor Series

34

17
 2/24/2020

The Taylor Theorem and Series

• The Taylor theorem states that any smooth
function can be approximated as a
polynomial.
• The Taylor series provides a means to
express this idea mathematically.

35

The Taylor Series

f '' (x i ) 2 f (3) (x i ) 3 f (n ) (x i ) n
f (x i+1) = f (x i ) + f (x i )h +
'
h + h + + h + Rn
2! 3! n!

𝑓 (𝑛+1) 𝑥𝑖 𝑛+1
𝑅𝑛 = ℎ
𝑛+1 !

36

18
 2/24/2020

Truncation Error
• In general, the 𝑛th order Taylor series
expansion will be exact for an nth order
polynomial.
• In other cases, the remainder term 𝑅𝑛 is of
the order of ℎ𝑛+1 , meaning:
▪ The more terms are used, the smaller the error,
and
▪ The smaller the spacing, the smaller the error for
a given number of terms.

37

Approximation of a Function with a

Taylor Series Expansion
• Example 4.3: Use Taylor series expansions
with n = 0 to 6 to approximate 𝑓(𝑥) = cos 𝑥 at
𝑥𝑖+1 =  / 3 on the basis of the value of 𝑓(𝑥)
and its derivatives at 𝑥𝑖 =  /4.

• Note that this means that h =  / 3 −  / 4

=  / 12.

38

19
 2/24/2020

• True value:
𝜋 𝜋
𝑓 = cos = 0.5
3 3
• Zero-order Approximation:
𝜋 𝜋 𝜋
𝑓 ≅𝑓 = cos = 0.707106781
3 4 4

0.5 − 0.707106781
𝜀𝑡 = × 100% = 41.4%
0.5

39

• First-order Approximation:
𝜋 𝜋 ′
𝜋
𝑓 ≅𝑓 + 𝑓 4 .ℎ
3 4
𝜋 𝜋 𝜋
= cos − sin = 0.521986659
4 4 12

0.5 − 0.521986659
𝜀𝑡 = × 100% = 4.40%
0.5

40

20
 2/24/2020

Order 𝑛 𝑓 (𝒏) (𝑥) 𝑓(/3) |𝑡 |

0 cos x 0.707106781 41.4

1 -sin x 0.521986659 4.40

2 -cos x 0.497754491 0.449

3 sin x 0.499869147 2.62x10-2

4 cos x 0.500007551 1.51x10-3

5 -sin x 0.500000304 6.08x10-5

6 -cos x 0.499999988 2.44x10-6

41

A Graphical Illustration of Error Rn

The derivative mean-value theorem states that if a function 𝑓(𝑥) and its first derivative are
continuous over an interval from 𝑥𝑖 to 𝑥𝑖+1 , then there exists at least one point on the function
that has a slope, designated by 𝑓′(ξ ), that is parallel to the line joining f (xi) and f (xi+1).

zero order: 𝑅0 = 𝑓 ′  ℎ

𝑓 ′′  2
first-order: 𝑅1 = ℎ
2!

42

21
 2/24/2020

Using the Taylor Series to Estimate

Truncation Errors
The objective of the bungee jumper examples was to predict
velocity as a function of time, 𝑣(𝑡) can be expanded in a Taylor
series: 𝑣 ′′ 𝑡 𝑖
𝑣 𝑡𝑖+1 = 𝑣 𝑡𝑖 + 𝑣 ′ 𝑡𝑖 𝑡𝑖+1 − 𝑡𝑖 + 𝑡𝑖+1 − 𝑡𝑖 2 + ⋯ + 𝑅𝑛
2!
Now let us truncate the series after the first derivative term:
𝑣 𝑡𝑖+1 = 𝑣 𝑡𝑖 + 𝑣 ′ 𝑡𝑖 𝑡𝑖+1 − 𝑡𝑖 + 𝑅1
Solving for 𝑣’(𝑡𝑖):
𝑣 𝑡𝑖+1 − 𝑣 𝑡𝑖 𝑅1
𝑣 ′ 𝑡𝑖 = +
𝑡𝑖+1 − 𝑡𝑖 𝑡𝑖+1 − 𝑡𝑖

First order Truncation

approximation error

43

• Recall that:
𝑓 (𝑛+1)  𝑛+1 𝑣 ′′  2
𝑅𝑛 = ℎ hence 𝑅1 = 𝑡𝑖+1 − 𝑡𝑖
𝑛+1 ! 2!

• We can rearrange to get:

𝑅1 𝑣 ′′ 
= 𝑡𝑖+1 − 𝑡𝑖 ≡ 𝑂 𝑡𝑖+1 − 𝑡𝑖
𝑡𝑖+1 − 𝑡𝑖 2!
• the error of our derivative approximation should be
proportional to the step size.
• Consequently, if we halve the step size, we would
expect to halve the error of the derivative.

44

22
 2/24/2020

Numerical Differentiation
• The first order Taylor series can be used to
calculate approximations to derivatives:
▪ Given: f (x i+1) = f (x i ) + f (x i )h + O(h )
' 2

f (x i+1 ) - f (x i )
▪ Then: f ' (x i ) = + O(h)
h

• This is termed a “forward” difference because
it utilizes data at 𝑖 and 𝑖 + 1 to estimate the
derivative.

45

Differentiation (cont)
• There are also backward and centered difference
approximations, depending on the points used:
• Forward:
f (x i+1 ) - f (x i )
f ' (x i ) = + O(h)
h
• Backward:
f (x i ) - f (x i-1)
 f ' (x i ) = + O(h)
h
• Centered:
f (x i+1 ) - f (x i-1)
f ' (x i ) = + O(h 2 )
 2h
46


23
 2/24/2020

Finite-Difference Approximations of
Derivatives
• Example 4.4:
Use forward and backward difference approximations of
O(h) and a centered difference approximation of O(h2) to
estimate the first derivative of:
𝑓 𝑥 = −0.1𝑥 4 − 0.15𝑥 3 − 0.5𝑥 2 − 0.25𝑥 + 1.2
at x = 0.5 using a step size h = 0.5. Repeat the
computation using h = 0.25. Note that the derivate can be
calculated directly as:
𝑓′ 𝑥 = −0.4𝑥 3 − 0.45𝑥 2 − 1.0𝑥 − 0.25
and can be used to compute the true value as
f ′(0.5)= – 0.9125

47

• At x = 0.5, with a step size of h = 0.5

𝑥𝑖−1 = 0 𝑓(𝑥𝑖−1 ) = 1.2
𝑥𝑖 = 0.5 𝑓(𝑥𝑖 ) = 0.925
𝑥𝑖+1 = 1.0 𝑓(𝑥𝑖+1 ) = 0.2

• We can use these values to find the

numerical derivatives as:
Forward 0.2 − 0.925 -1.45 𝜀𝑡 = 58.9%
Difference
0.5
Backward 0.925 − 1.2 -0.55 𝜀𝑡 = 39.7%
Difference
0.5
Centered 0.2 − 1.2 -1.00 𝜀𝑡 = 9.6%
Difference
1.0

48

24
 2/24/2020

• At x = 0.5, with a step size of h = 0.25

𝑥𝑖−1 = 0.25 𝑓(𝑥𝑖−1 ) = 1.10351563
𝑥𝑖 = 0.50 𝑓(𝑥𝑖 ) = 0.925
𝑥𝑖+1 = 0.75 𝑓(𝑥𝑖+1 ) = 0.63632813

• We can use these values to find the

numerical derivatives as:
0.63632813 − 0.925
Forward Difference -1.155 𝜀𝑡 = 26.5%
0.25
0.925 − 1.10351563
Backward Difference -0.714 𝜀𝑡 = 21.7%
0.25
0.63632813 − 1.10351563
Centered Difference -0.934 𝜀𝑡 = 2.4%
0.5

49

Total Numerical Error

• The total numerical error is the summation of the
truncation and roundoff errors.
• The truncation error generally increases as the step
size increases, while the roundoff error decreases
as the step size increases - this leads to a point of
diminishing returns for step size.

50

25
 2/24/2020

Control of Numerical Errors

• For most engineering applications, there is no exact solution. We
must settle for some estimate of the error in our calculations.
• There are no systematic and general approaches to evaluating
numerical errors for all problems. In many cases error estimates
are based on experience and judgment of the engineers.
• Taylor series is our primary tool for analysis of both truncation and
round-off errors. It is however suitable for small sized problems,
as the amount of work becomes unbearable even for moderate
sized problems.
• Finally, remember, errors are always a major concern in
numerical computations. There is no such a thing “100% exact” or
“100% correct” in numerical computation – every result is
measured by its error estimates.

51

Other Errors
• Blunders - errors caused by malfunctions of
the computer or human imperfection.
• Model errors - errors resulting from
incomplete mathematical models.
• Data uncertainty - errors resulting from the
accuracy and/or precision of the data.

52

26