Bindel, Spring 2012 Intro to Scientific Computing (CS 3220)

Week 1: Monday, Jan 23

Safe computing
If this class were about shooting rather than computing, we’d probably start
by talking about where the safety is. Otherwise, someone would shoot them-
self in the foot. The same is true of basic error analysis: even if it’s not
the most glamorous part of scientific computing, it comes at the start of the
course in order to keep students safe from themselves. For this, a rough-and-
ready discussion of sensitivity analysis, simplified models of rounding, and
exceptional behavior is probably sufficient. We’ll go a bit beyond that, but
you can think of most of the material in the first week in this light: basic
safety training for computing with real numbers.

Some basics
Suppose x̂ is an approximation for x. The absolute error in the approximation
is x̂ − x; the relative error is (x̂ − x)/x. Both are important, but we will use
relative error more frequently for this class.
Most measurements taken from the real world have error. Computations
with approximate inputs generally produce approximate outputs, even when
the computations are done exactly. Thus, we need to understand the propa-
gation of errors through a computation. For example, suppose that we have
two inputs x̂ and ŷ that approximate true values x and y. If ex and δx = ex /x
are the absolute and relative errors for x (and similarly for y), we can write
x̂ = x + ex = x(1 + δx )
ŷ = y + ey = y(1 + δy ).
When we add or subtract x̂ and ŷ, we add and subtract the absolute errors:
x̂ + ŷ = (x + y) + (ex + ey )
x̂ − ŷ = (x − y) + (ex − ey ).
Note, however, that the relative error in the approximation of x − y is (ex −
ey )/(x − y); and if x and y are close, this relative error may be much larger
than δx or δy . This effect is called cancellation, and it is a frequent culprit in
computations gone awry.
Bindel, Spring 2012 Intro to Scientific Computing (CS 3220)

From the perspective of relative error, multiplication and division are

more benign than addition and subtraction. The relative error of a product
is (approximately) the sum of the relative errors of the inputs:
x̂ŷ = xy(1 + δx )(1 + δy )
= xy(1 + δx + δy + δx δy )
≈ xy(1 + δx + δy ).
If δx and δy are small, then δx δy is absolutely miniscule, and so we typically
drop it when performing an error analysis1 Similiarly, using the fact that
(1 + δ)−1 = 1 − δ + O(δ 2 ),
the relative error in a quotient is approximately the difference of the relative
errors of the inputs:
 
x̂ x 1 + δx
=
ŷ y 1 + δy
x
≈ (1 + δx )(1 − δy )
y
x
≈ (1 + δx − δy ).
y
More generally, suppose f is some differentiable function of x, and we
approximate z = f (x) by ẑ = f (x̂). We estimate the absolute error in the
approximation by Taylor expansion:
ẑ − z = f (x̂) − f (x) ≈ f 0 (x)(x̂ − x).
The magnitude |f 0 (x)| tells us the relationship between the size of absolute
errors in the input and the output. Frequently, though, we care about how
relative errors are magnified by the computation:
 0 
ẑ − z xf (x) x̂ − x
≈ .
z z x
The size of the quantity xf 0 (x)/f (x) that tells us how relative errors on
input affect the relative errors of the output is sometimes called the condi-
tion number for the problem. Problems with large condition numbers (“ill-
conditioned” problems) are delicate: a little error in the input can destroy
the accuracy of the output.
1
This is a first-order error analysis or linearized error analysis, since we drop any terms
that involve products of small errors.
Bindel, Spring 2012 Intro to Scientific Computing (CS 3220)

A simple example
What’s the average velocity of the earth as it travels around the sun?
Yes, you can ask Google this question, and it will helpfully tell you that
the answer is about 29.783 km/s. But if you have a few facts at your finger-
tips, you can get a good estimate on your own:
• The earth is t ≈ 500 light-seconds from the sun.
• The speed of light is c ≈ 3 × 108 m/s.
• A year is about y ≈ π × 107 seconds.
So the earth follows a roughly circular orbit with radius of about r = ct ≈
1.5 × 1011 meters, traveling 2πr meters in a year. So our velocity is about
2πct 2π × 1.5 × 1011 m
≈ = 30 km/s.
y π × 107 s
How wrong is the estimate v̂ = 30 km/s? The absolute error compared
to the truth (v = 29.783 km/s a la Wiki answers) is about 217 m/s. Unfor-
tunately, it’s hard to tell without context whether this is a good or a bad
error. Compared to the speed of the earth going around the sun, it’s pretty
small; compared to my walking speed, it’s pretty big. It’s more satisfying to
know the relative error is
v̂ − v
≈ 0.0073,
v
or less than one percent. Informally, we might say that the answer has about
two correct digits.
What is the source of this slightly-less-than-a-percent error? Crudely
speaking, there are two sources:
1. We assumed the earth’s orbit is circular, but it’s actually slightly ellipti-
cal. This modeling error entered before we did any actual calculations.
2. We approximated all the parameters in our calculation: the speed of
light, the distance to the sun in light-seconds, and the length of a year.
More accurate values are:
t = 499.0 s
c = 299792458 m/s
y = 31556925.2 s
Bindel, Spring 2012 Intro to Scientific Computing (CS 3220)

If we plug these values into our earlier formula, we get a velocity of

29.786 km/s. This is approximated by 30 km/s with a relative error of
0.0072.

We’ll put aside for the moment the ellipticity of the earth’s orbit (and
the veracity of Wiki Answers) and focus on the error inherited from the
parameters. Denoting our earlier estimates by t̂, ĉ, and ŷ, we have the
relative errors

δt ≡ (t̂ − t)/t = 20 × 10−4

δc ≡ (ĉ − c)/c = 7 × 10−4
δy ≡ (ŷ − t)/y = −45 × 10−4

According to our error propagation formulas, we should have

 
2πĉt̂ 2πct
≈ (1 + δc + δt − δy ).
ŷ y

In fact, this looks about right:

20 × 10−4 + 7 × 10−4 + 45 × 10−4 = 72 × 10−4 .

An example of error amplification

Our previous example was benign; let us now consider a more sinister exam-
ple. Consider the integrals
Z 1
En = xn ex−1 dx
0

It’s easy to find E0 = 1 − 1/e; and for n > 0, integration by parts2 gives us
Z 1
n x−1 1

En = x e 0
−n xn−1 ex−1 dx = 1 − nEn−1 .
0

We can use the following MATLAB program to compute En for n = 1, . . . , nmax :

2
If you do not remember integration by parts, you should go to your calculus text and
refresh your memory.
Bindel, Spring 2012 Intro to Scientific Computing (CS 3220)

1.5 Eˆn
1/(n + 1)

0.5

0 2 4 6 8 10 12 14 16 18 20
n

Figure 1: Computed values Ên compared to a supposed upper bound.

%
% Compute the integral of xˆn exp(x−1) from 0 to 1
% for n = 1, 2, ..., nmax
%
function E = lec01int(nmax)

En = 1−1/e;
for n = 1:nmax
En = 1−n∗En;
E(n) = En;
end

The computed values of Ên according to our MATLAB program are

shown in Figure 1. A moment’s contemplation tells us that something is
amiss with these problems. Note that 1/e ≤ ex−1 ≤ 1 for 0 ≤ x ≤ 1, so that
1 1
≤ En ≤
e(n + 1) n+1

As shown in the figure, our computed values violate these bounds. What has
gone wrong?
Bindel, Spring 2012 Intro to Scientific Computing (CS 3220)

The integral is perfectly behaved; the problem lies in the algorithm. The
initial value E0 can only be represented approximately on the computer; let
us write the stored value as Ê0 . Even if this were the only error, we would
find the error in later calculations grows terribly rapidly:
Ê0 = E0 + 
Ê1 = 1 − Ê0 = E1 − 
Ê2 = 1 − 2Ê1 = E2 + 2
Ê3 = 1 − 3Ê2 = E3 − 3 · 2
..
.
Ên = En + (−1)n n!.
This is an example of an unstable algorithm.

Sources of error
We understand the world through simplified models. These models are, from
the outset, approximations to reality. To quote the statistician George Box:
All models are wrong. Some models are useful.
While modeling errors are important, we can rarely judge their quality with-
out the context of a specific application. In this class, we will generally take
the model as a given, and analyze errors from the input and the computation.
In addition to error intrinsic in our models, we frequently have error in
input parameters derived from measurements. In ill-conditioned problems,
small relative errors in the inputs can severely compromise the accuracy of
outputs. For example, if the input to a problem is known to within a millionth
of a percent and the condition number for the problem is 107 , the result might
only have one correct digit. This is a difficulty with the problem, not with
the method used to solve the problem. Ideally, a well-designed numerical
routine might warn us when we ask it to solve an ill-conditioned problem,
but asking for an accurate result may be asking too much. Instead, we will
seek stable algorithms that don’t amplify the error undeservedly. That is, if
the condition number for a problem is κ, a stable algorithm should amplify
the input error by at most Cκ, where C is some modest constant that does
not depend on the details of the input.
Bindel, Spring 2012 Intro to Scientific Computing (CS 3220)

In order to produce stable algorithms, we must control the sources of

error from within the computation itself. In the order we encounter them,
these include:

Roundoff error: IEEE floating point arithmetic is essentially binary sci-

entific notation. The number 1/3 cannot be represented exactly as a
finite decimal. It can’t be represented exactly in a finite number of
binary digits, either. We can, however, approximate 1/3 to a very high
degree of accuracy.

Termination of iterations: Nonlinear equations, optimization problems,

and even linear systems are frequently solved by iterations that produce
successively better approximations to a true answer. At some point,
we decide that we have an answer that is good enough, and stop.

Truncation error: We frequently approximate derivatives by finite differ-

ences and integrals by sums. The error in these approximations is
frequently called truncation error.

Stochastic error: Monte Carlo methods use randomness to compute ap-

proximations. The variance due to randomness typically dominates
the error in these methods.