Curve Fitting

ME 537 Numerical Methods for Engineers

University of Gaziantep
Faculty of Engineering
Dr. Mustafa Özakça
Lecture Notes, 1997
 f(x)

x
We want to find the best “fit” of a curve through the data.
 f(x)

x
We want to find the best “fit” of a curve through the data.
Here we see :
a) Least squares fit
 f(x)

We want to find the best “fit” of a curve through the data.

Here we see :
a) Least squares fit
b) Linear interpolation
 f(x)

Can you suggest another?

x
We want to find the best “fit” of a curve through the data.
Here we see :
a) Least squares fit
b) Linear interpolation
 Mathematical Background
• The prerequisite mathematical background
for interpolation is found in the material on
the Taylor series expansion and finite
divided differences
• Simple statistics
average
standard deviation
normal distribution
Normal Distribution

A histogram used
to depict the distributions
of an exam grade.
 x

x  2
95%

x 
68%
 Material to be Covered in Curve
Fitting
• Linear Regression
Polynomial Regression
Multiple Regression
General linear lease squares
Nonlinear regression
• Interpolation
Newton’s Polynomial
Lagrange polynomial
Coefficients of polynomials
 Specific Study Objectives
• Understand the fundamental difference between
regression and interpolation
• Understand the derivation of linear least squares
regression and be able to assess the reliability of
the fit using graphical and quantitative
assessments.
 Specific Study Objectives
• Know how to linearize data by transformation
• Understand situations where polynomial, multiple
and nonlinear regression are appropriate
• Understand the general matrix formulation of
linear least squares
• Understand that there is one and only one
polynomial of degree n or less that passes exactly
through n+1 points
 Specific Study Objectives
• Become familiar with the use of Newton’s and
Lagrange formulas for interpolating polynomials
• Recognize that the Newton and Lagrange
equations are merely different formulations of the
same interpolating polynomial
• Realize that more accurate results are obtained if
data used for interpolation is centered around and
close to the unknown point
 Specific Study Objectives
• Recognize the liabilities and risks associated with
extrapolation
• Understand why spline functions have utility for
data with local areas of abrupt change
 Least Squares Regression
• Simplest is fitting a straight line to a set of
paired observations
(x1,y1), (x2, y2).....(xn, yn)
• The resulting mathematical expression is
y = ao + a1x + e
• We will consider the error introduced at
each data point to develop a strategy for
determining the “best fit” equations
 n n 2

Sr   e2i    yi  a o  a1xi 
i 1 i 1

f(x)

x
 n n 2

Sr   e2i    yi  a o  a1xi 
i 1 i 1

f(x)

x
 n n 2

Sr   e2i    yi  a o  a1xi 
i 1 i 1

f(x)

yi  a o  a1xi

x
 n n 2

Sr   e2i    yi  a o  a1xi 
i 1 i 1

To determine the values for ao and a1, differentiate

with respect to each coefficient

Sr
 2  yi  ao  a1xi 
ao
Sr
 2  yi  ao  a1xi xi 
a1
Note: we have simplified the summation symbols.
What mathematics technique will minimize Sr?
Sr
 2  yi  ao  a1xi 
a o
Sr
 2  yi  ao  a1xi xi 
a1

Setting the derivative equal to zero will minimizing Sr.

If this is done, the equations can be expressed as:
0   yi   ao   a1xi
0   yi xi   ao xi   a1xi2
0   yi   ao   a1xi
0   yi xi   ao xi   a1xi2

If you recognize that: a o  nao

you have two equations, with two simultaneous equations

with two unknowns, ao and a1.

What are these equations? (hint: only place terms with

ao and a1 on the LHS of the equations)

What are the final equations for ao and a1?

nao   xi a1   yi These first two
equations are called
the normal equations
 i o  i a1   xi yi
x a  x 2

n xi yi   xi  y i
a1 
n x   xi 
2 2
i

ao  y  a1x
 Error
Recall:
n n 2
f(x)
Sr   e i    yi  a o  a1xi 
2

i 1 i 1

The most common measure of the “spread” of a sample is the

standard deviation about the mean:
S t    yi  y 
2

St
sy 
n 1
Introduce a term to measure the standard error of the estimate:

Sr
sy 
x n2

Coefficient of determination r2:

St  Sr
2
r 
St

r is the correlation coefficient

 2 St  Sr
r 
St

The following signifies that the line explains 100 percent

of the variability of the data:

Sr = 0
r = r2 = 1

If r = r2 = 0, then Sr = St and the fit is invalid.

Consider the following four sets of data

Data 1 Data 2 Data 3 Data 4

10 8.04 10 9.14 10 7.46 8 6.58
8 6.95 8 8.14 8 6.77 8 5.76
13 7.58 13 8.74 13 12.74 8 7.71
9 8.81 9 8.77 9 7.11 8 8.84
11 8.33 11 9.26 11 7.81 8 8.47
14 9.96 14 8.10 14 8.84 8 7.04
6 7.24 6 6.13 6 6.08 8 5.25
4 4.26 4 3.10 4 5.39 19 12.50
12 10.84 12 9.13 12 8.15 8 5.56
7 4.82 7 7.26 7 6.42 8 7.91
5 5.68 5 4.74 5 5.73 8 6.89
 Data Set 1 Data Set 2

12.00 10.00
10.00 8.00
8.00
6.00

f(x)
f(x)

6.00
4.00
4.00
2.00 2.00

0.00 0.00
0 5 10 15 0 5 10 15
x x

Data Set 3 Data Set 4

14.00 14.00
12.00 12.00
10.00 10.00
8.00 8.00
f(x)

f(x)
6.00 6.00
4.00 4.00
2.00 2.00
0.00 0.00
0 5 10 15 0 5 10 15 20
x x

GRAPHS OF FOUR DATA SETS

 Data Set 1 Data Set 2
Linear Regression Data Set 1
12.00
0.50 3.00 10.00
10.00 0.117906 1.124747 8.00
8.00 0.666542 1.236603
6.00
17.98994 9

f(x)
f(x)

6.00
4.00
4.00
2.00 2.00

0.00 0.00
0 5 10 15 0 5 10 15
x x

slope = 0.5
Results from Excel linest function
Data Set 3 Data Set 4

14.00 14.00
12.00 12.00
10.00 10.00
8.00 8.00
f(x)

f(x)
6.00 6.00
4.00 4.00
2.00 2.00
0.00 0.00
0 5 10 15 0 5 10 15 20
x x

GRAPHS OF FOUR DATA SETS

 Data Set 1 Data Set 2
Linear Regression Data Set 1
12.00
0.50 3.00 10.00
10.00 0.117906 1.124747 8.00
8.00 0.666542 1.236603
6.00
17.98994 9

f(x)
f(x)

6.00
4.00
4.00
2.00 2.00

0.00 0.00
0 5 10 15 0 5 10 15
x x

y - intercept = 3.00
Results from Excel linest function
Data Set 3 Data Set 4

14.00 14.00
12.00 12.00
10.00 10.00
8.00 8.00
f(x)

f(x)
6.00 6.00
4.00 4.00
2.00 2.00
0.00 0.00
0 5 10 15 0 5 10 15 20
x x

GRAPHS OF FOUR DATA SETS

 Data Set 1 Data Set 2
Linear Regression Data Set 1
12.00
0.50 3.00 10.00
10.00 0.117906 1.124747 8.00
8.00 0.666542 1.236603
6.00
17.98994 9

f(x)
f(x)

6.00
4.00
4.00
2.00 2.00

0.00 0.00
0 5 10 15 0 5 10 15
x x

R2 = 0.67 Results from Excel linest function

Data Set 3 Data Set 4

14.00 14.00
12.00 12.00
10.00 10.00
8.00 8.00
f(x)

f(x)
6.00 6.00
4.00 4.00
2.00 2.00
0.00 0.00
0 5 10 15 0 5 10 15 20
x x

GRAPHS OF FOUR DATA SETS

 Data Set 1 Data Set 2 Linear Regression Data Set 2
Linear Regression Data Set 1
0.50 3.00
12.00
0.50 3.00 10.00 0.117964 1.125302
10.00 0.117906 1.124747 8.00 0.666242 1.237214
8.00 0.666542 1.236603
6.00 17.96565 9
17.98994 9

f(x)
f(x)

6.00
4.00
4.00
2.00 2.00

0.00 0.00
0 5 10 15 0 5 10 15
x x

All equations are y = 0.5x + 3 R2 = 0.67

Data Set 3 Data Set 4 Linear Regression Data Set 4
Linear Regression Data Set 3
0.50 3.00
14.00 0.50 3.00 14.00 0.117819 1.123921
12.00 0.117878 1.124481 12.00
10.00 0.666707 1.235695
0.666324 1.236311 10.00
8.00 8.00 18.00329 9
17.97228 9
f(x)

f(x)
6.00 6.00
4.00 4.00
2.00 2.00
0.00 0.00
0 5 10 15 0 5 10 15 20
x x

GRAPHS OF FOUR DATA SETS

 Data Set 1 Data Set 2 Linear Regression Data Set 2
Linear Regression Data Set 1
0.50 3.00
12.00
0.50 3.00 10.00 0.117964 1.125302
10.00 0.117906 1.124747 8.00 0.666242 1.237214
8.00 0.666542 1.236603
6.00 17.96565 9
17.98994 9

f(x)
f(x)

6.00
4.00
4.00
2.00 2.00

0.00 0.00
0 5 10 15 0 5 10 15
x x

All equations are y = 0.5x + 3 R2 = 0.67

Data Set 3 Data Set 4 Linear Regression Data Set 4
Linear Regression Data Set 3
0.50 3.00
14.00 0.50 3.00 14.00 0.117819 1.123921
12.00 0.117878 1.124481 12.00
10.00 0.666707 1.235695
0.666324 1.236311 10.00
8.00 8.00 18.00329 9
17.97228 9
f(x)

f(x)
6.00 6.00
4.00 4.00
2.00 2.00
0.00 0.00
0 5 10 15 0 5 10 15 20
x x

GRAPHS OF FOUR DATA SETS

 Data Set 1 Data Set 2 Linear Regression Data Set 2
Linear Regression Data Set 1
0.50 3.00
12.00
0.50 3.00 10.00 0.117964 1.125302
10.00 0.117906 1.124747 8.00 0.666242 1.237214
8.00 0.666542 1.236603
6.00 17.96565 9
17.98994 9

f(x)
f(x)

6.00
4.00
4.00
2.00 2.00

0.00 0.00
0 5 10 15 0 5 10 15
x x

All equations are y = 0.5x + 3 R2 = 0.67

Data Set 3 Data Set 4 Linear Regression Data Set 4
Linear Regression Data Set 3
0.50 3.00
14.00 0.50 3.00 14.00 0.117819 1.123921
12.00 0.117878 1.124481 12.00
10.00 0.666707 1.235695
0.666324 1.236311 10.00
8.00 8.00 18.00329 9
17.97228 9
f(x)

f(x)
6.00 6.00
4.00 4.00
2.00 2.00
0.00 0.00
0 5 10 15 0 5 10 15 20
x x

GRAPHS OF FOUR DATA SETS

 Data Set 1 Data Set 2 Linear Regression Data Set 2
Linear Regression Data Set 1
0.50 3.00
12.00
0.50 3.00 10.00 0.117964 1.125302
10.00 0.117906 1.124747 8.00 0.666242 1.237214
8.00 0.666542 1.236603
6.00 17.96565 9
17.98994 9

f(x)
f(x)

6.00
4.00
4.00
2.00 2.00

0.00 0.00
0 5 10 15 0 5 10 15
x x

All equations are y = 0.5x + 3 R2 = 0.67

Data Set 3 Data Set 4 Linear Regression Data Set 4
Linear Regression Data Set 3
0.50 3.00
14.00 0.50 3.00 14.00 0.117819 1.123921
12.00 0.117878 1.124481 12.00
10.00 0.666707 1.235695
0.666324 1.236311 10.00
8.00 8.00 18.00329 9
17.97228 9
f(x)

f(x)
6.00 6.00
4.00 4.00
2.00 2.00
0.00 0.00
0 5 10 15 0 5 10 15 20
x x

GRAPHS OF FOUR DATA SETS

 Linearization of non-linear
relationships

Some data is simply ill-suited for f(x)

linear least squares regression....

or so it appears.

x
P EXPONENTIAL
EQUATIONS

P  Po e rt

Linearize
t
ln P

intercept = ln P0

slope = r
why?
t
 rt
Can you see the similarity
P  P0e with the equation for a line:
ln P  ln P0e rt 
y = b + mx
 ln P0   lne rt

 ln P0   rt where b is the y-intercept
and m is the slope?
lnP

intercept = ln Po slope = r

t
 P  P0e rt After taking the natural log
ln P  ln P0e rt  of the y-data, perform linear
regression.
 ln P0   lne rt  From this regression:
The value of b will give us
 ln P0   rt ln (P0). Hence, P0 = eb

The value of m will give us r

ln P0
directly.

intercept = ln P0 slope = r

t
 Q POWER EQUATIONS

Q  cH a
(Flow over a weir)

log Q H

Here we linearize
the equation by
taking the log of
H and Q data.
log H
What is the resulting
intercept and slope?
Q  cH a
log Q  logcH a 
 log c  log H a
 log c  a log H

log Q

slope = a

log H

intercept = log c
 Q  cH a So how do we get
log Q  logcH a  c and a from
performing regression
 log c  log H a on the log H vs log Q
 log c  a log H data?
From : y = mx + b

log Q intercept = b = log c c = 10b

slope = m = a
slope = a

log H

intercept = log c
 SATURATION-GROWTH
m
RATE EQUATION

S
   max
Ks  S

1/m S
Here, m is the growth rate
of a microbial population,
mmax is the maximum
growth rate, S is the
slope = Ks/mmax substrate or food
concentration, Ks is the
intercept = 1/mmax substrate concentration at a
value of m = mmax/2
1/ S
 General Comments of Linear
Regression
• You should be cognizant of the fact that
there are theoretical aspects of regression
that are of practical importance but are
beyond the scope of this book
• Statistical assumptions are inherent in the
linear least squares procedure
 General Comments of Linear
Regression
• x has a fixed value; it is not random and is
measured without error
• The y values are independent random
variable and all have the same variance
• The y values for a given x must be normally
distributed
 General Comments of Linear
Regression
• The regression of y versus x is not the same
as x versus y
• The error of y versus x is not the same as x
versus y
 General Comments of Linear
Regression
• The regression of y versus x is not the same
as x versus y
• The error of y versus x is not the same as x
versus y f(x)

x
 General Comments of Linear
Regression
• The regression of y versus x is not the same
as x versus y
• The error of y versus x is not the same as x
versus y f(x)

What error is associated with using this line to

model data if the regression analysis
included this point?

x
 General Comments of Linear
Regression
• The regression of y versus x is not the same
as x versus y
• The error of y versus x is not the same as x
versus y f(x)

x-direction

x
 General Comments of Linear
Regression
• The regression of y versus x is not the same
as x versus y
• The error of y versus x is not the same as x
versus y f(x)
y-direction

x-direction

x
 Polynomial Regression
• One of the reasons you were presented with
the theory behind linear regression was to
allow you the insight behind similar
procedures for higher order polynomials
• y = a 0 + a 1x
• mth - degree polynomial
y = a0 + a1x + a2x2 +....amxm + e
Based on the sum of the squares
of the residuals
Sr    yi  a o  a1xi  a x ...... a x
2 i
2
m i
m 2

 Based on the sum of the squares
of the residuals
Sr    yi  a o  a1xi  a x ...... a x
2 i
2
m i 
m 2

1. Take the derivative of the above equation with respect to

each of the unknown coefficients: i.e. the partial with respect
to a2

Sr
 2 xi  yi  ao  a1xi  a2 xi2 ..... amxim 
2

a 2
2. These equations are set to zero to minimize Sr., i.e.
minimize the error.
2. These equations are set to zero to minimize Sr., i.e.
minimize the error.

3. Set all unknowns values on the LHS of the equation.

Again, using the partial of Sr. wrt a2

ao  xi  a1  xi3  a2  xi4 ..... am  xi   xi yi

2 m 2 2
2. These equations are set to zero to minimize Sr., i.e.
minimize the error.

3. Set all unknowns values on the LHS of the equation.

Again, using the partial of Sr. wrt a2

ao  xi  a1  xi3  a2  xi4 ..... am  xi   xi yi

2 m 2 2

4. This set of normal equations result in m+1 simultaneous

equations which can be solved using matrix methods to
determine a0, a1, a2......am
 Multiple Linear Regression
• A useful extension of linear regression is
the case where y is a linear function of two
or more variables
y = ao + a1x1 + a2x2
• We follow the same procedure
y = ao + a1x1 + a2x2 + e
 Multiple Linear Regression
For two variables, we would solve a 3 x 3 matrix
in the following form:
 n

x 1i x 2i   a 0    yi 
   
  x1i x x x
1i 2 i   a1     x1i yi 
2
1i

 x2 i    x y
x x
1i 2 i x 2
2 i  a 2 
  2 i i 

[A] and {c}are clearly based on data given for x1, x2 and y
to solve for the unknowns in {x}.
 Interpolation
• General formula for an n-th order
polynomial
y = a0 + a1x + a2x2 +....amxm
• For m+1 data points, there is one, and only
one polynomial of order m or less that
passes through all points
• Example: y = a0 + a1x
– fits between 2 points
– 1st order
 Interpolation
• We will explore two mathematical methods
well suited for computer implementation
• Newton’s Divided Difference Interpolating
Polynomials
• Lagrange Interpolating Polynomial
 Newton’s Divided Difference
Interpolating Polynomials
• Linear Interpolation
• Quadratic Interpolation
• General Form
• Errors
 Linear Interpolation
3
Temperature, C Density, kg/m
0 999.9
5 1000.0
10 999.7
15 999.1
20 998.2

How would you approach estimating the density at 17 C?

 3
Temperature, C Density, kg/m
0 999.9
5 1000.0
10 999.7
15 999.1
20 998.2 ???
999.1 > r > 998.2
r
999.1
998.2

T
15 20
 r

999.1
998.2

T
15 20

998.2  999 .1  Assume a straight line

 between the known data.
20  15 T
Then calculate the slope.
 r

999.1
998.2

17 T
15 20

Assuming this linear

998.2    relationship is constant,

20  17 T the slope is the same
between the unknown
point and a known point.
 r

999.1
998.2

17 T
15 20

998.2  999 .1 998.2   

  Solve for r
20  15 20  17 T

Therefore, the slope of

one interval will equal the
slope of the other interval.
998.2  999 .1 998.2  

20  15 20  17
998.2  999 .1 998.2   Formula in text follows

20  15 20  17 from this analysis.

f x1   f  x0 
f1  x   f  xo   x  x0 
x1  x0
998.2  999 .1 998.2  

20  15 20  17

f x1   f  x0 
f1  x   f  xo   x  x0 
x1  x0

Note: The notation f1(x) designates that this is a first order

interpolating polynomial
 true solution 1

f(x)

x
 true solution 1
2
f(x)

smaller intervals
provide a better estimate

x
 true solution 1
2
f(x)

Alternative approach would be to include

a third point and estimate f(x) from a 2nd
order polynomial.

x
 true solution
f(x)

Alternative approach would be to include

a third point and estimate f(x) from a 2nd
order polynomial.

x
 Quadratic Interpolation

f 2 x  b0  b1 x  x0   b2  x  x0  x  x1 

Prove that this a 2nd order polynomial of

the form:
f x   a0  a1x  a2 x 2
First, multiply the terms

f 2 x  b0  b1  x  x0   b2  x  x0  x  x1 
First, multiply the terms

f 2 x  b0  b1  x  x0   b2  x  x0  x  x1 

f 2  x  b0  b1x  b1x0  b2 x 2  b2 x0 x1  b2 xx0  b2 xx1

First, multiply the terms

f 2 x  b0  b1  x  x0   b2  x  x0  x  x1 

f 2  x  b0  b1x  b1x0  b2 x 2  b2 x0 x1  b2 xx0  b2 xx1

Collect terms and recognize that:

a0  b0  b1 x0  b2 x0 x1
f  x  a0  a1x  a2 x 2
a1  b1  b2 x0  b2 x1
a2  b2
 x, f(x) x2, f(x2)

f(x)
Procedure for
x1, f(x1)
Quadratic
x0, f(x0)
Interpolation
x
b0  f  x0 
f  x1   f  x0 
b1 
x1  x0
f  x2   f  x1  f  x1   f  x0 

x2  x1 x1  x0
b2 
x2  x0
 Procedure for Quadratic
Interpolation
b0  f  x0 
f  x1   f  x0 
b1 
x1  x0
f  x2   f  x1  f  x1   f  x0 

x2  x1 x1  x0
b2 
x2  x0
 Procedure for Quadratic
Interpolation
b0  f  x0 
f  x1   f  x0 
b1 
x1  x0
f  x2   f  x1  f  x1   f  x0 

x2  x1 x1  x0
b2 
x2  x0

f 2  x   b0  b1  x  x0   b2  x  x0  x  x1 
 Example
1000 Include 10 degrees in
999.5 your calculation of the
density at 17 degrees.
Density

999

998.5
3
998
Temperature, C Density, kg/m
0 5 10 15 20 0 999.9
Tem p
5 1000.0
10 999.7
15 999.1
20 998.2
 Example
b0  f  x0 

b1 
f  x1   f  x0  Include 10 degrees in
x1  x0
f  x2   f  x1  f  x1   f  x0 
your calculation of the
b2 
x2  x1

x1  x0 density at 17 degrees.
x 2  x0

f 2  x   b0  b1  x  x0   b2  x  x0  x  x1 
3
Temperature, C Density, kg/m
0 999.9
5 1000.0
10 999.7
15 999.1
20 998.2
b0  f  x0   999.7
f  x1   f  x0  999.1  999.7
b1    012
.
x1  x0 15  10
f  x2   f  x1  f  x1   f  x0 

x2  x1 x1  x0
b2 
x2  x0
998.2  999.1
  012
. 
 20  15  0.006
20  10

f 2 17  998.776 ....end of problem

f 1 17  998.74
 General Form of Newton’s
Interpolating Polynomials
for the nth-order polynomial
f n  x   b0  b1  x  x0 ....bn  x  x0  x  x1   x  x n  1 

To establish a methodical approach to a solution define

the first finite divided difference as:
f  xi   f  x j 
 
f xi , x j 
xi  x j
 f  xi   f  x j 
 
f xi , x j 
xi  x j

if we let i=1 and j=0 then this is b1

f  x1   f  x0 
b1 
x1  x0

Similarly, we can define the second finite divided difference,

which expresses both b2 and the difference of the first two
divided difference
Similarly, we can define the second finite divided difference,
which expresses both b2 and the difference of the first two
divided difference

f  x2   f  x1  f  x1   f  x0 

x2  x1 x1  x0
b2 
x2  x0


f xi , x j , xk  
  
f xi , x j  f x j , xk 
xi  xk

Following the same scheme, the third divided difference is

the difference of two second finite divided difference.
This leads to a scheme that can easily lead to the
use of spreadsheets

i xi f(xi) first

0 x0 f(x0) f[x1,x0]

1 x1 f(x1)

2 x2 f(x2)

3 x3 f(x3)
This leads to a scheme that can easily lead to the
use of spreadsheets

i xi f(xi) first

0 x0 f(x0) f[x1,x0]

1 x1 f(x1) f[x2,x1]

2 x2 f(x2)

3 x3 f(x3)
This leads to a scheme that can easily lead to the
use of spreadsheets

i xi f(xi) first

0 x0 f(x0) f[x1,x0]

1 x1 f(x1) f[x2,x1]

2 x2 f(x2) f[x2,x3]

3 x3 f(x3)
This leads to a scheme that can easily lead to the
use of spreadsheets

i xi f(xi) first second

0 x0 f(x0) f[x1,x0] f[x2,x1,x0]

1 x1 f(x1) f[x2,x1]

2 x2 f(x2) f[x2,x3]

3 x3 f(x3)
This leads to a scheme that can easily lead to the
use of spreadsheets

i xi f(xi) first second

0 x0 f(x0) f[x1,x0] f[x2,x1,x0]

1 x1 f(x1) f[x2,x1] f[x3,x2,x0]

2 x2 f(x2) f[x2,x3]

3 x3 f(x3)
This leads to a scheme that can easily lead to the
use of spreadsheets

i xi f(xi) first second third

0 x0 f(x0) f[x1,x0] f[x2,x1,x0] f[x3,x2,x1,x0]

1 x1 f(x1) f[x2,x1] f[x3,x2,x0]

2 x2 f(x2) f[x2,x3]

3 x3 f(x3)
f n  x   b0  b1  x  x0  ....bn  x  x0  x  x1   x  x n  1 

These difference can be used to evaluate the b-coefficient s.

The result is the following interpolation polynomial called

the Newton’s Divided Difference Interpolating Polynomial

f n  x   f  x0   f  x1 , x0  x  x0 ....
 f  xn , x n  1 , , x0  x  x0  x  x1  x  xn  1 

To determine the error we need an extra point.

The error would follow a relationship analogous to the error
in the Taylor Series.
 Lagrange Interpolating
Polynomial
n
f n  x   Li x f  xi 
i0

n x  xj
Li  x  
j0 xi  x j
ji

where P designates the “product of”

The linear version of this expression is at n=1
Linear version: n=1
n
f n x   Li x f xi 
i 0

n
x  xj
Li x  
j 0 xi  x j
j i

x  x1 x  x0
f1  f  x0   f x1 
x0  x1 x1  x0

Your text shows you how to do n=2 (second order).

What would third order be?
 n
f n  x    Li  x  f  xi 
i0
n x  xj
Li  x   
j 0 xi  x j
j i

 x  x1  x  x2  x  x3 
f3  f  x0 
 x0  x1  x0  x2  x0  x3 
.......
 n
f n  x    Li  x  f  xi 
i0
n x  xj
Li  x   
j 0 xi  x j
j i
Note:
 x  x1  x  x2  x  x3  x0 is
f3  f  x0 
 x0  x1  x0  x2  x0  x3  not being subtracted
....... from the constant
term x
 n
f n  x    Li  x  f  xi 
i0
n x  xj
Li  x   
j 0 xi  x j
j i
Note:
 x  x1  x  x2  x  x3  x0 is
f3  f  x0 
 x0  x1  x0  x2  x0  x3  not being subtracted
....... from the constant
term x
 n
f n  x    Li  x  f  xi 
i0
n x  xj
Li  x   
j 0 xi  x j
j i
Note:
 x  x1  x  x2  x  x3  x0 is
f3  f  x0 
 x0  x1  x0  x2  x0  x3  not being subtracted
....... from the constant
term x
 n
f n  x    Li  x  f  xi 
i0
n x  xj
Li  x   
j 0 xi  x j
j i
Note:
 x  x1  x  x2  x  x3  x0 is
f3  f  x0 
 x0  x1  x0  x2  x0  x3  not being subtracted
....... from the constant
term x
or xi = x0 in
the numerator
or the denominator
j= 0
 n
f n  x    Li  x  f  xi 
i0
n x  xj
Li  x   
j 0 xi  x j
j i
Note:
 x  x1  x  x2  x  x3  x0 is
f3  f  x0 
 x0  x1  x0  x2  x0  x3  not being subtracted
....... from the constant
term x
or xi = x0 in
the numerator
or the denominator
j= 0
 n
f n  x    Li  x  f  xi 
i0
n x  xj
Li  x   
j 0 xi  x j
j i
Note:
 x  x1  x  x2  x  x3  x0 is
f3  f  x0 
 x0  x1  x0  x2  x0  x3  not being subtracted
....... from the constant
term x
or xi = x0 in
the numerator
or the denominator
j= 0
 n
f n  x    Li  x  f  xi 
i 0
n x  xj
Li  x   
j 0 xi  x j
j i

 x  x1  x  x2  x  x3 
f3  f  x0 
 x0  x1  x0  x2  x0  x3 
 x  x0  x  x2  x  x3 
 f  x1 
 x1  x0  x1  x2  x1  x3 
.......
 n
f n  x    Li  x  f  xi 
i 0
n x  xj
Li  x   
j 0 xi  x j
j i

 x  x1  x  x2  x  x3 
f3  f  x0 
 x0  x1  x0  x2  x0  x3  Note:
 x  x0  x  x2  x  x3  x1 is
 f  x1  not being subtracted
 x1  x0  x1  x2  x1  x3 
from the constant
....... term x
or xi = x1 in
the numerator
or the denominator
j= 1
 n
f n  x    Li  x  f  xi 
i 0
n x  xj
Li  x   
j 0 xi  x j
j i

 x  x1  x  x2  x  x3 
f3  f  x0 
 x0  x1  x0  x2  x0  x3 
 x  x0  x  x2  x  x3  Note:
 f  x1 
 x1  x0  x1  x2  x1  x3  x2 is
 x  x0  x  x1  x  x3  not being subtracted
 f  x2  from the constant
 x2  x0  x2  x1  x2  x3 
term x or xi = x2 in
......
the numerator
or the denominator
j= 2
 n
f n x   Li x f  xi 
i 0
n
x  xj
Li x  
j 0 xi  x j
j i

f3 
 x  x1 x  x2 x  x3 
f  x0 
x0  x1 x0  x2 x0  x3 

 x  x0  x  x2  x  x3 
f  x1  Note:
x1  x0 x1  x2 x1  x3  x3 is


 x  x0  x  x1  x  x3 
f  x2 
not being subtracted
x2  x0 x2  x1 x2  x3  from the constant
term x or xi = x3 in

 x  x0 x  x1  x  x2 
f  x3  the numerator
x3  x0 x3  x1 x3  x2  or the denominator
j= 3
 Example
1000 Determine the density
999.5 at 17 degrees.
Density

999

998.5
3
998
Temperature, C Density, kg/m
0 5 10 15 20 0 999.9
Tem p
5 1000.0
10 999.7
15 999.1
20 998.2
 n
Temperature, C Density, kg/m
3
f n  x   Li  x f  xi 
0 999.9 i0
5 1000.0
10 999.7
15 999.1
n x  xj
Li  x  
20 998.2

j0 xi  x j
ji

17  1517  20

f 2 17  999.7
10  1510  20
17  1017  20
 999.1
15  1015  20
17  1017  15
 998.2
20  1020  15
 1000

999.5
Density

999

998.5

998
0 5 10 15 20
Tem p

f 2 17  119.964  839.244  279.496  998.776

....end of problem
f 2 17  119.964  839.244  279.496  998.776

f 2 17  998.776
Using Newton’s
f 1 17  998.74 Interpolating Polynomial

In fact, you can derive Lagrange directly from

Newton’s Interpolating Polynomial
 Coefficients of an Interpolating
Polynomial
f n  x   b0  b1  x  x0 ....bn  x  x0  x  x1   x  x n  1 

y = a0 + a1x + a2x2 +....amxm

HOW CAN WE BE MORE STRAIGHT

FORWARD IN GETTING VALUES?
f x0   a0  a1 x0  a2 x02
f x1   a0  a1 x1  a2 x12
f x2   a0  a1 x2  a2 x22

This is a 2nd order polynomial.

We need three data points.

Plug the value of xi and f(xi)

directly into equations.

This gives three simultaneous equations

to solve for a0 , a1 , and a2
 Example
1000 Determine the density
999.5 at 17 degrees.
Density

999

998.5
3
998
Temperature, C Density, kg/m
0 5 10 15 20 0 999.9
Tem p
5 1000.0
10 999.7
15 999.1
20 998.2
 3
Temperature, C Density, kg/m
0 999.9
5 1000.0
10 999.7
15 999.1
20 998.2

1 10 10 2  a0  999.7 
 2    
 1 15 15 a
 1    999.1 
1 20 20 2  a2  998.2
a0   1000 
   
a 
 1  0.03 
a   0.006
 2  
17  1000  0.0317   0.00617 
2

 998.78
 Spline Interpolation
• Our previous approach was to derive an nth
order polynomial for n+1 data points.
• An alternative approach is to apply lower-
order polynomials to subset of data points
• Such connecting polynomials are called
spline functions
• Adaptation of drafting techniques
Spline interpolation is an adaptation of the
drafting technique of using a spline to draw smooth curves
through a series of points
Spline interpolation is an adaptation of the
drafting technique of using a spline to draw smooth curves
through a series of points
Spline interpolation is an adaptation of the
drafting technique of using a spline to draw smooth curves
through a series of points
Spline interpolation is an adaptation of the
drafting technique of using a spline to draw smooth curves
through a series of points
 Linear Splines

f  x  f  x0   m0 x  x0  x0  x  x1
f  x  f  x1   m1  x  x1  x1  x  x2

f  x  f  xn  1   mn  1  x  xn  1  xn  1  x  x n
where
f xi  1   f  xi 
mi 
xi  1  xi
Quadratic Spline
 Example
A well pumping at 250 gallons per minute has observation
wells located at 15, 42, 128, 317 and 433 ft away
along a straight line from the well.
After three hours of pumping, the following drawdowns
in the five wells were observed: 14.6, 10.7, 4.8
1.7 and 0.3 ft respectively. 16

Derive equations of 14

each quadratic spline.

12
10

Drawdown
8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
 15 14.6
42 10.7
128 4.8
317 1.7
433 0.3

16
14

12
10
Drawdown

8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
 15 14.6
42 10.7
128 4.8
317 1.7 a1 x 2  b1 x  c1  f x 
433 0.3

16
14

12
10
Drawdown

8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
 15 14.6
For all values of x between 15 and 42 we
42 10.7 want this equation
128 4.8
317 1.7 a1 x 2  b1 x  c1  f x 
433 0.3

16
14

12
10
Drawdown

8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
 15 14.6
For all values of x between 42 and 128 we
42 10.7 want this equation
128 4.8
317 1.7
433 0.3

16
14

12
10
Drawdown

8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
 15 14.6
For all values of x between 42 and 128 we
42 10.7 want this equation
128 4.8
317 1.7 a2 x 2  b2 x  c2  f x 
433 0.3

16
14

12
10
Drawdown

8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
 15 14.6
For all values of x between 128 and 317 we
42 10.7 want this equation
128 4.8
317 1.7
433 0.3

16
14

12
10
Drawdown

8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
 15 14.6
For all values of x between 128 and 317 we
42 10.7 want this equation
128 4.8
317 1.7 a3 x 2  b3 x  c3  f x 
433 0.3

16
14

12
10
Drawdown

8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
 15 14.6
For all values of x between 317 and 433 we
42 10.7 want this equation
128 4.8
317 1.7 a4 x 2  b4 x  c4  f x 
433 0.3

16
14

12
10
Drawdown

8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
 12 unknowns…….we need 12 equations
16
14
a1 x 2  b1 x  c1  f x 
12
Drawdown

10
a2 x 2  b2 x  c2  f x 
8
6
4
a3 x 2  b3 x  c3  f x 
2 a4 x 2  b4 x  c4  f x 
0
0 100 200 300 400 500
Distance from we ll
 15 14.6
42 10.7
128 4.8
317 1.7
433 0.3

(42)2 a2 + 42 b2 + c2 = 10.7
16
14

12
10
Drawdown

8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
 15 14.6
42 10.7
128 4.8
317 1.7
433 0.3

(42)2 a2 + 42 b2 + c2 = 10.7
16
14

12
16,384 a3 + 128 b3 + c3 = 4.8
10
Drawdown

8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
 15 14.6
42 10.7
128 4.8
317 1.7
433 0.3

(42)2 a2 + 42 b2 + c2 = 10.7
16
14

12
16,384 a3 + 128 b3 + c3 = 4.8
10
Drawdown

8
6 100,489a4 + 317b4 + c4 = 1.7
4

2
0
0 100 200 300 400 500
Distance from we ll
 15 14.6
42 10.7
128 4.8
317 1.7
433 0.3

(42)2 a2 + 42 b2 + c2 = 10.7
16
14

12
16,384 a3 + 128 b3 + c3 = 4.8
10
Drawdown

8
6 100,489a4 + 317b4 + c4 = 1.7
4

2
0
0 100 200 300 400 500
Distance from we ll

ai 1 x 2  bi 1 x  ci 1  f x 
Consider this
ai x 2  bi x  ci  f x 
 15 14.6 ai 1 x 2  bi 1 x  ci 1  f x 
42 10.7
128 4.8 ai x 2  bi x  ci  f x 
317 1.7
433 0.3

(42)2 a2 + 42 b2 + c2 = 10.7
16
14

12
16,384 a3 + 128 b3 + c3 = 4.8
10
Drawdown

8
6 100,489a4 + 317b4 + c4 = 1.7
4

2
0
0 100 200 300
Distance from we ll
400 500
(42)2 a1 + 42b1 + c1 = 10.7

Note: This point is in the first and second polynomial

 15 14.6 ai 1 x 2  bi 1 x  ci 1  f x 
42 10.7
128 4.8 ai x 2  bi x  ci  f x 
317 1.7
433 0.3

(42)2 a2 + 42 b2 + c2 = 10.7
16
14

12
16,384 a3 + 128 b3 + c3 = 4.8
10
Drawdown

8
6 100,489a4 + 317b4 + c4 = 1.7
4

2
0
0 100 200 300
Distance from we ll
400 500
(42)2 a1 + 42b1 + c1 = 10.7

16,384a2 + 128b2 + c2 = 4.8

 15 14.6 ai 1 x 2  bi 1 x  ci 1  f x 
42 10.7
128 4.8 ai x 2  bi x  ci  f x 
317 1.7
433 0.3

(42)2 a2 + 42 b2 + c2 = 10.7
16
14

12
16,384 a3 + 128 b3 + c3 = 4.8
10
Drawdown

8
6 100,489a4 + 317b4 + c4 = 1.7
4

2
0
0 100 200 300
Distance from we ll
400 500
(42)2 a1 + 42b1 + c1 = 10.7

16,384a2 + 128b2 + c2 = 4.8

100,489a3 + 317b3 + c3 = 1.7

Similarly, the equations 15 14.6
42 10.7
include the end points 128 4.8
317 1.7
433 0.3
a1 x02  b1 x0  c1  f x0 
an xn2  bn xn  cn  f xn 

16
14

12
10

Drawdown
8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
Similarly, the equations 15 14.6
42 10.7
include the end points 128 4.8
317 1.7
433 0.3
a1 x02  b1 x0  c1  f x0 
an xn2  bn xn  cn  f xn 

(15)2a1 + 15 b1 +c1 = 14.6

16
14

187,489a4 + 433b4 + c4 = 0.3 12

10

Drawdown
8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
The first derivative at the 15 14.6
42 10.7
interior knots must be equal. 128 4.8
317 1.7
433 0.3
2ai 1 xi 1  bi 1  2ai xi 1  b

16
14

12
10

Drawdown
8
6
4

2
0
0 100 200 300 400 500
Distance from we ll
The first derivative at the 15 14.6
42 10.7
interior knots must be equal. 128 4.8
317 1.7
433 0.3
2ai 1 xi 1  bi 1  2ai xi 1  b

2a1 (42) + b1 = 2a2 (42) + b2 16

14

12

2a2 (128) + b2 = 2a3 (128) + b3 10

Drawdown
8
6
4

2a3 (317) + b3 = 2a4 (317) + b4 2

0
0 100 200 300 400 500
Distance from we ll
Addition the last condition a1 = 0

You should be able to set these equations into a

matrix to solve for ai , bi, and ci for i = 1,3

....end of problem
 Splines
• To ensure that the mth derivatives are continuous
at the “knots”, a spline of at least m+1 order must
be used
• 3rd order polynomials or cubic splines that ensure
continuous first and second derivatives are most
frequently used in practice
• Although third and higher derivatives may be
discontinuous when using cubic splines, they
usually cannot be detected visually and
consequently are ignored.
 Splines
• The derivation of cubic splines is somewhat
involved
• First illustrate the concepts of spline interpolation
using second order polynomials.
• These “quadratic splines” have continuous first
derivatives at the “knots”
• Note: This does not ensure equal second
derivatives at the “knots”
 Quadratic Spline
1.The function must be equal at the interior
knots. This condition can be represented as:

ai 1xi21  bi 1xi 1  ci 1  f  xi 1 

note: we are referencing the same x and f(x)

ai xi21  bi xi 1  ci  f  xi 1 
ai  1xi2 1  bi  1xi  1  ci  1  f  xi  1 
ai xi2 1  bi xi  1  ci  f  xi  1 

This occurs between i = 2, n

Using the interior knots (n-1) this will provide

2n -2 equations.
2. The first and last functions must pass
through the end points.

This will add two more equations.

a1x02  b1x0  c1  f  x0 
an xn2  bn xn  cn  f  xn 

We now have 2n - 2 +2 = 2n equations.

How many do we need?

3. The first derivative at the interior knots must
be equal.

This provides another n-1 equations for

2n + n-1 =3n -1.

We need 3n
2 ai  1 xi  1  bi  1  2 ai xi  1  b
4. Unless we have some additional information
regarding the functions or their derivatives,
we must make an arbitrary choice in order
to successfully compute the constants.

5. Assume the second derivative is zero

at the first point. The visual interpretation
of this condition is that the first
two points will be connected by a
straight line.
a1 = 0
 Cubic Splines
• Third order polynomial
• Need n+1 = 3+1 = 4 intervals
• Consequently there are 4n unknown
constants to evaluate
• What are these equations?
f i  x  ai x 3  bi x 2  ci x  di
 Cubic Splines
• The function values must be equal at the interior
knots (2n -2)
• The first and last functions must pass through the
end points (2)
• The first derivatives at the interior knots must be
equal (n-1)
• The second derivatives at the interior knots must
be equal (n-1)
• The second derivative at the end knots are zero (2)
 Cubic Splines
• The function values must be equal at the interior
knots (2n -2)
• The first and last functions must pass through the
end points (2)
• The first derivatives at the interior knots must be
equal (n-1)
• The second derivatives at the interior knots must
be equal (n-1)
• The second derivative at the end knots are zero (2)
SPECIAL NOTE
On the surface it may appear that a third order
approximation
using splines would be inferior to higher order
polynomials.

Consider a situation where a spline may

perform better:
A generally smooth
function undergoes an
abrupt change
in a region of interest,
such as a step function
The abrupt change induces oscillations in
interpolating polynomials.
Here we see a 3rd
order
polynomial fit to
different
data points from
the function
Here we see even more oscillation by fitting
an even higher order polynomial.

Note: 6 points
Here we see even more oscillation by fitting
an even higher order polynomial.

Note: 6 points
The oscillation are continue to
amplify as we increase the order
of the polynomial

Note: 8 points
The oscillation are continue to
amplify as we increase the order
of the polynomial

Note: 8 points
Fitting a cubic spline provides
a much more acceptable
approximation
Fitting a cubic spline provides
a much more acceptable
approximation
Fitting a cubic spline provides
a much more acceptable
approximation
 Specific Study Objectives
• Understand the fundamental difference between
regression and interpolation
• Understand the derivation of linear least squares
regression and be able to assess the reliability of
the fit using graphical and quantitative
assessments.
 Specific Study Objectives
• Know how to linearize data by transformation
• Understand situations where polynomial, multiple
and nonlinear regression are appropriate
• Understand the general matrix formulation of
linear least squares
• Understand that there is one and only one
polynomial of degree n or less that passes exactly
through n+1 points
 Specific Study Objectives
• Become familiar with the use of Newton’s and
Lagrange formulas for interpolating polynomials
• Recognize that the Newton and Lagrange
equations are merely different formulations of the
same interpolating polynomial
• Realize that more accurate results are obtained if
data used for interpolation is centered around and
close to the unknown point
 Specific Study Objectives
• Recognize the liabilities and risks associated with
extrapolation
• Understand why spline functions have utility for
data with local areas of abrupt change
Example
(Previous Exam Question)

A past issue of PCWORD advertised the following

486-DX computer with 4 MB of Ram.

Price Speed Memory

Set up the matrix equation
(MHz) (MB)
1795 33 85
[A]{x} ={c}
2190 33 210 needed to estimate the price of
2490 50 210 a 486-DX based on
2299 33 122 speed and memory.
1995 33 200
1599 33 120
SOLUTION
y x1 x2
2 2
Price Speed Memory x1 x2 x1x2 x1y x2y
(MHz) (MB)
1795 33 85 1089 7225 2805 59235 152575
2190 33 210 1089 44100 6930 72270 459900
2490 50 210 2500 44100 10500 124500 522900
2299 33 122 1089 14884 4026 75867 280478
1995 33 200 1089 40000 6600 65835 399000
1599 33 120 1089 14400 3960 52767 191880
 12368 215 947 7945 164709 34821 450474 2006733

 6 215 947   a0   12368 

 215 7945 34821   a    450474 
  1   
 947 34821 164709 a2  20006733
Example
(Previous Exam Question)

Given the following data, develop the simultaneous

equations for a quadratic spline. Express your final
answers in matrix form.

x f(x) 7.00
6.00

1 0.50 5.00
4.00
f(x)

4 4.60
3.00
2.00
1.00

6 1.50 0.00
0 2 4 6 8

7 3.00 x
 (4, 4.6)
7.00
Interior knots:
6.00
5.00 16a1 + 4b1 + c1 = 4.6
4.00
16a2 + 4b2 + c2 = 4.6
f(x)

3.00
2.00
1.00
0.00 36a2 + 6b2 + c2 = 1.5
0 2 4 6 8
x 36a3 + 6b3 + c3 = 1.5

End conditions
x f(x) a1 + b1 + c1 = 0.5
1 0.50 49a3 + 7b3 + c3 = 3.0

4 4.60 First derivative cot. at interior knots

6 1.50 8a1 + b1 = 8a2 + b2
7 3.00 12a2 + b2 = 12a3 + b3

Extra equation
a1 =0
Interior knots: End conditions
16a1 + 4b1 + c1 = 4.6 16a2 + 4b2 + c2 = 4.6 a1 + b1 + c1 = 0.5 49a3 + 7b3 + c3 = 3.0
36a2 + 6b2 + c2 = 1.5 36a3 + 6b3 + c3 = 1.5 First derivative cont. at interior knots
8a1 + b1 = 8a2 + b2 12a2 + b2 = 12a3 + b3
Extra equation
a1 =0

b1 c1 a2 b2 c2 a3 b3 c3
4 1 0 0 0 0 0  b1  4.6
0
0 0 16 4 1 0 0 0  c1  4.6
    
0 0 36 6 1 0 0 0 a2   15 . 
0 0 0 0 0 36 6 1 b2   15. 
     
1 1 0 0 0 0 0 0  c2   0.5
 
0 0 0 0 0 49 7 1  a3   3.0 
   
1 0 8 1 0 0 0 0  b3   0 
 
0 0 12 1 0 12 1 0  c3   0 
…end of lecture