1 / 60

Chapter 3: Curve Fitting

Vu Van Thieu, Dinh Viet Sang, Ban Ha Bang

SoICT
Hanoi University of Science and Technology

2 / 60
Introduction

1 Problem

2 Interpolation
Lagrange interpolation
Spline interpolation

3 Regression
Linear regression
High-order curve fitting
General curve fitting

3 / 60
Problem

The problem is:

Suppose we need to approximate the function f (x) without knowing the formula, but by
experiment we can determine the value of f (x) at n + 1 at discrete points

fi = f (xi ) i = 0, 1, · · · , N

Given the dataset {(xi , fi ), i = 0, 1, ..., N}, we need to find a way to approximate the function
f (x) by a function (which can be calculated using an analytic formula) p(x).

4 / 60
Problem (continued)

continue...
To solve the problem
We will learn two methods used to construct curve fitting.
Interpolation: function p(x) must go through all points in the dataset.
Regression: given the form p(x) with parameters, we must determine the parameters to
minimize a certain error criterion (usually the least squares criterion is used).

5 / 60
Interpolation

Introduction
The possible interpolation with functions p(x) is
Polynomial function
Rational function
Fourier series function

6 / 60
Interpolation

Interpolation
Definition: We say the function p(x) is an interpolation function of the dataset
{(xi , fi ), i = 0, 1, ..., N} if the following conditions are satisfied:

pi = fi , i = 0, 1, ..., N

This system of N + 1 of equations is called interpolation condition. The reason that the
polynomial function was chosen is because it is easy to calculate values, derivatives, and
differentials (the analytic properties of the function are obvious).
The polynomial that interpolates the data set is called the interpolating polynomial.
Definition: We call a polynomial of degree of no more than K has form of:

pK (x) = a0 + a1 x + · · · + aK x K

where a0 , a1 , · · · , aK are constants.

7 / 60
Interpolation

Linear interpolation
We have two data points (x0 , f0 ) and (x1 , f1 ) which need interpolation by polynomial
p1 (x) ≡ a0 + a1 x
p1 (x) satisfies the system of equations

p1 (x0 ) ≡ a0 + a1 x0 = f0
p1 (x1 ) ≡ a0 + a1 x1 = f1

Solve this system, we have

When x0 ̸= x1 occurs two cases
−x0 f1
▶ if f0 ̸= f1 then p1 (x) = x1xf10 −x 0
+ xf11 −f
−x0 x
0

▶ if f0 = f1 then the solution is constant ⇒ polynomial of 0-th degree.

8 / 60
Interpolation
Linear interpolation (continued)
f

(9,6)
p1 (x) = 0.6 + 0.6x

(4,3)

9 / 60
Interpolation
Lagrange Interpolation Formula
It is necessary to construct a curve that passes through 3, 4 or more points (see figure).

(9,6)

(7,4)
(4,3)

x 10 / 60
Interpolation

Lagrange interpolation formula (continued)

Consider polynomial
pM (x) = a0 + a1 x + a2 x 2 + · · · + aM x M
The polynomial pM (x) interpolates the dataset {(xi , fi ), i = 0, 1, ..., N} if we have the
interpolation conditions:

pM (x0 ) ≡ a0 + a1 x0 + a2 x02 + · · · + aM x0M = f0

pM (x1 ) ≡ a0 + a1 x1 + a2 x12 + · · · + aM x1M = f1
···
pM (xN ) ≡ aN + a1 xN + a2 xN2 + · · · + aM xNM = fN

This is a system of linear equations, so we can represent it using a matrix formula.

11 / 60
Interpolation

Lagrange interpolation formula (continued)

Thus the matrix of the interpolation conditions is as follows:
    
1 x0 · · · x0M a0 f0
M
1 x1 · · · x1   a1   f1 
    
  ..  =  .. 
. . .. 
. . ..
   
. . . .  .   . 
1 xN · · · xNM aM fN

Assuming that x0 , x1 , · · · , xN is distinct, then according to the theory of solving the system of
linear equations in the previous chapter, we have
N=M: system of equations with unique solution
M<N: can choose the dataset so that it has no solution
M>N: if the system has a solution, it has infinitely many solutions
12 / 60
Interpolation

Lagrange interpolation formula (continued)

When N = M the coefficient matrix of the above system of equations is a Vandermonde
matrix (Not Voldemort)
 
1 x0 · · · x0M
1 x1 · · · x1M 
 
VN =   .. .. .. . 
. . . .. 


1 xN · · · xNM
The determinant of Vn is: Y
VN = (xi − xj )
i>j

infer that the system has a solution when x0 , x1 , · · · , xN is pairwise distinct.

13 / 60
Interpolation

Lagrange interpolation formula (continued)

Theorem (Uniqueness of Interpolated Polynomials): If the data nodes x0 , x1 , · · · , xN are
pairwise different, then there exists a unique interpolation polynomial pN (x) of no more than
N degree that interpolates the dataset {(xi , fi ), i = 0, 1, ..., N}.

The degree of the interpolated polynomial can be less than N

For example, when the three data points (x0 , f0 ), (x1 , f1 ), (x3 , f3 ) line up, the interpolation
equation becomes of no more than two degree because it is a straight line.

14 / 60
Interpolation

Example 1:
Determine the coefficients of the interpolation polynomial p2 (x) = a0 + a1 x + a2 x 2
interpolating the dataset {(-1,0),(0,1),(1,3 )}, we have interpolation condition

a0 + (−1)a1 + 1a2 = 0
a0 + 0a1 + 0a2 = 1
a0 + 1a1 + 1a2 = 3

Solving the system of equations we have a0 = 1, a1 = 1.5 and a2 = 0.5. So the interpolated
polynomial is
p2 (x) = 1 + 1.5x + 0.5x 2

15 / 60
Interpolation

Lagrange interpolation formula (continued)

Directly solving a system of interpolated conditional equations using Gaussian elimination, for
example, requires a lot of computations O(N 3 ). In addition, because the power of the
coefficient x can be very high, the error is even larger due to rounding effects.
⇒ we can use Lagrange interpolation formula to calculate the constant of the interpolated
polynomial without solving the above system of linear equations.

16 / 60
Interpolation

Lagrange interpolation formula (continued)

Consider the dataset {(xi , fi ), i = 0, 1, ..., N}. The Lagrangian form of the interpolation
polynomial for the dataset is as follows:

pN (x) = f0 V0 (x) + f1 V1 (x) + · · · + fN VN (x)

where Vi (x) : i = 0, 1, · · · , N are polynomials of degree N satisfying the condition:

(
1, i = j
Vi (xj ) =
0, i ̸= j

The family of polynomials Vi (x) : i = 0, 1, · · · , N is called the family of basis polynomials. It is

easy to check that:
pN (xi ) = fi i = 0, 1, · · · , N

17 / 60
Interpolation

Lagrange interpolation formula (continued)

One of the base polynomial class that can be built according to the following formula:

(x − x0 )(x − x1 ) · · · (x − xi−1 )(x − xi+1 ) · · · (x − xn )

Vi (x) =
(xi − x0 )(xi − x1 ) · · · (xi − xi−1 )(xi − xi+1 ) · · · (xi − xn )

It is clearly satisfied (
1, i = j
Vi (xj ) =
0, i ̸= j
So we can summarize the above formula as follows:
Q
i̸=j (x − xj )
Vi (x) = Q
i̸=j (xi − xj )

18 / 60
Interpolation

Lagrange interpolation formula (continued)

So the Lagrange interpolation formula

(x − x1 )(x − x2 ) · · · (x − xN )
pN (x) = f0
(x0 − x1 ) · · · (x0 − xN )
···
(x − x0 )(x − x1 ) · · · (x − xi−1 )(x − xi+1 ) · · · (x − xn )
+ fi (1)
(xi − x0 )(xi − x1 ) · · · (xi − xi−1 )(xi − xi+1 ) · · · (xi − xn )
···
(x − x0 )(x − x1 ) · · · (x − xN−1 )
+ fN
(xN − x0 ) · · · (xN − xN−1 )

Formula (1) is long but easier to remember and calculate.

19 / 60
Interpolation
Install Vi (x) using Matlab
v = polyinterp(x, y , u) calculates v (j) = P(u(j)) where P is a polynomial of degree
d = length(x) − 1 such that P(x(i)) = y (i). Use Lagrange interpolation formula. Two vectors
x, y represent the coordinates of the data points.

function v = polyinterp(x,y,u)
n = length(x);
v = zeros(size(u));
for k = 1:n
w = ones(size(u));
for j = [1:k-1 k+1:n]
w = (ux(j))./(x(k)-x(j)).*w;
end
v = v + w*y(k);
end
20 / 60
Interpolation
Install Vi (x) using Matlab (continued)
function v = polyinterp(x,y,u)
n = length(x);
v = zeros(size(u));
for k = 1:n
w = ones(size(u));
for j = [1:k-1 k+1:n]
w = (ux(j))./(x(k)-x(j)).*w;
end
v = v + w*y(k);
end
The two vectors x,y represent the coordinates of the data points.
u is a vector containing the coordinates of the points to be calculated according to the
Lagrange interpolation formula.
21 / 60
Interpolation

Example 2:
x = [0 1 2 3];
y = [-5 -6 -1 16];
u= [0.25:0.01:3.25];
v = polyinterp(x,y,u);
plot(x,y,’o’,u,v,’-’)

22 / 60
Interpolation

23 / 60
Interpolation
Interpolation error
Definition: The error of the interpolated polynomial g (x) is determined by the formula:

e(x) = f (x) − g (x)

The magnitude of e(x) depends on

The magnitude of the data points.
Size of the interpolation domain D = xn − x0 .
Degree of interpolation polynomial
For Lagrange interpolated polynomials, the error is determined by the formula:

e(x) = f (x) − g (x) = L(x)f (N+1) (ξ)

where f (N+1) is the derivative of N + 1 degree of the function f (x) and

L(x) = (x−x0 )(x−x1 )···(x−xn )
(N+1)! 24 / 60
Interpolation

Interpolation error (continued)

formula:
e(x) = f (x) − g (x) = L(x)f (N+1) (ξ)
ξ depends on x but satisfies the condition x0 < ξ < xN . According to the above formula, if f(x)
is a polynomial of degree no more than N, then the derivative f (N+1) will be zero, that is, the
error is zero. Since L(x) = (x−x0 )(x−x1 )···(x−xn )
(N+1)! we have the following conclusions
With an equidistant grid, the amplitude of oscillations of L(x) is smallest at the center
and increases towards the boundary of the interpolation domain.
The interpolation size increases, the magnitude of the oscillation increases rapidly.

25 / 60
Lagrange interpolation

26 / 60
Spline interpolation

Problem
In this case we consider interpolation by a set of low-order polynomials instead of a single
higher-order polynomial
Interpolation by linear spline
Interpolation by cubic spline
Recall the Lagrange interpolation error evaluation formula

f (x) − pN (x) = [wN+1 (x)/(N + 1)!]f (N+1) (ξx )

where wN+1 (x) = (x − x0 )(x − x1 ) · · · (x − xN ) and ξx is a value between (x0 , xN ).

27 / 60
Spline interpolation

Problem (continued)
Assuming every data point xi ∈ [a, b] then we have

maxx∈[a,b] |f (N+1) (x)|

max |f (x) − pN (x)| ≤ |b − a|N+1 ×
x∈[a,b] (N + 1)!

This evaluation indicates that if we want to reduce the error while keeping the number of data
points N, we need to find a way to reduce the size of |b − a|.
Our approach is piecewise polynominal approximation: Thus the segment [a, b] will be divided
into many small non-intersecting segments and different polynomials will approximated on
each sub-segment.

28 / 60
Spline interpolation

Snippet Linear Spline

For the dataset
{(xi , fi ) : i = 0, 1, ..., N}
Inside
a = x0 < x1 < · · · < xN = b, h ≡ max |xi − xi−1 |,
i

linear spline S1,N (x) is a continuous function that interpolates given data and is built from
linear functions defined by two data point interpolation polynomials:

29 / 60
Spline interpolation

Snippet Linear Spline (continued)


f1 −f0
x1 −x0 (x − x1 ) + f1 , if x ∈ [x0 , x1 ]


..



.



f −f
S1,N (x) = xii −xi−1
i−1
(x − xi ) + fi , if x ∈ [xi−1 , xi ]



.
..


 f −f
 N N−1 (x − xN ) + fN , if x ∈ [xN−1 , xN ]

xN −xN−1

Easy to see
fi − fi−1
Li (x) = (x − xi ) + fi
xi − xi−1
is the equation of the line through two points (xi , fi ) and (xi−1 , fi−1 )

30 / 60
Spline interpolation

Snippet Linear Spline (continued)

Using error formula for interpolated polynomial with x ∈ [a, b] for each segment [xi−1 , xi ] we
get

|xi − xi−1 |2
max |f (z) − S1,N (z)| ≤ × max |f (2) (x)|
z∈[xi−1 ,xi ] 8 x∈[xi−1 ,xi ]

h2
≤ × max |f (2) (x)|
8 x∈[xi−1 ,xi ]

where h ≡ maxi |xi − xi−1 |.

31 / 60
Spline interpolation

Example 3:
Given the dataset {(-1,0),(0,1),(1,3)}, we build a spline calculated as follows
(
f1 −f0 f1 −f0
−x0 (x − x1 ) + f1 = x1 −x0 (x − x0 ) + f0 , if x ∈ [x0 , x1 ]
S1,2 (x) = xf21 −f f2 −f1
x2 −x1 (x − x2 ) + f2 = x2 −x1 (x − x1 ) + f1 , if x ∈ [x1 , x2 ]
1

infer with the corresponding data set

(
1−0
0−(−1) x + 1 = x + 1, if x ∈ [−1, 0]
S1,2 (x) = 3−1
1−0 (x − 1) + 3 = 2x + 1, if x ∈ [0, 1]

32 / 60
Spline interpolation

Example 4:
» clear;
» x=1:6;
» y=[16 18 21 17 15 12];
» plot(x,y,’o’)
» hold on; grid on;
» axis([-1 7 10 22])
» xx = [1:0.01:6];
» yy = piecelin(x,y,xx);
» plot(xx,yy,’r’)
» hold off
» xlabel(’x-axis’);ylabel(’y-axis’);
» title(’Linear Spline Interpolation’);

33 / 60
Spline interpolation

34 / 60
Spline interpolation

Third-order splines
Instead of line segments, third-order splines use 3rd degree polynomials to approximate the
segment polynomial. I also have the dataset

{(xi , fi ) : i = 0, 1, ..., N}

Inside
a = x0 < x1 < · · · < xN = b, h ≡ max |xi − xi−1 |,
i

35 / 60
Spline interpolation

Third-order spline (continued)

Definition: A third-order spline S3,N (x) interpolates a given dataset
S3,N (x) is a 3rd order function between two nodes [xi−1 , xi ]


 p1 (x) = a1,0 + a1,1 x + a1,2 x 2 + a1,3 x 3 , if x ∈ [x0 , x1 ]

 ..
.



S3,N (x) = pi (x) = ai,0 + ai,1 x + ai,2 x 2 + ai,3 x 3 , if x ∈ [xi−1 , xi ]
..




 .

pN (x) = aN,0 + aN,1 x + aN,2 x 2 + aN,3 x 3 , ifx ∈ [xN−1 , xN ]


S3,N (x) is a continuous function with continuous first and second derivatives on the
segment [x0 , xN ] (including boundaries)

36 / 60
Spline interpolation

Third-order spline (continued)

For xi−1 < x < xi , S3,N (x) has value For xi < x < xi+1 , S3,N (x) has value
pi (x) = ai,0 + ai,1 x + ai,2 x 2 + ai,3 x 3 pi+1 (x) = ai+1,0 + · · · + ai+1,3 x 3
Value of S3,N (x) when x → xi− Value of S3,N (x) when x → xi+
Comment:
pi (xi ) pi+1 (xi )
pi′ (xi ) ′
pi+1 (xi )
pi ”(xi ) pi+1 ”(xi )
The smoothness condition at the point xi then

pi′ (xi ) = pi+1

′
(xi ); pi ”(xi ) = pi+1 ”(xi ); i = 1, 2, · · · , N − 1

Guaranteed data interpolation condition

pi (xi ) = pi+1 (xi ) = fi ; i = 0, 1, 2, · · · , N − 1

37 / 60
Spline interpolation

Third-order spline (continued)

Natural Boundary Condition

p1 ”(x0 ) = 0; pN ”(xN ) = 0;

Second Derivative Condition

p1 ”(x0 ) = f ”(x0 ); pN ”(xN ) = f ”(xN );

Not-a-knot Condition

p1′′′ (x1 ) = p2′′′ (x1 ); pN−1

′′′ ′′′
(xN−1 ) = pN (xN−1 );

38 / 60
Spline interpolation

Spline function for 3rd order interpolation in MatLab:

p=spline(x,y): returns the segmented polynomial of the third order spline for the dataset
(x,y).
v=ppval(p,x): Calculates the value of the interpolated polynomial at input x (p is determined
by the spline function).
yy=spline(x,y,xx): Defines the 3rd order interpolated spline for the dataset (x,y) and returns
yy as the vector of the values of the spline interpolated at points in the vector xx .

39 / 60
Spline interpolation

Example 4:
» x = 0:10; y = sin(10*x);
» p = spline(x,y);
» xx = linspace(0,10,200);
» yy = ppval(p,xx);
» plot(x,y,’o’,xx,yy)

40 / 60
Level 3 spline interpolation

41 / 60
Spline interpolation

Example 5:
» x = 0:10; y = sin(10*x);
» xx = linspace(0,10,200);
» coef = polyfit(x,y,10);
Warning: Polynomial is badly conditioned. Remove repeated data points or try
centering and scaling as described in HELP POLYFIT. (Type "warning off
MATLAB:polyfit:RepeatedPointsOrRescale" to suppress this warning.)
» yy = polyval(coef,xx);
» plot(x,y,’o’,xx,yy,’r’)

42 / 60
Level 3 spline interpolation

So when we interpolate with spline of order 3, it is better than polynomial interpolation with43 / 60
Spline interpolation

Example 6:
The power consumption of a device in 12 hours is as follows

1 2 3 4 5 6
54.4 54.6 67.1 78.3 85.3 88.7
7 8 9 10 11 12
96.9 97.6 84.1 80.1 68.8 61.1

Using the third-order spline to interpolate the above data point set.

44 / 60
Third-order spline interpolation

45 / 60
Level 3 spline interpolation

46 / 60
Regression

Problem
Given the dataset {(xi , fi ), i = 1, · · · , N}, we need to find a way to approximate the function
f (x) by a function p(x). We need to take care of measuring the total error over the entire
segment.
Maximum error E∞ (f ) = max1≤k≤n |f (xk ) − yk |
Average error E1 (f ) = n1 nk=1 |f (xk ) − yk |
P

Square root error (least square)

v
u n
u1 X
E2 (f ) = t (f (xk ) − yk )2
n
k=1

47 / 60
Regression

Linear regression
Linear regression is finding a line that fits the data points in the least squares sense.
Given a set of N pairs of data points {(xi , fi ) : i = 1, · · · , N}
Find the coefficient m and the free constant b of the line

y (x) = mx + b

such that this line fits the data according to the least squares criterion.
X N
L(m, b) = (fi − (mxi + b))2
i=1

48 / 60
Regression
Linear regression (continued)
That is, we need to find m and b to minimize the function
N
X
L(m, b) = (fi − (mxi + b))2
i=1

To find this minimum, we need to solve the system of equations to find the stopping point
N
∂L X
= 2(fi − (mxi + b))(−xi ) = 0
∂m
i=1
N
∂L X
= 2(fi − (mxi + b))(−1) = 0
∂b
i=1

49 / 60
Regression

Linear regression (continued)

" N # " N
# N
X X X
xi2 m+ xi b = xi fi
i=1 i=1 i=1
N N N
" # " #
X X X
xi m + 1 b= fi
i=1 i=1 i=1

or in matrix form, ! ! !
PN 2 PN PN
i=1 xi i=1 xi m i=1 xi fi
PN = PN
i=1 xi N b i=1 fi

Solving the system of equations we get the coefficients m and b.

50 / 60
Regression

Example 6:
Construct the fitting curve for the following dataset

i 1 2 3 4
xi 1 3 4 5
fi 2 4 3 1

So the system of equations to be solved is

13m + 4b = 10
51m + 13b = 31

then find b = 107/35 and m = −6/35, the line to find is y = (−6/35)x + 107/35
51 / 60
Regression

High-order curve fitting

Build the curve fitting
pM (x) = a0 + a1 x + a2 x 2 + · · · + aM x M
matches the dataset {(xi , fi )i = 1, · · · , N} according to least squares criteria

σM ≡ (pM (x1 ) − f1 )2 + (pM (x2 ) − f2 )2 + · · · + (pM (xN ) − fN )2

Solve the system of equations to find the stopping point

∂ ∂ ∂
σM = 0; σM = 0; · · · σM = 0;
∂a0 ∂a1 ∂aM

52 / 60
Regression

High-order curve fitting (continued)

Write in matrix form the system of standard equations
 PN PN PN M     PN 
i=1 1 xi ··· x a 0 fi
 PN PNi=1 2 PNi=1 M+1 i
    PN
i=1
 i=1 xi i=1 xi ··· x a i=1 fi xi

i=1 i   1   
.. .. .. ..   ..  =  ..
     
. . . .  .   .
 
 
PN M PN M+1 PN 2M PN C
i=1 xi i=1 xi ··· i=1 xi am i=1 fi xi ODE

The special feature of the above matrix is that it is definite symmetry, so we can use Gaussian
elimination without changing the line to solve the system of standard equations.

53 / 60
Regression

Example 7:
Determine the parameters a, b, cc of the curve

y = ax 2 + bx + c

matches the following dataset

x 1 2 3 4 5 6 7 8
y 1 8 27 64 125 216 350 560

54 / 60
Regression
Example 7: (continued)
x = [1 2 3 4 5 6 7 8];
y = [1 8 27 64 125 216 350 560];
s0 = length(x); s1 = sum(x); s2 = sum(x.ˆ2);
s3 = sum(x.ˆ3); s4 = sum(x.ˆ4);
A = [s4 s3 s2; s3 s2 s1; s2 s1 s0];
b = [sum(x.ˆ2.*y); sum(x.*y);sum(y)];
c0 = A\b;
c = polyfit(x,y,2); % Find a regression polynomial that matches the data
c0 % Returns the calculated coefficient
c % Returns the coefficient found by polyfit
xx = 1:0.1:8;
yy = polyval(c0,xx);
plot(x,y,’o’,xx,yy)
55 / 60
Regression

56 / 60
Regression

General curve fitting

Suppose we want to match N multipoint for {(xi , fi ) : i = 1, · · · , N} because they include m
linearly independent function φj (x), j = 1, 2, · · · , m, i.e. the function f (x) to look for has the
form
X m
f (x) = cj φj (x)
j=1

For example, the following family of functions

φj (x) = x j , j = 1, 2, · · · , m
φj (x) = sin(jx), j = 1, 2, · · · , m
φj (x) = cos(jx), j = 1, 2, · · · , m

57 / 60
Regression

General curve fitting (continued)

We need to determine the numbers c1 , c2 , · · · , cm to minimize the error of the square root of
the sum of squares of errors:
  2
n
X n
X m
X
E (c1 , c2 , · · · , cm ) = (f (xk ) − fk )2 =  cj φj (x) − fk 
k=1 k=1 j=1

To find the coefficients c1 , c2 , · · · , cm , we need to solve the system of equations

∂E
= 0, j = 1, 2, · · · , m
∂cj

58 / 60
Regression

59 / 60
Summary

Summary
Curve fitting problem
Definition of interpolation
▶ Lagrange Interpolation
▶ Spline Interpolation
Definition of regression
▶ Linear Regression
▶ Higher order polynomial regression

60 / 60