Best approximation in the 2-norm

Elena Celledoni

Department of Mathematical Sciences, NTNU

september 26th 2012

Elena Celledoni NA
Vector space

A real vector space V is a set with a 0 element and three

operations:
• Addition: x, y ∈ V then x + y ∈ V
• Inverse: for all x ∈ V there is −x ∈ V such that x + (−x) = 0
• Scalar multiplication: λ ∈ R, x ∈ V then λ x ∈ V

1 x +y =y +x
2 (x + y ) + z = x + (y + z)
3 0+x =x +0=x
4 0x = 0
5 1x = x
6 (λ µ) x = λ (µ x)
7 λ (x + y ) = λx + λy
8 (λ + µ) x = λx + µx

Elena Celledoni NA
Inner product spaces: 9.2
Definition of inner product space
Let V be a linear space over R.
A function h·, ·i : V × V → R, is called an inner product on V if:

1 hf + g , hi = hf , hi + hg , hi for all f , g , h in V ;
2 hλf , g i = λhf , g i for all f , g in V and λ ∈ R;
3 hf , g i = hg , f i for all f , g in V ;
4 hf , f i > 0 if f 6= 0, f ∈ V .

Elena Celledoni NA
Inner product spaces: 9.2
Definition of inner product space
Let V be a linear space over R.
A function h·, ·i : V × V → R, is called an inner product on V if:

1 hf + g , hi = hf , hi + hg , hi for all f , g , h in V ;
2 hλf , g i = λhf , g i for all f , g in V and λ ∈ R;
3 hf , g i = hg , f i for all f , g in V ;
4 hf , f i > 0 if f 6= 0, f ∈ V .
A linear space with an inner product is an inner product space.

Elena Celledoni NA
Inner product spaces: 9.2
Definition of inner product space
Let V be a linear space over R.
A function h·, ·i : V × V → R, is called an inner product on V if:

1 hf + g , hi = hf , hi + hg , hi for all f , g , h in V ;
2 hλf , g i = λhf , g i for all f , g in V and λ ∈ R;
3 hf , g i = hg , f i for all f , g in V ;
4 hf , f i > 0 if f 6= 0, f ∈ V .
A linear space with an inner product is an inner product space.
Example
The n-dimensional Euclidean space Rn is an inner product space
n
X
hu, bi = ui vi , u, v ∈ Rn
i=0

and u = [u1 , . . . , un ]T , v = [v1 , . . . , vn ]T .

Elena Celledoni NA
Inner product spaces: orthogonality and norm

Orthogonality
If in an inner product space V we have

hf , g i = 0

then we say that f is orthogonal to g .

Elena Celledoni NA
Inner product spaces: orthogonality and norm

Orthogonality
If in an inner product space V we have

hf , g i = 0

then we say that f is orthogonal to g .

Induced norm
In an inner product space we can define the norm
1
kf k := hf , f i 2 ,

this is called norm induced by the inner product.

Elena Celledoni NA
Inner product spaces: orthogonality and norm

Orthogonality
If in an inner product space V we have

hf , g i = 0

then we say that f is orthogonal to g .

Induced norm
In an inner product space we can define the norm
1
kf k := hf , f i 2 ,

this is called norm induced by the inner product.

Theorem An inner product space V over R with the induced
norm defined above is a normed space.
Proof. Need to prove that the induced norm satisfies the axioms of
norm, the proof makes use of the Cauchy-Schwarz inequality.

Elena Celledoni NA
Lemma: Cauchy-Schwarz inequality

|hf , g i| ≤ kf k kg k, ∀f , g ∈ V .

Proof Similar to the one seen in chapter 2 for the inner product in Rn .

Elena Celledoni NA
Lemma: Cauchy-Schwarz inequality

|hf , g i| ≤ kf k kg k, ∀f , g ∈ V .

Proof Similar to the one seen in chapter 2 for the inner product in Rn .

• Using the lemma we prove the triangular inequality for the induced
norm:

kf + g k2 = hf + g , f + g i = kf k2 + 2hf , g i + kg k2

from Cauchy-Schwarz we get

kf + g k2 ≤ kf k2 + 2kf kkg k + kg k2 = (kf k + kg k)2

so
kf + g k ≤ kf k + kg k.

Elena Celledoni NA
Lemma: Cauchy-Schwarz inequality

|hf , g i| ≤ kf k kg k, ∀f , g ∈ V .

Proof Similar to the one seen in chapter 2 for the inner product in Rn .

• Using the lemma we prove the triangular inequality for the induced
norm:

kf + g k2 = hf + g , f + g i = kf k2 + 2hf , g i + kg k2

from Cauchy-Schwarz we get

kf + g k2 ≤ kf k2 + 2kf kkg k + kg k2 = (kf k + kg k)2

so
kf + g k ≤ kf k + kg k.

• The other two axioms of norm are directly proved using the
definition of induced norm.
Elena Celledoni NA
Space of continuous functions

Example (9.3)
C [a, b] (continuous functions defined on the closed interval [a, b])
is an inner product space with the inner product
Z b
hf , g i = w (x)f (x)g (x)dx,
a

and w is a weight function defined, positive, continuous and

integrable on (a, b). The induced norm is
Z b  12
kf k2 = w (x) |f (x)|2 dx .
a

This norm is called “the 2-norm on C [a, b]”.

Elena Celledoni NA
Space of square integrable functions: L2
We do not need f to be continuous in order for kf k2 to be finite, it is
sufficient that |f |2 w is integrable on [a, b].

Elena Celledoni NA
Space of square integrable functions: L2
We do not need f to be continuous in order for kf k2 to be finite, it is
sufficient that |f |2 w is integrable on [a, b].
Example
L2w (a, b) := {f : (a, b) → R | w (x)|f (x)|2 is integrable on (a, b)}.

This is an inner product space with the inner product

Z b
hf , g i = w (x)f (x)g (x)dx,
a

and w is a weight function defined, positive, continuous and integrable

on (a, b). The induced norm is
! 12
Z b
kf k2 = w (x) |f (x)|2 dx .
a

This norm is called the L2 -norm.

Note C [a, b] ⊂ L2w (a, b). In the case w ≡ 1 then we write L2 (a, b).
Elena Celledoni NA
Best approximation in the 2-norm: ch 9.3
The problem of best approximation in the 2-norm can be
formulated as follows:
Best approximation problem in the 2-norm
Given that f ∈ L2w (a, b), find pn ∈ Πn such that

kf − pn k2 = inf kf − qk2
q∈Πn

such pn is called a polynomial of best approximation of degree

n to the function f in the 2-norm on (a, b).

Elena Celledoni NA
Best approximation in the 2-norm: ch 9.3
The problem of best approximation in the 2-norm can be
formulated as follows:
Best approximation problem in the 2-norm
Given that f ∈ L2w (a, b), find pn ∈ Πn such that

kf − pn k2 = inf kf − qk2
q∈Πn

such pn is called a polynomial of best approximation of degree

n to the function f in the 2-norm on (a, b).

Theorem 9.2
Let f ∈ L2w (a, b), there exists a unique polynomial pn ∈ Πn such
that
kf − pn k2 = min kf − qk2 .
q∈Πn

Elena Celledoni NA
Example: best approximation in k · k2 vs k · k∞

Let  > 0 fixed

x
f (x) = 1 − e −  , x ∈ [0, 1]

find
p02−norm and p0∞−norm

Elena Celledoni NA
In general, to find pn (x) = c0 + c1 x + · · · + cn x n best
approximation in the two norm on [0, 1]:

Elena Celledoni NA
In general, to find pn (x) = c0 + c1 x + · · · + cn x n best
approximation in the two norm on [0, 1]:

Solve
Mc=b
where M is the Hilbert matrix:
Z 1
1
Mj,k = x k+j dx = j, k = 0, . . . , n,
0 k +j +1

and  
b0 Z 1
bT =  ...  , bj = f (x)x j dx.
 
0
bn

Elena Celledoni NA
Alternatively write pn (x) = γ0 ϕ0 + γ1 ϕ1 (x) + · · · + γn ϕn (x)
best approximation in the two norm on [a, b]:
and
Πn = span{1, x, . . . , x n } = span{ϕ0 , ϕ1 . . . , ϕn },
and find γ0 , . . . , γn minimizing
Z b n
X
E (γ0 , . . . , γn ) := |f (x) − γj ϕj |2 .
a j=0

Elena Celledoni NA
Alternatively write pn (x) = γ0 ϕ0 + γ1 ϕ1 (x) + · · · + γn ϕn (x)
best approximation in the two norm on [a, b]:
and
Πn = span{1, x, . . . , x n } = span{ϕ0 , ϕ1 . . . , ϕn },
and find γ0 , . . . , γn minimizing
Z b n
X
E (γ0 , . . . , γn ) := |f (x) − γj ϕj |2 .
a j=0

This leads to
Mγ =b
where γ = [γ0 , . . . , γn ]T , M is the matrix:
Mj,k = hϕk , ϕj i j, k = 0, . . . , n,
and
b = [b0 , . . . , bn ]T bj = hf , ϕj i.
We’d like M diagonal (i.e. hϕk , ϕj i = δk,j ).
Elena Celledoni NA
Def 9.4: Orthogonal polynomials

Given a weight function w on (a, b) we say that the sequence of

polynomials
ϕj , j = 0, 1, . . .
is a system of orthogonal polynomials on (a, b) with respect to
w , if each ϕj is of exact degree j and

hϕj , ϕk i = δj,k .

To construct a system of orthogonal polynomials we can use a

Gram-Schmidt process.

Elena Celledoni NA
Gram-Schmidt orthogonalization
Let ϕ0 ≡ 1 and assume ϕj j = 0, . . . , n n ≥ 0 are already
orthogonal so
Z b
hϕk , ϕj i = w (x)ϕk (x)ϕj (x) dx = 0, k, j ∈ {0, . . . , n}, k 6= j
a
consider
q(x) = x n+1 − a0 ϕ0 (x) − · · · − an ϕn (x) ∈ Πn+1 ,
and take
hx n+1 , ϕj i
aj = , j = 0, . . . , n
hϕj , ϕj i
THEN

Elena Celledoni NA
Gram-Schmidt orthogonalization
Let ϕ0 ≡ 1 and assume ϕj j = 0, . . . , n n ≥ 0 are already
orthogonal so
Z b
hϕk , ϕj i = w (x)ϕk (x)ϕj (x) dx = 0, k, j ∈ {0, . . . , n}, k 6= j
a
consider
q(x) = x n+1 − a0 ϕ0 (x) − · · · − an ϕn (x) ∈ Πn+1 ,
and take
hx n+1 , ϕj i
aj = , j = 0, . . . , n
hϕj , ϕj i
THEN

hq, ϕj i = hx n+1 , ϕj i − aj hϕj , ϕj i = 0,

Elena Celledoni NA
Gram-Schmidt orthogonalization
Let ϕ0 ≡ 1 and assume ϕj j = 0, . . . , n n ≥ 0 are already
orthogonal so
Z b
hϕk , ϕj i = w (x)ϕk (x)ϕj (x) dx = 0, k, j ∈ {0, . . . , n}, k 6= j
a
consider
q(x) = x n+1 − a0 ϕ0 (x) − · · · − an ϕn (x) ∈ Πn+1 ,
and take
hx n+1 , ϕj i
aj = , j = 0, . . . , n
hϕj , ϕj i
THEN

hq, ϕj i = hx n+1 , ϕj i − aj hϕj , ϕj i = 0,

and we can take
ϕn+1 = α q
with α a scaling constant.
Elena Celledoni NA
Example (9.6 Legendre polynomials)
(a, b) = (−1, 1) w (x) ≡ 1:

ϕ0 (x) ≡ 1
ϕ1 (x) = x
3 2 1
ϕ2 (x) = x −
2 2
5 3 3
ϕ2 (x) = x − x
2 2

Example (9.6 Chebishev polynomials)

(a, b) = (−1, 1) w (x) ≡ (1 − x 2 )−1/2

Tn (x) = cos(n arcos(x)), n = 0, 1, . . . .

Elena Celledoni NA
Characterization of the best approximation in the 2-norm

Theorem 9.3
pn ∈ Πn is such that

kf − pn k2 = min kf − qk2 , f ∈ L2w (a, b),

q∈Πn

if and only if
hf − pn , qi = 0, ∀q ∈ Πn

Remark: This theorem relates best approximation to orthogonality.

Elena Celledoni NA
Example
On (0, 1) and w (x) ≡ 1 on (0, 1) by Gram-Schmidt orthogonalization
we get
1 1
ϕ0 = 1, ϕ1 = x − , ϕ2 (x) = x 2 − x + .
2 6
So for
x
f (x) = 1 − e −  , x ∈ [0, 1]
the best approximation p3 is p3 (x) = γ0 + γ1 ϕ1 (x) + γ2 ϕ2 (x)

Elena Celledoni NA
Example
On (0, 1) and w (x) ≡ 1 on (0, 1) by Gram-Schmidt orthogonalization
we get
1 1
ϕ0 = 1, ϕ1 = x − , ϕ2 (x) = x 2 − x + .
2 6
So for
x
f (x) = 1 − e −  , x ∈ [0, 1]
the best approximation p3 is p3 (x) = γ0 + γ1 ϕ1 (x) + γ2 ϕ2 (x)
Z 1
x
γ0 = (1 − e −  ) dx,
0

R1 x
0
(1 − e −  )(x − 12 ) dx
γ1 = R1 ,
0
(x − 12 )2 dx
R1 x
0
(1 − e −  )(x 2 − x + 16 ) dx
γ1 = R1 ,
0
(x 2 − x + 16 )2 dx

Elena Celledoni NA