Orthogonal Bases Orthogonal Bases

The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥

Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Outline

Math 221: LINEAR ALGEBRA

1 Orthogonal Bases
Chapter 8. Orthogonality 2 The Orthogonal Complement U ⊥
§8-1. Orthogonal Complements and Projections
3 Definition of Orthogonal Projection

Tianshi Xu1 4 The Projection Theorem and its Implications

Emory University, 2024 Fall
(last updated on November 30, 2024) 5 Projection as a Linear Transformation

Creative Commons License

(CC BY-NC-SA)

1
Slides are adapted from those by Karen Seyffarth from University of Calgary.
1 / 32 2 / 32

Orthogonal Bases Orthogonal Bases

The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Orthogonality Basis
Lemma (Independent Lemma)
Let V be a vector space and S = {v1 , v2 , . . . , vk } an independent subset of V . If u is a
Definition (Orthogonality) vector in V , but u ̸∈ span(S), then S ′ = {v1 , v2 , . . . , vk , u} is independent.
Let ⃗x , ⃗y ∈ Rn . We say the ⃗x and ⃗y are orthogonal if ⃗x · ⃗y = 0.
More generally, X = {⃗x1 , ⃗x2 , . . . , ⃗xk } ⊆ Rn is an orthogonal set if each x⃗i is
nonzero, and every pair of distinct vectors of X is orthogonal, i.e., ⃗xi · ⃗xj = 0 for all — v.s. —
i ̸= j, 1 ≤ i, j ≤ k .
A set X = {⃗x1 , ⃗x2 , . . . , ⃗xk } ⊆ Rn is an orthonormal set if X is an orthogonal set of
Lemma (Orthogonal Lemma)
unit vectors, i.e., ||⃗xi || = 1 for all i, 1 ≤ i ≤ k .
Suppose {⃗f1 , ⃗f2 , . . . , ⃗fm } is an orthogonal subset of Rn , and suppose ⃗x ∈ Rn . Define
\
⃗ ⃗ ⃗
⃗fm+1 = ⃗x − ⃗x · f1 ⃗f1 − ⃗x · f2 ⃗f2 − · · · − ⃗x · fm ⃗fm .
||⃗f1 ||2 ||⃗f2 ||2 ||⃗fm ||2
Definition (Linearly Independence) Then
Let V be a vector space and S = {⃗x1 , ⃗x2 , . . . , ⃗xk } a subset of V . The set S is linearly 1 ⃗
fm+1 · ⃗fj = 0 for all j, 1 ≤ j ≤ m.
independent if the following condition holds: 2 If ⃗x ̸∈ span{⃗f1 , ⃗f2 , . . . , ⃗fm }, then ⃗fm+1 ̸= ⃗0, and {⃗f1 , ⃗f2 , . . . , ⃗fm , ⃗fm+1 } is an
orthogonal set.
s1⃗x1 + s2⃗x2 + · · · + sk ⃗xk = ⃗0 ⇒ s1 = s2 = · · · = sk = 0.

3 / 32 4 / 32
 Orthogonal Bases Orthogonal Bases
The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Problem
Let    
1 1 1
 0   0   1 
⃗x1 = 
 1 ,
 ⃗x2 = 
 1 ,
 and ⃗x3 = 
 0 ,

Theorem (Gram-Schmidt Orthogonalization Algorithm)
0 1 0
Let U be a subset of Rn and let {⃗x1 , ⃗x2 , . . . , ⃗xm } be a basis of U. Let ⃗f1 = ⃗x1 , and for
each j, 2 ≤ j ≤ m, let and let U = span{⃗x1 , ⃗x2 , ⃗x3 }. We use the Gram-Schmidt Orthogonalization Algorithm
to construct an orthogonal basis B of U.
⃗ ⃗ ⃗ ⃗ ⃗ ⃗
⃗fj = ⃗xj − xj · f1 ⃗f1 − xj · f2 ⃗f2 − · · · − xj · fj−1 ⃗fj−1 .
||⃗f1 ||2 ||⃗f2 ||2 ||⃗fj−1 ||2

Then {⃗f1 , ⃗f2 , . . . , ⃗fm } is an orthogonal basis of U, and

span{⃗f1 , ⃗f2 , . . . , ⃗fj } = span{⃗x1 , ⃗x2 , . . . , ⃗xj } ∀j = 1, · · · , m.

5 / 32 6 / 32

Orthogonal Bases Orthogonal Bases

The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Problem Problem
Let     Let    
1 1 1 1 1 1
 0   0   1   0   0   1 
⃗x1 = 
 1 ,
 ⃗x2 = 
 1 ,
 and ⃗x3 = 
 0 ,
 ⃗x1 = 
 1 ,
 ⃗x2 = 
 1 ,
 and ⃗x3 = 
 0 ,


0 1 0 0 1 0
and let U = span{⃗x1 , ⃗x2 , ⃗x3 }. We use the Gram-Schmidt Orthogonalization Algorithm and let U = span{⃗x1 , ⃗x2 , ⃗x3 }. We use the Gram-Schmidt Orthogonalization Algorithm
to construct an orthogonal basis B of U. to construct an orthogonal basis B of U.

Proof. Proof.

First ⃗f1 = ⃗x1 . First ⃗f1 = ⃗x1 . Next,      

1 1 0
⃗f2 =  0  − 2
   0   0 
 1 = 0
   .
 1  2 
1 0 1

6 / 32 6 / 32
 Orthogonal Bases Orthogonal Bases
The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Problem
Let    
1 1 1 Proof. (continued)
 0   0   1 
⃗x1 = 
 1 ,
 ⃗x2 = 
 1 ,
 and ⃗x3 = 
 0 ,
 Therefore,      
1 0 1/2
0 1 0 
 

 0 , 0 1
     
, 
and let U = span{⃗x1 , ⃗x2 , ⃗x3 }. We use the Gram-Schmidt Orthogonalization Algorithm  1   0   −1/2 
 
to construct an orthogonal basis B of U. 0 1 0
 

is an orthogonal basis of U. However, it is sometimes more convenient to deal with

Proof. vectors having integer entries, in which case we take
First ⃗f1 = ⃗x1 . Next,      

1
 
0
  
1 
1 1 0

 
0 , 0 , 2  .
    
⃗f2 =  0  − 2 B= 
   0   0 
 = .  1   0   −1 
 1  2  1   0  
 
0 1 0
 
1 0 1
Finally, (Orthogonality of the set is not affected by multiplying vectors in the set by nonzero
scalars.)
      
1 1 0 1/2
⃗f3 =  1  − 1
   0  0  0   1 
 1 − 1
   = .
 0  2  0   −1/2 
0 0 1 0
6 / 32 7 / 32

Orthogonal Bases Orthogonal Bases

The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Outline The Orthogonal Complement U ⊥

Definition
1 Orthogonal Bases Let U be a subspace of Rn . The orthogonal complement of U, called U perp, is
denoted U ⊥ and is defined as
2 The Orthogonal Complement U ⊥ U ⊥ = {⃗x ∈ Rn | ⃗x · ⃗y = 0 for all ⃗y ∈ U}.

3 Definition of Orthogonal Projection

4 The Projection Theorem and its Implications

5 Projection as a Linear Transformation

8 / 32 9 / 32
 Orthogonal Bases Orthogonal Bases
The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

The Orthogonal Complement U ⊥

Definition
Let U be a subspace of Rn . The orthogonal complement of U, called U perp, is Theorem (Properties of the Orthogonal Complement)
denoted U ⊥ and is defined as
Let U be a subspace of Rn .
⊥
U = {⃗x ∈ Rn | ⃗x · ⃗y = 0 for all ⃗y ∈ U}. 1 U ⊥ is a subspace of Rn .
2 {⃗0}⊥ = Rn and (Rn )⊥ = {⃗0}.
3 If U = span{⃗y1 , ⃗y2 , . . . , ⃗ym }, then

U ⊥ = {⃗x ∈ Rn | ⃗x · ⃗yj = 0 for j = 1, 2, . . . , m}.

9 / 32 10 / 32

Orthogonal Bases Orthogonal Bases

The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Problem
Let    
0 2 
Example

 
−1  1 
  
      U = span 
 3 ,
   .
 −2 5  a 
 0 

Let U = span  3  ,  −1  , and suppose ⃗v =  b  ∈ U ⊥ . Then 2 4
 
1 2 c
 
Find U ⊥ by finding a basis of U ⊥ .
−2a + 3b + c = 0 and 5a − b + 2c = 0.

This system of two equations in three variables has solution

 
−7
⃗v = −9 t, ∀t ∈ R,
13

which is noting but a line passing through origin and perpendicular with the plane U.

11 / 32 12 / 32
 Orthogonal Bases Orthogonal Bases
The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Problem Solution (continued)

   
Let 0 −1 3 2 0 1 0 3/2 3 0
→ ··· →
   

 0 2  2 1 0 4 0 0 1 −3 −2 0
−1  1 
  
U = span  ,  .
Therefore,


 3   0 

2 4
 
− 32 s − 3t − 32
      

 
 
 −3  
Find U ⊥ by finding a basis of U ⊥ . 3s + 2t 3   2 
  
U⊥  ∈ R4

=  s, t ∈ R = span   ,   .


 s  
 
 1   0  
t 0 1
   
Solution
− 32
   
          
 −3 
a a 0 a 2

3   2 
Since the set B =  , is independent and spans U ⊥ , B is a basis

 
    
b   b   −1   b   1  1   0 
 
⊥  ∈ R4
 c  ·  3  = 0 and
U =       · =0 . 
 
c   c   0  0 1
  

 


d d 2 d 4
 ⊥
of U .

This leads to the system of two equation in four variables

−b + 3c + 2d = 0
2a + b + 4d = 0

12 / 32 13 / 32

Orthogonal Bases Orthogonal Bases

The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Solution (continued)
Outline
   
0 −1 3 2 0 1 0 3/2 3 0
A= → ··· →
2 1 0 4 0 0 1 −3 −2 0
Therefore,
1 Orthogonal Bases
− 32 s − 3t − 32
      

 
 
 −3 
3s + 2t , 2  .
3 
   
U⊥  ∈ R4 The Orthogonal Complement U ⊥
  2
=  s, t ∈ R = span 


 s  
 

 1   0 
t 0 1
   
3 Definition of Orthogonal Projection
− 32
   
 −3 
The Projection Theorem and its Implications
  4
 ,  2  is independent and spans U ⊥ , B is a basis
3 
  
Since the set B = 


 1   0  
0 1 5 Projection as a Linear Transformation
 
⊥
of U .

Remark
Notice that U ⊥ = null(A), where A is the matrix whose rows are a spanning subset of
U.

13 / 32 14 / 32
 Orthogonal Bases Orthogonal Bases
The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Definition of Orthogonal Projection

Definition

Let {⃗f1 , ⃗f2 , . . . , ⃗fm } be an orthogonal basis for a subspace U of Rn , and let ⃗x ∈ Rn . The
projection of ⃗x on U is defined as
Theorem (Projection Formula)
! ! !
Suppose ⃗u and ⃗v are vectors in R3 , ⃗v ̸= ⃗0. Then the projection of ⃗u on ⃗v , denoted as ⃗x · ⃗f1 ⃗ ⃗x · ⃗f2 ⃗ ⃗x · ⃗fm ⃗
⃗
projU (x ) = f1 + f2 + · · · + fm .
proj⃗v (⃗u ), is equal to ||⃗f1 ||2 ||⃗f2 ||2 ||⃗fm ||2
⃗u · ⃗v
 
proj⃗v (⃗u ) = ⃗v .
||⃗v ||2

⃗
u
⃗
u

⃗
v

proj⃗v (⃗
u)

15 / 32 16 / 32

Orthogonal Bases Orthogonal Bases

The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Definition Example ( Generalizing to Rn )

Let {⃗f1 , ⃗f2 , . . . , ⃗fm } be an orthogonal basis for a subspace U of Rn , and let ⃗x ∈ Rn . The
Suppose U is a subspace of Rn , ⃗x ∈ Rn , and that {⃗f1 , ⃗f2 , . . . , ⃗fm } and {⃗g1 , ⃗g2 , . . . , ⃗gm }
projection of ⃗x on U is defined as
are orthogonal bases of U. Define
! ! !
⃗x · ⃗f1 ⃗ ⃗x · ⃗f2 ⃗ ⃗x · ⃗fm ⃗ ! ! !
projU (⃗x ) = f1 + f2 + · · · + fm . ⃗x · ⃗f1 ⃗ ⃗x · ⃗f2 ⃗ ⃗x · ⃗fm ⃗
⃗pf = f1 + f2 + · · · + fm and
||⃗f1 ||2 ||⃗f2 ||2 ||⃗fm ||2
||⃗f1 ||2 ||⃗f2 ||2 ||⃗fm ||2
⃗x · ⃗g1 ⃗x · ⃗g2 ⃗x · ⃗gm
     
⃗pg = ⃗g1 + ⃗g2 + · · · + ⃗gm .
Remark ||⃗g1 ||2 ||⃗g2 ||2 ||⃗gm ||2

1 if U = {⃗0}, then proj{⃗0} (⃗x ) = ⃗0 for any ⃗x ∈ Rn ;

2 if ⃗x ∈ U, then projU (⃗x ) is also called the Fourier Expansion of ⃗x .
3 In Orthogonal Lemma
!
⃗fm+1 = ⃗x − ⃗x · ⃗f1 ⃗ ⃗x · ⃗f2 ⃗ ⃗x · ⃗fm ⃗
f1 + f2 − · · · + fm .
||⃗f1 ||2 ||⃗f2 ||2 ||⃗fm ||2
| {z }
= projU (⃗x )

16 / 32 17 / 32
 Orthogonal Bases Orthogonal Bases
The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Example ( Generalizing to Rn ) Example ( Generalizing to Rn )

Suppose U is a subspace of Rn , ⃗x ∈ Rn , and that {⃗f1 , ⃗f2 , . . . , ⃗fm } and {⃗g1 , ⃗g2 , . . . , ⃗gm } Suppose U is a subspace of Rn , ⃗x ∈ Rn , and that {⃗f1 , ⃗f2 , . . . , ⃗fm } and {⃗g1 , ⃗g2 , . . . , ⃗gm }
are orthogonal bases of U. Define are orthogonal bases of U. Define
! ! ! ! ! !
⃗x · ⃗f1 ⃗ ⃗x · ⃗f2 ⃗ ⃗x · ⃗fm ⃗ ⃗x · ⃗f1 ⃗ ⃗x · ⃗f2 ⃗ ⃗x · ⃗fm ⃗
⃗pf = f1 + f2 + · · · + fm and ⃗pf = f1 + f2 + · · · + fm and
||⃗f1 ||2 ||⃗f2 ||2 ||⃗fm ||2 ||⃗f1 ||2 ||⃗f2 ||2 ||⃗fm ||2
⃗x · ⃗g1 ⃗x · ⃗g2 ⃗x · ⃗gm ⃗x · ⃗g1 ⃗x · ⃗g2 ⃗x · ⃗gm
           
⃗pg = ⃗g1 + ⃗g2 + · · · + ⃗gm . ⃗pg = ⃗g1 + ⃗g2 + · · · + ⃗gm .
||⃗g1 ||2 ||⃗g2 ||2 ||⃗gm ||2 ||⃗g1 ||2 ||⃗g2 ||2 ||⃗gm ||2

Then ⃗pf , ⃗pg ∈ U (since they are linear combinations of vectors of U) and Then ⃗pf , ⃗pg ∈ U (since they are linear combinations of vectors of U) and
⃗x − ⃗pf , ⃗x − ⃗pg ∈ U ⊥ (by the Orthogonal Lemma). ⃗x − ⃗pf , ⃗x − ⃗pg ∈ U ⊥ (by the Orthogonal Lemma). This implies that ⃗pf − ⃗pg ∈ U, and
(⃗x − ⃗pg ) − (⃗x − ⃗pf ) ∈ U ⊥ . However,

(⃗x − ⃗pg ) − (⃗x − ⃗pf ) = ⃗pf − ⃗pg ,

and thus ⃗pf − ⃗pg is in both U and U ⊥ . This is possible if and only if ⃗pf − ⃗pg = ⃗0, i.e.,
⃗pf = ⃗pg . This means that the computation of ⃗pf and ⃗pg does not depend on which
orthogonal basis is used.

17 / 32 17 / 32

Orthogonal Bases Orthogonal Bases

The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Outline The Projection Theorem and its Implications

Theorem (Projection Theorem)

Let U be a subspace of Rn , ⃗x ∈ Rn , and ⃗p = projU (⃗x ). Then
1 Orthogonal Bases 1 ⃗p ∈ U and ⃗x − ⃗p ∈ U ⊥ ;
2 ⃗p is the vector in U closest to ⃗x , meaning that for any ⃗y ∈ U, ⃗y ̸= ⃗p,
2 The Orthogonal Complement U ⊥
||⃗x − ⃗p|| < ||⃗x − ⃗y ||.
3 Definition of Orthogonal Projection

4 The Projection Theorem and its Implications

5 Projection as a Linear Transformation

18 / 32 19 / 32
 Orthogonal Bases Orthogonal Bases
The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

The Projection Theorem and its Implications

Proof.
1 By definition, ⃗p ∈ U, and by the Orthogonal Lemma, ⃗x − ⃗p ∈ U ⊥ .
Theorem (Projection Theorem)
Let U be a subspace of Rn , ⃗x ∈ Rn , and ⃗p = projU (⃗x ). Then
1 ⃗p ∈ U and ⃗x − ⃗p ∈ U ⊥ ;
2 ⃗p is the vector in U closest to ⃗x , meaning that for any ⃗y ∈ U, ⃗y ̸= ⃗p,

||⃗x − ⃗p|| < ||⃗x − ⃗y ||.

⃗x
⃗x

⃗x − ⃗p
⃗x − ⃗p
O
⃗p
U
⃗p O U : ⃗n · ⃗x = d

19 / 32 20 / 32

Orthogonal Bases Orthogonal Bases

The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Proof. Proof.
1 By definition, ⃗p ∈ U, and by the Orthogonal Lemma, ⃗x − ⃗p ∈ U ⊥ . 1 By definition, ⃗p ∈ U, and by the Orthogonal Lemma, ⃗x − ⃗p ∈ U ⊥ .
2 Let ⃗y ∈ U, ⃗y ̸= ⃗p. By the properties of vector addition/subtraction 2 Let ⃗y ∈ U, ⃗y ̸= ⃗p. By the properties of vector addition/subtraction

⃗x − ⃗y = (⃗x − ⃗p) + (⃗p − ⃗y ). ⃗x − ⃗y = (⃗x − ⃗p) + (⃗p − ⃗y ).

Since ⃗x − ⃗p ∈ U ⊥ and ⃗p − ⃗y ∈ U, Since ⃗x − ⃗p ∈ U ⊥ and ⃗p − ⃗y ∈ U,

(⃗x − ⃗p) · (⃗p − ⃗y ) = 0. (⃗x − ⃗p) · (⃗p − ⃗y ) = 0.

Hence, by Pythagoras’ Theorem,

||⃗x − ⃗y ||2 = ||⃗x − ⃗p||2 + ||⃗p − ⃗y ||2 .

Since ⃗y ̸= ⃗p, ||⃗p − ⃗y || > 0, so

||⃗x − ⃗y ||2 > ||⃗x − ⃗p||2 .

Taking square roots (since ||⃗x − ⃗y || and ||⃗x − ⃗p|| are nonnegative),

||⃗x − ⃗y || > ||⃗x − ⃗p||.

20 / 32 20 / 32
 Orthogonal Bases Orthogonal Bases
The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Example Example
Let         Let        
1 1 1 4 1 1 1 4
 0   0
 , ⃗x3 =  1  ,
 3   0   0
 , ⃗x3 =  1  ,
       3 
⃗x1 = 
 1  , ⃗x2 =  1
 
  0  and ⃗v = 
 −2  .
 ⃗x1 = 
 1  , ⃗x2 =  1
 
  0  and ⃗v = 
 −2  .


0 1 0 5 0 1 0 5
We want to find the vector in U = span{⃗x1 , ⃗x2 , ⃗x3 } closest to ⃗v . We want to find the vector in U = span{⃗x1 , ⃗x2 , ⃗x3 } closest to ⃗v .

In a previous example, we used the Gram-Schmidt Orthogonalization Algorithm to

construct the orthogonal basis, B, of U:
     

 1 0 1 
0   0   2 
     
B=  , ,  .


 1   0   −1 


0 1 0
 

21 / 32 21 / 32

Orthogonal Bases Orthogonal Bases

The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Problem
Let      
1 1 1
 0   1  1 
Example (continued)

⃗x1 =  , ⃗x = , and ⃗x3 = 
 0 ,
1  2  1
   

By the Projection Theorem, 0 0 0
       
1 0 1 3 and let U = span{⃗x1 , ⃗x2 , ⃗x3 }. Find an orthogonal basis of U, and find the vector in U
2 0 
+ 5
 0  12  2   4  closest to
projU (⃗v ) =   0 + 6
   =
 −1   −1
 
2

2 1  1 
0 1 0 5  0 
⃗v = 
 −1  .


is the vector in U closest to ⃗v . 3

22 / 32 23 / 32
 Orthogonal Bases Orthogonal Bases
The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Solution ( continued )

Problem Gram-Schmidt orthogonalization with

Let      ⃗f1 = ⃗x1 ,

1 1 1
 0   1  1  ⃗f2 ⃗x2 · ⃗f1 ⃗
= ⃗x2 − f1 ,

⃗x1 = 
 1  , ⃗x2 =  1
  , and ⃗x3 = 
 0 ,

 ||⃗f1 ||2
0 0 0
⃗f3 ⃗x3 · ⃗f1 ⃗ ⃗x3 · ⃗f2 ⃗
= ⃗x3 − f1 − f2
and let U = span{⃗x1 , ⃗x2 , ⃗x3 }. Find an orthogonal basis of U, and find the vector in U ||⃗f1 ||2 ||⃗f2 ||2
closest to  
2 yields an orthogonal basis
 0 
⃗v = 
 −1  .
      

 1 0 1 
3 
0 
, 1
   0 
B= 
 1   0
,  .
  −1 

 
0 0 0
 
Solution ( Outline )
Thus the vector in U closest of ⃗v is
First use the Gram-Schmidt Orthogonalization Algorithm to construct an orthogonal
basis of of U, and then find the projection of ⃗v on U.
     
1 1 2
+ 3
1 0   0   0 
projU (⃗v ) =   =
 −1   −1
.
2 1  2 
0 0 0
23 / 32 24 / 32

Orthogonal Bases Orthogonal Bases

The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Problem Problem
Find the point q in the plane 3x + y − 2z = 0 that is closest to the point p0 = (1, 1, 1). Find the point q in the plane 3x + y − 2z = 0 that is closest to the point p0 = (1, 1, 1).

Solution
Recall that any plane in R3 that contains the origin is a subspace of R3 .
1 Find a basis X of the subspace U of R3 defined by the equation 3x + y − 2z = 0.
2 Orthogonalize the basis X to get an orthogonal basis B of U.
3 Find the projection on U of the position vector of the point p0 .

25 / 32 25 / 32
 Orthogonal Bases Orthogonal Bases
The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Solution (continued)
1 3x + y − 2z = 0 is a system of one equation in three variables. Putting the Solution (continued)
augmented matrix in reduced row-echelon form
h i 1 Use the Gram-Schmidt Orthogonalization Algorithm to get an orthogonal basis
3 1 −2 0 → 1 13 − 23 0 of U:
 
       
−1 2 −1 9
⃗f1 =  3  and ⃗f2 =  0  − −2  3  = 1  3  .
1
gives general solution x = 3
s + 23 t, y = s, z = t for any s, t ∈ R. Then
0 3 10 0 5 15
 − 31
   2 
3 
U = span  1 , 0  . Therefore,
0 1
     
 −1 3 
Let B =  3 , 1 
0 5
 
   
 −1 2 
X =  3 , 0  is an orthogonal basis of U.
0 3
 

Then X is linearly independent and span(X ) = U, so X is a basis of U.

26 / 32 27 / 32

Orthogonal Bases Orthogonal Bases

The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Outline

Solution (continued)
3 To find the point q on U closest to p0 = (1, 1, 1), compute 1 Orthogonal Bases
     
1 −1 3 The Orthogonal Complement U ⊥
2 9 2
projU  1  =  3 +  1 
1 10 0 35 5
  3 Definition of Orthogonal Projection
4
1
= 6 .
7 4 The Projection Theorem and its Implications
9

4 6 9
 5 Projection as a Linear Transformation
Therefore, q = , ,
7 7 7
.

28 / 32 29 / 32
 Orthogonal Bases Orthogonal Bases
The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Projection as a Linear Transformation Projection as a Linear Transformation

Definition Definition
Let V and W be vector spaces, and T : V → W a linear transformation. Let V and W be vector spaces, and T : V → W a linear transformation.
1 The kernel of T (sometimes called the null space of T ) is defined to be the set 1 The kernel of T (sometimes called the null space of T ) is defined to be the set

ker(T ) = {⃗v ∈ V | T (⃗v ) = ⃗0}. ker(T ) = {⃗v ∈ V | T (⃗v ) = ⃗0}.

2 The image of T is defined to be the set 2 The image of T is defined to be the set

im(T ) = {T (⃗v ) | ⃗v ∈ V }. im(T ) = {T (⃗v ) | ⃗v ∈ V }.

Theorem
Let U be a fixed subspace of Rn , and define T : Rn → Rn by

T (⃗x ) = projU (⃗x ) for all ⃗x ∈ Rn .

Then
1 T is a linear operator on Rn ;
2 im(T ) = U and ker(T ) = U ⊥ ;
3 dim(U) + dim(U ⊥ ) = n.
30 / 32 30 / 32

Orthogonal Bases Orthogonal Bases

The Orthogonal Complement U ⊥ The Orthogonal Complement U ⊥
Definition of Orthogonal Projection Definition of Orthogonal Projection
The Projection Theorem and its Implications The Projection Theorem and its Implications
Projection as a Linear Transformation Projection as a Linear Transformation

Takeaway Thank You

Orthogonal Bases
The Orthogonal Complement U ⊥
Thank you.
Definition of Orthogonal Projection
The Projection Theorem and its Implications
Projection as a Linear Transformation

31 / 32 32 / 32