DA241M Mathematical Foundations of Data Science

Course Instructor: Teena Sharma, Ph.D. (Email: teena@iitg.ac.in)

Singular Value Decomposition

Any square matrix that is diagonalizable may be expressed as

C = V ΛV −1

where V = (v1 , v2 , . . . , vn ) is a square matrix made of the column eigenvectors v1 , v2 , . . . of the

matrix C.
Eigendecomposition allows for an easy way of inverting a diagonalizable square matrix C that can
be inverted.
C −1 = V Λ−1 V −1

You may feel this is not easy since it involves inverting V anyway. The important point here is that
inverting V is trivial. The reason is because V is made up of mutually orthogonal vectors. So the
inversion is as simple as (for real matrices C and V ),

V −1 = V T

Thus V is an orthogonal matrix (prove this). Inverse of Λ is even simpler, it is just the reciprocal
of the diagonal entries.
For rectangular matrices, eigen-decomposition does not make sense. However, it is possible to have
an analogous decomposition called Singular Value Decomposition.
Singular Value Decomposition (SVD) is a powerful matrix factorization technique used in linear
algebra and data analysis. It decomposes a matrix into three other matrices, which helps in under-
standing the inherent structure and properties of the original matrix. SVD is widely used in various
applications, such as image compression, data compression, collaborative filtering, and machine
learning.
Let’s consider a real-valued matrix A of size m × n. SVD breaks down this matrix into three
component matrices:
1. U : The left singular vectors matrix - This matrix contains the eigenvectors of AAT (where
AT is the transpose of A). The columns of U are orthogonal (unit length and perpendicular
to each other). The number of columns in U is equal to m, and the matrix has dimensions
m × m.
2. Σ: The singular values matrix - This matrix is a diagonal matrix that contains the square root
of the eigenvalues of both AAT and AT A. The diagonal elements of Σ are known as singular
values and are arranged in descending order. The number of non-zero singular values is equal
to the rank of the matrix A. The dimensions of Σ are m × n.
3. V T : The right singular vectors matrix - This matrix contains the eigenvectors of AT A. The
columns of V T are orthogonal (unit length and perpendicular to each other). The number of
columns in V T is equal to n, and the matrix has dimensions n × n.

1
Mathematically, the SVD of matrix A can be represented as follows:

A=U ·Σ·VT

Example: Consider the following matrix A:

 
1 2 3
A=
4 5 6

Step 1: Calculate AT A and AAT .

 
    17 22 27
1 2 3 1 2 3
AT A = · = 22 29 36
4 5 6 4 5 6
27 36 45

     
T 1 2 3 1 2 3 14 32
AA = · =
4 5 6 4 5 6 32 77

Step 2: Calculate the eigenvalues and eigenvectors of both AT A and AAT . For AT A:
 
17 − λ 22 27
|AT A − λI| =  22 29 − λ 36  = −λ3 + 91λ2 − 54λ = 0
27 36 45 − λ

Solving this cubic equation, we get the eigenvalues for AT A as λ1 ≈ 90.4027, λ2 ≈ 0.597327, and
λ3 ≈ 0.
For AAT :  
T 14 − λ 32
|AA − λI| = = λ2 − 91λ + 54 = 0
32 77 − λ

Solving this equation, we get the eigenvalues for AAT as λ4 = 90.4027 and λ5 = 0.597327.
Step 3: Calculate the singular values and singular vectors. The singular values (σ) are the square
root of the eigenvalues of both AAT and AT A.
For AT A: p √
σ1 = λ1 ≈ 90.4027 ≈ 9.50803
p √
σ2 = λ2 ≈ 0.597327 ≈ 0.772869
p √
σ3 = λ3 ≈ 0 ≈ 0

For AAT : p √
σ4 = λ4 ≈ 90.4027 ≈ 9.50803
p √
σ5 = |λ5 | ≈ 0.597327 ≈ 0.772869

Next, we find the eigenvectors corresponding to each eigenvalue:

For AT A:

2
For λ1 ≈ 90.4027:
  
17 − 90.4027 22 27 v11
T
(A A − λ1 I)v1 = 0 ⇒  22 29 − 90.4027 36   v12  = 0
27 36 45 − 90.4027 v13

Solving the system of equations, we get

 
−0.428667
v1 ≈ −0.566307
−0.703947

For λ2 ≈ 0.597327:
  
17 − 0.597327 22 27 v21
T
(A A − λ2 I)v2 = 0 ⇒  22 29 − 0.597327 36   v22  = 0
27 36 45 − 0.597327 v23

Solving the system of equations, we get

 
0.805964
v2 ≈  0.112382 
−0.581199

For λ3 ≈ 0:   
17 22 27 v31
T
(A A − λ3 I)v3 = 0 ⇒ 22 29 36
   v32  = 0
27 36 45 v33

Solving the system of equations, we get

 
0.408248
v3 ≈ −0.816497
0.408248

For AAT :
For λ4 = 90.4027:
  
T 14 − 90.4027 32 v41
(AA − λ4 I)v4 = 0 ⇒ =0
32 77 − 90.4027 v42

Solving the system of equations, we get

 
−0.386318
v4 ≈
−0.922366

3
For λ5 = 0.597327:
  
T 14 − 0.597327 32 v51
(AA − λ5 I)v5 = 0 ⇒ =0
32 77 − 0.597327 v52

Solving the system of equations, we get

 
−0.922366
v5 ≈
0.386318

Step 4: Assemble the SVD. Now, we put everything together to form the SVD of matrix A:

A=U ·Σ·VT
√ 
λ4 √0
Σ=
0 λ5

√ 
λ4 √0
A = (v4 , v5 ) · · (v1 , v2 , v3 )T
0 λ5
 
  √  −0.428667 −0.566307 −0.703947
−0.386318 −0.922366 90.4027 √ 0
A≈ · ·  0.805964 0.112382 −0.581199
−0.922366 0.386318 0 0.597327
0.408248 −0.816497 0.408248
 
1 2 3
A≈
4 5 6

This completes the SVD decomposition of matrix A.

SVD is particularly useful in data compression and dimensionality reduction because it allows us to
approximate the original matrix using only a subset of its singular values and vectors. By keeping
only the largest singular values and their corresponding vectors, we can capture the most important
information of the original matrix while discarding the noise or less relevant information. This
property is utilized in various applications, including image compression and collaborative filtering
for recommendation systems.