Questions for the course

Numerical Linear Algebra

TMA265/MMA600

Date: 2013 October 24, Time: 14.00 - 18.00, Place: at CTH,

Maskinhuset

Question 1
• 1. Find eigenvalues for a matrix A such that
 
4 1 0
A = 2 0 0
0 0 5
(1p)
• 2. Compute ||A||∞ , ||A||1 . Find conjugate transpose matrix A∗ to the matrix A.
(1p)
• 3. Find inverse matrix A−1 to the matrix A via the matrix of cofactors. (2p)

Question 2
• 1. Write algorithm of LU factorization of the matrix A with pivoting using con-
ventional programming language notation. (1p)
• 2. Prove that following two statements are equivalent:
1. There exists a unique unit lower triangular matrix L and nonsingular upper
triangular U such that A = LU .
2. All leading principal submatrices of A are nonsingular.
(3p)

Question 3
• 1. Derive the condition number k(A) of the matrix A. (2p)
• 2. Derive practical error bounds
||e
x − x||∞
error =
||e
x||∞
of Ax = b in the terms of residual r and the approximate solution xe of Ax = b.
(2p)

Question 4
• 1. Derive normal equations AT Ax = AT b in the method of normal equations. (2p)
• 2. Derive formula for x that minimizes the functional F (x) = ||Ax − b||22 using the
QR decomposition of the matrix A = QR. You can present any one of the three
derivations of this formula. (3p)
• 3. Let A will be m × n matrix with m ≥ n. Dene the SVD decomposition of a
matrix A. (1p)

1
2

Question 5
• Compute QR decomposition of the matrix A using Householder reections:
 
4 4 3
A = 0 3 1
3 4 7
(2p)
• Compute QR decomposition of the matrix A using Given's rotations
 
4 4 3
A = 3 3 1
0 4 7
(1p)

Question 6
• 1. Give denitions of the Schur canonical form and the Real Schur canonical form.
(1p)
• 2. Let the matrix A is diagonalizible such that S −1 AS = Λ, where Λ = diag(λ1 , . . . , λn )
are eigenvalues. Prove that the S = [x1 , . . . , xn ] be the nonsingular matrix of right
eigenvectors, and rows of S −1 are conjugate transposes of the left eigenvectors yi .
(1p)
• 3. Let λ be a simple eigenvalue of A with right eigenvector x and left eigenvector y ,
normalized so that ||x||2 = ||y||2 = 1. Dene condition number for the eigenvalue
λ.
(1p)

Question 7
• 1. Let A = U ΣV be the SVD decomposition of the m-by-n matrix A with m ≥ n.
T

Dene the Moore-Penrose pseudoinverse matrix A+ of the matrix A.

(1p)
• 2. Let A = U ΣV T be the SVD decomposition of the m-by-n matrix A with m ≥ n.
Using denitions of A and A+ prove that AA+ A = A.
(2p)
 3

Numerical Linear Algebra

TMA265/MMA600
Solutions to the examination at 24 October 2013
Question 1
1. We should solve characteristic equation det(A − λI) = 0 :
 
4−λ 1 0
det  2 −λ 0  = 0.
0 0 5−λ
√
Solving
√
above equation for λ we get three eigenvalues λ1 = 5, λ2 = 4+2 24 ≈ 4.4495, λ3 =
4− 24
2
≈ −0.4495 which are solutions to the equation (5 − λ)(λ2 − 4λ − 2) = 0.
3. We use denition of A∗ :
A∗ = AT .
 
4 2 0
AT = 1 0 0 .
0 0 5
||A||∞ = 5, ||A||1 = 6.
4. By denition of an inverse matrix we have:
1
A−1 = (C T )ij
det A
Thus,
 
0 −10 0
C = −5 20 0 ,
0 0 −2
 
0 −5 0
C T = −10 20 0  ,
0 0 −2
   
0 −5 0 0 0.5 0
and thus A−1 = 1
−10
· −10 20 0 = 1
   −2 0 
0 0 −2 0 0 0.2.

Question 2
1. See Lecture 3 and the course book.
1. See Theorem 2.4 of the course book.
Proof.
We rst show that (1) implies (2). A = LU may also be written
       
A11 A12 L 0 U U L U L11 U12
= 11 × 11 12 = 11 11
A21 A22 L21 L22 0 U22 L21 U11 L21 U12 + L22 U22

where A11 is a j-by-j leading principal submatrix, as well as L11 and U11 . Therefore
detA11 = det(L11 U11 ) = detL11 detU11 = 1 · k=1 (U11 )kk 6= 0, since L is unit triangular
Qj
and U is triangular.
4

(2) implies (1) is proved by induction on n. It is easy for 1-by-1 matrices: a = 1 · a. To

prove it for n-by-n matrices Ã, we need to nd unique (n-1)-by-(n-1) triangular matrices
L and U , unique (n-1)-by-1 vectors l and u, and unique nonzero scalar η such that
       
A b L 0 U u LU Lu
à = T = T × = T
c δ l 1 0 η l U lT u + η

By induction unique L and U exist such that A = LU . Now let u = L−1 b, lT = cT U −1 , and
η = δ − lT u, all of which are unique. The diagonal entries of U are nonzero by induction,
and η 6= 0 since 0 6= detà = det(U ) · η .

Question 3

See Lectures 2,3 and the course book.

1.
• Consider linear system Ax = b ,
• x̂ such that x̂ = δx + x is its computed solution.
• Suppose (A + δA)x̂ = b + δb.
• Goal: to bound the norm of δx ≡ x̂ − x .
• Subtract the equalities and solve them for δx
• Rearranging terms we get:

δx = A−1 (−δAx̂ + δb)

Taking norms and triangle inequality leads us to

kδxk ≤ kA−1 k(kδAk · kx̂k + kδbk)

Rearranging inequality gives us

kδxk kδAk kδbk
≤ kA−1 k · kAk · ( + )
kx̂k kAk kAk · kx̂k

where k(A) = kA−1 k · kAk is the condition number of the matrix A

2.
||e
x − x||∞ ||r||∞
error = ≤ ||A−1 ||∞ · , (2.13)
||e
x||∞ ||e
x||∞

where r = Ae x − b is the residual. We estimate ||A−1 ||∞ by applying Algorithm 2.5 to

B = A−T , estimating ||B||1 = ||A−T ||1 = ||A−1 ||∞ (see parts 5 and 6 of Lemma 1.7).

Question 4

1. See Lecture 6 and the course book.

2. We will derive the formula for the x that minimizes ||Ax − b||2 using the decompo-
sition A = QR in three slightly dierent ways. First, we can always choose m − n more
orthonormal vectors Q̃ so that [Q, Q̃] is a square orthogonal matrix (for example, we can
choose any m − n more independent vectors X̃ that we want and then apply Algorithm
 5

3.1 to the n-by-n nonsingular matrix [Q, X̃]). Then

||Ax − b||22 = ||[Q, Q̃]T (Ax − b)||22
 T  2
Q
=
Q̃T (QRx − b)

 2  2
QT b
n×n

I
= (m−n)×n Rx −
O Q̃T b
2
Rx − QT b 2
 
=
−Q̃T b
2
2
= Rx − QT b + kQ̃T bk22
2
≥ kQ̃T bk22 .
We can solve Rx − QT b = 0 for x, since A and R have the same rank, n, and so R is
nonsinsular. Then x = R−1 QT b, and the minimum value of ||Ax − b||2 is ||Q̃T b||2 .
Here is a second, slightly dierent derivation that does not use the matrix Q̃. Rewrite
Ax − b as
Ax − b = QRx − b = QRx − (QQT + I − QQT )b
= Q(Rx − QT b) − (I − QQT )b.
Note that the vectors Q(Rx − QT b) and (I − QQT )b are orthogonal, because (Q(Rx −
QT b))T ((I − QQT )b) = (Rx − QT b)T [QT (I − QQT )]b = (Rx − QT b)T [0]b = 0. Therefore,
by the Pythagorean theorem,
kAx − bk22 = kQ(Rx − QT b)k22 + k(I − QQT )bk22
= kRx − QT bk22 + k(I − QQT )bk22 .
where we have used part 4 of Lemma 1.7 in the form This sum of squares is minimized
when the rst term is zero, i.e., x = R−1 QT b.
Finally, here is a third derivation that starts from the normal equations solution:
x = (AT A)−1 AT b
= (RT QT QR)−1 RT QT b = (RT R)−1 RT QT b
= R−1 R−T RT QT b = R−1 QT b.
3. See Lecture 7 and the course book.
Question 5
• 1. First, we need to nd a reection that transforms the rst column of matrix A
 
4 4 3
A = 0 3 1
3 4 7
We have:
u = x + αe1 ,
where x = (4, 0, 3) , α = −sign(4) · ||x||
T

u
v= .
kuk
Here,
α = −5.
Therefore √
u = (−1, 0, 3)T , ||u|| = 10.
6

and v = √1 (−1, 0, 3)T

10
, and then
 
−1
2

P1 = I − √ √  0  −1 0 3
10 10 3
 
1 0 −3
1
=I−  0 0 0 
5 −3 0 9
 
4/5 0 3/5
= 0 1 0 .
3/5 0 −4/5
Now observe:  
5 5.6 6.6
P1 A =  0 3 1 ,
0 −0.8 −3.8
so we already have almost a triangular matrix. We only need to zero the (3, 2)
entry.
Take the (1, 1) minor, and then apply the process again to
 
0 3 1
A = M11 = .
−0.8 −3.8
We have:
u = x + αe1 ,
where x = (3, −0.8)T , α = −sign(3) · ||x||
u
v= .
kuk
Here,
α = −3.1048.
Therefore
u = (−0.1048, −0.8)T , ||u|| = 0.8068.
and v = 1
0.8068
(−0.1048, −0.8)T , and then
 
2 −0.1048
P20


=I− −0.1048 −0.8
0.651 −0.8
 
2 0.011 0.0838
=I−
0.651 0.0838 0.64
 
0.9662 −0.2575
= .
0.2575 −0.9662
Then the second matrix of the Householder transformation is
 
1 0 0
P2 = 0 0.9662 −0.2575
0 −0.2575 −0.9662
 7

Now, we nd
 
5 5.6 6.6
R = P2 P1 A = 0 3.1046 1.9447 .
0 0.0005 3.4141

The matrix P is orthogonal and R is upper triangular, so A = QR is the required

QR-decomposition with P = P1T P2T .
• 2. To obtain QR decomposition of the matrix A
 
4 4 3
A = 3 3 1
0 4 7
using Given's rotation we have to zero out (2, 1) and (3, 2) elements of the matrix
A.
1. First, we zero out element (2, 1) of the matrix A.
To do that we compute c, s from the known a = 4 and b = 3 as
     
c −s a r
· =
s c b 0

to get formulas:
√ √
r= a2 + b 2 = 42 + 32 = 5,
a
c= = 0.8,
r
−b
s= = −0.6.
r
The rst Given's matrix will be

c −s 0
G1 = s c 0
0 0 1
or
 
0.8 0.6 0
G1 = −0.6 0.8 0
0 0 1
Then
 
5 5 3
G1 · A = 0 0 −1
0 4 7
2. Next step is to construct second Given's matrix G2 in order to zero out (3, 2)
element of the matrix G1 · A.
To do that we compute c, s from the known a = 0 and b = 4 as
     
c −s a r
· =
s c b 0
8

to get formulas:
√ √
r= a2 + b 2 = 02 + 42 = 4,
a
c= = 0,
r
−b
s= = −1.
r
The second Given's matrix will be  
1 0 0
G2 = 0 c −s
0 s c
or  
1 0 0
G2 = 0 0 1
0 −1 0
Then upper triangular matrix R in the QR decomposition will be
 
5 5 3
R = G2 · G1 · A = 0 4 7
0 0 1
Then A = GT1 · GT2 · R = QR will be QR decomposition of the matrix A with
Q = GT1 · GT2 given by  
0.8 0 0.6
Q = 0.6 0 −0.8
0 1 0

Question 6
• 1. Schur canonical form. Given A, there exists a unitary matrix Q and an upper
triangular matrix T such that Q∗ AQ = T . The eigenvalues of A are the diagonal
entries of T .
Real Schur canonical form. If A is real, there exists a real orthogonal matrix V
such that V T AV = T is quasi-upper triangular. This means that T is block upper
triangular with 1-by-1 and 2-by-2 blocks on the diagonal. Its eigenvalues are the
eigenvalues of its diagonal blocks. The 1-by-1 blocks correspond to real eigenvalues,
and the 2-by-2 blocks to complex conjugate pairs of eigenvalues.
• 2. Let S = [x1 , . . . , xn ] the nonsingular matrix of right eigenvectors. and we know
that A is diagonalizible and thus AS = SΛ, where Λ = diag(λ1 , . . . , λn ), since the
columns xi of S are eigenvectors. This is equivalent to AS −1 = S −1 Λ, so the rows
of S −1 are conjugate transposes of the left eigenvectors yi .
• 3. The expression secΘ(y, x) = 1/|y ∗ x| is the condition number of the eigenvalue
λ.
Question 7
• 1. Let A be the m-by-n matrix with m ≥ n and has a full rank such that A = U ΣV T .
Then Moore-Penrose pseudoinverse of A is A+ = (AT A)−1 AT . If m < n then
A+ = AT (AAT )−1 .
• 2.
AA+ A = U ΣV T (V ΣU T U ΣV T )−1 V ΣU T U ΣV T
= U ΣV T (V −1 Σ−2 V −T )Σ2 = U ΣV −1 = U ΣV T = A.