2.

MATRIX ALGEBRA AND RANDOM VECTORS

2.1 Review of Matrices and its Operation

▪ A matrix is a rectangular array of real numbers. The matrix is denoted by a

capital letter. The numbers in the array are called the entries or elements.
Any matrix A may be expressed as

 a11 a12 a1n 

a 
 21 a22 a23 .... an 
A 
 
 
 am1 am 2 .... amn 

where the brackets are used to emphasise that the entire array is to be regarded
as a single entity.

▪ The order or size of the matrix A above is m  n . That is, ' m ' the number of
rows and ' n ' the number of columns.

▪ The element aij we call the first subscript the i th row and the second

subscript the j th column. For example a23 is the second row and third
column.

1
2.1.2 Types of Matrices

(i). A row matrix is one whose order is of the form 1 n . It has only one row. For
example, G  1 2 3 is of the order 1 3 .

(ii). A column matrix is one whose order is of the form m1 . It has only one column.
5
Example, A    is of the order 2  1 .
 2

(iii). Two matrices are said to be equal if they have the same size and their
corresponding entries are equal.

(iv). The matrix N is a null matrix (or zero matrix) if aij  0 for all i , j . That is, a

0 0
zero matrix has zero entries only. Example, N   
0 0

(v). A square matrix is one whose order is of the form n  n . It has the same number
of rows and columns.

 3 5 7
 0 1
Examples: A    B   6 4 1  C   4 are square matrices.
8 1
 2 3 4 

(vi). If the only nonzero elements of a square matrix lie on the main diagonal, is
said to be a diagonal matrix.
7 0 0
3 0
Examples M    and N   0 2 0 
0 1 

 0 0 1 

2
 Thus, any diagonal matrix of order n can be denoted as

 d11 0 0 
 0 d 0 
D 22
 
 
 0 0 d nn 

(vii). If d11  d 22   d nn  1 then D, is called the identity matrix or unit matrix

denoted by I

Thus
1 0 0
0 1 0 
I 
 
 
  ij
 
0 0 1

where ij is the Kronecker delta symbol defined as

1 if i  j
 ij  
0 if i  j

It sometimes convenient to include a subscript n to indicate the order of I.

1 0 0 
For example, I 3  0 1 0  is a 3  3 identity matrix.
 
 0 0 1 

3
 (viii). A square matrix
A   aij  is upper triangular if aij  0 for  j  i
and lower triangular if aij  0 for  j  i

 a11 a12 a1n 

0 a 
U  22  upper triangular matrix
 
 
0 ann 

 a11 0 0 
a a 
L   21 22  lower triangular matrix
 
 
 an1 ann 

If a matrix is upper triangular or lower triangular it is said to be triangular.

2.1.3 Matrix Operation

If A , B and C are m  n matrices, 0 is an m  n matrix, and  ,  are scalars, then:

(a) A B  B  A (b)  A  B  C  A   B  C 

(c) A 0  A (d) A    A  0

(e)    A    A (f)     A   A   A

(g) 1A  A (h) 0A  0

4
2.1.4 Matrix Multiplication

If A   aij  is any m  n matrix and B  bij  is any n  p matrix; so that the

number of columns of A is equal to the number of rows of B, then AB is the product.

Properties of Matrix Multiplication

If  ,   and the matrices A, B and C are suitably conformable, then

(a)  A B  A  B     AB 

(b) A  BC    AB  C

(c)  B  C  A  BA  CA

(d) A  B   C    AB   AC

2.1.5 Trace of a Matrix

If A is a square matrix, then the trace of A, denoted by tr  A , is defined to be the
sum of the entries on the main diagonal of A .
i.e.
tr  A   Aii
i

▪ If AB is a square matrix then tr  AB   tr  BA

▪ The trace is undefined if A is not a square matrix.

5
2.1.6 Transpose of Matrices
Given any m  n matrix A   aij  , we define the transpose of A,

denoted as AT
 a11 a21 am1 
a a22 am 2 
A   a ji    12
T  
 
 
 a1n a2n amn 

That is, we interchange the rows and columns of A, so that, the first column of AT is
the first row of A, and the second column of AT is the second row, and so forth.

Properties of Transpose
(a) Suppose A and B are both n  m matrices then

 AT 
T
(i). A

(ii).  A  B T  AT  BT

(iii).  A T   AT where  

(b) Suppose A is n  m and B is n  p then  AB T  BT AT i.e. transposing reverses

the order of products

(c) I T  I

(d)  ABC T  C T BT AT

6
2.1.6 Determinants
In general, if A is a 3  3 matrix, then its determinant is

a11 a12 a13

det  A   a21 a22 a23
a31 a32 a33

a22 a23 a a a a
 a11  a12 21 23  a13 21 22
a32 a33 a31 a33 a31 a32

2.1.7 Inverse
If A is a square matrix, and if a matrix B of the same size can be found such that
AB  BA  I , then A is said to be invertible and B is called an inverse of A .
The inverse of A is denoted by A1

If no such matrix B can be found, then A is said to be singular, i.e., A  0 .

Properties of Inverses
(a) The matrix A is invertible if and only if A is non-singular, ie, A  0

1
(b) If A is non-singular A1 
A

   
1 T
(c) If A is square and non-singular, then, AT  A1

7
(d) If A and B are invertible matrices of the same size, then AB is invertible and
 AB 1  B 1 A1

 A1 
1
(e) A

1 1 1
(f) For any nonzero scalar k , the matrix kA is invertible and  kA   A
k

(g) A diagonal matrix is invertible if and only if all of its diagonal entries are nonzero

 d1 0 0
 1 
0 1
d2
0
 
D 1   
 
 
0 0 dn 
1
 

(h) Powers of diagonal matrices are easy to compute; if k is a positive integers then

 d1k 0 0
 
0 d 2k 0
D 
k 
 
 
 
 0 0 d nk 

8
2.1.8 Symmetric matrix
A square matrix is said to be symmetric if AT  A , i.e., aij  a ji
▪ In general, a symmetric matrix need not be invertible, for example, a square zero
matrix is symmetric, but not invertible

▪ If A and B are symmetric matrices with the same dimension, and if k 

(i) AT is symmetric
(ii) A  B and A  B are symmetric
(iii) kA is symmetric

▪ The square matrix A is said to be skew symmetric or anti-symmetric

if AT   A , i.e. aii  aiiT  aii

▪ It is skew-symmetric if the diagonal entries are zero and the off-diagonal

entries are equal but opposite in sign

9
2.1.9 Orthogonal Matrices
A square matrix Q is orthogonal if its transpose is the same as its inverse.
That is, if Q 1  QT

Or, equivalently, if QQ  Q Q  I
T T

Example 2.1.5

The matrix
 73 2
7
6 
7
 2 
Q   76 3
7 7 
2 6 3 
7 7 7 

is orthogonal since

 73 6 2  3 6 
7 7 7
2
7 7 1 0 0 
 6   6 2 
QT Q   72 7
3
7  7
3
7 7 
 0 1 0 
6 2 3   2 6 3  0 0 1 
7 7 7  7 7 7 

10
2.2 Vectors
A vector is a quantity that has a magnitude (size) and a direction. We denote such
vectors by bold type, e.g. a or, when hand written, by a .

Definitions
(i). The size, magnitude, norm or modulus of a vector a is denoted by a is
the length of the line representing a

(ii). Two vectors are equal if they have the same magnitude and same direction.

(iii). Negative vectors: If two vectors a and b have the same magnitude but
opposite directions.

(iv). A scalar quantity is one that is defined by magnitude alone and can be
represented by a real number, i.e.  

2.2.1 Vector Operation

n
Let u , v and w be vectors in (the plane), and let k , m

(a) u  v  v u

(b) k  mu    km  u

(c)  k  m u  ku  mu

(d) u   u   0

11
2.2.2 Norm of a vector
The length, or magnitude of a vector u is often called the norm of u and is denoted
by u .

(a) It follows from the theorem of Pythagoras that the norm of a vector u   u1,u 2 

in 2
is u  u12  u 22

(b) The norm of a vector u   u1, u 2 , u 3  in 3

is u  u12  u 22  u 32

(c) If v  1 , then the vector v is called a unit vector

2.2.3 Euclidean inner product

n
If u and v are vectors in then the dot(scalar) product or Euclidean inner
product is a scalar quantity and is denoted by
u  v  u1v1  u2v2   unvn

n
More formally: If u and v are vectors in and  is the angle between u and v,
then the dot product is defined by

 u v cos if u  0 and v  0
uv  
0 or u  0 or v  0

Perpendicular (orthogonal) Vectors

If u and v are perpendicular then  = 2 thus, u  v  u v cos 2  0  

12
2.2.4 Orthogonal Projection
At times it is of interest to “decompose” a vector u into sum of two terms: one parallel
to a specified nonzero vector a and the other perpendicular to a .

Gram-Schmidt Process
Given linearly independent vectors, x1, x2 ,..., xk , there exist mutually perpendicular
vectors u1, u2 ,..., uk with the same linear span. These may be constructed sequentially
by setting:

u1  x1

x2T u1
u2  x2  T u1
u1 u1

xkT u1 xkT uk 1
uk  xk  T u1   T uk 1
u1 u1 uk 1uk 1

uj
We usually convert u ' s to unit length by setting z j 
u Tj u j

k 1
This implies that  
xkT z j z j is the projection of xk on z j and   xkT z j  z j is the
j 1

project of xk on the linear span of x1, x2 ,..., xk 1

13
 Example 2.2.1
 4 3 
   
 0 1
Construct perpendicular vectors from x1  and x2   
0 0 
   
 2  1 

2.2.5 Linear Independence

A set of vectors x1, x2 , , xk is linearly independent if and only if the coefficients
satisfying the vector equation

a1 x1  a2 x2   ak xk  0

are a1  0, a2  0, , ak  0

2.2.6 Eigenvalues and Eigenvectors

If A is an n  n matrix, then a nonzero vector x in n
is called an eigenvector of
A if Ax is a scalar multiple of x , that is, Ax   x for some scalar 

The scalar  is called an eigenvalue of A , and x is said to be an eigenvector

corresponding to 

14
Eigenvalue Calculation
If A is an n  n matrix, then  is an eigenvalue of A if and only if it satisfies the
equation
det   I  A  0 This is called the characteristic equation of A

Example 2.2.2
3 0 
Given the matrix A 
8 1

Find the eigenvalues of A

Eigenvectors Calculation
By definition, the eigenvectors of A corresponding to an eigenvalue of  are the
nonzero vectors that satisfy
  I  A x  0

Example 2.2.3
 1 3 
Find the eigenvectors of the matrix A  
2 0 

15
2.3 Quadratic Forms
The study of the variation and interrelationships in multivariate data is often based
upon distances and the assumption that the data are multivariate normally distributed.

Squared distances and the multivariate normal density can be expressed in terms of
matrix products called quadratic forms.

If A is a symmetric n  n matrix and x is an n1 column vector of variables, then

we call the function
QA  x   xT Ax is called a quadratic form associated with A

T
If the case where A , is a diagonal matrix, the quadratic form x Ax has no cross
product terms, then
1 0 0   x1 
0  0   x2 
xT Ax   x1 x2 xn   2
  
  
0 0 n   xn 

T
A quadratic form x Ax is said to be:-

if x Ax  0 for x  0
T
(i). positive definite

if x Ax  0 for x  0
T
(ii). negative definite

T
(iii). Indefinite if x Ax has both positive and negative values.

16
If A is a symmetric matrix, then:
(a) xT Ax is positive if and only if all eigenvalues of A are positive

(b) xT Ax is negative definite if and only if all eigenvalues of A are negative

(c) xT Ax is indefinite if and only if A has at least one positive eigenvalue and
at least one negative eigenvalue

Also a quadratic form for which:

(a) xT Ax  0 if x  0 is called positive semi-definite

(b) xT Ax  0 if x  0 is called negative semi-definite

Every positive definite form is positive semidefinite, but not conversely; and every
negative definite form is negative semidefinite, but not conversely.

Example 2.3.1
Express the quadratic forms in matrix notations 2 x 2  6 xy  5 y 2

Example 2.3.2
(a) Express the quadratic forms in matrix notations x12  2 x1 x2  x22

(b) Express x12  6 x1x2  x22  4 x2 x3  2 x32 in matrix notation

17
2.4 Decomposition of Matrix
(a) Every symmetric matrix A can be expressed as
A  PDPT (1)

where P is an n  n orthogonal matrix of eigenvectors of A , and D is the

diagonal matrix whose diagonal entries are the eigenvalues corresponding to the
column vectors of P . Equation (1) is called the Eigenvalue Decomposition.

(b) The Spectral Decomposition of a p  p matrix A is given by

p
A   i ei eiT
i 1

where ei ' s are eigenvectors correspond to i

The spectral decomposition allows us to express the inverse of a square matrix

in terms of its eigenvalues and eigenvectors. This leads to a square-root matrix.

▪ Suppose that the spectral decomposition of A is

p
A   i ei eiT  PDPT
i 1

where P is the orthogonal matrix of eigenvectors and D is a diagonal

matrix with eigenvalues on the diagonal

▪ It can be useful to have a square-root matrix, for a positive definite

matrix, satisfying

A 2   PD 2 PT 
1 1

 

18
(c) Singular Value Decomposition(SVD)
Let A be an m  k matrix of real numbers. Then there exists an m  m
orthogonal matrix U and a k  k orthogonal matrix V such that

A  UDV T

where the m  k matrix D is the diagonal matrix, whose diagonal entries are
the eigenvalues  i  0  for i  1, 2,..., min  m, k  .

The positive constants i , that is, 1  1 ,  2  2 , ….,  n  n

are called the singular values of A

Example 2.4.1
1 2 
Find the spectral decomposition for the matrix A   
 2 2 

Example 2.4.2
1 1 
Find the singular value decomposition of the matrix A   0 1 
1 0 

19
Solution 2.4.2
Here we have and m  n matrix. To let it be symmetric or orthogonal matrix

1 1 
1 0 1  0 1    2 1 
Q  AT A       
1 1 0  1 0  1 2 
 

The characteristic equation:  2  4  3     3   1

Eigenvalues: 1  3 and 2  1

Singular values of A :  1  3 and  2  1  1

 2  2 
Eigenvectors: 1  3 : v1    2  1: v2   
2 2
and
 2  2 
 2   2

Orthonormal for columns of A :

 6
1 1   2  3

3
0 1  
1  6
u1  Av1 
2
1 3   2  6 
1 0   2   6
 6 

 
1 1   2   0 
Av2  1 0 1  
1    2 
u2 
2
2  2   2 
1 0   2   2 
2 

Note that u1 and u2 are orthonormal. Also, we need to find a vector, u3 that is an
unit vector and is orthogonal to u1 and u2 .

20
 2 0 
So 6  u1  1  and 2  u2   1
1  1 

The vector u3 must be a solution of the homogeneous system

 x1 
 2 1 1   0 
 0 1 1  x2   0 
    
 x3 

 x1  1
The general solution is  x   t 1
 2 
 x3  1

 1 
 3
Normalising the vector: u3   1 
 3 
1 
 3 

Hence, the singular value decomposition of A is

 6
0  1 
1 1   3 3 3 1  2 
 
2
0 1    6
 2 1 0 1  2 2

      22 
6 2 3 2
1 0    1 0   2
 
6 2 1
 6 2 3 VT
A D
U

21
2.5 Random Vectors and Matrices
A random vector is a vector whose elements are random variables.

A random matrix is a matrix whose elements are random variables.

The expected value of a random matrix (or vector), is the matrix (vector) consisting
of the expected values of each of its elements.

That is, let X   X ij  be an n  p random matrix

Then, the expected value if they exists, is denoted by

 E  X11  E  X12 


E X1 p 


E  X  E  X 
EX    21 22 
E X2p 


 
 
 E  X n1  E  X n 2  
E X np 

For each element of the matrix


   
  xij fij xij dxij if X ij is a continuous r.v. with pdf fij xij
 

 
E X ij 

 
  xij pij xij  
if X ij is a discrete r.v. with pdf pij xij
 ij
all x


Let X and Y be random matrices of the same dimension, and let A and B be
conformable matrices of constants. Then

E  X  Y   E  X   E Y 
E  AXB   AE  X  B

22
Example 2.5.1
Suppose p  2 and n  1 and the let random vector X T   X1, X 2  be discrete and
have the following probability distribution:

x1 1 0 1
p1  x1  0.3 0.3 0.4
and

x2 0 1
p2  x2  0.8 0.2

(a) For each distribution find the mean

(b) Find E  X  , the mean of the random vector

23
2.6 Mean Vectors and Covariance Matrices
Suppose X T   X1 X2 X T  is a p  1 random vector.

Then, each element of X is a random variable with its own marginal probability
distribution.

The marginal means  i and variances  i2 are defined as

i  E  X i  ,  i2  E  X i  i 
2
i  1, 2,..., p respectively.

The measure of linear association between random variables, X i and X k is given by

the covariance
 ik  E  X i  i  X k  k 

The means and covariances of the p  1 random vector X can be set out as matrices.

The expected value is contained in the vector means   EX 

and the p variances,  ii and the p p1

2
distinct covariances,  ik  i  k 

are contained in the symmetric variance-covariance matrix

  E  X    X   
T

So, the population mean

 E  X 1    1 
   
 E  X    2 
EX   
2
  (2.3.1)
   
 
 E X p    p 

24
population variance-covariance

  E  X    X   
T

 X 1  1  
  
 X 2  2  
 E
 
X   , X  2 ,
 1 1 2
, X p  p  
  
 X p   p  
  



 X 1  1 
2
 X1  1  X 2  2   X1  1   X p   p  
 X  X 
 2 2  1 1   X 2  2 2  X 2  2   X p   p 
 E
 
 

 X  X  
  X p   p   X 2  2   X p  p  
2
 p p 1 1 

11 12 1 p 
 
12  22 2p 
  (2.3.2)
 
1 p  2 p  pp 

 Cov  X 

25
Let the population correlation matrix be p  p symmetric matrix

  11  12 1 p 
 
  11  11  11  22  11  pp 
 2p 
  12  22 
    11  22  22  22  22  pp 
 
 
 
 1 p 2p  pp 
  11  pp  22  pp  pp  pp 
 

 1 12 1 p 
 
 12 1 2 p 
  (2.3.3)
 
 1 p 2 p 1 

And let the p  p standard deviation matrix be

  11 0 0 
 
1  0  22 0 
V2   (2.3.4)
 
 0 0  pp 


1 1
   V 2    V 2 
1 1 1 1
Thus, V 2 V 2   and
   

26
Example 2.6.1
Find the covariance matrix for the two random variables X 1 and X 2 given in example
2.5.1, when their joint probability distribution p x1x2  x1 , x2  is outlined in the table
below

x2
x1 0 1 p1  x1 
1 0.24 0.06 0.3
0 0.16 0.14 0.3
1 0.4 0.0 0.4
p2  x2  0.8 0.2 1

Example 2.6.2
4 1 2 
Suppose Cov  X     1 9 3
 2 3 25 
1
Obtain V 2 and 

27
2.7 Linear Combinations of Random Variables
Recall that in the univariate case, if a, b are scalar, and X and Y are random
variables, then:
▪ E  aX   aE  X 
▪ E  aX  bY   aE  X   bE Y 
▪ Var  aX   a 2Var  X 

▪ Cov  aX , bY   abCov  X , Y 
▪ Var  aX  bY   a 2Var  X   b2Var Y  2abCov  X ,Y 

In the multivariate case: cT X  c1 X1  ....  c p X p

▪  
E cT X  c T E  X   c T 

▪ Var  c X   c  c
T T

In general , consider q linear combinations of the p random variables, X1,..., X p

Z1  c11 X1  c12 X 2   c1 p X p
Z 2  c21 X1  c22 X 2   c2 p X p

Z q  cq1 X 1  cq 2 X 2   cqp X p

Alternate
 Z1   c11 c12 c1 p   X 1 
 Z  c 
c2 p   X 2 
 21 c22
Z  
2
      CX
    
 Z q   cq1 cq 2 cqp   X q 
q1 q p p1

28
 ▪ The linear combinations Z  CX we have

Z  E  Z   E  CX   C  X

Z  Cov  Z   Cov  CX   C X CT

Example 2.7.1
Find the mean vector and covariance matrix for the linear combinations

Z1  X1  X 2 and Z 2  X1  X 2
where
X1 ~ N  1,11  and X 2 ~ N  2 , 22 

29