MATH2142/MATH1309: Lecture Note 3 1

MULTIVARIATE NORMAL DISTRIBUTION

Reference: Johnson & Wichern(2007) Applied Multivariate Statistical

Analysis Chapter 4.

1 Univariate Normal Distribution

We begin with the definition of univariate normal distribuion.

(a) Standard normal distribution N (0, 1). The probability density func-
tion f (z) of a standard normal random variable Z is given by
1 1 2
f (z) = √ e− 2 z for − ∞ < z < ∞
2π
Notation: Z ∼ N (0, 1).

(b) Normal distribution with mean µ and variance σ 2 N (µ, σ 2 ). The proba-
bility density function(pdf) f (x : µ, σ 2 ) of the normal random variable
with mean µ and variance σ 2 is
2
1 1 1 (x−µ)
f (x : µ, σ 2 ) = (2π)− 2 (σ 2 )− 2 e− 2 σ2 for − ∞ < x < ∞

Notation: X ∼ N (µ, σ 2 ).
Note: X = σZ + µ.

2 Multivariate Normal Distribution

Let Y T = (y1 y2 . . . yp ) be a random vector of p variate with mean vector

µ and covariance matrix σ. Now we can obtain the pdf of Y replacing x, µ
and σ 2 in f (x : µ, σ 2 ) respectively by vectors y, vector µ and matrix Σ.
p
fp (y : µ, Σ) = (2π)− 2 |Σ|− 2 e− 2 (y −µ) y −µ) for y ∈ <p
1 1 T Σ−1 (

where <p is the p-dimensional real space. Here fp (y : µ, Σ) is called multi-

variate normal probability density function.
Notation: Y ∼ Np (µ, Σ).
2 MATH2142/MATH1309: Lecture Note 3

Special Cases

(a) If Y has Np (µ, Σ), then the distribution of Z = (Y − µ) is Np (0, Σ).

(b) If Σ = diag( σ11 σ22 . . . σpp ) then Y1 , Y2 , . . . , Yp are independent

normal random variables and

fp (y : µ, Σ) = f (y1 : µ1 , σ11 )f (y2 : µ2 , σ22 ) . . . f (yp : µp , σpp )

(c) if µ = 0 and Σ = I, then Y1 , Y2 , . . . , Yp , are independent identically

distributed (standard) normal random variables.

(d) if Z = A−1 (Y − µ) where A is a nonsingular matrix such that AAT =

Σ then Z ∼ Np (0, I). Note that Y = µ + AZ.

(e) Bivariate Normal Distribuion.

Consider a bivariate normal random vector X. That is,

X ∼ N2 (µ, Σ)

where    
µ1 σ11 σ12
µ=  and Σ =  .
µ2 σ21 σ22
Excercise: Compute the inverse of Σ and write down the probability
density function of bivariate normal distribution.

3 Properties of the Multivariate Normal Distribu-

tion

(a) Suppose X ∼ Np (µ, Σ), we have E(X) = µ, Cov(X) = Σ.

(b) Consider the transformations, Z = AX + b, where Ap×p is a constant

matrix,bq×1 is a constant vector, then

Z ∼ Nq (Aµ + b, AΣAT ).
MATH2142/MATH1309: Lecture Note 3 3

(c) Sums of independent random vectors:

Let X 1 , X 2 , . . . , X k , be independent random vectors with distribuion
X j ∼ Np (µj , Σ) for j = 1, 2, . . . , k and a1 , a2 , . . . , ak are real numbers.
Define the random vector Y such that
k
X
Y = a1 X 1 + a2 X 2 + . . . + ak X k = aj X j ,
j=1

then  
k
X k
X
Y ∼ Np  aj µj , a2j Σ
j=1 j=1

(d) Relation of the Multivariate Normal Distribution to χ2 Distribuion:

For the univariate case, X ∼ N1 (µ, σ 2 ), where σ > 0, we know that
 2
2 X −µ
Z = ∼ χ21 .
σ

Similarly, suppose Σ is nonsingular and the random variable Y is given

by
Y = (X − µ)T Σ−1 (X − µ)

then Y ∼ χ2p .
Proof: Since Σ is positive deinite and symmetric, there exist a non-
singular matrix B such that Σ = BB T .
Let Z be the random vector defined by Z = B −1 (X − µ) then

Y = (X − µ)T Σ−1 (X − µ)

= (X − µ)T (BB T )−1 (X − µ)

= (X − µ)T (B T )−1 (B)−1 (X − µ)

= (B −1 (X − µ))T (B −1 (X − µ))

= Z T Z = Z12 + Z22 + . . . + Zp2

where Z = ( Z1 Z2 . . . Zp ).
Since Z = B −1 (X − µ), E(Z) = 0 and
¯
4 MATH2142/MATH1309: Lecture Note 3

Cov(Z) = Cov(B −1 (X − µ)) = B −1 Cov(X − µ)(B −1 )T

= B −1 Cov(X)(B −1 )T = B −1 Σ(B −1 )T

= B −1 (BB T )(B −1 ) = (B −1 B)(B T (B −1 )T ) = I.

Hence, Z ∼ Np (0, I) and thus Z1 , Z2 , . . . , Zp , are i.i.d N (0, 1). There-

fore by univariate result Y = pj=1 Zj2 ∼ χ2p .
P

4 Sampling Distributions

Consider a random sample X 1 , X 2 , . . . , X n from a multivariate normal pop-

ulation with mean µ and covariance Σ. Let X be the sample mean vector
and Sn be the sample covariance matrix computed using the above sample.
Then
n n
1X 1 X
Xn = X j and Sn = (X j − X n )(X j − X n )T .
n n−1
j=1 j=1

(a) Distribution of X n .

Since E(X j ) = µ and Cov(X) = Σ,

1
E(X n ) = µ and Cov(X n ) = Σ.
n

Further, since the population distribution is multivariate normal(that

is, X ,j s are multivariate normal random vectors,

1
X n ∼ Np (µ, Σ).
n

(b) Distribuion of Sn .

(n−1)Sn has a Wishart distribution with (n−1) degree of freedom(df).

For more details of the Wishart distribution see Johnson & Wichern
page 174.
MATH2142/MATH1309: Lecture Note 3 5

Note: If Z1 , Z2 , . . . , Zn are iid normal random vectors with E(Z i ) = 0

and Cov(Z i ) = Σ for all i, then ni=1 Z j Z Tj has a Wishart distribu-
P

tion with n df.

(c) Distribution of the Quatratic.

Let Z 2 = n(X n − µ)T Σ−1 (X n − µ). Then Z has a chi-square dis-

tribuion with p df.

(d) Independence of Sample Mean and Covariance Matrix.

X n and Sn are independent. Recall that in univariate case, the sample

mean and the sample variance are independent. This is refer to as
Basu’s theorem.

(e) Hotelling T 2 Distribuion.

The distribuion of the statistics, T 2 where

T 2 = n(X n − µ)T Sn−1 (X n − µ).

is called Hotelling’s T 2 distribution.(For more details of the Hotelling’s

T 2 distribuion see Johnson & Wichern page 212.) Further we can
(n−p) 2
prove that p(n−1) T is distributed as Fp,n−p (F distribution with p
and (n − p)df ).

(f) Mahalanobis Distance

Mahalanobis distance, D2 is a standardised form of Euclidean distance.

Mahalanobis distance of the obervation X i in the sample given above
is
Di2 = (X i − X n )T Sn−1 (X i − X n )

for i = 1, 2, . . . , n.

If both n and n − p are respectively greater than 30 and 25, then

Mahalanobis distances, Di2 (i = 1, 2, . . . , n) are chi-square distributed
with p degree of freedom.
6 MATH2142/MATH1309: Lecture Note 3

5 Assessing the Normality of Multivariate Popu-

lation

Let X be the random vector representing the multivariate population of

interest and
 
X1
 
X2
 
 
X=
 ..



 . 

Xp

Theorem 1: if X is multivariate normal random vector then

(a) all the marginals are normally distributed, that is, Xj for all j, are
normal random variables, and

(b) if a is a constant vector, then

aT X ∼ N (aT µ, aT Σa)

where µ is the mean and Σ is the covariance of X.

Using the above themrem, we can prove the following theorem:

Theorem 2: A random vector X has a p-variate normal distribution if and

only if (iff) aT X is normal for every constant vector a.

Note:

(a) Theorem 2 is not use to prove that X is a normal random vector

because it is difficult to prove that aT X is normal for all a.

(b) Note that if X1 , X2 , . . . , Xp are normal random variables, it does not

imply that X is multivariate normal. Prove this using the Example
4.8 page 202 in Johnson & Wichern.
MATH2142/MATH1309: Lecture Note 3 7

6 Normal Probability Plot

Normal probability plot is also called a Q-Q(quantile quantile) plot.

(a) Univariate Case

Step 1: Arrange the observations X1 , X2 , . . . , Xn in increasing order.

That is,

X(1) ≤ X(2) ≤ . . . ≤ X(n) .

Step 2: Obtain zi from a Normal Table, such that P (Z ≤ zi ) =

(i − 0.5)/n where Z ∼ N (0, 1).

Step 3: Plot the points (zi , X(i) ), i = 1, 2, . . . , n.

If the points approximately follow a straight line, then the observations

are from a normal distribution.

(b) Multivariate Case

Step 1: Compute the Mahalanobis distances, Di2 of all the observa-

tions X 1 , X 2 , . . . , X n ,. Arrange the distances are in increasing order.
That is,
2 2 2
D(1) ≤ D(2) ≤ . . . ≤ D(n) .

Step 2: Obtain ξi from a Chi-Square Table, such that

P(χ2p ≤ ξi ) = (i − 0.5)/n

for i = 1, 2, . . . , n, where χ2p is the chi-square distribution with p de-

grees of freedom.
 
2
Step 3: Plot the points ξi , D(i) , i = 1, 2, . . . , n.

If the points approximately follow a straight line, then the observations

are from a multivariate normal distribution.
8 MATH2142/MATH1309: Lecture Note 3

7 Large Sample Distribution of X n and Sn

Now we assume that the random sample, X 1 , X 2 , . . . , X n , from a unknown

multivariate population with mean µ and covariance Σ.

(a) Law of Large Numbers.

1. X n converges in probability to µ: that is, P(|X n − µ| < ) → 1

as n → ∞ and

2. Sn converges in probability to Σ.

(b) Central Limit Theorem.

If n is large relative to p, then

√
n(X n − µ) ∼ Np (0, Σ).

(c) Important Results.

If the sample covariance matrix is finite and non-singular and n is large

relative to p, then the distribution of
√
• n(X n − µ) is approximately Np (0, Σ) and

• n(X n − µ)T Sn−1 (X n − µ) is approximately χ2p .