Multivariate Data Analysis

Lecture 4

School of Mathematics
Univariate Normal Distribution

• For x ⇠ N(µ, 2
) the density function is
✓ ◆2 !
1 1 x µ
f (x) = p exp
2⇡ 2

1
Univariate Normal Distribution

• For x ⇠ N(µ, 2
) the density function is
✓ ◆2 !
1 1 x µ
f (x) = p exp
2⇡ 2

x µ 2
• Write = (x µ)( 2
) 1
(x µ)

1
Univariate Normal Distribution

• For x ⇠ N(µ, 2
) the density function is
✓ ◆2 !
1 1 x µ
f (x) = p exp
2⇡ 2

x µ 2
• Write = (x µ)( 2
) 1
(x µ)
• For generalised vector x this is

(x µ)0 ⌃ 1
(x µ)

1
Mahalanobis distance

• The Mahalanobis distance is

2
= (x µ)0 ⌃ 1
(x µ)

2
Mahalanobis distance

• The Mahalanobis distance is

2
= (x µ)0 ⌃ 1
(x µ)

• Also written as
2
= dM (x, µ)

2
Multivariate Normal Distribution

0
Definition 6.1 A random vector x = (x1 , . . . , xp ) follows a multivariate
normal distribution (MVN; or a p-dimensional normal distribution) of
parameters µ and ⌃, x ⇠ Np (µ, ⌃), if its density function is

⇢
1 1
f (x) = exp (x µ)0 ⌃ 1
(x µ)
(2⇡)p/2 |⌃|1/2 2

3
Multivariate Normal Distribution

0
Definition 6.1 A random vector x = (x1 , . . . , xp ) follows a multivariate
normal distribution (MVN; or a p-dimensional normal distribution) of
parameters µ and ⌃, x ⇠ Np (µ, ⌃), if its density function is

⇢
1 1
f (x) = exp (x µ)0 ⌃ 1
(x µ)
(2⇡)p/2 |⌃|1/2 2

• E [x] = µ and cov (x) = ⌃, with µ 2 Rp and ⌃ being a symmetric

square positive definite matrix.
• Here increases with p

3
Multivariate Normal Distribution

0
Definition 6.1 A random vector x = (x1 , . . . , xp ) follows a multivariate
normal distribution (MVN; or a p-dimensional normal distribution) of
parameters µ and ⌃, x ⇠ Np (µ, ⌃), if its density function is

⇢
1 1
f (x) = exp (x µ)0 ⌃ 1
(x µ)
(2⇡)p/2 |⌃|1/2 2

• E [x] = µ and cov (x) = ⌃, with µ 2 Rp and ⌃ being a symmetric

square positive definite matrix.
• Here increases with p
p
• |⌃| 1/2
is an analogue to 2 in the univariate case

3
Multivariate Normal Distribution

0
Definition 6.1 A random vector x = (x1 , . . . , xp ) follows a multivariate
normal distribution (MVN; or a p-dimensional normal distribution) of
parameters µ and ⌃, x ⇠ Np (µ, ⌃), if its density function is

⇢
1 1
f (x) = exp (x µ)0 ⌃ 1
(x µ)
(2⇡)p/2 |⌃|1/2 2

• E [x] = µ and cov (x) = ⌃, with µ 2 Rp and ⌃ being a symmetric

square positive definite matrix.
• Here increases with p
p
• |⌃| 1/2
is an analogue to 2 in the univariate case
• |⌃|

3
Generalised Population Variance

• |⌃| is the generalised population variance

4
Generalised Population Variance

• |⌃| is the generalised population variance

p
• |⌃| 1/2
is an analogue to 2 in the univariate case

4
Generalised Population Variance

• |⌃| is the generalised population variance

p
• |⌃| 1/2
is an analogue to 2 in the univariate case

• Multicollinearity indicates variables are highly intercorrelated:

4
Generalised Population Variance

• |⌃| is the generalised population variance

p
• |⌃| 1/2
is an analogue to 2 in the univariate case

• Multicollinearity indicates variables are highly intercorrelated:

• Eigenvalues of ⌃ close to 0
• |⌃| will be small

4
Example:

Consider the bivariate normal distribution:

5
Example:

Consider the bivariate normal distribution:

5
Example:

Consider the bivariate normal distribution:

Left: large |⌃|

Right: small |⌃|

5
Example: Contour plots

6
Example: Contour plots

Left: large |⌃|

Right: small |⌃|

6
Example: Finding contours

https://datasciencegenie.com/3d-contour-plots-of-bivariate-normal-distribution/

7
MVN Properties:

Normality of Linear combinations:

1. If a is a vector of constants (p ⇥ 1) so the rank(a) = 1,

y = a1 x1 + . . . + ap xp of the components of x ⇠ Np (µ, ⌃) follows a
univariate normal distribution.

8
MVN Properties:

Normality of Linear combinations:

1. If a is a vector of constants (p ⇥ 1) so the rank(a) = 1,

y = a1 x1 + . . . + ap xp of the components of x ⇠ Np (µ, ⌃) follows a
univariate normal distribution.
i.e.
If x is Np (µ, ⌃), then a0 x is N (a0 µ, a0 ⌃a) .

8
MVN Properties:

Normality of Linear combinations:

1. If a is a vector of constants (p ⇥ 1) so the rank(a) = 1,

y = a1 x1 + . . . + ap xp of the components of x ⇠ Np (µ, ⌃) follows a
univariate normal distribution.
i.e.
If x is Np (µ, ⌃), then a0 x is N (a0 µ, a0 ⌃a) .

2. If yq⇥1 = Aq⇥p xp⇥1 , being x ⇠ Np (µ, ⌃) and rank(A) = q, q  p;

then
y ⇠ Nq (Aµ, A⌃A0 ) .

8
MVN Properties (cont.):

Standardized variables:

1. Any p-dimensional normal random vector x ⇠ Np (µ, ⌃) can be

transformed into a standard normal random vector z with mean
vector 0p and covariance matrix Ip by the following transformation:

1/2
z=⌃ (x µ) ⇠ Np (0, I)

9
MVN Properties (cont.):

Chi-square distribution:

• Recall 2p is a chi-square random variable with p degrees of freedom

is defined as the sum of squares of p independent standard normal
random variables.
• If z is ⇠ Np (0, I) then
p
X
zj2 = z0 z
j=1

10
MVN Properties (cont.):

Chi-square distribution:

• Recall 2p is a chi-square random variable with p degrees of freedom

is defined as the sum of squares of p independent standard normal
random variables.
• If z is ⇠ Np (0, I) then
p
X
zj2 = z0 z
j=1

And it holds that z0 z = (x µ)0 ⌃ 1

(x µ) ⇠ 2
p.

10
MVN Properties (cont.):

Normality of MVN distributions

1. All marginal and conditional distributions are also normal

distributions.

11
MVN Properties (cont.):

Normality of MVN distributions

1. All marginal and conditional distributions are also normal

distributions.
2. Any subvector of x of dimension k  p follows a k-dimensional
normal distribution:
! ! !
x1 µ1 ⌃11 ⌃12
x= , µ= , ⌃=
x2 µ2 ⌃21 ⌃22

11
MVN Properties (cont.):

Normality of MVN distributions

1. All marginal and conditional distributions are also normal

distributions.
2. Any subvector of x of dimension k  p follows a k-dimensional
normal distribution:
! ! !
x1 µ1 ⌃11 ⌃12
x= , µ= , ⌃=
x2 µ2 ⌃21 ⌃22

Here x1 and µ1 are r ⇥ 1 and ⌃11 is r ⇥ r where r < p

11
MVN Properties (cont.):

Normality of MVN distributions

1. All marginal and conditional distributions are also normal

distributions.
2. Any subvector of x of dimension k  p follows a k-dimensional
normal distribution:
! ! !
x1 µ1 ⌃11 ⌃12
x= , µ= , ⌃=
x2 µ2 ⌃21 ⌃22

Here x1 and µ1 are r ⇥ 1 and ⌃11 is r ⇥ r where r < p

x1 ⇠ Nr (µ1 , ⌃11 )

11
MVN Properties (cont.):

Partition the observations into x(p⇥1) and y(q⇥1) , then

! ! ! !
x µx x ⌃xx ⌃xy
E = , cov =
y µy y ⌃yx ⌃yy

12
MVN Properties (cont.):

Partition the observations into x(p⇥1) and y(q⇥1) , then

! ! ! !
x µx x ⌃xx ⌃xy
E = , cov =
y µy y ⌃yx ⌃yy
For the next three slides we assume
! " ! !#
x µx ⌃xx ⌃xy
is Np+q ,
y µy ⌃yx ⌃yy

12
MVN Properties (cont.):

Independence:

1. x and y are independent if ⌃xy = 0

13
MVN Properties (cont.):

Independence:

1. x and y are independent if ⌃xy = 0

2. Two components xi and xj are independent if ij =0

13
MVN Properties (cont.):

Independence:

1. x and y are independent if ⌃xy = 0

2. Two components xi and xj are independent if ij =0
3. If the components xi of x are all uncorrelated to each other
(⇢ = Ip ), then they are independent. That is, the joint density
function can be obtained as product of the marginals:
p
Y
xi uncorrelated normal variables , f (x1 , . . . , xp ) = f (xi ) .
i=1

13
MVN Properties (cont.):

Conditional distributions:

1. If x and y are not independent, then ⌃xy 6= 0

then the conditional distribution f (x|y) is multivariate normal with

14
MVN Properties (cont.):

Conditional distributions:

1. If x and y are not independent, then ⌃xy 6= 0

then the conditional distribution f (x|y) is multivariate normal with

E (x | y) = µx + ⌃xy ⌃yy1 y µy
cov(x | y) = ⌃xx ⌃xy ⌃yy1 ⌃yx

Distribution of sum of two subvectors:

14
MVN Properties (cont.):

Conditional distributions:

1. If x and y are not independent, then ⌃xy 6= 0

then the conditional distribution f (x|y) is multivariate normal with

E (x | y) = µx + ⌃xy ⌃yy1 y µy
cov(x | y) = ⌃xx ⌃xy ⌃yy1 ⌃yx

Distribution of sum of two subvectors:

1. If x and y are the same size (both p ⇥ 1) and independent

14
MVN Properties (cont.):

Conditional distributions:

1. If x and y are not independent, then ⌃xy 6= 0

then the conditional distribution f (x|y) is multivariate normal with

E (x | y) = µx + ⌃xy ⌃yy1 y µy
cov(x | y) = ⌃xx ⌃xy ⌃yy1 ⌃yx

Distribution of sum of two subvectors:

1. If x and y are the same size (both p ⇥ 1) and independent

then
x + y is Np µx + µy , ⌃xx + ⌃yy
x y is Np µx µy , ⌃xx + ⌃yy

14
Bivariate Normal Distribution

! !!
2
0 µx x xy
Given a random vector (x, y) ⇠ N , 2
µy yx y

15
Bivariate Normal Distribution

! !!
2
0 µx x xy
Given a random vector (x, y) ⇠ N , 2
µy yx y

15
Joint density function

1. Joint density function:

✓ ◆
1 1
f (x, y ) = p exp Q(x, y ) ,
2⇡ x y 1 ⇢2 2

16
Joint density function

1. Joint density function:

✓ ◆
1 1
f (x, y ) = p exp Q(x, y ) ,
2⇡ x y 1 ⇢2 2
where

"✓ ◆2 ✓ ◆2 #
1 x µx y µy (x µx ) (y µy )
Q(x, y ) = + 2⇢
1 ⇢2 x y x y

16
Marginal density functions

2. Marginal density functions:

( ✓ ◆2 )
1 1 x µx
fx (x) = p exp
2⇡ x 2 x

and ( ✓ ◆2 )
1 1 y µy
fy (y ) = p exp .
2⇡ y 2 y

(i.e. univariate normal distributions)

17
Independence

3. If ⇢ = 0, that is, if x and y independent, then

18
Independence

3. If ⇢ = 0, that is, if x and y independent, then

✓ ◆2 ✓ ◆2
x µx y µy
Q(x, y ) = + ,
x y

18
Independence

3. If ⇢ = 0, that is, if x and y independent, then

✓ ◆2 ✓ ◆2
x µx y µy
Q(x, y ) = + ,
x y

and, consequently, f (x, y ) = fx (x)fy (y ).

18
Conditional Distributions

4. If ⇢ 6= 0, the conditional distributions can be obtained as:

8 !2 9
1 < 1 y µy ⇢ yx (x µx ) =
fy|x (y | x) = p p exp p
2⇡ y 1 ⇢ 2 : 2 y 1 ⇢2 ;

19
Conditional Distributions

4. If ⇢ 6= 0, the conditional distributions can be obtained as:

8 !2 9
1 < 1 y µy ⇢ yx (x µx ) =
fy|x (y | x) = p p exp p
2⇡ y 1 ⇢ 2 : 2 y 1 ⇢2 ;

and
8 !2 9
1 < 1 x µx ⇢ yx (y µy ) =
fx|y (x | y ) = p p exp p .
2⇡ x 1 ⇢ 2 : 2 x 1 ⇢2 ;

19
Conditional Distributions

4. If ⇢ 6= 0, the conditional distributions can be obtained as:

8 !2 9
1 < 1 y µy ⇢ yx (x µx ) =
fy|x (y | x) = p p exp p
2⇡ y 1 ⇢ 2 : 2 y 1 ⇢2 ;

and
8 !2 9
1 < 1 x µx ⇢ yx (y µy ) =
fx|y (x | y ) = p p exp p .
2⇡ x 1 ⇢ 2 : 2 x 1 ⇢2 ;

Thus
✓ ◆
y 2
y | x = x ⇠ N µy + ⇢ (x µx ) , y 1 ⇢2
x

19
Conditional Distributions

4. If ⇢ 6= 0, the conditional distributions can be obtained as:

8 !2 9
1 < 1 y µy ⇢ yx (x µx ) =
fy|x (y | x) = p p exp p
2⇡ y 1 ⇢ 2 : 2 y 1 ⇢2 ;

and
8 !2 9
1 < 1 x µx ⇢ yx (y µy ) =
fx|y (x | y ) = p p exp p .
2⇡ x 1 ⇢ 2 : 2 x 1 ⇢2 ;

Thus
✓ ◆
y 2
y | x = x ⇠ N µy + ⇢ (x µx ) , y 1 ⇢2
x
and
✓ ◆
x 2
x | y = y ⇠ N µx + ⇢ (y µy ) , x 1 ⇢2
y
19