Part IIA Paper 3 Econometrics

Lecture 0
Revision of Part I Statistics

Oleg I. Kitov
oik22@cam.ac.uk

Faculty of Economics and Selwyn College

Michaelmas Term 2021

1/19
Outline


I Random sampling, iid random variables Xi ∼ iid µ, σ 2 and estimation
I Properties of estimator θ̂ for parameter θ:
 
- unbiasedness: E θ̂ = θ,
- consistency: θ̂ → θ as n → ∞,



- efficiency: Var θ̂ ≤ Var θ̃ then θ̂ is more efficient,
a

- asymptotic normality: θ̂ ∼ N θ, Sθ̂2 as n → ∞.


I Asymptotic results for Xi ∼ iid µ, σ 2 as n → ∞:
−1
Pn
- law of large numbers (LLN): X̄n = n i=1
Xi −→ E [Xi ] = µ.
 
σ2
−1
Pn
- central limit theorem (CLT): X̄n = n X −→ N µ,
i=1 i n
.

I Sample mean X̄n as an estimator of population mean µ:

- sampling distribution and hypothesis testing about population mean.
2
I Sample variance Sn−1 as an estimator of population variance σ 2
- χ2n -distribution and hypothesis testing about the variance
- F -distribution and hypothesis testing about equality two variances
2/19
Random sample

I Random sample: a collection of random variables X1 , . . . , Xn

[1] independent: write Xi ⊥ Xj for i 6= j, implies Cov (Xi , Xj ) = 0
[2] identically distributed: drawn from the same distribution

I Note 1: Cov (Xi , Xj ) = 0 does not imply that Xi and Xj are independent.
I Note 2: Cov (Xi , Xj ) 6= 0 does imply that Xi and Xj are not independent.


I Notation: random sample Xi ∼ iid µ, σ 2 , i = 1, . . . , n.
I Population parameters: mean µ and variance σ 2 .
I Example 1: Bernoulli distribution, write Xi ∼ iid Ber(p).


I Example 2: normal distribution, write Xi ∼ iid N µ, σ 2 .
I Random sample: X = (X1 , . . . , Xn ).
I Observed sample: x = (x1 , . . . , xn ), realization of X = (X1 , . . . , Xn ).

3/19
Estimation


I Consider a random sample Xi ∼ iid µ, σ 2 .
I Want to estimate population parameters µ and σ 2 from sample data.
I Estimator [random variables]: a rule for computing population
parameters, uses random sample (X1 , . . . , Xn ).

n n
1X 2 1 X
2
X̄n = Xi , Sn−1 = Xi − X̄n
n i=1 n − 1 i=1

I Estimate [number]: a particular value of an estimator, uses observed

sample x = (x1 , . . . , xn ).

n n
1X 2 1 X 2
x̄n = xi , sn−1 = (xi − x̄n ) .
n i=1 n − 1 i=1

I Estimator is a random variable so has a (sampling) distribution.

I Estimate is an observed realization from that distribution. 4/19
X̄n is unbiased for µ

I Definition. Estimator θ̂n is unbiased for parameter θ, if E[θ̂n ] = θ.

Pn
I Estimator X̄n = n−1 i=1 Xi is unbiased for population mean µ.
I Note: for random variables X , Y and constants a, b ∈ R

E [aX + bY ] = aE [X ] + bE [Y ]

5/19
Variance of X̄n


I Show that Var X̄n = σ 2 /n.
I Note: for random variables X , Y and constants a, b ∈ R

Var (aX + bY ) = a2 Var (X ) + b 2 Var (Y ) + 2abCov (X , Y )

6/19
Law of large numbers (LLN)
I LLN informally. For a sample of iid random variables, as the sample size
increases, the sample mean converges (in probability) to the expected
value (population mean).


I LLN formally. Xi ∼ iid µ, σ 2 , i = 1, . . . , n, and E [Xi ] < ∞, as n → ∞:

n
1X
Xi −→ E [Xi ] .
n i=1
1.0

X̄n
µ=0
0.5
X̄n
0.0
-0.5
-1.0

0 200 400 600 800 1000

n 7/19
X̄n is consistent for µ

I Definition. Estimator θ̂n is consistent for θ if θ̂n → θ as n → ∞.

I Estimator X̄n is consistent for µ by LLN, as n → ∞
n
1X LLN
X̄n = Xi −→ E [Xi ] = µ
n i=1

I Note that Var(X̄n ) = σ 2 /n → 0 as n → ∞.

I The distribution of X̄n around µ becomes so narrow that it effectively
converges to a single point.

8/19
Central limit theorem (CLT)
I CLT informally. For a sample of iid random variables, as the sample size
increases, the sample mean approaches (in distribution) a normal
distribution centered around the population mean.


I CLT formally. Xi ∼ iid µ, σ 2 , i = 1, . . . , n, and σ 2 < ∞, as n → ∞,

σ2
 
X̄n −→ N µ,
n
σ2
 
X̄n − µ −→ N 0,
n
X̄n − µ
√ −→ N (0, 1)
σ/ n

I Asymptotic normality: X̄n is approximately normal for large n:

σ2
 
a X̄n − µ a
X̄n ∼ N µ, ⇐⇒ √ ∼ N (0, 1) .
n σ/ n
9/19
Unbiasedness, consistency and asymptotic normality

Illustrating unbiasedness Illustrating consistency

pdf

pdf
µ µ

10/19
t-distribution


I Xi ∼ iid N µ, σ 2 , i = 1, . . . , n, σ 2 unknown, estimated using Sn−1
2
.
I Standardized X̄n follows a t-distribution with n − 1 degrees of freedom:

X̄n − µ
T = √ ∼ tn−1 .
Sn−1 / n

I t-distribution converges to the standard normal, tn → N (0, 1) as n → ∞.

Probability density function Cumulative distribution function

0.4

1.0
t1
t5

0.8
0.3

t10
0.6 N (0, 1)
F (x)
f (x)
0.2

0.4
0.1

0.2
0.0

0.0

-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3
x x
11/19
Sampling distribution of X̄n


Case 1. Xi ∼ iid N µ, σ 2 and σ 2 is known.
I Standardized X¯n is exactly normal:

X̄n − µ
√ ∼ N (0, 1) .
σ/ n


Case 2. Xi ∼ iid N µ, σ 2 and σ 2 is estimated using Sn−1
2
.
¯
I Standardized Xn is exactly t-distributed with n − 1 degrees of freedom:

X̄n − µ
√ ∼ tn−1 .
Sn−1 / n


Case 3. Xi ∼ iid µ, σ 2 and σ 2 is estimated using Sn−1
2
.
¯
I Standardized Xn is approximately normal by CLT:

X̄n − µ a
√ ∼ N (0, 1) .
Sn−1 / n
12/19
Hypothesis testing about population mean


Consider Case 3: Xi ∼ iid µ, σ 2 and σ 2 is estimated using Sn−1
2
.
1. Hypothesis. H0 : µ = µ0 and H1 : µ 6= µ0 , where µ0 ∈ R.
2. Significance level. α ∈ (0, 1).
3. Assume H0 is true. Find test statistic and its distribution under H0 :

X̄n − µ0 a
H0 : µ = µ0 =⇒ T = √ ∼ N (0, 1)
Sn−1 / n

4. Rejection rule. Reject H0 at α if sample test statistic t is |t| > z1− α2 .

5. Computation. Compute sample test statistic from observed sample:

x̄n − µ0
t= √
sn−1 / n

6. Conclusion. Compare t with critical value z1− α2 = Φ−1 1 − α

2 .
13/19
Chi-squared distribution χ2n
I Definition. Let Zi ∼ iid N (0, 1) for i = 1, . . . , n. Then the random
variable U that is defined as the sum of squares of Zi has a chi-squared
distribution with n degrees of freedom (DoF):

n
X
U= Zi2 ∼ χ2n .
i=1

I E [U] = n, expectation of a χ2n random variable is equal to its DoF= n.

14/19
 2
Estimator of variance Sn−1
2
I Estimator Sn−1 of the population variance σ 2 :

n
2 1 X
2
Sn−1 = Xi − X̄n
n − 1 i=1

2
I Sn−1 is unbiased for σ 2 [adjusts for DoF lost via estimation of X̄n ].
2
I Standardized version of Sn−1 has χ2n−1 distribution:

2
(n − 1) Sn−1
∼ χ2n−1
σ2

I Can use this to test hypothesis about σ 2 . Test statistic under H0 : σ 2 = σ02

2
(n − 1) Sn−1
U= ∼ χ2n−1
σ02

I Reject H0 in favour of H1 : σ 2 > σ02 at α if sample statistic u > χ21−α,n−1 .

15/19
Testing equality of population variances
I Two independent normal samples: X1 , . . . , Xn , and Y1 , . . . , Ym .
I Population means µx , µy , variances σx2 , σy2 , sample sizes n and m.
I Unbiased estimators of the population means:
n m
1X 1 X
X̄n = Xi , Ȳm = Yi
n i=1 m i=1

I Unbiased estimators of the population variances:

n m
1 X
2 1 X
2
Sx2 = Xi − X̄n , Sy2 = Yi − Ȳm
n − 1 i=1 m − 1 i=1

I Want to test H0 : σx2 = σy2 . Suppose σx2 = σy2 = σ02 , where σ02 ∈ R.
I Under the null, individual variance estimators are χ2 -distributed:

(n − 1) Sx2 (m − 1) Sy2
Ux = ∼ χ2n−1 , Uy = ∼ χ2m−1 .
σ02 σ02 16/19
F -distribution

I Definition. If U1 ∼ χ2d1 and U2 ∼ χ2d2 such that U1 ⊥U2 , then

U1 /d1
V = ∼ Fd1 ,d2 .
U2 /d2

I Construct an F -distributed test statistic from Ux ∼ χ2n−1 and Uy ∼ χ2m−1 .

17/19
F -test for equality of population variances

I Under H0 : σx2 = σy2 , use F -test statistic V = Sx2 /Sy2 ∼ Fn−1,m−1 .

I Two-sided alternative: H1 : σx2 6= σy2 , reject H0 at α if sample test
statistic v = sx2 /sy2 is such that v < F α2 ,n−1,m−1 or v > F1− α2 ,n−1,m−1 .
I One-sided alternative: suppose sx2 > sy2 and so pick H1 : σx2 > σy2 , reject
H0 at α if v > F1−α,n−1,m−1 .
I Note: will use F -test for joint hypothesis testing in multivariate models!
Probability density function Cumulative distribution
1.5

1.0
F20,5
F50,50

0.8
F2,50
1.0

0.6
F (x)
f (x)

0.4
0.5

0.2
0.0

0.0
0.0 0.5 1.0 1.5 2.0 2.5 3.0 0.0 0.5 1.0 1.5 2.0

18/19
t-distribution formally defined
I Definition. Let Z ∼ N (0, 1) and U ∼ χ2n , Z ⊥ U. Then
Z
T =p ∼ tn .
U/n

I Intuition. tn is a wider version of N (0,1) [fatter tails / less confidence].

I Recall that if Xi ∼ iid N µ, σ 2 and σ 2 is estimated using Sn−12
, then:
X̄n − µ
T = √ ∼ tn−1 .
Sn−1 / n

I Recall the distributions of the standardized estimators of µ and σ 2 :

2
X̄n − µ (n − 1) Sn−1
Z= √ ∼ N (0, 1) , U= ∼ χ2n−1
σ/ n σ2

I Construct T ∼ tn−1 from Z ∼ N (0, 1) and U ∼ χ2n−1 using the definition:

 
 s 2
X̄n − µ  (n − 1) Sn−1

T =p
Z
= √ /  = X̄n −√µ
∼ tn−1
σ/ n σ 2 (n − 1) S / n
U/ (n − 1) n−1 19/19