Finite Sample Properties of Least Squares

Sisir Debnath
Indian Institute of Technology Delhi

September 3, 2021
Introduction

Estimates and estimators

Properties of an estimator - the sampling distribution

“Finite sample” properties as opposed to “asymptotic” or “large sample”

properties

HSL719 2021-22 IITD Sisir Debnath

Application: Health Care Panel Data

German health care usage data.

7,293 Individuals.
Varying numbers of periods.
Data downloaded from Journal of Applied Econometrics Archive.
There are altogether 27,326 observations.
The number of observations ranges from 1 to 7. (Frequencies are: 1=1525, 2=2158,
3=825, 4=926, 5=1051, 6=1000, 7=987).
Variables description
DOCVIS number of doctor visits in last three months
HOSPVIS number of hospital visits in last calendar year
DOCTOR 1(Number of doctor visits > 0)
HOSPITAL 1(Number of hospital visits > 0)
HSAT health satisfaction, coded 0 (low) - 10 (high)
PUBLIC insured in public health insurance = 1; otherwise = 0
ADDON insured by add-on insurance = 1; otherswise = 0
HHNINC household nominal monthly net income in German marks / 10000.
(4 observations with income=0 were dropped)
HHKIDS children under age 16 in the household = 1; otherwise = 0
EDUC years of schooling
AGE age in years
MARRIED marital status
For now, treat this sample as if it were a cross section, and as if it were the full
population.
HSL719 2021-22 IITD Sisir Debnath
Application: Health Care Panel Data, Population Regression
. reg hhninc educ

Source SS df MS Number of obs = 27,326

F(1, 27324) = 2019.63
Model 58.8590627 1 58.8590627 Prob > F = 0.0000
Residual 796.318636 27,324 .02914356 R-squared = 0.0688
Adj R-squared = 0.0688
Total 855.177698 27,325 .031296531 Root MSE = .17071

hhninc Coef. Std. Err. t P>|t| [95% Conf. Interval]

educ .019963 .0004442 44.94 0.000 .0190923 .0208336

_cons .1260903 .0051337 24.56 0.000 .116028 .1361526

HSL719 2021-22 IITD Sisir Debnath

Application: A Sampling Experiment

A sampling experiment:
Draw 30 observations at random from the population. Compute the regression. Repeat
100 times. Display estimates.

Stata Program:
set seed 2038947
mat M = J(100,1,.)
forvalues i = 1(1)100{
insheet using healthcare.csv, comma clear
sample 30
reg hhninc educ
matrix M[‘i’,1]= b[educ]
}
svmat M, names(b educ)
save b educ, replace
#delimit;
twoway (hist b educ, xline(.019963, lc(red) lw(thick)) bin(40))
(kdensity b edu, xtitle(‘‘Coefficient on Education")
ytitle(‘‘Frequency")
legend(label(1 ‘‘Density") label(2
‘‘Distribution of b")));
#delimit cr

HSL719 2021-22 IITD Sisir Debnath

Application: A Sampling Experiment

1000
Frequency
500 0

.018 .019 .02 .021 .022

Coefficient on Education

Density Distribution of b

HSL719 2021-22 IITD Sisir Debnath

Motivating Least Square

The sample of data from the population: Data generating process is y = Xβ + ϵ.

The stochastic specification of the regression model: Assumptions about the

random ϵ.

Endowment of the stochastic properties of the model upon the least squares
estimator. The estimator is a function of the observed (realized) data.

−1
b = X′X X′y

−1
= X′X X ′ (Xβ + ϵ)

−1 ′
= β + X′X X ϵ
|{z} | {z }
The true parameter Sampling error

HSL719 2021-22 IITD Sisir Debnath

Properties of b

Therefore, b is a vector of random variables.

The assumption of nonstochastic regressors. How it is used at this point.

We do the analysis conditional on an X, then show that results do not depend on

the particular X in hand, so the result must be general – i.e., independent of X.

Properties of a Least Square Estimator (b):

b is a linear.
b is unbiased.
b is the most efficient (best) linear estimator.
b is consistent (will discuss this property later).

HSL719 2021-22 IITD Sisir Debnath

Linearity of b

−1
b = X′X X′y

−1
= X′X X ′ (Xβ + ϵ)

−1 ′
= β + X′X X ϵ
|{z} | {z }
The true parameter Sampling error
n
= β + ∑ vi ϵi
i=1
|{z}
The true parameter (constant) | {z }
Linear function of the errors

HSL719 2021-22 IITD Sisir Debnath

Unbiasedness of b

Crucial assumption: ϵi is uncorrelated to X

−1
b = X′X X′y

−1
= X′X X ′ (Xβ + ϵ)

−1 ′
′
= β+ X X X ϵ

Now take expectations iterating over X

−1
E[b|X ] = β + E[ X ′ X X ′ ϵ]
=β
Therefore,
E[b] = EX (E[b|X ])
=β

HSL719 2021-22 IITD Sisir Debnath

Efficiency of b
Crucial assumption: ϵi has zero mean and it is is uncorrelated to every other ϵj .
Var[ϵi |X ] = σ2 .
Var[ϵ|X ] = σ2 I .

−1 ′
b = β + X′X X ϵ
′

−1 ′
b−β = X X X ϵ

Var[b|X ] = E[(b − β)(b − β)′ |X ]

−1 ′
−1 ′ ′
= E[( X ′ X X ϵ)( X ′ X X ϵ ) |X ]
′

−1 ′ ′ ′

−1
= XX X E[ϵϵ |X ]X X X

−1 ′ 2
−1
= X′X X σ IX X ′ X

−1 ′
−1
= σ2 X ′ X X X X′X

−1
= σ2 X ′ X

Var[b] = E (Var[b|X ]) + Var[E[b|X ]]

−1
= σ2 E X ′ X + Var[ β]

−1
= σ2 E X ′ X +0

HSL719 2021-22 IITD Sisir Debnath

Efficiency of b
An estimator is efficient if it is the minimum variance unbiased estimator.
How do we know that b is the most efficient estimator of β
The Cramer-Rao inequality provides verification of efficiency, since it establishes
the lower bound for the variance-covariance matrix of any unbiased estimator.
This lower bound is given by the corresponding element of the diagonal of the
inverse of the information matrix (or sample information matrix) In (θ ), which is
defined as:
In (θ ) = −E[H (θ )]
where H denotes the hessian matrix, i.e., the matrix of the second partial
derivatives of the log-likelihood function (?).
We will come back to this later. For time being let’s consider a simpler proof.
Let, b0 = Cy be another linear unbiased estimator of β
This immediately implies CX = I (prove this)
−1
C being a matrix of constants we can write C = A + D where A = (X ′ X ) X′.

Var[b0 ] = E[(b0 − β)(b0 − β)′ ]

= Var[C(Xβ + ϵ)]
= E[Cϵϵ′ C′ ]
= σ2 CC′
HSL719 2021-22 IITD Sisir Debnath
Efficiency of b

CC′ = (A + D)(A′ + D′ )
= (AA′ + DA′ + AD′ + DD′ )

I = CX
I = (A + D)X
I = AX + DX

−1 ′
I = X′X X X + DX
I = I + DX
DX = 0


−1 ′
DA′ = D X′X X′

=0
= AD′

−1
Therefore, CC′ = (AA′ + DD′ ) = (X ′ X ) + DD′
HSL719 2021-22 IITD Sisir Debnath
Efficiency of b

Var[b0 ] = σ2 CC′

−1 
= σ2 X ′ X + DD′
= Var[b] + σ2 DD′

HSL719 2021-22 IITD Sisir Debnath

Gauss-Markov Theorem

Gauss-Markov Theorem:

In the classical linear regression model with regressor matrix X, the least square
estimator b is the minimum variance linear unbiased estimator of β. For any vector of
constants w, the minimum variance linear unbiased estimator of w′ β is given by w′ b,
where b is the least square estimator.

HSL719 2021-22 IITD Sisir Debnath