STATISTICAL INFERENCE

POPULATION AND SAMPLE

Population = all elements of interest

Characterized by a distribution F
with some parameter θ

Sample = the data X1, . . . , Xn,

selected subset of the population

n = sample size

Examples of F : Bernoulli(p), Normal(µ, σ),

Gamma(n, λ), Poisson(λ), etc.

1
 Statistical Inference

Statistical Inference
= inference about the population based on a
sample

• Parameter estimation

• Confidence intervals

• Hypothesis testing

• Model fitting

2
 Statistical Inference

Parameter Estimation

Statistic = any function of data W (X1, ..., Xn)

Estimator of θ = any statistic used

to estimate parameter θ

Estimator θ̂ is unbiased if E(θ̂) = θ

Standard error of an estimator is its standard

deviation Std(θ̂)

d θ̂). It shows the accu-

It is estimated by Std(
racy, reliability of estimator θ̂.

3
 Parameter estimation

Estimation of a mean

Sample (X1, . . . , Xn) is collected from a popu-

lation with E(X) = µ and Var(X) = σ 2.

Estimate the population mean θ = µ = E(Xi)

by a sample mean
n
1 ∑
X̄ = Xi
n i=1
Properties:
n n
1 ∑ 1 ∑
E(X̄) = EX i = θ=θ
n i=1 n i=1
n
1 ∑ nσ 2 σ2
Var(X̄) = 2 VarXi = 2 =
n i=1 n n

So, X̄ is unbiased, and its standard error is

σ
SE(X̄) = Std(X̄) = √
n
4
 Parameter Estimation

Estimation of a variance

Estimate the population variance

θ = σ 2 = Var(Xi)
by a sample variance
n
1 ∑
S2 = (Xi − X̄)2
n − 1 i=1

It is also unbiased: E(S 2) = σ 2.

Then, the standard error of X̄ is estimated by

v
u∑
S u (X − X̄)2
d X̄) = √
Std( = t i
n n(n − 1)

5
 Parameter estimation

Estimation of a proportion

Sample (X1, . . . , Xn) is collected from Bernoulli

population with parameter p.

Estimate the population proportion p = E(Xi)

by a sample proportion
n
1 ∑ number of Xi = 1
p̂ = Xi =
n i=1 n
Special case of a sample mean X̄
σ2 p(1 − p)
E(p̂) = p, Var(p̂) = =
n n

So, p̂ is unbiased;
√
p(1 − p)
its standard error is SE(p̂) =
n
√
d p̂(1 − p̂)
typically estimated by SE(p̂) =
n
6
 Parameter Estimation

General Methods of Estimation

1. Method of Moments

kth population moment µk = EX k

n
1 ∑
kth sample moment Mk = Xik
n i=1

To estimate d parameters, solve the system of

d equations


 M1 = µ1
...

 M
d = µd

M1, . . . , Md are known from the sample; µ1, . . . , µd

are functions of unknown parameters
7
 Method of moments

Example: X1, . . . , Xn are Exponential(λ)

Estimate λ.

The number of parameters is d = 1. So, we

need 1 equation.

1
M1 = X̄; µ1 =
λ

Solve
1 1
M1 = µ 1 ⇒ X̄ = ⇒ λ̂mom =
λ X̄

8
2. Method of Maximum Likelihood

Maximize the probability (pmf, pdf) of seeing

the really observed data

Implementation

Observe X1, ..., Xn from pdf or pmf f (x | θ).

Maximize
n
∏
f (X1, ..., Xn | θ) = f (Xi | θ)
i=1
in θ.

Simplification: maximize
n
∑
ln f (X1, ..., Xn | θ) = ln f (Xi | θ)
i=1

Typically, compute
n
∑
∂ ∂
ln f (X1, ..., Xn | θ) = ln f (Xi | θ)
∂θ i=1 ∂θ
equate to 0 and solve in θ.
9
 Method of maximum likelihood

Example: X1, . . . , Xn are Exponential(λ)

n
∏
f (X1, ..., Xn | θ) = λe−λXi
i=1

n
∑ ( )
ln f (X1, ..., Xn | θ) = ln λe−λXi
i=1
n
∑
= n ln λ − λ Xi
i=1
n
∑
∂ n
ln f (X1, ..., Xn | θ) = − Xi =: 0
∂λ λ i=1
Solve for λ,
n 1
λ̂mle = ∑ =
Xi X̄

10
 Maximum likelihood

6
- 2h 

f (x) This area

= P {x − h ≤ x ≤ x + h}
≈ (2h)f (x)

-
x−h x x+h

Probability of observing “almost” X = x

11
 Statistical Inference

Confidence Intervals

100 (1−α) %-confidence interval is an inter-

val that contains parameter θ with probability
γ.

That is,

P {a ≤ θ ≤ b} = 1 − α
where

a = a(X1, ..., Xn) and b = b(X1, ..., Xn)

are statistics. So, a and b are random, θ is not.

12
 Confidence intervals

for the same parameter θ



obtained from different




Confidence intervals





samples of data


























-
θ
Confidence intervals and coverage of parameter θ.

13
 Confidence intervals

Example: X1, ..., Xn from Normal(µ, σ) with

unknown µ, known σ

1 ∑
1. Estimate θ = µ by its estimator X̄ = n Xi.

2. Find its distribution: Normal with

E(X̄) = µ
n
1 ∑ nσ 2 σ2
Var(X̄) = 2 Var(Xi) = 2 =
n 1 n n
Therefore,
X̄ − µ
Z= √ is Normal(0,1)
σ/ n

3. Find critical values ±zα/2 such that

{ }
P −zα/2 < Z < zα/2
for Z ∼ Normal(0,1).
14
 Confidence intervals

4. Then we have
{ }
X̄ − µ
P −zα/2 < √ < zα/2 =1−α
σ/ n

Solve for µ:
{ }
zα/2σ zα/2σ
P X̄ − √ < µ < X̄ + √ =1−α
n n

5. Hence,
[ ]
zα/2σ zα/2σ zα/2σ
X̄ ± √ = X̄ − √ , X̄ + √
n n n
is a (1 − α)100% confidence interval for µ.

X̄ is approximately Normal for large n and any

distribution of X1, . . . , Xn.
15
 Confidence intervals

When σ is unknown

Data X1, ..., Xn from Normal(µ, σ) with

unknown µ, unknown σ
v
u n (
u 1 ∑ )
1. Estimate σ by S = t Xi − X̄ 2
n−1 1

2. Use t-distribution with (n − 1) degrees of

freedom instead of Normal.

For large n, use Normal approximation.

Result:
tα/2,n−1S
X̄ ± √
n

16
 TESTING HYPOTHESES

Hypothesis H0 and alternative HA = mutually

exclusive statements about the unknown pa-
rameter θ.

Collect data
⇓
Conduct a test
⇓
State if there is sufficient evidence to reject
H0 in favour of HA.

Conclusion Reject H0 Accept H0

H0 is true Type I error correct
H0 is false correct Type II error

Control the significance level

α = P { Type I error }

17
 Hypotheses testing

Data: X1, ..., Xn from Normal(µ, σ) with

unknown µ, known σ

Test H0 : µ = µ0 vs HA : µ ̸= µ0.

1. Find ±zα/2. Acceptance region: [−zα/2, zα/2].

2. Compute the test statistic

X̄ − µ0
Z= √ .
σ/ n

3. If Z belongs to the acceptance region, do

not reject H0.
Otherwise, reject H0.

If H0 is true, Z has Normal(0,1) distribution,

and
{ }
P { Type I error } = P |Z| > zα/2 = α

18
 Hypotheses testing

One-sided, right-tail tests

Test H0 : µ = µ0 vs HA : µ > µ0.

1. Find zα. The acceptance region is (−∞, zα].

2. Compute the test statistic

X̄ − µ0
Z= √ .
σ/ n

3. If Z belongs to the acceptance region, do

not reject H0.
Otherwise, reject H0.

19
 Hypotheses testing

One-sided, left-tail tests

Test H0 : µ = µ0 vs HA : µ < µ0.

1. Find zα. The acceptance region is [−zα, +∞).

2. Compute the test statistic

X̄ − µ0
Z= √ .
σ/ n

3. If Z belongs to the acceptance region, do

not reject H0.
Otherwise, reject H0.

20
 Hypotheses testing

Case of unknown variance

v
u n (
u 1 ∑ )2
1. Estimate σ by S = t Xi − X̄
n−1 1

2. Use t-distribution with (n − 1) degrees of

freedom.

For large n, use Normal approximation.

21
 Hypotheses testing, Z-tests

Null Parameter,
hypothesis estimator If H0 is true: Test statistic

θ̂ − θ0
H0 θ, θ̂ E(θ̂) Var(θ̂) Z= √
Var(θ̂)

One-sample Z-tests for means and proportions, based on a sample of size n

σ2 X̄ − µ0
µ = µ0 µ, X̄ µ0 √
n σ/ n

p0 (1 − p0 ) p̂ − p0
p = p0 p, p̂ p0 √
n p̂(1−p̂)
n

Two-sample Z-tests comparing means and proportions of two populations,

based on independent samples of size n and m

µX − µY µX − µY , 2
σX σY2 X̄ − Ȳ − D
D + √
=D X̄ − Ȳ n m 2
σX σY2
n
+ m

p1 − p2 p1 − p2 , p1 (1−p1 )
p̂1 − p̂2 − D
D n √
=D p̂1 − p̂2 + m p2 (1−p2 )
p̂1 (1−p̂1 ) p̂2 (1−p̂2 )
n
+ m

22
 Hypothesis testing, t-tests

Hypothesis Test statistic Degrees of

H0 Conditions t freedom

Sample size n; X̄ − µ0
µ = µ0 unknown σ t= √ n−1
s/ n

Sample sizes n, m; X̄ − Ȳ − D
µX − µY = D unknown but equal t= √1 1
n+m−2
σX = σy sp n
+ m

Sample sizes n, m; X̄ − Ȳ − D Special

µX − µY = D unknown, unequal t= √ formula
σX ̸= σy s2X s2Y
n
+ m

23