CHAPTER 3

ESTIMATION: POINT ESTIMATION OF POPULATION

PARAMERTERS

Introduction
Point estimation is concerned with finding a single value which we think best represents the
unknown population parameter. Suppose we want to a value which best represents the
proportion of the Zimbabwean population owing a vehicle. The best value is the sample
proportion. The population variance is best represented by the sample variance.

If we are to make some inference about some population parameters, we have to have some
observations or sample, 𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 , from the population of interest. Then based on these
values we can then find approximate probability distribution which we can then use to address
questions relating to the population of interest. The major aim of having these values is to
obtain a statistics 𝛳̂ = 𝛳̂ (𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 ) which we think is good enough to estimate the
population parameter 𝛳 . Once 𝛳 has been estimated then the underlying probability
distribution can be used for further inference about the population or probability distribution.
Definition: An estimator

Definition: Let 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 be a random sample from a population with probability density

function given by 𝑓(𝑥, 𝛳), i.e. the function depends on the unknown parameter 𝛳. Then the
statistic 𝛳̂ = 𝛳̂ (𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 ) is called an estimator of 𝛳 if it is “close” in some sense to the
true value of 𝛳. A possible value of ̂
𝛳 = 𝛳̂ (𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 ) is called an estimate of 𝛳.

There are several methods used to find these point estimates. The following section deals with
all the methods used to find these estimates.

Methods of Finding Estimates

The following methods are usually used to find estimators of population parameters.

(a) Judgemental method

(b) Method of Moments
(c) Maximum Likelihood Method
(d) Method of Least Squares
Judgemental method

In this method of estimation personal expertise, experience or judgement is used to determine

an estimate 𝛳̂ of the population parameter 𝜃. This method is commonly used in Economics
were personal judgement and experience are required to come up with sound judgements about
the levels of various parameters of an economic system.

Example
If you are asked to estimate the average household income for the population in Harare. What
statistic would you use and why?

Solution
The sample mean is the best estimate. A parameter being estimated is the population mean
therefore the sample mean would be the best natural estimate of 𝜇.

Example
Let 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 be a random sample from the uniform distribution

1
𝑓(𝑥) = , ∝≤ 𝑥 ≤ 𝛽
𝛽−∝

(a) Find estimators of ∝ 𝑎𝑛𝑑 𝛽 .

(b) If the random sample 1.2, 2.4, 3.1, 0.9, 2.0, 1.8, 4.2 is from the uniform distribution, find
the estimates of ∝ 𝑎𝑛𝑑 𝛽.

Solution

(a) Since all values of 𝑋 are equal to or greater than ∝, the most sensible estimator of ∝ is
̂ = 𝑚𝑖𝑛𝑖𝑚𝑢𝑚(𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 ) = 𝑋(1)
∝
Since all values of 𝑋 are equal to or less than 𝛽, the most sensible estimator of 𝛽 is 𝛽̂ =
𝑚𝑎𝑥𝑖𝑚𝑢𝑚(𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 ) = 𝑋(𝑛)

̂ = 𝑚𝑖𝑛𝑖𝑚𝑢𝑚(𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 ) = 0.9 and 𝛽̂ = 𝑚𝑎𝑥𝑖𝑚𝑢𝑚(𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 ) = 4.2

(b) Thus ∝
Method of Moments
This methods is based on both sample and population moments.

Definition: Let 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 be a random sample from a distribution with density function

𝑓(𝑥, 𝜃). The method of moments estimator (MME) of 𝜃 is obtained by equating population
moments to sample moments, i.e.

𝑛
1
𝐸(𝑋) = ∑ 𝑋𝑖
𝑛
𝑖=1

𝑛
1
𝐸(𝑋 2)
= ∑ 𝑋𝑖2
𝑛
𝑖=1

⋮
𝑛
1
𝐸(𝑋 𝑛)
= ∑ 𝑋𝑖𝑛
𝑛
𝑖=1

then solve the resulting equations for 𝜃. The solution(s) to the equations are the estimate(s)
of 𝜃.

The population parameter can be vector or single parameter. One can use moments about zero
or moments about the mean, the results obtained are the same.

Example
Let 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 be a random sample from a distribution with density function

𝑓(𝑥, 𝜔) = 𝜔𝑥 𝜔−1 , 0 < 𝑥 < 1, 𝜔>0

(a) Use the method of moments to find an estimator of 𝜔.

(b) If the sample 0.8, 0.2, 0.1, 0.6, 0.7, 0.5, 0.4, 0.9 and 0.1 is from the population with the
distribution, find the estimate of 𝜔.

Solution
1
(a) 𝐸(𝑋) = ∫0 𝑥. 𝜔𝑥 𝜔−1 𝑑𝑥
1
= ∫ 𝜔𝑥 𝑤 𝑑𝑥
0
𝜔
= [𝑥 𝜔+1 ]10
𝜔+1
 𝜔
= 𝜔+1 is the first population moment

1
𝑋̅ = 𝑛 ∑𝑛𝑖=1 𝑋𝑖 is the first sample moment

Equate the moments and solve for 𝜔, i.e.

𝜔
𝑋̅ = 𝜔+1 solving for 𝜔

𝑋̅
𝜔
̂=
1 − 𝑋̅
1 1
(b) 𝑋̅ = 9 ∑𝑛𝑖=1 𝑋𝑖 = 9 (4.3) = 0.4778

0.4778 0.4778
𝜔
̂= = = 0.9149
1 − 0.4778 0.5222

Exercises
1. Find the method of moments estimators of 𝜃 based on a random sample 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛
from each of the following density functions:
(a) 𝑓(𝑥, 𝜃) = 𝜃𝑥 𝜃−1 , 0 < 𝑥 < 1.
(b) 𝑓(𝑥, 𝜃) = (𝜃 + 1)𝑥 −𝜃−2 , 𝑥 > 1, 𝜃 > 0.
(c) 𝑓(𝑥, 𝜃) = 𝜃 2 𝑥𝑒 −𝜃𝑥 , 𝑥 > 0, 𝜃 > 0.

Maximum Likelihood Method

This is the most common method of estimation parameters. This is based on a function called
the likelihood function.

Definition: Let 𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 be possible values of a random sample 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 from a

population with a density 𝑓(𝑥, 𝜃), then the likelihood function of the observed sample
𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 is defined by

𝐿(𝜃, 𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 ) = 𝑓(𝑥1 , 𝜃)𝑓(𝑥2 , 𝜃) ⋯ 𝑓(𝑥𝑛 , 𝜃) = ∏ 𝑓(𝑥𝑖 , 𝜃)

𝑖=1

The likelihood function is the joint density of 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 it represents the probability of

selecting a sample equal to the observed sample in the case of discrete variable. Since this
probability depends on the parameter 𝜃, we have to chose a value of 𝜃 which maximizes the
probability of obtaining the observed values.
Definition: Let 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 be a random sample from a distribution with density
function 𝑓(𝑥, 𝜃) , an estimator of 𝜃 that maximizes the likelihood function 𝐿(𝜃, 𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 )
is called the Maximum Likelihood Estimator (MLE) for 𝜃.

If the likelihood function is defined on 𝓡𝒏 and if 𝐿(𝜃) is differentiable and assumes a

maximum on 𝓡𝒏 , then the MLE will be a solution of the equation (maximum likelihood
equation)

𝑑 𝐿(𝜃)
=0
𝑑𝜃

If one or more solutions exist, then it should be verified which ones maximises 𝐿(𝜃). The value
of 𝜃 that maximises 𝐿(𝜃) also maximises the log-likelihood, ln 𝐿(𝜃). So for computational
convenience the alternative form of the maximum likelihood equation

𝑑
𝑙𝑛𝐿(𝜃) = 0
𝑑𝜃

can be used.

Example
Let 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 be a random sample from a Poisson distribution

𝑒 −𝜇 𝜇 𝑥
𝑃(𝑥, 𝜇) = , 𝑥 = 1, 2, 3, ⋯
𝑥!

(a) Find the MLE for 𝜇.

(b) Use the observed values 2, 4, 6, 4, 7, 10, 3, 5, 5, 4, 3 and 7 to find an estimate of 𝜇.

Solution
(a) The likelihood function is given by
𝑒 −𝜇 𝜇 𝑥𝑗
𝐿(𝜇) = ∏
𝑥𝑗 !
𝑒 −𝑛𝜇 𝜇 ∑ 𝑥𝑗
=
(∏ 𝑥𝑗 !)

Taking natural logarithms and then differentiate with respect to 𝜇 and set the derivative
to 0, we obtain
∑ 𝑥𝑗
𝑒 −𝑛𝜇 𝜇
𝑙𝑛 𝐿(𝜇) = 𝑙𝑛 ( (∏ 𝑥𝑗 !)
)
= 𝑙𝑛 𝑒 −𝑛𝜇 + 𝑙𝑛 𝜇 ∑ 𝑥𝑗 − 𝑙𝑛 (∏ 𝑥𝑗 !)

= −𝑛𝜇 + ∑ 𝑥𝑗 − 𝑙𝑛 (∏ 𝑥𝑗 !)

= 𝑛𝜇 + 𝑛𝑥̅ 𝑙𝑛𝜇 − 𝑙𝑛 (∏ 𝑥𝑗 !)

𝑑 ln 𝐿(𝜇) 𝑛𝑥̅
= −𝑛 + =0
𝑑𝜇 𝜇

Thus the MLE for 𝜇 𝑖𝑠 𝜇̂ = 𝑋̅

(b) The estimate of 𝜇 is

1
𝜇̂ = 12 (2 + 4 + 6 + 4 + 7 + 10 + 3 + 5 + 5 + 4 + 3 + 7) = 5.

Example
Let 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 be a random sample from an exponential distribution

1 −𝑥⁄
𝑓(𝑥, 𝜃) = 𝑒 𝜃, 0 < 𝑥 < ∞
𝜃
Find the likelihood function 𝐿(𝜃) = 𝐿(𝜃, 𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 ).

Solution

1 𝑥⁄
(a) 𝐿(𝜃) = ∏𝑛𝑗=1 (𝜃 𝑒 − 𝜃)

1 𝑛 − ∑𝑥𝑗⁄
=( ) 𝑒 𝜃
𝜃
Taking natural logarithms differentiate and equate to 0 and solve the resulting equation
for 𝜃
∑𝑥 𝑑𝑙𝑛 𝐿(𝜃) −𝑛 ∑𝑥
𝑙𝑛 𝐿(𝜃) = −𝑛𝑙𝑛𝜃 − and = + =0
𝜃 𝑑𝜃 𝜃 𝜃2

𝜃̂ = 𝑥̅

There are cases where the MLE exists but cannot be obtained as a solution of the maximum
likelihood equation. The example below illustrates such a scenario.

Example

Let 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 be a random sample from a population with a density function

 𝑥−𝜃
𝑓(𝑥, 𝜃) = {𝑒 , 𝑥 ≥ 𝜃, −∞ < 𝜃 < ∞
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

(a) Find the MLE for 𝜃.

(b) Sketch the graph of 𝐿(𝜃) as a function of = 𝑒 ∑ 𝑥𝑗 −𝑛𝜃 .
(c) Hence or otherwise find the MLE for 𝜃.

Solution

(a) The likelihood function for the sample values is

𝐿(𝜃) = ∏ 𝑒 𝑥𝑗 −𝜃

= 𝑒 ∑ 𝑥𝑗−𝑛𝜃
= 𝑒 𝑛(𝑥̅ −𝜃)

(b) 𝐿(𝜃)

𝒙(𝟏)

(c) From the graph the likelihood attains its maximum at 𝒙(𝟏) . Thus the MLE for 𝜃 is
̂ = 𝒙(𝟏) .
𝜽

Exercises
1. Let 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 be a random sample from a geometric distribution

1 1 𝑥
𝑓(𝑥, 𝜃) = (1 − ) ( ) , 𝑥 = 1, 2, ⋯
𝜃 𝜃
Find the MLE for 𝜃.
 2. Let 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 be a random sample from a gamma distribution
1 𝑥⁄
𝑓(𝑥, 𝜃) = 𝜃2 𝑥𝑒 − 𝜃, 𝑥 > 0.

(a) Find the MLE for 𝜃.

(b) Find the MME for 𝜃.

3. Let 𝑋1 , 𝑋2 , ⋯ , 𝑋𝑛 be a random sample from a normal distribution

1 1 𝑥−𝜇 2
𝑓(𝑥, 𝜇, 𝜎 2 ) = 𝑒𝑥𝑝 {− ( ) }, −∞ < 𝑥 < ∞
𝜎√2𝜋 2 𝜎
Find the MLE for 𝜇 and 𝜎 2 .

Method of Least Squares

In certain types of models, the principle of least squares is very important. We assume that the
mean of the random variable 𝑌 is a linear function of 𝑝 unknown vector of parameters 𝜷 =
(𝛽1 , 𝛽2 , ⋯ , 𝛽𝑝 ) and 𝑝 factors 𝒙 = (𝑥1 , 𝑥2 , ⋯ , 𝑥𝑝 ) that can be fixed and measured without error.
We assume that 𝑉𝑎𝑟(𝑌) = 𝜎 2 , where 𝜎 2 is constant and does not depend on 𝑥𝑖 ′𝑠.

The linear model can be written as

𝑌𝑖 = 𝛽0 +𝛽1 𝑥1𝑖 + 𝛽2 𝑥2𝑖 + ⋯ + 𝛽𝑝 𝑥𝑝𝑖 + 𝑒𝑖

The least squares estimate of 𝜷 = (𝛽1 , 𝛽2 , ⋯ , 𝛽𝑝 ) based on the sample 𝒙𝟏 , 𝒙𝟐 , 𝒙𝟑 , ⋯ , 𝒙𝒏

where 𝒙𝒊 = (𝑥1𝑖 , 𝑥2𝑖 , ⋯ , 𝑥𝑝𝑖 ), for 𝑖 = 1,2,3, ⋯ , 𝑛, is that ̂ = (𝛽̂1 , 𝛽̂2 , ⋯ , 𝛽̂𝑝 ) which
𝜷
minimizes the sum of squares

𝑛 𝑛
2
∑𝑒 = ∑( 𝑌𝑖 − 𝛽0 −𝛽1 𝑥1𝑖 − 𝛽2 𝑥2𝑖 − ⋯ 𝛽𝑝 𝑥𝑝𝑖 )2
𝑖=1 𝑖 𝑖=1

We differentiate ∑𝑛𝑖=1 𝑒 2 𝑖 with respect to all the 𝛽𝑖 ′𝑠 then solve the resulting equations for ̂
𝜷=

(𝛽̂1 , 𝛽̂2 , ⋯ , 𝛽̂𝑝 ).

Example

Let 𝑌 be a random variable mean 𝜇(𝛼, 𝛽) = 𝛼 + 𝛽𝑥𝑖 . Given sample of size n,

(𝑥1 , 𝑦1 ), (𝑥2 , 𝑦2 ), ⋯ , (𝑥,𝑛 𝑦𝑛 ), find the least squares estimates of 𝛼 and 𝛽.
Solution

Let 𝑄 = ∑𝑛𝑖=1(𝑦𝑖 − 𝛼 + 𝛽𝑥𝑖 )2

𝑛
𝛿𝑄
= −2 ∑(𝑦𝑖 − 𝛼 + 𝛽𝑥𝑖 )
𝛿𝛼
𝑖=1

𝑛
𝛿𝑄
= −2 ∑ 𝑥𝑖 (𝑦𝑖 − 𝛼 + 𝛽𝑥𝑖 )
𝛿𝛽
𝑖=1

Set the derivatives to zero and solve the simultaneous equations for 𝛼 and 𝛽 we obtain

∑ 𝑥𝑖 𝑦𝑖 − (∑ 𝑥𝑖 )(∑ 𝑦𝑖 )
𝛽̂ =
∑ 𝑥𝑖2 − (∑ 𝑥𝑖 )2

and

∑ 𝑦𝑖 − 𝛽̂ ∑ 𝑥𝑖
𝛼̂ =
𝑛

Exercises

Consider the data on monthly Income ($) and Savings ($) given in the table below.

Income 980 1500 420 198 785 2800 3845 1890 4210

Savings 180 315 80 42 165 520 760 366 800

Suppose that Income (X) and savings (Y) are related through a relationship with the form

𝑦 =∝ +𝛽𝑥.

(a) Find the least squares estimates of ∝ and 𝛽.

(b) Estimate 𝑦 when 𝑥 = 7200.