Statistics 5350/7110

Forecasting

Lecture 15
Estimated ARMA Models
Con dence Intervals and Diagnostics

Professor Robert Stine

fi
 Preliminaries
• Questions?

• Midterm ex m
• Tuesd y, October 22.
• During regul r cl ss period (1.5 hours)
• One p ge of notes, closed book. No R coding
• Multiple choice, short nswer

• Quick review
• Estim tion for ARMA models
• Condition l sum-of-squ res
• M ximum likelihood

2
a
a
a
a
a
a
a
a
a
a
 Today’s Topics
Text, §4.3

• Closer h nds-on look t estim tion

Peek t wh t h ppens when
• Compute the CSS estim tor “by h nd” building l rge l ngu ge models
• Iter tive p th of p r meter estim tes

• Properties of the m ximum likelihood estim tor

• St nd rd errors nd con idence interv ls
• Colline rity

• Routine di gnostics
• Residu ls: sequence nd qu ntile plots
• ACF of residu ls

• Further regression di gnostics

• Addition l plots to check the model
• Mix Durbin procedure with ML estim tes

3
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
f
a
a
a
a
a
a
a
a
a
 Properties of an MLE
• Ex mple: Me n of norm l distribution
• Log of the likelihood s function of μ
∑i (Xi − X)2 + n(X − μ)2
( (2πσ ) )
1 ∑ (X − μ)2
− i i2
ℓ(μ) = log(X1, …, Xn) = log e 2σ =c−
2 n/2 2σ 2
• First deriv tive of the log-likelihood reve ls MLE
n(X − μ)
ℓ′(μ) = ⇒ μ̂ = X
σ2
• Second deriv tive is the neg tive of the inverse (reciproc l) of its v ri nce
n σ2
ℓ′′(μ) = − 2 ⇒ Var(μ)̂ =
σ n
• Gener lize
• Typic lly for l rge n, n MLE is norm lly distribution nd unbi sed with sm llest possible v ri nce
−1

( ∂β 2 )
2
∂ ℓ(β)
β ̂ ∼ AN(β, I(β)) where I(β) = −

4



a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Likelihood for AR(1)
• Joint distribution
• F ctors into product of m rgin l of X1 times condition l distributions of X2 … Xn
• X1 ∼ N(0,σw2 /(1 − ϕ 2)) nd Xt | Xt−1 ∼ N(ϕ Xt−1, σw2 )

f(X1, X2, …, Xn) = f(X1) f(X2 | X1)⋯f(Xn | Xn−1)

n 2 2
1 − ϕ 2 −(1−ϕ 2)(X )2/(2σ 2) e − ∑t=2 (Xt−ϕXt−1) /(2σw)
= e 1 w
2πσw
2 (2π σw2 )n/2

• M ximize w.r.t. ϕ
• T king logs, converting product into sum
• Second term is simple: le st squ res
• First term is messy, but only dds little bit (1 p rt out of n)
• Hence MLE is very simil r to the le st squ res estim te

5
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Uncertainty of Estimates Property 4.29

• Estim tes re pproxim tely norm lly distributed

• Typic l result for MLE
• “Asymtotic lly unbi sed” (i.e., bi s goes w y s length of series grows)
• Expression for the st nd rd error depends on the model

• S mpling v ri nce for sm ll utoregressions

• AR(1)
n Var(ϕ1̂ ) ≈ (1 − ϕ12)
• AR(2) Softw re typic lly computes

n Var(ϕ1̂ ) ≈ (1 − ϕ22), n Var(ϕ2̂ ) ≈ (1 − ϕ22)

st nd rd errors from the curv ture
of the log likelihood r ther th n
using these expressions.
• Simil r expressions for sm ll moving ver ge models
• MA(1)
n Var(θ1̂ ) ≈ (1 − θ12)
• MA(2)
n Var(θ1̂ ) ≈ (1 − θ22), n Var(θ2̂ ) ≈ (1 − θ22)
6
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Diagnostics for ARMA
• Speci ic plots for time series
• Sequence plot of residu ls
• Autocorrel tion of residu ls
• Ljung-Box-Pierce st tistic for
cumul tive sum of utocorrel tions
h
̂ 2
∑
ρ(h)
k=1

• R function s rim
• Tr ces iter tive c lcul tions of CSS nd
then the likelihood.
• Autom tic residu l plots

7
a
a
a
f
a
a
a
a
a
a
a
a
a
a
a
a
a
 Additional Diagnostics for ARMA
• Where re the other residu l plots?
• Lever ge plots for outliers
• Added v ri ble plots

• Procedure
• Durbin procedure: Fit AR(p) model with l rge p to estim te white noise
• E sier: Since h ve the MLE, use residu ls from the MLE s white noise
• L gged ML residu ls repl ce unknown wt in regression model

• Ex mple
• ARMA(2,2) process, Xt = xt-1 - 0.4 Xt-2 - 0.6 wt-1 - 0.3 wt-2
MLE OLS, using ML residu ls

8
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Diagnostic Plots ARMA s Regression

Introduce outlier in time series t t = 33

s rim plots

9
a
a
a
a
a
 What’s next?
• One more thing before forec sting: ARIMA models
• Further gener liz tion th t incorpor tes di erencing/trend remov l into the time series model.

• Forec sting ARIMA processes

10
a
a
a
a
a
a
ff