Part II – TIME SERIES ANALYSIS

C5 ARIMA (Box-Jenkins) Models

© Angel A. Juan & Carles Serrat - UPC 2007/2008

2.5.1: Introduction to ARIMA models
 The Autoregressive Integrated Moving Average (ARIMA) Recall that stationary processes
Recall that stationary processes
vary about a fixed level, and
vary about a fixed level, and
models, or Box-Jenkins methodology, are a class of linear nonstationary processes have no
nonstationary processes have no
natural constant mean level.
models that is capable of representing stationary as well as natural constant mean level.
nonstationary time series. The ACF and PACF associated to the TS
The ACF and PACF associated to the TS
are matched with the theoretical
are matched with the theoretical
 ARIMA models rely heavily on autocorrelation patterns in autocorrelation pattern associated with a
autocorrelation pattern associated with a
particular ARIMA model.
particular ARIMA model.
data  both ACF and PACF are used to select an initial
model.
 The Box-Jenkins methodology uses an iterative approach:
1. An initial model is selected, from a general class of ARIMA
models, based on an examination of the TS and an
examination of its autocorrelations for several time lags
2. The chosen model is then checked against the historical data
to see whether it accurately describes the series: the model
fits well if the residuals are generally small, randomly
distributed, and contain no useful information.
3. If the specified model is not satisfactory, the process is
repeated using a new model designed to improve on the
original one.
4. Once a satisfactory model is found, it can be used for
forecasting.
2.5.2: Autoregressive Models AR(p)
 A pth-order autoregressive model, or AR(p), takes the form:
YYt 0 1YYt 1 2YYt  2 ......pYYt  p t YYt response
responsevariable
variableatattime
timet t
t 0 1 t 1 2 t 2 p t p t t
YYt  k observation
observation(predictor
(predictorvariable) timet tkk
variable)atattime
t k
An AR(p) model is a regression model with lagged
An AR(p) model is a regression model with lagged
values of the dependent variable in the independent i regression
regressioncoefficients
coefficientstotobe
beestimated
estimated
values of the dependent variable in the independent i
variable positions, hence the name autoregressive model.
variable positions, hence the name autoregressive model. t  error
errorterm
termatattime
timet t
t

 Autoregressive models are appropriate for stationary time series, and the
coefficient Ф0 is related to the constant level of the series.
 Theoretical behavior of the ACF and PACF for AR(1) and AR(2) models:

AR(1) AR(2)
AR(1) AR(2)
ACF  0 ACF  0
ACF  0 ACF  0
PACF = 0 for lag > 1 PACF = 0 for lag > 2
PACF = 0 for lag > 1 PACF = 0 for lag > 2
2.5.3: Moving Average Models MA(q)
 A qth-order moving average model, or MA(q), takes the form: The term Moving
The term Moving
Average is
Average is
YYt t 1t 1 2t  2 ......qt q YYt tresponse
responsevariable
variableatattime
timet t
historical and
historical and
should not be
t t 1 t 1 2 t 2 q t q should not be
constant
constantmean
meanofofthe
theprocess
process
confused with the
confused with the
moving average
An MA(q) model is a regression model with the dependent moving average
An MA(q) model is a regression model with the dependent
variable, Yt, depending on previous values of the errors i regression
regressioncoefficients
coefficientstotobebeestimated
estimated
smoothing
smoothing
procedures.
variable, Yt, depending on previous values of the errors i procedures.
rather than on the variable itself.
rather than on the variable itself. t  k error
errorinintime
timeperiod
periodt t- -kk
t k

 MA models are appropriate for stationary time series. The weights ωi do not
necessarily sum to 1 and may be positive or negative.
 Theoretical behavior of the ACF and PACF for MA(1) and MA(2) models:

MA(1) MA(2)
MA(1) MA(2)
ACF = 0 for lag > 1; PACF  0 ACF = 0 for lag > 2; PACF  0
ACF = 0 for lag > 1; PACF  0 ACF = 0 for lag > 2; PACF  0
2.5.4: ARMA(p,q) Models
 A model with autoregressive terms can be combined with a model Note that:
Note that:
having moving average terms to get an ARMA(p,q) model: • ARMA(p,0) = AR(p)
• ARMA(p,0) = AR(p)
• ARMA(0,q) = MA(q)
• ARMA(0,q) = MA(q)
YYt 0 1YYt 1 2YYt 2 ......pYYt  p t 1t 1 2t 2 ......q t q
t 0 1 t 1 2 t 2 p t p t 1 t 1 2 t 2 q t q In practice, the values
In practice, the values
of p and q each rarely
of p and q each rarely
exceed 2.
exceed 2.
 ARMA(p,q) models can describe a wide variety of behaviors for
stationary time series.
 Theoretical behavior of the ACF and PACF for autoregressive-
moving average processes:

ACF PACF
Die out Cut off after the order
AR(p) p of the process

Cut off after the order Die out

MA(q) q of the process
In this context…
In this context…
• “Die out” means “tend
• “Die out” means “tend
to zero gradually”
to zero gradually”
Die out Die out
ARMA(p,q) • “Cut off” means
• “Cut off” means
“disappear” or “is zero”
“disappear” or “is zero”
2.5.5: Building an ARIMA model (1/2)
 The first step in model identification is to determine whether the A nonstationary TS is
A nonstationary TS is
indicated if the series
series is stationary. It is useful to look at a plot of the series indicated if the series
appears to grow or decline
appears to grow or decline
over time and the sample
along with the sample ACF. over time and the sample
ACF fail to die out rapidly.
ACF fail to die out rapidly.

 If the series is not stationary, it can often be converted to a

stationary series by differencing: the original series is replaced In some cases, it may be
In some cases, it may be
necessary to difference
necessary to difference
by a series of differences and an ARMA model is then specified the differences before
the differences before
stationary data are
for the differenced series (in effect, the analyst is modeling stationary data are
obtained.
obtained.
changes rather than levels)
 Models for nonstationary series are called Autoregressive
Integrated Moving Average models, or ARIMA(p,d,q), where d Note that:
Note that:
indicates the amount of differencing. ARIMA(p,0,q) = ARMA(p,q)
ARIMA(p,0,q) = ARMA(p,q)

 Once a stationary series has been obtained, the analyst must By counting the number of
By counting the number of
significant sample
identify the form of the model to be used by comparing the significant sample
autocorrelations and partial
autocorrelations and partial
autocorrelations, the orders
sample ACF and PACF to the theoretical ACF and PACF for the autocorrelations, the orders
of the AR and MA parts can
of the AR and MA parts can
be determined.
various ARIMA models. be determined.

 Principle of parsimony: “all things being equal, simple models Advice: start with a model
Advice: start with a model
containing few rather than
containing few rather than
are preferred to complex models” many parameters. The
many parameters. The
need for additional
need for additional
parameters will be evident
 Once a tentative model has been selected, the parameters for parameters will be evident
from an examination of the
from an examination of the
residual ACF and PACF.
that model are estimated using least squares estimates. residual ACF and PACF.
2.5.5: Building an ARIMA model (2/2)
Many of the same
 Before using the model for forecasting, it must be checked for Many of the same
residual plots that are
residual plots that are
useful in regression
adequacy. Basically, a model is adequate if the residuals cannot be useful in regression
analysis can be
analysis can be
used to improve the forecasts, i.e., developed for the
developed for the
residuals from an
residuals from an
ARIMA model
 The residuals should be random and normally distributed ARIMA model
(histogram, normal
(histogram, normal
probability plot, time
probability plot, time
 The individual residual autocorrelations should be small. Significant sequence plot, etc.)
sequence plot, etc.)
residual autocorrelations at low lags or seasonal lags suggest the
model is inadequate
In general, the
In general, the
longer the forecast
 After an adequate model has been found, forecasts can be made. longer the forecast
lead time, the larger
lead time, the larger
the prediction
Prediction intervals based on the forecasts can also be constructed. the prediction
interval (due to
interval (due to
greater uncertainty)
greater uncertainty)
 As more data become available, it is a good idea to monitor the
forecast errors, since the model must need to be reevaluated if:
 The magnitudes of the most recent errors tend to be consistently
larger than previous errors, or
 The recent forecast errors tend to be consistently positive or negative
In addition, for
In addition, for
nonstationary
 Seasonal ARIMA (SARIMA) models contain: nonstationary
seasonal series,
seasonal series,
an additional
an additional
 Regular AR and MA terms that account for the correlation at low lags seasonal
seasonal
difference is often
difference is often
required
 Seasonal AR and MA terms that account for the correlation at the required

seasonal lags
2.5.6: ARIMA with Minitab – Ex. 1 (1/4)
 File: PORTFOLIO_INVESTMENT.MTW A consulting corporation wants
A consulting corporation wants
to try the Box-Jenkins
to try the Box-Jenkins
 Stat > Time Series > … technique for forecasting the
technique for forecasting the
Transportation Index of the
Transportation Index of the
Dow Jones.
Dow Jones.
Time Series Plot of Index
290

280
The series show an
The series show an
upward trend.
upward trend.
270

260
Index

250

240

230

220 Autocorrelation Function for Index

(with 5% significance limits for the autocorrelations)
210
1 6 12 18 24 30 36 42 48 54 60 1,0
Index 0,8
0,6
0,4
A utocorrelation

The first several autocorrelations are persistently 0,2

The first several autocorrelations are persistently
large and trailed off to zero rather slowly  a trend
large and trailed off to zero rather slowly  a trend 0,0
exists and this time series is nonstationary (it does
exists and this time series is nonstationary (it does -0,2
not vary about a fixed level)
not vary about a fixed level)
-0,4
-0,6
-0,8
-1,0
Idea: to difference the data to see if we could 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Idea: to difference the data to see if we could
eliminate the trend and create a stationary series. Lag
eliminate the trend and create a stationary series.
2.5.6: ARIMA with Minitab – Ex. 1 (2/4)
Time Series Plot of Diff1
First order differences.
First order differences.
5

4
A plot of the
A plot of the 3
differenced data
differenced data 2
appears to vary about
appears to vary about
a fixed level.

Diff1
1
a fixed level.
0

-1

-2

-3

-4
1 6 12 18 24 30 36 42 48 54 60
Comparing the autocorrelations with their error limits, the only significant
Comparing the autocorrelations with their error limits, the only significant Index
autocorrelation is at lag 1. Similarly, only the lag 1 partial autocorrelation is
autocorrelation is at lag 1. Similarly, only the lag 1 partial autocorrelation is
significant. The PACF appears to cut off after lag 1, indicating AR(1) behavior.
significant. The PACF appears to cut off after lag 1, indicating AR(1) behavior.
The ACF appears to cut off after lag 1, indicating MA(1) behavior  we will A constant term in each model will be included to allow for the fact that
The ACF appears to cut off after lag 1, indicating MA(1) behavior  we will A constant term in each model will be included to allow for the fact that
try: ARIMA(1,1,0) and ARIMA(0,1,1) the series of differences appears to vary about a level greater than zero.
try: ARIMA(1,1,0) and ARIMA(0,1,1) the series of differences appears to vary about a level greater than zero.

Autocorrelation Function for Diff1 Partial Autocorrelation Function for Diff1

(with 5% significance limits for the autocorrelations) (with 5% significance limits for the partial autocorrelations)

1,0 1,0

0,8 0,8

0,6 0,6
Partial Autocorrelation

0,4 0,4
A utocorrelation

0,2 0,2
0,0 0,0

-0,2 -0,2

-0,4 -0,4

-0,6 -0,6
-0,8 -0,8
-1,0 -1,0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Lag Lag
2.5.6: ARIMA with Minitab – Ex. 1 (3/4)

ARIMA(1,1,0) ARIMA(0,1,1)
ARIMA(1,1,0) ARIMA(0,1,1)

The LBQ statistics are not significant as indicated by the large p-

The LBQ statistics are not significant as indicated by the large p-
values for either model.
values for either model.
2.5.6: ARIMA with Minitab – Ex. 1 (4/4)
Autocorrelation Function for RESI1
(with 5% significance limits for the autocorrelations)

1,0
0,8
0,6
0,4
Autocorrelation

0,2
0,0
-0,2
-0,4
-0,6
-0,8
-1,0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Lag

Autocorrelation Function for RESI2

(with 5% significance limits for the autocorrelations)

1,0
0,8
Finally, there is no significant residual 0,6
Finally, there is no significant residual
autocorrelation for the ARIMA(1,1,0) model.
autocorrelation for the ARIMA(1,1,0) model. 0,4
Autocorrelation

The results for the ARIMA(0,1,1) are similar.

The results for the ARIMA(0,1,1) are similar. 0,2
0,0
-0,2
-0,4
-0,6
-0,8
Therefore, either model is adequate and provide
Therefore, either model is adequate and provide -1,0
nearly the same one-step-ahead forecasts.
nearly the same one-step-ahead forecasts.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Lag
2.5.7: ARIMA with Minitab – Ex. 2 (1/3)
 File: READINGS.MTW A consulting corporation wants
A consulting corporation wants
to try the Box-Jenkins
to try the Box-Jenkins
technique for forecasting a
technique for forecasting a
 Stat > Time Series > … process.
process.

Time Series Plot of Readings

110
The time series of readings appears to vary
The time series of readings appears to vary
100 about a fixed level of around 80, and the
about a fixed level of around 80, and the
autocorrelations die out rapidly toward zero
90 autocorrelations die out rapidly toward zero
 the time series seems to be stationary.
80  the time series seems to be stationary.
Readings

70

60
The first sample ACF coefficient is significantly different form zero. The
The first sample ACF coefficient is significantly different form zero. The
autocorrelation at lag 2 is close to significant and opposite in sign from the lag 1
50 autocorrelation at lag 2 is close to significant and opposite in sign from the lag 1
autocorrelation. The remaining autocorrelations are small. This suggests either an
autocorrelation. The remaining autocorrelations are small. This suggests either an
40 AR(1) model or an MA(2) model.
AR(1) model or an MA(2) model.
30 The first PACF coefficient is significantly different from zero, but none of the other
The first PACF coefficient is significantly different from zero, but none of the other
20 partial autocorrelations approaches significance, This suggests an AR(1) or
partial autocorrelations approaches significance, This suggests an AR(1) or
1 7 14 21 28 35 42 49 56 63 70 ARIMA(1,0,0)
Index ARIMA(1,0,0)

Autocorrelation Function for Readings Partial Autocorrelation Function for Readings

(with 5% significance limits for the autocorrelations) (with 5% significance limits for the partial autocorrelations)

1,0 1,0
0,8 0,8
0,6 0,6

Partial Autocorrelation
0,4 0,4
Autocorrelation

0,2 0,2
0,0 0,0
-0,2 -0,2
-0,4 -0,4
-0,6 -0,6
-0,8 -0,8
-1,0 -1,0

2 4 6 8 10 12 14 16 18 2 4 6 8 10 12 14 16 18
Lag Lag
2.5.7: ARIMA with Minitab – Ex. 2 (2/3)
A constant term is
A constant term is
included in both MA(2) =
included in both MA(2) =
AR(1) = models to allow for ARIMA(0,0,2)
AR(1) = models to allow for ARIMA(0,0,2)
ARIMA(1,0,0) the fact that the
ARIMA(1,0,0) the fact that the
readings vary about a
readings vary about a
level other than zero.
level other than zero.

Both models appear

Both models appear
to fit the data well.
to fit the data well.
The estimated
The estimated
coefficients are
coefficients are
significantly different
significantly different
from zero and the
from zero and the
mean square (MS)
mean square (MS)
errors are similar.
errors are similar.

Let’s take a look at the residuals ACF…

Let’s take a look at the residuals ACF…
2.5.7: ARIMA with Minitab – Ex. 2 (3/3)
Autocorrelation Function for RESI1
(with 5% significance limits for the autocorrelations)

1,0
0,8
0,6
0,4
Autocorrelation

0,2
0,0
-0,2
-0,4
-0,6
-0,8
-1,0

2 4 6 8 10 12 14 16 18
Lag

Autocorrelation Function for RESI2

(with 5% significance limits for the autocorrelations)

1,0
Finally, there is no significant residual 0,8
Finally, there is no significant residual
autocorrelation for the ARIMA(1,0,0) model. 0,6
autocorrelation for the ARIMA(1,0,0) model.
The results for the ARIMA(0,0,2) are similar.
The results for the ARIMA(0,0,2) are similar. 0,4
A utocorrelation

0,2
0,0
-0,2
-0,4
Therefore, either model is adequate and provide nearly the same three- -0,6
Therefore, either model is adequate and provide nearly the same three-
step-ahead forecasts. Since the AR(1) model has two parameters
step-ahead forecasts. Since the AR(1) model has two parameters -0,8
(including the constant term) and the MA(2) model has three parameters,
(including the constant term) and the MA(2) model has three parameters, -1,0
applying the principle of parsimony we would use the simpler AR(1)
applying the principle of parsimony we would use the simpler AR(1)
model to forecast future readings. 2 4 6 8 10 12 14 16 18
model to forecast future readings.
Lag