Rob J Hyndman

Forecasting using

7. Transformations and adjustments

OTexts.com/fpp/2/4/

Forecasting using R

Outline

1 Exponential smoothing

2 Transformations

3 Adjustments

Forecasting using R

Exponential smoothing

Exponential smoothing
ets() function
Automatically chooses a model by default using
the AIC, AICc or BIC.
Can handle any combination of trend,
seasonality and damping
Produces prediction intervals for every model
Ensures the parameters are admissible
(equivalent to invertible)
Produces an object of class ets.

Forecasting using R

Exponential smoothing

Exponential smoothing
ets() function
Automatically chooses a model by default using
the AIC, AICc or BIC.
Can handle any combination of trend,
seasonality and damping
Produces prediction intervals for every model
Ensures the parameters are admissible
(equivalent to invertible)
Produces an object of class ets.

Forecasting using R

Exponential smoothing

Exponential smoothing
ets() function
Automatically chooses a model by default using
the AIC, AICc or BIC.
Can handle any combination of trend,
seasonality and damping
Produces prediction intervals for every model
Ensures the parameters are admissible
(equivalent to invertible)
Produces an object of class ets.

Forecasting using R

Exponential smoothing

Exponential smoothing
ets() function
Automatically chooses a model by default using
the AIC, AICc or BIC.
Can handle any combination of trend,
seasonality and damping
Produces prediction intervals for every model
Ensures the parameters are admissible
(equivalent to invertible)
Produces an object of class ets.

Forecasting using R

Exponential smoothing

Exponential smoothing
ets() function
Automatically chooses a model by default using
the AIC, AICc or BIC.
Can handle any combination of trend,
seasonality and damping
Produces prediction intervals for every model
Ensures the parameters are admissible
(equivalent to invertible)
Produces an object of class ets.

Forecasting using R

Exponential smoothing

Exponential smoothing
ets objects
Methods: coef(), plot(), summary(),
residuals(), fitted(), simulate()
and forecast()
plot() function shows time plots of the
original time series along with the
extracted components (level, growth
and seasonal).
Forecasting using R

Exponential smoothing

Exponential smoothing
ets objects
Methods: coef(), plot(), summary(),
residuals(), fitted(), simulate()
and forecast()
plot() function shows time plots of the
original time series along with the
extracted components (level, growth
and seasonal).
Forecasting using R

Exponential smoothing

Exponential smoothing
600
400
350
1.005
1.1 0.990
0.9

season

slope

250

level

450200

observed

plot(fit)
Decomposition by ETS(M,Md,M) method

1960

1970

Forecasting using R

1980

1990

2000

Exponential smoothing

2010
5

Goodness-of-fit

> accuracy(fit)
ME
RMSE
MAE
0.17847 15.48781 11.77800

MPE
0.07204

MAPE
2.81921

MASE
0.20705

> accuracy(fit2)
ME
RMSE
MAE
MPE
-0.11711 15.90526 12.18930 -0.03765

MAPE
2.91255

MASE
0.21428

Forecasting using R

Exponential smoothing

Forecast intervals
> plot(forecast(fit,level=c(50,80,95)))

300

350

400

450

500

550

600

Forecasts from ETS(M,Md,M)

1995

Forecasting using R

2000

2005

2010

Exponential smoothing

Forecast intervals
> plot(forecast(fit,fan=TRUE))

300

350

400

450

500

550

600

Forecasts from ETS(M,Md,M)

1995

Forecasting using R

2000

2005

2010

Exponential smoothing

Exponential smoothing
ets() function also allows refitting model to new
data set.
> usfit <- ets(usnetelec[1:45])
> test <- ets(usnetelec[46:55], model = usfit)
> accuracy(test)
ME
RMSE
MAE
MPE
-3.35419 58.02763 43.85545 -0.07624

MAPE
1.18483

MASE
0.52452

> accuracy(forecast(usfit,10), usnetelec[46:55])

ME
RMSE
MAE
MPE
MAPE
40.7034
61.2075
46.3246
1.0980
1.2620

Forecasting using R

Exponential smoothing

MASE
0.6776

Unstable models
ETS(M,M,A)
ETS(M,Md ,A)
ETS(A,N,M)
ETS(A,A,M)
ETS(A,Ad ,M)
ETS(A,M,N)
ETS(A,M,A)
ETS(A,M,M)
ETS(A,Md ,N)
ETS(A,Md ,A)
ETS(A,Md ,M)
Forecasting using R

Exponential smoothing

Unstable models
ETS(M,M,A)
ETS(M,Md ,A)
ETS(A,N,M)
ETS(A,A,M)
ETS(A,Ad ,M)
ETS(A,M,N)
ETS(A,M,A)
ETS(A,M,M)

In practice, the models

work fine for short- to
medium-term forecasts
provided the data are
strictly positive.

ETS(A,Md ,N)
ETS(A,Md ,A)
ETS(A,Md ,M)
Forecasting using R

Exponential smoothing

Forecastability conditions
ets(y, model="ZZZ", damped=NULL, alpha=NULL,
beta=NULL, gamma=NULL, phi=NULL,
additive.only=FALSE,
lower=c(rep(0.0001,3),0.80),
upper=c(rep(0.9999,3),0.98),
opt.crit=c("lik","amse","mse","sigma"),
nmse=3,
bounds=c("both","usual","admissible"),
ic=c("aic","aicc","bic"), restrict=TRUE)

Forecasting using R

Exponential smoothing

10

The magic forecast() function

forecast returns forecasts when applied to an

ets object (or the output from many other time
series models).
If you use forecast directly on data, it will
select an ETS model automatically and then
return forecasts.

Forecasting using R

Exponential smoothing

11

The magic forecast() function

forecast returns forecasts when applied to an

ets object (or the output from many other time
series models).
If you use forecast directly on data, it will
select an ETS model automatically and then
return forecasts.

Forecasting using R

Exponential smoothing

11

Outline

1 Exponential smoothing

2 Transformations

3 Adjustments

Forecasting using R

Transformations

12

Transformations to stabilize the variance

If the data show different variation at different levels of the
series, then a transformation can be useful.
Denote original observations as y1 , . . . , yn and transformed
observations as w1 , . . . , wn .
Mathematical transformations for stabilizing variation

Square root

wt =

yt

Cube root

wt =

Increasing

Logarithm

wt = log(yt )

strength

yt

Logarithms, in particular, are useful because they are more

interpretable: changes in a log value are relative (percent)
changes on the original scale.
Forecasting using R

Transformations

13

Transformations to stabilize the variance

If the data show different variation at different levels of the
series, then a transformation can be useful.
Denote original observations as y1 , . . . , yn and transformed
observations as w1 , . . . , wn .
Mathematical transformations for stabilizing variation

Square root

wt =

yt

Cube root

wt =

Increasing

Logarithm

wt = log(yt )

strength

yt

Logarithms, in particular, are useful because they are more

interpretable: changes in a log value are relative (percent)
changes on the original scale.
Forecasting using R

Transformations

13

Transformations to stabilize the variance

If the data show different variation at different levels of the
series, then a transformation can be useful.
Denote original observations as y1 , . . . , yn and transformed
observations as w1 , . . . , wn .
Mathematical transformations for stabilizing variation

Square root

wt =

yt

Cube root

wt =

Increasing

Logarithm

wt = log(yt )

strength

yt

Logarithms, in particular, are useful because they are more

interpretable: changes in a log value are relative (percent)
changes on the original scale.
Forecasting using R

Transformations

13

Transformations to stabilize the variance

If the data show different variation at different levels of the
series, then a transformation can be useful.
Denote original observations as y1 , . . . , yn and transformed
observations as w1 , . . . , wn .
Mathematical transformations for stabilizing variation

Square root

wt =

yt

Cube root

wt =

Increasing

Logarithm

wt = log(yt )

strength

yt

Logarithms, in particular, are useful because they are more

interpretable: changes in a log value are relative (percent)
changes on the original scale.
Forecasting using R

Transformations

13

Transformations

40

60

80

100

120

Square root electricity production

1960

1970

1980

1990

Year
Forecasting using R

Transformations

14

Transformations

12

14

16

18

20

22

24

Cube root electricity production

1960

1970

1980

1990

Year
Forecasting using R

Transformations

14

Transformations

7.5

8.0

8.5

9.0

9.5

Log electricity production

1960

1970

1980

1990

Year
Forecasting using R

Transformations

14

Transformations

8e04

6e04

4e04

2e04

Inverse electricity production

1960

1970

1980

1990

Year
Forecasting using R

Transformations

14

Box-Cox transformations
Each of these transformations is close to a member
of the family of Box-Cox transformations:


wt =

log(yt ),
(yt 1)/,

= 0;
6= 0.

= 1: (No substantive transformation)

= 12 : (Square root plus linear transformation)
= 0: (Natural logarithm)
= 1: (Inverse plus 1)
Forecasting using R

Transformations

15

Box-Cox transformations
Each of these transformations is close to a member
of the family of Box-Cox transformations:


wt =

log(yt ),
(yt 1)/,

= 0;
6= 0.

= 1: (No substantive transformation)

= 12 : (Square root plus linear transformation)
= 0: (Natural logarithm)
= 1: (Inverse plus 1)
Forecasting using R

Transformations

15

Box-Cox transformations
Each of these transformations is close to a member
of the family of Box-Cox transformations:


wt =

log(yt ),
(yt 1)/,

= 0;
6= 0.

= 1: (No substantive transformation)

= 12 : (Square root plus linear transformation)
= 0: (Natural logarithm)
= 1: (Inverse plus 1)
Forecasting using R

Transformations

15

Box-Cox transformations
Each of these transformations is close to a member
of the family of Box-Cox transformations:


wt =

log(yt ),
(yt 1)/,

= 0;
6= 0.

= 1: (No substantive transformation)

= 12 : (Square root plus linear transformation)
= 0: (Natural logarithm)
= 1: (Inverse plus 1)
Forecasting using R

Transformations

15

Box-Cox transformations
Each of these transformations is close to a member
of the family of Box-Cox transformations:


wt =

log(yt ),
(yt 1)/,

= 0;
6= 0.

= 1: (No substantive transformation)

= 12 : (Square root plus linear transformation)
= 0: (Natural logarithm)
= 1: (Inverse plus 1)
Forecasting using R

Transformations

15

Box-Cox transformations

10000
6000
2000

Monthly electricity production

14000

= 1.00

1960

1970

1980

1990

Year
Forecasting using R

Transformations

16

Box-Cox transformations
yt for close to zero behaves like logs.
If some yt = 0, then must have > 0
if some yt < 0, no power transformation is
possible unless all yt adjusted by adding a
constant to all values.
Choose a simple value of . It makes
explanation easier.
Results are relatively insensitive to value of
Often no transformation ( = 1) needed.
Transformation often makes little difference to
forecasts but has large effect on PI.
Choosing = 0 is a simple way to force
forecasts to be positive
Forecasting using R

Transformations

17

Box-Cox transformations
yt for close to zero behaves like logs.
If some yt = 0, then must have > 0
if some yt < 0, no power transformation is
possible unless all yt adjusted by adding a
constant to all values.
Choose a simple value of . It makes
explanation easier.
Results are relatively insensitive to value of
Often no transformation ( = 1) needed.
Transformation often makes little difference to
forecasts but has large effect on PI.
Choosing = 0 is a simple way to force
forecasts to be positive
Forecasting using R

Transformations

17

Box-Cox transformations
yt for close to zero behaves like logs.
If some yt = 0, then must have > 0
if some yt < 0, no power transformation is
possible unless all yt adjusted by adding a
constant to all values.
Choose a simple value of . It makes
explanation easier.
Results are relatively insensitive to value of
Often no transformation ( = 1) needed.
Transformation often makes little difference to
forecasts but has large effect on PI.
Choosing = 0 is a simple way to force
forecasts to be positive
Forecasting using R

Transformations

17

Box-Cox transformations
yt for close to zero behaves like logs.
If some yt = 0, then must have > 0
if some yt < 0, no power transformation is
possible unless all yt adjusted by adding a
constant to all values.
Choose a simple value of . It makes
explanation easier.
Results are relatively insensitive to value of
Often no transformation ( = 1) needed.
Transformation often makes little difference to
forecasts but has large effect on PI.
Choosing = 0 is a simple way to force
forecasts to be positive
Forecasting using R

Transformations

17

Box-Cox transformations
yt for close to zero behaves like logs.
If some yt = 0, then must have > 0
if some yt < 0, no power transformation is
possible unless all yt adjusted by adding a
constant to all values.
Choose a simple value of . It makes
explanation easier.
Results are relatively insensitive to value of
Often no transformation ( = 1) needed.
Transformation often makes little difference to
forecasts but has large effect on PI.
Choosing = 0 is a simple way to force
forecasts to be positive
Forecasting using R

Transformations

17

Box-Cox transformations
yt for close to zero behaves like logs.
If some yt = 0, then must have > 0
if some yt < 0, no power transformation is
possible unless all yt adjusted by adding a
constant to all values.
Choose a simple value of . It makes
explanation easier.
Results are relatively insensitive to value of
Often no transformation ( = 1) needed.
Transformation often makes little difference to
forecasts but has large effect on PI.
Choosing = 0 is a simple way to force
forecasts to be positive
Forecasting using R

Transformations

17

Box-Cox transformations
yt for close to zero behaves like logs.
If some yt = 0, then must have > 0
if some yt < 0, no power transformation is
possible unless all yt adjusted by adding a
constant to all values.
Choose a simple value of . It makes
explanation easier.
Results are relatively insensitive to value of
Often no transformation ( = 1) needed.
Transformation often makes little difference to
forecasts but has large effect on PI.
Choosing = 0 is a simple way to force
forecasts to be positive
Forecasting using R

Transformations

17

Box-Cox transformations
yt for close to zero behaves like logs.
If some yt = 0, then must have > 0
if some yt < 0, no power transformation is
possible unless all yt adjusted by adding a
constant to all values.
Choose a simple value of . It makes
explanation easier.
Results are relatively insensitive to value of
Often no transformation ( = 1) needed.
Transformation often makes little difference to
forecasts but has large effect on PI.
Choosing = 0 is a simple way to force
forecasts to be positive
Forecasting using R

Transformations

17

Back-transformation
We must reverse the transformation (or
back-transform) to obtain forecasts on the original
scale. The reverse Box-Cox transformations are
given by


yt =

exp(wt ),
(Wt + 1)1/ ,

= 0;
6= 0.

plot(BoxCox(elec,lambda=1/3))
fit <- snaive(elec, lambda=1/3)
plot(fit)
plot(fit, include=120)
Forecasting using R

Transformations

18

Back-transformation
We must reverse the transformation (or
back-transform) to obtain forecasts on the original
scale. The reverse Box-Cox transformations are
given by


yt =

exp(wt ),
(Wt + 1)1/ ,

= 0;
6= 0.

plot(BoxCox(elec,lambda=1/3))
fit <- snaive(elec, lambda=1/3)
plot(fit)
plot(fit, include=120)
Forecasting using R

Transformations

18

Automated Box-Cox transformations

BoxCox.lambda(elec)
This attempts to balance the seasonal
fluctuations and random variation across the
series.
Always check the results.
A low value of can give extremely large
prediction intervals.

Forecasting using R

Transformations

19

Automated Box-Cox transformations

BoxCox.lambda(elec)
This attempts to balance the seasonal
fluctuations and random variation across the
series.
Always check the results.
A low value of can give extremely large
prediction intervals.

Forecasting using R

Transformations

19

Automated Box-Cox transformations

BoxCox.lambda(elec)
This attempts to balance the seasonal
fluctuations and random variation across the
series.
Always check the results.
A low value of can give extremely large
prediction intervals.

Forecasting using R

Transformations

19

Automated Box-Cox transformations

BoxCox.lambda(elec)
This attempts to balance the seasonal
fluctuations and random variation across the
series.
Always check the results.
A low value of can give extremely large
prediction intervals.

Forecasting using R

Transformations

19

ETS and transformations

A Box-Cox transformation followed by an

additive ETS model is often better than an ETS
model without transformation.
A Box-Cox transformation followed by STL +
ETS is often better than an ETS model without
transformation.
It makes no sense to use a Box-Cox
transformation and a non-additive ETS model.

Forecasting using R

Transformations

20

ETS and transformations

A Box-Cox transformation followed by an

additive ETS model is often better than an ETS
model without transformation.
A Box-Cox transformation followed by STL +
ETS is often better than an ETS model without
transformation.
It makes no sense to use a Box-Cox
transformation and a non-additive ETS model.

Forecasting using R

Transformations

20

ETS and transformations

A Box-Cox transformation followed by an

additive ETS model is often better than an ETS
model without transformation.
A Box-Cox transformation followed by STL +
ETS is often better than an ETS model without
transformation.
It makes no sense to use a Box-Cox
transformation and a non-additive ETS model.

Forecasting using R

Transformations

20

Outline

1 Exponential smoothing

2 Transformations

3 Adjustments

Forecasting using R

Adjustments

21

Calendar adjustments

Some of the variation in a time series may be due to

the variation in the number of trading days each
month. It is a good idea to adjust for this known
source of variation to allow study of other
interesting features.
Month length
Trading day

Forecasting using R

Adjustments

22

Calendar adjustments

Some of the variation in a time series may be due to

the variation in the number of trading days each
month. It is a good idea to adjust for this known
source of variation to allow study of other
interesting features.
Month length
Trading day

Forecasting using R

Adjustments

22

Month length adjustment

If this is not removed, it shows up as a seasonal
effect, which may not cause problems though it
does make any seasonal pattern hard to interpret. It
is easily adjusted for:
yt = yt

= yt

no. of days in an average month

no. of days in month t
365.25/12
no. of days in month t

where yt has already been transformed if necessary.

monthdays gives the number of days in each month
or quarter.
Forecasting using R

Adjustments

23

Month length adjustment

If this is not removed, it shows up as a seasonal
effect, which may not cause problems though it
does make any seasonal pattern hard to interpret. It
is easily adjusted for:
yt = yt

= yt

no. of days in an average month

no. of days in month t
365.25/12
no. of days in month t

where yt has already been transformed if necessary.

monthdays gives the number of days in each month
or quarter.
Forecasting using R

Adjustments

23

Trading day adjustment

occurs in monthly data when there is also a
weekly cycle, since proportions of various days
in given month vary from year to year.
number of trading days is predictable, but
effects of various days are unknown.
Simplest case: All trading days assumed to
have same effect.
no. of trading days in average month
yt = yt
.
no. of trading days in month t
where yt has already been adjusted for month
length and transformed if necessary.
If weekly cycle more complex, these effects
must be estimated from data.
Forecasting using R

Adjustments

24

Trading day adjustment

occurs in monthly data when there is also a
weekly cycle, since proportions of various days
in given month vary from year to year.
number of trading days is predictable, but
effects of various days are unknown.
Simplest case: All trading days assumed to
have same effect.
no. of trading days in average month
yt = yt
.
no. of trading days in month t
where yt has already been adjusted for month
length and transformed if necessary.
If weekly cycle more complex, these effects
must be estimated from data.
Forecasting using R

Adjustments

24

Trading day adjustment

occurs in monthly data when there is also a
weekly cycle, since proportions of various days
in given month vary from year to year.
number of trading days is predictable, but
effects of various days are unknown.
Simplest case: All trading days assumed to
have same effect.
no. of trading days in average month
yt = yt
.
no. of trading days in month t
where yt has already been adjusted for month
length and transformed if necessary.
If weekly cycle more complex, these effects
must be estimated from data.
Forecasting using R

Adjustments

24

Trading day adjustment

occurs in monthly data when there is also a
weekly cycle, since proportions of various days
in given month vary from year to year.
number of trading days is predictable, but
effects of various days are unknown.
Simplest case: All trading days assumed to
have same effect.
no. of trading days in average month
yt = yt
.
no. of trading days in month t
where yt has already been adjusted for month
length and transformed if necessary.
If weekly cycle more complex, these effects
must be estimated from data.
Forecasting using R

Adjustments

24

Explainable variation
Examples:
Calendar variation
Increasing population
Inflation
Strikes
Changes in government
Changes in law
Try and understand all possible sources of variation
before modelling the time series.
Forecasting using R

Adjustments

25

Explainable variation
Examples:
Calendar variation
Increasing population
Inflation
Strikes
Changes in government
Changes in law
Try and understand all possible sources of variation
before modelling the time series.
Forecasting using R

Adjustments

25

Explainable variation
Examples:
Calendar variation
Increasing population
Inflation
Strikes
Changes in government
Changes in law
Try and understand all possible sources of variation
before modelling the time series.
Forecasting using R

Adjustments

25

Explainable variation
Examples:
Calendar variation
Increasing population
Inflation
Strikes
Changes in government
Changes in law
Try and understand all possible sources of variation
before modelling the time series.
Forecasting using R

Adjustments

25

Explainable variation
Examples:
Calendar variation
Increasing population
Inflation
Strikes
Changes in government
Changes in law
Try and understand all possible sources of variation
before modelling the time series.
Forecasting using R

Adjustments

25

Explainable variation
Examples:
Calendar variation
Increasing population
Inflation
Strikes
Changes in government
Changes in law
Try and understand all possible sources of variation
before modelling the time series.
Forecasting using R

Adjustments

25

Explainable variation
Examples:
Calendar variation
Increasing population
Inflation
Strikes
Changes in government
Changes in law
Try and understand all possible sources of variation
before modelling the time series.
Forecasting using R

Adjustments

25

Explainable variation
Examples:
Calendar variation
Increasing population
Inflation
Strikes
Changes in government
Changes in law
Try and understand all possible sources of variation
before modelling the time series.
Forecasting using R

Adjustments

25