Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: THEend8_
Time Series Analysis
STA457H1
ARIMA Models
Lecture 5
Learning Objectives
By the end of this chapter, you should be able to do the following:
Understand the sufficient and necessarily conditions for AR,MA, and ARMA
models.
Evaluate stationary, causality, and invertibility in time series models.
Use R with real time series data and interpret outcomes of analyses.
Linear Processes
The Box-Jenkins methodology requires that the model used in describing and forecasting a time
series be both stationary and invertible.
Definition
The process {xt} is a stationary if it remains in statistical equilibrium with probabilistic
properties that do not change over time, in particular varying about a fixed constant mean
level and with constant variance.
The process {xt} is an invertible if its weights decline (does not depend on time), and hence
the Box-Jenkins model can be used to express xt as a function of past x observations (that
is, xt−1, xt−2, . . . ).
For a stationary time series without trends or seasonal effects. That is; if necessary, any
trends, seasonal or cyclical effects have already been removed from the series, we might
construct a linear model for a time series with autocorrelation.
The most important special cases of linear processes are:
Autoregressive (AR) Model.
Moving-Average (MA) Model.
Autoregressive Moving Average (ARMA) Model.
Adding nonstationary models to the mix ARMA models leads to the autoregressive
integrated moving average (ARIMA) model.
Autoregressive (AR) Models with Zero Mean
Definition
A time series {xt}, with zero mean, is called autoregressive process of order p
and denoted by AR(p), if it can be written in the form:
xt = ϕ1xt−1 + ϕ2xt−2 + · · ·+ ϕpxt−p + wt (1)
where wt is a random shock, usually assumed to be white noise, i.e.,
wt ∼ wn(0, σ2w ), and ϕ1, · · · , ϕp(ϕp ̸= 0) are parameters.
With backshift operator, the AR process can be written as
Φp(B)xt = wt , (2)
where Φp(B) = 1− ϕ1B − ϕ2B2 − · · · − ϕpBp is a polynomial of degree p.
Autoregressive (AR) Models with Mean µ
If the mean, µ, of xt is not zero, replace xt by xt − µ to get:
xt − µ = ϕ1(xt−1 − µ) + ϕ2(xt−2 − µ) + · · ·+ ϕp(xt−p − µ) + wt (3)
or write
xt = δ + ϕ1xt−1 + ϕ2xt−2 + · · ·+ ϕpxt−p + wt , (4)
where
δ = µ(1− ϕ1 − ϕ2 − · · · − ϕp)
For example, the AR(2) model
xt = 1.5 + 1.2xt−1 − 0.5xt−2 + wt
is
xt − µ = 1.2(xt−1 − µ)− 0.5(xt−2 − µ) + wt
where 1.5 = µ(1− 1.2− (−0.5))⇒ 1.5 = 0.3µ.
Thus, the model has mean µ = 5 and the model can be written as
xt − 5 = 1.2(xt−1 − 5)− 0.5(xt−2 − 5) + wt
A Simulated AR(1) Series: xt = 0.8xt−1 + wt
set.seed(12345)
ar1.sim<-arima.sim(n=100,list(order=c(1,0,0),ar=0.8))
par(mfrow=c(1,3))
plot(ar1.sim, ylab="x(t)", main="Simulated AR(1)",lwd=2)
acf(ar1.sim, 50, main="Autocorrelation Function",lwd=2)
acf(ar1.sim, type="partial", main="Partial ACF",lwd=2)
Simulated AR(1)
Time
x(t
)
0 20 40 60 80 100
−
2
0
2
4
0 10 20 30 40 50
−
0.
2
0.
0
0.
2
0.
4
0.
6
0.
8
1.
0
Lag
AC
F
Autocorrelation Function
5 10 15 20
−
0.
2
0.
0
0.
2
0.
4
0.
6
Lag
Pa
rti
al
A
CF
Partial ACF
Figure: A simulated AR(1) process: (1− 0.8B)xt = wt
A Simulated AR(1) Series: xt = −0.8xt−1 + wt
set.seed(12345)
ar11.sim<-arima.sim(n=100,list(order=c(1,0,0),ar=-0.8))
par(mfrow=c(1,3))
plot(ar11.sim, ylab="x(t)", main="Simulated AR(1)",lwd=2)
acf(ar11.sim, main="Autocorrelation Function",lwd=2)
acf(ar11.sim, type="partial", main="Partial ACF",lwd=2)
Simulated AR(1)
Time
x(t
)
0 20 40 60 80 100
−
4
−
2
0
2
0 5 10 15 20
−
0.
5
0.
0
0.
5
1.
0
Lag
AC
F
Autocorrelation Function
5 10 15 20
−
0.
6
−
0.
4
−
0.
2
0.
0
0.
2
Lag
Pa
rti
al
A
CF
Partial ACF
Figure: A simulated AR(1) process: (1 + 0.8B)xt = wt
A Simulated AR(2) Series: xt = 1.1xt−1 − 0.3xt−2 + wt
set.seed(1965)
ar2<-arima.sim(n=200,list(order=c(2,0,0),ar=c(1.1,-0.3)))
par(mfrow=c(1,3))
plot(ar2, ylab="x(t)", main="Simulated AR(2)",lwd=2)
acf(ar2, main="Autocorrelation Function",lwd=2)
acf(ar2, type="partial", main="Partial ACF",lwd=2)
Simulated AR(2)
Time
x(t
)
0 50 100 150 200
−
4
−
2
0
2
4
0 5 10 15 20
0.
0
0.
2
0.
4
0.
6
0.
8
1.
0
Lag
AC
F
Autocorrelation Function
5 10 15 20
−
0.
2
0.
0
0.
2
0.
4
0.
6
0.
8
Lag
Pa
rti
al
A
CF
Partial ACF
Figure: A simulated AR(2) process: (1− 1.1B + 0.3B2)xt = wt
Causal Conditions for an of AR(1) Process
The autoregressive process of order 1, AR(1), xt = ϕxt−1 + wt (equivalently
(1− ϕB)xt = wt ) is a causal process if it is stationary with values that are not
depending on the future. In this case, the absolute value of the root of (1− ϕz) = 0
must lie outside the unit circle.
That is, the AR(1) is causal process if
|z| = |1
ϕ
| > 1 or equivalently if |ϕ| < 1 (5)
Note that causality implies stationarity but not the other way around.
If the process is not stationary then it will not causal.
Examples
1.) (1− 0.4B)xt = wt is a causal process because the absolute value of the root of
(1− 0.4z) = 0 is |z| = |1/0.4| = 2.5 > 1.
2.) (1 + 1.8B)xt = wt is not stationary process (not casual) because |root| of
(1 + 1.8z) = 0 is |z| = | 1−1.8 | = 0.56 < 1.
3.) xt = 0.5xt−1 + wt is a causal process because |0.5| < 1.
Causal Conditions for an of AR(2) Process
The AR(2) model,
xt = ϕ1xt−1 + ϕ2xt−2 + wt ,
is causal when the two roots of 1− ϕ1z − ϕ2z2 lie outside of the unit circle. Using the quadratic
formula, this requirement can be written as
∣∣∣∣ϕ1 ±
√
ϕ21 + 4ϕ2
−2ϕ2
∣∣∣∣ > 1.
In this case, the necessary and sufficient conditions for causality are:
|ϕ2| < 1, ϕ1 + ϕ2 < 1, and ϕ2 − ϕ1 < 1
Recall the solution for quadratic equation p(z) = a+ bz + cz2 is
z =
−b ±
√
b2 − 4ac
2a
Examples
1.) xt = 1.1xt−1 − 0.4xt−2 + wt is causal.
2.) xt = 0.6xt−1 − 1.3xt−2 + wt is not stationary (|ϕ2| ≮ 1).
3.) xt = 0.6xt−1 + 0.8xt−2 + wt is not stationary (ϕ1 + ϕ2 ≮ 1).
4.) xt = −0.4xt−1 + 0.7xt−2 + wt is not stationary (ϕ2 − ϕ1 ≮ 1).
Causal Conditions for an of AR(p) Process
The Autoregressive process of order p, AR(p)
xt = ϕ1xt−1 + · · ·+ ϕpxt−p + wt ,
is said to be casual if the all absolute values of the roots of
1− ϕ1z − ϕ2z2 − · · · − ϕpzp = 0 lie outside the unit circle.
Examples
1.) xt = −0.4xt−1 + 0.21xt−2 + wt is a casual process as |roots| of
1 + 0.4z − 0.21z2 = (1− 0.3z)(1 + 0.7z) = 0 are | 10.3 | = 3.3 > 1 and
| 1−0.7 | = 1.4 > 1.
2.) xt = 1.7xt−1 − 0.42xt−2 + wt is not a stationary process because the
absolute value of one of the two roots of
1− 1.7z + 0.42z2 = (1− 0.3z)(1− 1.4z) = 0 lies inside the unit circle,
which is | 11.4 | = 0.71 < 1.
3.) xt = −0.25xt−2 + wt is a casual process because |roots| of
1 + 0.25z2 = 0 are | ± 2i | = 2 > 1, where i = √−1.
Finding the Roots in R
In R, the function polyroot(a), where a is the vector of polynomial coefficients
in increasing order can be used to find the zeros of a real or complex polynomial
of degree n − 1
p(x) = a1 + a2x + · · ·+ anxn−1.
The function Mod() can be used to check whether the roots of p(x) have
modulus > 1 or not.
Example
The AR(4) process xt = −0.3xt−1 + 0.7xt−2 − 1.2xt−3 + 0.1xt−4 + wt is not stationary
as
roots <- polyroot(c(1,-0.3, 0.7,-1.2,0.1))
Mod(roots)
## [1] 0.8829242 0.8829242 1.1250090 11.4024243
Moving Average (MA) Models
Definition
A time series {xt}, with zero mean, is called moving Average process of order
q and denoted by MA(q), if it can be written in the form:
xt = wt + θ1wt−1 + θ2wt−2 + · · ·+ θqwt−q (6)
where wt is a white noise, wt ∼ wn(0, σ2w ), and θ1, · · · , θq(θq ̸= 0) are
parameters.
With backshift operator, the MA process can be written as
xt = Θq(B)wt , (7)
where Θq(B) = 1 + θ1B + θ2B2 + · · ·+ θqBq is polynomial of degree q.
The MA(q), ∀q ≥ 1, process is always stationary (regardless of the
values of θj , j = 1,2, · · · ,q).
If the |roots| of 1 + θ1z + θ2z2 + · · ·+ θqzq = 0 > 1, the MA(q) process is
said to be invertible.
A Simulated MA(1) Series: xt = wt + 0.8wt−1
set.seed(6436)
ma1.sim<-arima.sim(n=200,list(order=c(0,0,1),ma=0.8))
par(mfrow=c(1,3))
plot(ma1.sim, ylab="x(t)", main="Simulated MA(1)",lwd=2)
acf(ma1.sim, main="Autocorrelation Function",lwd=2)
acf(ma1.sim, type="partial", main="Partial ACF",lwd=2)
Simulated MA(1)
Time
x(t
)
0 50 100 150 200
−
3
−
2
−
1
0
1
2
3
4
0 5 10 15 20
−
0.
2
0.
0
0.
2
0.
4
0.
6
0.
8
1.
0
Lag
AC
F
Autocorrelation Function
5 10 15 20
−
0.
2
0.
0
0.
2
0.
4
Lag
Pa
rti
al
A
CF
Partial ACF
Figure: A simulated MA(1) process: xt = (1 + 0.8B)wt
A Simulated MA(1) Series: xt = wt − 0.8wt−1
set.seed(6436)
ma11.sim<-arima.sim(n=200,list(order=c(0,0,1),ma=-0.8))
par(mfrow=c(1,3))
plot(ma11.sim, ylab="x(t)", main="Simulated MA(1)",lwd=2)
acf(ma11.sim, main="Autocorrelation Function",lwd=2)
acf(ma11.sim, type="partial", main="Partial ACF",lwd=2)
Simulated MA(1)
Time
x(t
)
0 50 100 150 200
−
4
−
2
0
2
0 5 10 15 20
−
0.
5
0.
0
0.
5
1.
0
Lag
AC
F
Autocorrelation Function
5 10 15 20
−
0.
4
−
0.
3
−
0.
2
−
0.
1
0.
0
0.
1
Lag
Pa
rti
al
A
CF
Partial ACF
Figure: A simulated MA(1) process: xt = (1− 0.8B)wt
A Simulated MA(2) Series: xt = wt +0.8wt−1 +0.6wt−2
set.seed(14762)
ma2<-arima.sim(n=200,list(order=c(0,0,2),ma=c(0.8,0.6)))
par(mfrow=c(1,3))
plot(ma2, ylab="x(t)", main="Simulated MA(2)",lwd=2)
acf(ma2, main="Autocorrelation Function",lwd=2)
acf(ma2, type="partial", main="Partial ACF",lwd=2)
Simulated MA(2)
Time
x(t
)
0 50 100 150 200
−
4
−
2
0
2
4
0 5 10 15 20
0.
0
0.
2
0.
4
0.
6
0.
8
1.
0
Lag
AC
F
Autocorrelation Function
5 10 15 20
−
0.
2
0.
0
0.
2
0.
4
0.
6
Lag
Pa
rti
al
A
CF
Partial ACF
Figure: A simulated MA(2) process: xt = (1 + 0.8B + 0.6B2)wt
Non-uniqueness of MA Models and Invertibility
If we replace θ by 1/θ, the autocorrelation function for MA(1), ρ(h) = θ1+θ2
(studied later), will not be changed. There are thus two processes,
xt = wt + θwt−1
and
xt = wt +
1
θ
wt−1,
show identical autocorrelation pattern, and hence the MA(1) coefficient is
not uniquely identified.
For the general moving average process, MA(q), there is a similar
identifiability problem.
The problem can be resolved by assuming that the following operator is
invertible
1 + θ1B + θ2B2 + · · ·+ θqBq
i.e., all absolute roots of 1 + θ1z + θ2z2 + · · ·+ θqzq = 0 lie outside the
unit circle.
Examples - Invertibility of MA Models
Example
1.) The MA(1) process xt = wt + 0.4wt−1 is an invertible process because
|θ| = |0.4| < 1 (or equivalently the root of 1 + 0.4z = 0 in absolute value
is z = 1/0.4 = 2.5 > 1).
2.) The MA(1) process xt = w1 + 1.8wt−1, is not invertible process because
the root of 1 + 1.8z = 0 in absolute value is z = 1/1.8 = 0.56 < 1.
3.) The MA(3) process xt = wt − 0.3wt−1 + 0.7wt−2 − 1.2wt−3 is not
invertible as
roots <- polyroot(c(1,-0.3, 0.7,-1.2))
Mod(roots)
## [1] 0.8810509 0.8810509 1.0735363
Mixed Autoregressive-Moving Average (ARMA)
Models
Definition
A time series {xt}, with zero mean, is called Autoregressive-Moving Average
process of order (p,q) and denoted by ARMA(p,q), if it can be written as
xt =
p∑
i=1
ϕixt−i︸ ︷︷ ︸
Autoregressive Part
+
q∑
j=0
θjwt−j︸ ︷︷ ︸
Moving Average Part
, (8)
where wt ∼ wn(0, σ2w ) is white noise, ϕi & θj for i = 1,2, · · · ,p, j = 0,1, · · · ,q,
are the parameters of the Autoregressive & Moving Average parts
respectively, and θ0 = 1.
Mixed Autoregressive-Moving Average (ARMA)
Models
With backshift operator, the process can be rewritten as:
Φp(B)xt︸ ︷︷ ︸
Autoregressive Part
= Θq(B)wt︸ ︷︷ ︸
Moving Average Part
, (9)
where Φp(B) = 1−∑pi=1 ϕiB i and Θq(B) = 1 +∑qj=1 θjB j .
If the mean, µ, of xt is not zero, replace xt by xt − µ to get:
Φp(B)(xt − µ) = Θq(B)wt .
can also be written as:
xt = δ + ϕ1xt−1 + · · ·+ ϕpxt−p︸ ︷︷ ︸
Autoregressive Part
+wt + θ1wt−1 + · · ·+ θqwt−q︸ ︷︷ ︸
Moving Average Part
, (10)
where δ = µ(1− ϕ1 − ϕ2 − · · · − ϕp) and µ is called the intercept obtained from
the output of the R function arima().
Three Conditions for ARMA Models
The ARMA model is assumed to be stationary, invertible and identifiable:
The condition for the stationarity is the same for the pure AR(p) process:
i.e., the roots of 1− ϕ1z − ϕ2z2 − · · · − ϕpzp lie outside the unit circle.
The condition for the invertibility is the same for the pure MA(q) process:
i.e., the roots of 1 + θ1z + θ2z2 + · · ·+ θqzq lie outside the unit circle.
The identifiability condition means that the model is not redundant: i.e.,
Φp(B) = 0 and Θq(B) = 0 have no common roots.
Stationary and Invertibility Conditions for ARMA
Models
Model Stationary conditions Invertibility conditions
MA(1) None |θ1| < 1
xt = δ + at + θ1at−1
MA(2): None θ1 + θ2 < 1
xt = δ + at + θ1at−1 + θ2at−2 θ2 − θ1 < 1
|θ2| < 1
AR(1)
xt = δ + ϕ1xt−1 + at |ϕ1| < 1 None
AR(2) ϕ1 + ϕ2 < 1 None
xt = δ + ϕ1xt−1 + ϕ2xt−2 + at ϕ2 − ϕ1 < 1
|ϕ2| < 1
ARMA(1, 1)
xt = δ + ϕ1xt−1 + at + θ1at−1 |ϕ1| < 1 |θ1| < 1
Example: Model Redundancy (Shared Common
Roots in ARMA Models)
Consider the ARMA(1,2):
xt = 0.2xt−1 + wt − 1.1wt−1 + 0.18wt−2,
this model can be written as
(1− 0.2B)xt = (1− 1.1B + 0.18B2)wt
or equivalently
(1− 0.2B)xt = (1− 0.2B)(1− 0.9B)wt
cancelling (1− 0.2B) from both sides to get:
xt = (1− 0.9B)wt
Thus, the process is not really an ARMA(1,2), but it is a MA(1) ≡ ARMA(0,1).