Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: THEend8_
Time Series Analysis
STA457H1 ARIMA Models Lecture 5
Learning Objectives By the end of this chapter, you should be able to do the following: Understand the sufficient and necessarily conditions for AR,MA, and ARMA models. Evaluate stationary, causality, and invertibility in time series models. Use R with real time series data and interpret outcomes of analyses.
Linear Processes The Box-Jenkins methodology requires that the model used in describing and forecasting a time series be both stationary and invertible. Definition The process {xt} is a stationary if it remains in statistical equilibrium with probabilistic properties that do not change over time, in particular varying about a fixed constant mean level and with constant variance. The process {xt} is an invertible if its weights decline (does not depend on time), and hence the Box-Jenkins model can be used to express xt as a function of past x observations (that is, xt−1, xt−2, . . . ). For a stationary time series without trends or seasonal effects. That is; if necessary, any trends, seasonal or cyclical effects have already been removed from the series, we might construct a linear model for a time series with autocorrelation. The most important special cases of linear processes are: Autoregressive (AR) Model. Moving-Average (MA) Model. Autoregressive Moving Average (ARMA) Model. Adding nonstationary models to the mix ARMA models leads to the autoregressive integrated moving average (ARIMA) model. Autoregressive (AR) Models with Zero Mean Definition A time series {xt}, with zero mean, is called autoregressive process of order p and denoted by AR(p), if it can be written in the form: xt = ϕ1xt−1 + ϕ2xt−2 + · · ·+ ϕpxt−p + wt (1) where wt is a random shock, usually assumed to be white noise, i.e., wt ∼ wn(0, σ2w ), and ϕ1, · · · , ϕp(ϕp ̸= 0) are parameters. With backshift operator, the AR process can be written as Φp(B)xt = wt , (2) where Φp(B) = 1− ϕ1B − ϕ2B2 − · · · − ϕpBp is a polynomial of degree p.
Autoregressive (AR) Models with Mean µ If the mean, µ, of xt is not zero, replace xt by xt − µ to get: xt − µ = ϕ1(xt−1 − µ) + ϕ2(xt−2 − µ) + · · ·+ ϕp(xt−p − µ) + wt (3) or write xt = δ + ϕ1xt−1 + ϕ2xt−2 + · · ·+ ϕpxt−p + wt , (4) where δ = µ(1− ϕ1 − ϕ2 − · · · − ϕp) For example, the AR(2) model xt = 1.5 + 1.2xt−1 − 0.5xt−2 + wt is xt − µ = 1.2(xt−1 − µ)− 0.5(xt−2 − µ) + wt where 1.5 = µ(1− 1.2− (−0.5))⇒ 1.5 = 0.3µ. Thus, the model has mean µ = 5 and the model can be written as xt − 5 = 1.2(xt−1 − 5)− 0.5(xt−2 − 5) + wt
A Simulated AR(1) Series: xt = 0.8xt−1 + wt set.seed(12345) ar1.sim<-arima.sim(n=100,list(order=c(1,0,0),ar=0.8)) par(mfrow=c(1,3)) plot(ar1.sim, ylab="x(t)", main="Simulated AR(1)",lwd=2) acf(ar1.sim, 50, main="Autocorrelation Function",lwd=2) acf(ar1.sim, type="partial", main="Partial ACF",lwd=2) Simulated AR(1) Time x(t ) 0 20 40 60 80 100 − 2 0 2 4 0 10 20 30 40 50 − 0. 2 0. 0 0. 2 0. 4 0. 6 0. 8 1. 0 Lag AC F Autocorrelation Function 5 10 15 20 − 0. 2 0. 0 0. 2 0. 4 0. 6 Lag Pa rti al A CF Partial ACF Figure: A simulated AR(1) process: (1− 0.8B)xt = wt
A Simulated AR(1) Series: xt = −0.8xt−1 + wt set.seed(12345) ar11.sim<-arima.sim(n=100,list(order=c(1,0,0),ar=-0.8)) par(mfrow=c(1,3)) plot(ar11.sim, ylab="x(t)", main="Simulated AR(1)",lwd=2) acf(ar11.sim, main="Autocorrelation Function",lwd=2) acf(ar11.sim, type="partial", main="Partial ACF",lwd=2) Simulated AR(1) Time x(t ) 0 20 40 60 80 100 − 4 − 2 0 2 0 5 10 15 20 − 0. 5 0. 0 0. 5 1. 0 Lag AC F Autocorrelation Function 5 10 15 20 − 0. 6 − 0. 4 − 0. 2 0. 0 0. 2 Lag Pa rti al A CF Partial ACF Figure: A simulated AR(1) process: (1 + 0.8B)xt = wt
A Simulated AR(2) Series: xt = 1.1xt−1 − 0.3xt−2 + wt set.seed(1965) ar2<-arima.sim(n=200,list(order=c(2,0,0),ar=c(1.1,-0.3))) par(mfrow=c(1,3)) plot(ar2, ylab="x(t)", main="Simulated AR(2)",lwd=2) acf(ar2, main="Autocorrelation Function",lwd=2) acf(ar2, type="partial", main="Partial ACF",lwd=2) Simulated AR(2) Time x(t ) 0 50 100 150 200 − 4 − 2 0 2 4 0 5 10 15 20 0. 0 0. 2 0. 4 0. 6 0. 8 1. 0 Lag AC F Autocorrelation Function 5 10 15 20 − 0. 2 0. 0 0. 2 0. 4 0. 6 0. 8 Lag Pa rti al A CF Partial ACF Figure: A simulated AR(2) process: (1− 1.1B + 0.3B2)xt = wt
Causal Conditions for an of AR(1) Process The autoregressive process of order 1, AR(1), xt = ϕxt−1 + wt (equivalently (1− ϕB)xt = wt ) is a causal process if it is stationary with values that are not depending on the future. In this case, the absolute value of the root of (1− ϕz) = 0 must lie outside the unit circle. That is, the AR(1) is causal process if |z| = |1 ϕ | > 1 or equivalently if |ϕ| < 1 (5) Note that causality implies stationarity but not the other way around. If the process is not stationary then it will not causal. Examples 1.) (1− 0.4B)xt = wt is a causal process because the absolute value of the root of (1− 0.4z) = 0 is |z| = |1/0.4| = 2.5 > 1. 2.) (1 + 1.8B)xt = wt is not stationary process (not casual) because |root| of (1 + 1.8z) = 0 is |z| = | 1−1.8 | = 0.56 < 1. 3.) xt = 0.5xt−1 + wt is a causal process because |0.5| < 1.
Causal Conditions for an of AR(2) Process The AR(2) model, xt = ϕ1xt−1 + ϕ2xt−2 + wt , is causal when the two roots of 1− ϕ1z − ϕ2z2 lie outside of the unit circle. Using the quadratic formula, this requirement can be written as ∣∣∣∣ϕ1 ± √ ϕ21 + 4ϕ2 −2ϕ2 ∣∣∣∣ > 1. In this case, the necessary and sufficient conditions for causality are: |ϕ2| < 1, ϕ1 + ϕ2 < 1, and ϕ2 − ϕ1 < 1 Recall the solution for quadratic equation p(z) = a+ bz + cz2 is z = −b ± √ b2 − 4ac 2a Examples 1.) xt = 1.1xt−1 − 0.4xt−2 + wt is causal. 2.) xt = 0.6xt−1 − 1.3xt−2 + wt is not stationary (|ϕ2| ≮ 1). 3.) xt = 0.6xt−1 + 0.8xt−2 + wt is not stationary (ϕ1 + ϕ2 ≮ 1). 4.) xt = −0.4xt−1 + 0.7xt−2 + wt is not stationary (ϕ2 − ϕ1 ≮ 1).
Causal Conditions for an of AR(p) Process The Autoregressive process of order p, AR(p) xt = ϕ1xt−1 + · · ·+ ϕpxt−p + wt , is said to be casual if the all absolute values of the roots of 1− ϕ1z − ϕ2z2 − · · · − ϕpzp = 0 lie outside the unit circle. Examples 1.) xt = −0.4xt−1 + 0.21xt−2 + wt is a casual process as |roots| of 1 + 0.4z − 0.21z2 = (1− 0.3z)(1 + 0.7z) = 0 are | 10.3 | = 3.3 > 1 and | 1−0.7 | = 1.4 > 1. 2.) xt = 1.7xt−1 − 0.42xt−2 + wt is not a stationary process because the absolute value of one of the two roots of 1− 1.7z + 0.42z2 = (1− 0.3z)(1− 1.4z) = 0 lies inside the unit circle, which is | 11.4 | = 0.71 < 1. 3.) xt = −0.25xt−2 + wt is a casual process because |roots| of 1 + 0.25z2 = 0 are | ± 2i | = 2 > 1, where i = √−1.
Finding the Roots in R In R, the function polyroot(a), where a is the vector of polynomial coefficients in increasing order can be used to find the zeros of a real or complex polynomial of degree n − 1 p(x) = a1 + a2x + · · ·+ anxn−1. The function Mod() can be used to check whether the roots of p(x) have modulus > 1 or not. Example The AR(4) process xt = −0.3xt−1 + 0.7xt−2 − 1.2xt−3 + 0.1xt−4 + wt is not stationary as roots <- polyroot(c(1,-0.3, 0.7,-1.2,0.1)) Mod(roots) ## [1] 0.8829242 0.8829242 1.1250090 11.4024243
Moving Average (MA) Models Definition A time series {xt}, with zero mean, is called moving Average process of order q and denoted by MA(q), if it can be written in the form: xt = wt + θ1wt−1 + θ2wt−2 + · · ·+ θqwt−q (6) where wt is a white noise, wt ∼ wn(0, σ2w ), and θ1, · · · , θq(θq ̸= 0) are parameters. With backshift operator, the MA process can be written as xt = Θq(B)wt , (7) where Θq(B) = 1 + θ1B + θ2B2 + · · ·+ θqBq is polynomial of degree q. The MA(q), ∀q ≥ 1, process is always stationary (regardless of the values of θj , j = 1,2, · · · ,q). If the |roots| of 1 + θ1z + θ2z2 + · · ·+ θqzq = 0 > 1, the MA(q) process is said to be invertible.
A Simulated MA(1) Series: xt = wt + 0.8wt−1 set.seed(6436) ma1.sim<-arima.sim(n=200,list(order=c(0,0,1),ma=0.8)) par(mfrow=c(1,3)) plot(ma1.sim, ylab="x(t)", main="Simulated MA(1)",lwd=2) acf(ma1.sim, main="Autocorrelation Function",lwd=2) acf(ma1.sim, type="partial", main="Partial ACF",lwd=2) Simulated MA(1) Time x(t ) 0 50 100 150 200 − 3 − 2 − 1 0 1 2 3 4 0 5 10 15 20 − 0. 2 0. 0 0. 2 0. 4 0. 6 0. 8 1. 0 Lag AC F Autocorrelation Function 5 10 15 20 − 0. 2 0. 0 0. 2 0. 4 Lag Pa rti al A CF Partial ACF Figure: A simulated MA(1) process: xt = (1 + 0.8B)wt
A Simulated MA(1) Series: xt = wt − 0.8wt−1 set.seed(6436) ma11.sim<-arima.sim(n=200,list(order=c(0,0,1),ma=-0.8)) par(mfrow=c(1,3)) plot(ma11.sim, ylab="x(t)", main="Simulated MA(1)",lwd=2) acf(ma11.sim, main="Autocorrelation Function",lwd=2) acf(ma11.sim, type="partial", main="Partial ACF",lwd=2) Simulated MA(1) Time x(t ) 0 50 100 150 200 − 4 − 2 0 2 0 5 10 15 20 − 0. 5 0. 0 0. 5 1. 0 Lag AC F Autocorrelation Function 5 10 15 20 − 0. 4 − 0. 3 − 0. 2 − 0. 1 0. 0 0. 1 Lag Pa rti al A CF Partial ACF Figure: A simulated MA(1) process: xt = (1− 0.8B)wt
A Simulated MA(2) Series: xt = wt +0.8wt−1 +0.6wt−2 set.seed(14762) ma2<-arima.sim(n=200,list(order=c(0,0,2),ma=c(0.8,0.6))) par(mfrow=c(1,3)) plot(ma2, ylab="x(t)", main="Simulated MA(2)",lwd=2) acf(ma2, main="Autocorrelation Function",lwd=2) acf(ma2, type="partial", main="Partial ACF",lwd=2) Simulated MA(2) Time x(t ) 0 50 100 150 200 − 4 − 2 0 2 4 0 5 10 15 20 0. 0 0. 2 0. 4 0. 6 0. 8 1. 0 Lag AC F Autocorrelation Function 5 10 15 20 − 0. 2 0. 0 0. 2 0. 4 0. 6 Lag Pa rti al A CF Partial ACF Figure: A simulated MA(2) process: xt = (1 + 0.8B + 0.6B2)wt
Non-uniqueness of MA Models and Invertibility If we replace θ by 1/θ, the autocorrelation function for MA(1), ρ(h) = θ1+θ2 (studied later), will not be changed. There are thus two processes, xt = wt + θwt−1 and xt = wt + 1 θ wt−1, show identical autocorrelation pattern, and hence the MA(1) coefficient is not uniquely identified. For the general moving average process, MA(q), there is a similar identifiability problem. The problem can be resolved by assuming that the following operator is invertible 1 + θ1B + θ2B2 + · · ·+ θqBq i.e., all absolute roots of 1 + θ1z + θ2z2 + · · ·+ θqzq = 0 lie outside the unit circle.
Examples - Invertibility of MA Models Example 1.) The MA(1) process xt = wt + 0.4wt−1 is an invertible process because |θ| = |0.4| < 1 (or equivalently the root of 1 + 0.4z = 0 in absolute value is z = 1/0.4 = 2.5 > 1). 2.) The MA(1) process xt = w1 + 1.8wt−1, is not invertible process because the root of 1 + 1.8z = 0 in absolute value is z = 1/1.8 = 0.56 < 1. 3.) The MA(3) process xt = wt − 0.3wt−1 + 0.7wt−2 − 1.2wt−3 is not invertible as roots <- polyroot(c(1,-0.3, 0.7,-1.2)) Mod(roots) ## [1] 0.8810509 0.8810509 1.0735363
Mixed Autoregressive-Moving Average (ARMA) Models Definition A time series {xt}, with zero mean, is called Autoregressive-Moving Average process of order (p,q) and denoted by ARMA(p,q), if it can be written as xt = p∑ i=1 ϕixt−i︸ ︷︷ ︸ Autoregressive Part + q∑ j=0 θjwt−j︸ ︷︷ ︸ Moving Average Part , (8) where wt ∼ wn(0, σ2w ) is white noise, ϕi & θj for i = 1,2, · · · ,p, j = 0,1, · · · ,q, are the parameters of the Autoregressive & Moving Average parts respectively, and θ0 = 1.
Mixed Autoregressive-Moving Average (ARMA) Models With backshift operator, the process can be rewritten as: Φp(B)xt︸ ︷︷ ︸ Autoregressive Part = Θq(B)wt︸ ︷︷ ︸ Moving Average Part , (9) where Φp(B) = 1−∑pi=1 ϕiB i and Θq(B) = 1 +∑qj=1 θjB j . If the mean, µ, of xt is not zero, replace xt by xt − µ to get: Φp(B)(xt − µ) = Θq(B)wt . can also be written as: xt = δ + ϕ1xt−1 + · · ·+ ϕpxt−p︸ ︷︷ ︸ Autoregressive Part +wt + θ1wt−1 + · · ·+ θqwt−q︸ ︷︷ ︸ Moving Average Part , (10) where δ = µ(1− ϕ1 − ϕ2 − · · · − ϕp) and µ is called the intercept obtained from the output of the R function arima().
Three Conditions for ARMA Models The ARMA model is assumed to be stationary, invertible and identifiable: The condition for the stationarity is the same for the pure AR(p) process: i.e., the roots of 1− ϕ1z − ϕ2z2 − · · · − ϕpzp lie outside the unit circle. The condition for the invertibility is the same for the pure MA(q) process: i.e., the roots of 1 + θ1z + θ2z2 + · · ·+ θqzq lie outside the unit circle. The identifiability condition means that the model is not redundant: i.e., Φp(B) = 0 and Θq(B) = 0 have no common roots.
Example: Model Redundancy (Shared Common Roots in ARMA Models) Consider the ARMA(1,2): xt = 0.2xt−1 + wt − 1.1wt−1 + 0.18wt−2, this model can be written as (1− 0.2B)xt = (1− 1.1B + 0.18B2)wt or equivalently (1− 0.2B)xt = (1− 0.2B)(1− 0.9B)wt cancelling (1− 0.2B) from both sides to get: xt = (1− 0.9B)wt Thus, the process is not really an ARMA(1,2), but it is a MA(1) ≡ ARMA(0,1).