LECTURE 5

Linear Stochastic Models

Autcovariances of a Stationary Process

A temporal stochastic process is simply a sequence of random variables
indexed by a time subscript. Such a process can be denoted by x(t). The
element of the sequence at the point t = τ is xτ = x(τ ).
Let {xτ +1 , xτ +2 , . . . , xτ +n } denote n consecutive elements of the sequence.
Then the process is said to be strictly stationary if the joint probability distri-
bution of the elements does not depend on τ regardless of the size of n. This
means that any two segments of the sequence of equal length have identical
probability density functions. In consequence, the decision on where to place
the time origin is arbitrary; and the argument τ can be omitted. Some further
implications of stationarity are that

(5.1) E(xt ) = µ < ∞ for all t and C(xτ +t , xτ +s ) = γ|t−s| .

The latter condition means that the covariance of any two elements depends
only on their temporal separation |t − s|. Notice that, if the elements of the
sequence are normally distributed, then the two conditions are sufficient to
establish strict stationarity. On their own, they constitute the conditions of
weak or 2nd-order stationarity.
The condition on the covariances implies that the dispersion matrix of the
vector [x1 , x2 , . . . , xn ] is a bisymmetric Laurent matrix of the form
 γ γ1 γ2 ... γn−1 
0
 γ1 γ0 γ1 ... γn−2 
 
(5.2) Γ= γ2
 .
γ1 γ0 ... γn−3  ,
 . .. .. .. .. 

. . . . .
γn−1 γn−2 γn−3 ... γ0

wherein the generic element in the (i, j)th position is γ|i−j| = C(xi , xj ). Given
that a sequence of observations of a time series represents only a segment of
a single realisation of a stochastic process, one might imagine that there is
little chance of making valid inferences about the parameters of the process.

66
 D.S.G. POLLOCK : LINEAR STOCHASTIC MODELS

However, provided that the process x(t) is stationary and provided that the
statistical dependencies between widely separated elements of the sequence are
weak, it is possible to estimate consistently those parameters of the process
which express the dependence of proximate elements of the sequence. If one
is prepared to make sufficiently strong assumptions about the nature of the
process, then a knowledge of such parameters may be all that is needed for a
complete characterisation of the process.

Moving-Average Processes
The qth-order moving average process, or MA(q) process, is defined by the
equation
(5.3) y(t) = µ0 ε(t) + µ1 ε(t − 1) + · · · + µq ε(t − q),
where ε(t), which has E{ε(t)} = 0, is a white-noise process consisting of a
sequence of independently and identically distributed random variables with
zero expectations. The equation is normalised either by setting µ0 = 1 or by
setting V {ε(t)} = σε2 = 1. The equation can be written in summary notation
as y(t) = µ(L)ε(t), where µ(L) = µ0 + µ1 L + · · · + µq Lq is a polynomial in the
lag operator.
A moving-average process is clearly stationary since any two elements
yt and ys represent the same function of the vectors [εt , εt−1 , . . . , εt−q ] and
[εs , εs−1 , . . . , εs−q ] which are identically distributed. In addition to the condi-
tion of stationarity, it is usually required that a moving-average process should
be invertible such that it can be expressed in the form of µ−1 (L)y(t) = ε(t)
where the LHS embodies a convergent sum of past values of y(t). This is an
infinite-order autoregressive representation of the process. The representation
is available only if all the roots of the equation µ(z) = µ0 + µ1 z + · · · + µq z q = 0
lie outside the unit circle. This conclusion follows from our discussion of partial
fractions.
As an example, let us consider the first-order moving-average process which
is defined by
(5.4) y(t) = ε(t) − θε(t − 1) = (1 − θL)ε(t).
Provided that |θ| < 1, this can be written in autoregressive form as
ε(t) = (1 − θL)−1 y(t)
(5.5) © ª
= y(t) + θy(t − 1) + θ2 y(t − 2) + · · · .
Imagine that |θ| > 1 instead. Then, to obtain a convergent series, we have to
write
y(t + 1) = ε(t + 1) − θε(t)
(5.6)
= −θ(1 − L−1 /θ)ε(t),

67
 D.S.G. POLLOCK : TIME SERIES AND FORECASTING

where L−1 ε(t) = ε(t + 1). This gives

ε(t) = −θ−1 (1 − L−1 /θ)−1 y(t + 1)

(5.7) © ª
= −θ−1 y(t + 1)/θ + y(t + 2)/θ2 + y(t − 3)/θ3 + · · · .

Normally, an expression such as this, which embodies future values of y(t),

would have no reasonable meaning.
It is straightforward to generate the sequence of autocovariances from a
knowledge of the parameters of the moving-average process and of the variance
of the white-noise process. Consider

γτ = E(yt yt−τ )
nX X o
=E µi εt−i µj εt−τ −j
(5.8) i j
XX
= µi µj E(εt−i εt−τ −j ).
i j

Since ε(t) is a sequence of independently and identically distributed random

variables with zero expectations, it follows that
½
0, if i 6= τ + j;
(5.9) E(εt−i εt−τ −j ) =
σε2 , if i = τ + j.

Therefore
X
(5.10) γτ = σε2 µj µj+τ .
j

Now let τ = 0, 1, . . . , q. This gives

γ0 = σε2 (µ20 + µ21 + · · · + µ2q ),

γ1 = σε2 (µ0 µ1 + µ1 µ2 + · · · + µq−1 µq ),
(5.11) ..
.
γq = σε2 µ0 µq .

Also, γτ = 0 for all τ > q.

The first-order moving-average process y(t) = ε(t) − θε(t − 1) has the
following autocovariances:

γ0 = σε2 (1 + θ2 ),
(5.12) γ1 = −σε2 θ,
γτ = 0 if τ > 1.

68
 D.S.G. POLLOCK : LINEAR STOCHASTIC MODELS

Thus, for a vector y = [y1 , y2 , . . . , yT ]0 of T consecutive elements from a first-

order moving-average process, the dispersion matrix is
 
1 + θ2 −θ 0 ... 0
 −θ 1 + θ2 −θ ... 0 
2
 
(5.13) D(y) = σε  0 −θ 1 + θ 2
. . . 0 .
 ... .. .. .. .. 
. . . . 
0 0 0 . . . 1 + θ2

In general, the dispersion matrix of a qth-order moving-average process has q

subdiagonal and q supradiagonal bands of nonzero elements and zero elements
elsewhere.
It is also helpful to define an autocovariance generating function which is a
power series whose coefficients are the autocovariances γτ for successive values
of τ . This is denoted by
X
(5.14) γ(z) = γτ z τ ; with τ = {0, ±1, ±2, . . .} and γτ = γ−τ .
τ

The generating function is also called the z-transform of the autocovariance

function.
The autocovariance generating function of the qth-order moving-average
process can be found quite readily. Consider the convolution
X X
µ(z)µ(z −1 ) = µi z i µj z −j
i j
XX
(5.15) = µi µj z i−j
i j
X³X ´
= µi µj+τ z τ , τ = i − j.
τ j

By referring to the expression for the autocovariance of lag τ of a moving-

average process given under (10), it can be seen that the autocovariance gen-
erating function is just

(5.16) γ(z) = σε2 µ(z)µ(z −1 ).

Autoregressive Processes
The pth-order autoregressive process, or AR(p) process, is defined by the
equation

(5.17) α0 y(t) + α1 y(t − 1) + · · · + αp y(t − p) = ε(t).

69
 D.S.G. POLLOCK : TIME SERIES AND FORECASTING

This equation is invariably normalised by setting α0 = 1, although it would

be possible to set σε2 = 1 instead. The equation can be written in summary
notation as α(L)y(t) = ε(t), where α(L) = α0 + α1 L + · · · + αp Lp . For the
process to be stationary, the roots of the equation α(z) = α0 + α1 z + · · · +
αp z p = 0 must lie outside the unit circle. This condition enables us to write
the autoregressive process as an infinite-order moving-average process in the
form of y(t) = α−1 (L)ε(t).
As an example, let us consider the first-order autoregressive process which
is defined by

ε(t) = y(t) − φy(t − 1)

(5.18)
= (1 − φL)y(t).

Provided that the process is stationary with |φ| < 1, it can be represented in
moving-average form as

y(t) = (1 − φL)−1 ε(t)

(5.19) © ª
= ε(t) + φε(t − 1) + φ2 ε(t − 2) + · · · .

The autocovariances of the process can be found by using the formula of (10)
which is applicable to moving-average process of finite or infinite order. Thus

γτ = E(yt yt−τ )
nX X o
=E φi εt−i φj εt−τ −j
(5.20) i j
XX
= φi φj E(εt−i εt−τ −j );
i j

and the result under (9) indicates that

X
γτ = σε2 φj φj+τ
j
(5.21)
σε2 φτ
= .
1 − φ2
For a vector y = [y1 , y2 , . . . , yT ]0 of T consecutive elements from a first-order
autoregressive process, the dispersion matrix has the form
 
1 φ φ2 . . . φT −1
 φ 1 φ . . . φT −2 
2
σε  φ2  
(5.22) D(y) = φ 1 . . . φT −3  .
1 − φ2   .. .. .. .. .. 
. . . . .
φT −1 φT −2 φT −3 . . . 1

70
 D.S.G. POLLOCK : LINEAR STOCHASTIC MODELS

To find the autocovariance generating function for the general pth-order

P i
autoregressive process, we may consider again the function α(z) = i αi z .
Since an autoregressive process may be treated as an infinite-order moving-
average process, it follows that
σε2
(5.23) γ(z) = .
α(z)α(z −1 )
Pof finding the autocovariances of the pth-order process,
For an alternative way
consider multiplying i αi yt−i = εt by yt−τ and taking expectations to give
X
(5.24) αi E(yt−i yt−τ ) = E(εt yt−τ ).
i
Taking account of the normalisation α0 = 1, we find that
(
σε2 , if τ = 0;
(5.25) E(εt yt−τ ) =
0, if τ > 0.
Therefore, on setting E(yt−i yt−τ ) = γτ −i , equation (24) gives
(
X σε2 , if τ = 0;
(5.26) αi γτ −i =
i 0, if τ > 0.
The second of these is a homogeneous difference equation which enables us to
generate the sequence {γp , γp+1 , . . .} once p starting values γ0 , γ1 , . . . , γp−1 are
known. By letting τ = 0, 1, . . . , p in (26), we generate a set of p + 1 equations
which can be arrayed in matrix form as follows:
γ γ1 γ2 ... γp   1   σε2 
0
 γ1 γ0 γ1 . . . γp−1   α1   0 
    
(5.27)  γ 2 γ 1 γ 0 . . . γp−2   α2  =  0  .
 . .. .. ..   .   . 
 . ..
.   ..   .. 
. . . .
γp γp−1 γp−2 . . . γ0 αp 0
These are called the Yule–Walker equations, and they can be used either for
generating the values γ0 , γ1 , . . . , γp from the values α1 , . . . , αp , σε2 or vice versa.
For an example of the two uses of the Yule–Walker equations, let us con-
sider the second-order autoregressive process. In that case, we have
 
     γ2
γ0 γ1 γ2 α0 α2 α1 α0 0 0  γ1 
 γ1 γ0 γ1   α1  =  0 α2 α1 α0 0   
 γ0 
 
γ2 γ1 γ0 α2 0 0 α2 α1 α0 γ1
(5.28) γ2
    2 
α0 α1 α2 γ0 σε

= α1 α0 + α2 0    
γ1 = 0  .
α2 α1 α0 γ2 0

71
 D.S.G. POLLOCK : TIME SERIES AND FORECASTING

Given α0 = 1 and the values for γ0 , γ1 , γ2 , we can find σε2 and α1 , α2 . Con-
versely, given α0 , α1 , α2 and σε2 , we can find γ0 , γ1 , γ2 . It is worth recalling at
this juncture that the normalisation σε2 = 1 might have been chosen instead
of α0 = 1. This would have rendered the equations more easily intelligible.
Notice also how the matrix following the first equality is folded across the axis
which divides it vertically to give the matrix which follows the second equality.
Pleasing effects of this sort often arise in time-series analysis.

The Partial Autocorrelation Function

Let αr(r) be the coefficient associated with y(t − r) in an autoregres-

sive process of order r whose parameters correspond to the autocovariances
γ0 , γ1 , . . . , γr . Then the sequence {αr(r) ; r = 1, 2, . . .} of such coefficients, whose
index corresponds to models of increasing orders, constitutes the partial auto-
correlation function. In effect, αr(r) indicates the role in explaining the variance
of y(t) which is due to y(t − r) when y(t − 1), . . . , y(t − r + 1) are also taken
into account.
Much of the theoretical importance of the partial autocorrelation function
is due to the fact that, when γ0 is added, it represents an alternative way of
conveying the information which is present in the sequence of autocorrelations.
Its role in identifying the order of an autoregressive process is evident; for, if
αr(r) 6= 0 and if αp(p) = 0 for all p > r, then it is clearly implied that the
process has an order of r.
The sequence of partial autocorrelations may be computed efficiently via
the recursive Durbin–Levinson Algorithm which uses the coefficients of the AR
model of order r as the basis for calculating the coefficients of the model of
order r + 1.
To derive the algorithm, let us imagine that we already have the values
α0(r) = 1, α1(r) , . . . , αr(r) . Then, by extending the set of rth-order Yule–Walker
equations to which these values correspond, we can derive the system

 γ   
0 γ1 ... γr γr+1  1 2 
σ(r)
 γ1 γ0 ... γr−1 γr   
 α1(r)   0 
 . ..   ..  = 
 
(5.29)  . .. .. ..
   ...  ,
 . . . . .  .   
   0 
γr γr−1 ... γ0 γ1 αr(r) 
γr+1 γr ... γ1 γ0 0 g

wherein

X
r
(5.30) g= αj(r) γr+1−j with α0(r) = 1.
j=0

72
 D.S.G. POLLOCK : LINEAR STOCHASTIC MODELS

The system can also be written as

 γ   
0 γ1 ... γr γr+1  0 g

 γ1 γ0 ... γr−1 γr   
 αr(r)   0 
 . ..   ..  = 
 .. 
 . .. .. ..
  . 
(5.31)  . . . . .  .  .
   
 α1(r)  0 
γr γr−1 ... γ0 γ1
2
γr+1 γr ... γ1 γ0 1 σ(r)

The two systems of equations (29) and (31) can be combined to give
 γ    2
0 γ1 ... γr γr+1  1 σ(r) + cg

 γ1 γ0 ... γr−1 γr   
 α1(r) + cαr(r)   0 
 . ..     . 
(5.32) 
 ..
..
.
..
.
..
.

. 
..
.  = 

.. .

    
γr γr−1 ... γ0 γ1 αr(r) + cα1(r)  0
2
γr+1 γr ... γ1 γ0 c g + cσ(r)

If we take the coefficient of the combination to be

g
(5.33) c=− 2 ,
σ(r)

then the final element in the vector on the RHS becomes zero and the system
becomes the set of Yule–Walker equations of order r + 1. The solution of the
equations, from the last element αr+1(r+1) = c through to the variance term
2
σ(r+1) is given by

½ r ¾
1 X
αr+1(r+1) = 2 αj(r) γr+1−j
σ(r) j=0

     
α1(r+1) α1(r) αr(r)
(5.34)  ..   .   . 
 .  =  ..  + αr+1(r+1)  .. 
αr(r+1) αr(r) α1(r)
© ª
2
σ(r+1) 2
= σ(r) 1 − (αr+1(r+1) )2 .

Thus the solution of the Yule–Walker system of order r + 1 is easily derived

from the solution of the system of order r, and there is scope for devising a
recursive procedure. The starting values for the recursion are
© ª
(5.35) α1(1) = −γ1 /γ0 and 2
σ(1) = γ0 1 − (α1(1) )2 .

73
 D.S.G. POLLOCK : TIME SERIES AND FORECASTING

Autoregressive Moving Average Processes

The autoregressive moving-average process of orders p and q, which is
referred to as the ARMA(p, q) process, is defined by the equation

α0 y(t) + α1 y(t − 1) + · · · + αp y(t − p)

(5.36)
= µ0 ε(t) + µ1 ε(t − 1) + · · · + µq ε(t − q).

The equation is normalised by setting α0 = 1 and by setting either µ0 = 1

or σε2 = 1. A more summary expression for the equation is α(L)y(t) = µ(L)ε(t).
Provided that the roots of the equation α(z) = 0 lie outside the unit circle,
the process can be represented by the equation y(t) = α−1 (L)µ(L)ε(t) which
corresponds to an infinite-order moving-average process. Conversely, provided
the roots of the equation µ(z) = 0 lie outside the unit circle, the process can
be represented by the equation µ−1 (L)α(L)y(t) = ε(t) which corresponds to an
infinite-order autoregressive process.
By considering the moving-average form of the process, and by noting the
form of the autocovariance generating function for such a process which is given
by equation (16), it can be seen that the autocovariance generating function
for the autoregressive moving-average process is

µ(z)µ(z −1 )
(5.37) γ(z) = σε2 .
α(z)α(z −1 )

This generating function, which is of some theoretical interest, does not

provide a practical means of findingPthe autocovariances.
P To find these, let us
consider multiplying the equation i αi yt−i = i µi εt−i by yt−τ and taking
expectations. This gives
X X
(5.38) αi γτ −i = µi δi−τ ,
i i

where γτ −i = E(yt−τ yt−i ) and δi−τ = E(yt−τ εt−i ). Since εt−i is uncorrelated
with yt−τ whenever it is subsequent to the latter, it follows that δi−τ = 0 if
τ > i. Since the index i in the RHS of the equation (38) runs from 0 to q, it
follows that
X
(5.39) αi γi−τ = 0 if τ > q.
i

Given the q +1 nonzero values δ0 , δ1 , . . . , δq , and p initial values γ0 , γ1 , . . . , γp−1

for the autocovariances, the equations can be solved recursively to obtain the
subsequent values {γp , γp+1 , . . .}.

74
 D.S.G. POLLOCK : LINEAR STOCHASTIC MODELS

P To find the P requisite values δ0 , δ1 , . . . , δq , consider multiplying the equation

i αi yt−i = i µi εt−i by εt−τ and taking expectations. This gives
X
(5.40) αi δτ −i = µτ σε2 ,
i

where δτ −i = E(yt−i εt−τ ). The equation may be rewritten as

1 ³ X ´
(5.41) δτ = µτ σε −
2
δτ −i ,
α0 i=1

and, by setting τ = 0, 1, . . . , q, we can generate recursively the required values

δ0 , δ1 , . . . , δq .

Example. Consider the ARMA(2, 2) model which gives the equation

(5.42) α0 yt + α1 yt−1 + α2 yt−2 = µ0 εt + µ1 εt−1 + µ2 εt−2 .
Multiplying by yt , yt−1 and yt−2 and taking expectations gives
     
γ0 γ1 γ2 α0 δ 0 δ1 δ2 µ0
(5.43)  γ1 γ0 γ1    
α1 = 0 δ0 δ1   µ1  .
γ2 γ1 γ0 α2 0 0 δ0 µ2
Multiplying by εt , εt−1 and εt−2 and taking expectations gives
    2  
δ0 0 0 α0 σε 0 0 µ0
(5.44)  δ 1 δ0 0    
α1 = 0 σε 0 2   µ1  .
δ 2 δ1 δ0 α2 0 0 σε2 µ2
When the latter equations are written as
    
α0 0 0 δ0 µ0
(5.45)  α1 α0 0   δ1  = σε2  µ1  ,
α2 α1 α0 δ2 µ2
they can be solved recursively for δ0 , δ1 and δ2 on the assumption that that
the values of α0 , α1 , α2 and σε2 are known. Notice that, when we adopt the
normalisation α0 = µ0 = 1, we get δ0 = σε2 . When the equations (43) are
rewritten as
     
α0 α1 α2 γ0 µ0 µ1 µ2 δ0
(5.46)  α1 α0 + α2 0    
γ1 = µ1 µ2 0   δ1  ,
α2 α1 α0 γ2 µ2 0 0 δ2
they can be solved for γ0 , γ1 and γ2 . Thus the starting values are obtained
which enable the equation
(5.47) α0 γτ + α1 γτ −1 + α2 γτ −2 = 0; τ >2
to be solved recursively to generate the succeeding values {γ3 , γ4 , . . .} of the
autocovariances.

75