APM 4AI02 TP – Booklet of Exercises

Marco Cagnazzo, Mathieu Fontaine, François Roueff

Contents
1 General reminders and notation 1
1.1 Gaussian r.v.’s, vectors, processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Functions of r.v.’s and of random processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2.1 Example: inversion of a FIR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Autocovariance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Gaussian vectors 4

3 Stationarity 5

4 Covariance, spectral measure and spectral density 5

5 Linear filtering, ARMA processes 6

6 Solutions 9

A Annals 24
A.1 Exam of 2021 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
A.2 Exam of 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

B Solutions of annals 27

1 General reminders and notation

1.1 Gaussian r.v.’s, vectors, processes
Except for the zero-variance case, a real valued Gaussian random variable X has the following probability
density function (pdf):
1 (x−µ)2
pX (x) = √ e− 2σ2 .
2πσ 2
T n
Pn vector X = [X1 , X2 , . . . , Xn ] is a Gaussian random vector if and only if, ∀u ∈ R , Y =
A random
uT X = i=1 ui Xi is a Gaussian r.v. The pdf h of a Gaussian
i vector is completely defined by the mean vector
µ = E [X] and the covariance matrix Γ = E Xc Xc T
A random process {Xt , t ∈ Z} is a Gaussian random process if and only if for all finite set of indexes
T
I ⊂ Z, I = {t1 , t2 , . . . , tn }, the random vector [Xt1 , Xt2 , . . . , Xtn ] is a Gaussian vector.

1.2 Functions of r.v.’s and of random processes

Let X be a real-valued r.v. and let g a real function. Let us suppose that g is derivable over R, except for a
set whose measure is zero, e.g., a numerable set of points. If we define a new r.v. Y = g(X), the pdf of Y is

1
 Shortcut Meaning
X Conjugate of X
AT Transpose pf A
AH Hermitian of A, i.e. AT
N0 Natural numbers including zero
R+ Positive real numbers: {x ∈ R|x > 0}
R+0 Non-negative real numbers: {x ∈ R|x ≥ 0}
1A (x) Indicator function of set A: 1A (x) = 1 if and only if x ∈ A; otherwise, 1A (x) = 0
r.v. random variable
pdf probability density function
X∼P
X is a r.v. distributed with law P
N µ, σ 2 Gaussian r.v. with mean µ and variance σ 2
E [X] Expectation of the r.v. X
Xc Centered version of X: X c = X − E [X] 
c 2
Var (X) Variance
h iof the r.v. X: Var (X) = E |X |
Cov (X, Y ) E X cY c
{Xt , t ∈ Z} Discrete random process
s.o.1 A process {Xt , t ∈ Z} is stationary at order 1 if and only if E [Xt ] does not
 depend on t
A process {Xt , t ∈ Z} is stationary at order 2, if and only if ∀t ∈ Z, E |Xt |2 < +∞ and

s.o.2
∀t, h ∈ Z, Cov (Xt , Xt+h ) does not depend on t
w.s. weakly stationary, i.e., s.o.1 and s.o.2
γX (h) For {Xt , t ∈ Z} s.o.2, γX (h) = Cov (Xt+h , Xt ) = Cov (Xh , X0 )
δh The Kronecker’s delta: δ : h ∈ Z → δh ; if h = 0, δh = 1; otherwise, δh = 0

Table 1: Shortcuts and notation used throughout this document.

related to that of X as follows:

(
0 if the equation in the variable x, g(x) = y, has no solution
pY (y) = PNy pX (xi (y))
i=1 |g ′ (xi (y))| if g(x) = y has Ny ≥ 1 solutions, referred to as {xi (y)}i=1,...,Ny

We can also consider function of multiple r.v.’s. A particularly interesting case is when a random process
is obtained by applying a function to another random process:

Xt = gt ({Zs , s ∈ Z})

A special case is when the transformation is the same at each time (i.e. g does not depend on t) and it has
a finite number of inputs. Apart from a time shift, this can be written as:

Xt = g(Zt , Zt−1 , . . . , Zt−k+1 )

This is called a moving transformation. It can be shown that, for a moving transformation, if g is measurable
and {Zt , t ∈ Z} i.i.d., then {Xt , t ∈ Z} is strictly stationary.
A particularly interesting case of moving transformation is a linear filter:
X
Yt = αn Xt−n
n∈Z

If the support of α is finite, this filter is called Finite Impulse Response (FIR); otherwise it is an Infinite
Impulse Response (IIR).

2
1.2.1 Example: inversion of a FIR
Let us remember a particularly simple case of invertible filter. Let θ ∈ C and |θ| < 1. We introduce the
following L1 sequences:

a : n ∈ Z → δn − θδn−1
(
θn if n ≥ 0
b:n∈Z→
0 otherwise
X
c = (a ∗ b) : n ∈ Z → ak bn−k
k∈Z

It is easy to find that (a ∗ b) = δ. In that case, we say that a FIR having a as impulse response, can be
inverted by an IIR having b as impulse response, since the cascade of a and b will not change an input signal.
Let us show that c = δ.

X 0 − θ · 0 = 0
 if n < 0
cn = ak bn−k = bn − θ · bn−1 = 1 − θ · 0 = 1 if n = 0 = δn

 n
k∈Z θ − θ · θn−1 = 0 if n > 0

1.3 Autocovariance
h i
Cov (X, Y )=E X c Y c
h i h i
Cov (X, Y )=E XY − E [X] E Y
Cov (X, Y )=Cov (Y, X)
Cov (X + a, Y )=Cov (X, Y )
Cov (X, Y + a)=Cov (X, Y )
Cov (aX, Y )=aCov (X, Y )
Cov (X, aY )=aCov (X, Y )
Cov (X1 + X2 , Y )=Cov (X1 , Y ) + Cov (X2 , Y )
Cov (X, Y1 + Y2 )=Cov (X, Y1 ) + Cov (X, Y2 )

Table 2: Covariance properties. X, X1 , X2 , Y are complex or real r.v.’s; a ∈ C.

The covariance of two r.v.’s has several interesting properties resumed in Tab. 2. Two real r.v. with null
covariance are said to be uncorrelated. Two complex r.v. with null covariance are said to be orthogonal, while
if also the pseudo-covariance E [XY ] is null, they are said uncorrelated. Independent r.v.’s are uncorrelated
while the converse is not true in general. A notable exception is when (X, Y ) is a Gaussian vector (but not
when X and Y are marginally Gaussian and not jointly Gaussian): in that case, uncorrelatedness implies
independence.
The covariance allows to define a scalar product between two r.v.’s: ⟨Xt , Xs ⟩ = Cov (Xt , Xs ). The
(squared) norm of a r.v.’s is then its variance. Note that this scalar product is not affected by the mean of
the r.v.’s, since neither the covariance is. For example, a zero-norm r.v. has a null variance, but can have
any mean.
We can also introduce the concept of linear independent r.v.’s.
P (X1 , . .. , Xk ) is a set of linearly indepen-
Pk k
dent r.v.’s if and only if ∀a ∈ Rk − {0}, ∥ i=1 ai Xi ∥2 = Var i=1 ai Xi > 0.
Note also that, if (X1 , . . . , Xk ) are not linearly independent, this means that one of the Xi can be
expressed as a linear combination of the other r.v.’s, up to an additive constant, which does not affect the
covariance. This constant is null in the case E [(X1 , . . . , Xk )] = 0.
For the sake of simplicity, let us prove that for some i, Xi is a linear combinationof the otherr.v.’s only
Pk
in the case of a centered vector. In this case it must exist a ∈ Rk − {0} such that Var i=1 ai Xi = 0. The
Pk
vector a must have at least one non-zero component, let it be aj . Let also Y = i=1 ai Xi ; since its variance

3
is zero, Y = E [Y ] = 0. This implies:
k
X X
0= ai Xi = aj Xj + ai Xi
i=1 i̸=j
X
aj Xj = − ai Xi
i̸=j
X ai
Xj = − Xi
aj
i̸=j

Then Xj is a linear combination of other r.v.’s. It can be shown that, if the Xi are not centered, the
Pk
same result holds up to a constant: Xj = − i̸=j aaji Xi + i=1 aaji E [Xi ].
P
h i
T
The covariance matrix of a complex-valued random vector X = [X1 , X2 , . . . , Xn ] is Γ = E Xc Xc H .
In other words, Γi,j = Cov (Xi , Xj ). It is an Hermitian, non-negative matrix, since for all u ∈ Cn the random
variable Y = uH X shall have a non negative variance:
0 ≤ Var (Y ) = E ∥uH X − E uH X ∥2 = E ∥uH (X − E [X] ∥2
     
h i
= E ∥uH Xc ∥2 = E uH Xc Xc H u
 
h i
= uH E Xc Xc H u = uH Γu

The autocovariance function (acf) of a random process {Xt , t ∈ Z} is a function of two discrete variables
t and s:
γ(t, s) = Cov (Xt , Xs )
A weakly stationary process is a process s.o.1 and s.o.2, therefore, all Xt have finite quadratic mean,
the mean of Xt is the same for all t and the autocovariance function only depend on the delay t − s:
γ(t, s) = γ(t − s) = Cov (Xt−s , X0 )
In that case, we use a single-parameter notation for γ:
γ(h) = Cov (Xh , X0 )
The acf of weakly stationary processes is an Hermitian and non-negative function. The maximum of |γ| is
in 0. The normalized acf, ρ(h) = γ(h)
γ(0) is referred to as autocorrelation function.

1.4 Noise
A weak white noise is a real-valued, weakly stationary process {Xt , t ∈ Z}, with zero-mean and impulsive
acf: γX (h) = σ 2 δ(h). In other words, for all t ̸= s, Xt and Xs are uncorrelated variables.
A strong white noise is a real-valued, zero-mean, i.i.d. process. Note that a strong white noise is also
a weak white noise, since i.i.d. implies weak stationarity and impulsive acf. On the contrary, a weak white
noise is not necessarily a strong one, since uncorrelated r.v.’s may be dependent.
In both cases, we usually consider finite, positive variance σ 2 = Var (Xt ).

2 Gaussian vectors
Exercise 2.1 (Functions of Gaussian random variables). Let X ∼ N (0, 1), a ∈ R+ and Y a = X1{|X|<a} −
X1{|X|≥a} .
1. Give the law of Y a
2. Compute Cov (X, Y a ). For which value a0 of a the covariance is null? Are X and Y a0 independent?
3. Is (X, Y a0 ) a Gaussian vector?
4. For a ̸= a0 , is (X, Y a ) a Gaussian vector?

4
3 Stationarity
Exercise 3.1 (Uncorrelated processes). Let {Xt , t ∈ Z} and {Yt , t ∈ Z} be two weakly stationary (w.s.),
uncorrelated random processes. Show that {Zt = Xt + Yt , t ∈ Z} is weakly stationary. Find the covariance
function of Zt from those of Xt and Yt and the spectral measure of Zt from those of Xt and Yt
 
Exercise 3.2 (Functions of strong white noise). Let {ϵt , t ∈ Z} be a strong white noise with E ϵ20 < ∞.
For each of the following processes (functions of the white noise), find out if they are weakly stationary or
strictly stationary.
1. Wt = a + bϵt + cϵt−1 , with a, b, c real numbers
2. Xt = ϵt ϵt−1

3. Yt = (−1)t ϵt
4. Zt = ϵt + Yt
 
1 ρ
Exercise 3.3 (Structured covariance matrix). Let us consider a real number ρ; we define Σ2 = .
ρ 1
Moreover, let ∀t ∈ Z, Σt be a t × t matrix with diagonal elements equal to 1, and out-of-diagonal elements
equal to ρ:  
1 ρ ρ ... ρ
ρ 1 ρ ... ρ 
 
Σt =  ρ
 ρ 1 ... ρ  
. . . . . . . . . . . . . . .
ρ ρ ρ ... 1
1. Which condition on ρ must be imposed such that Σt is a covariance matrix for all t? Suggestion:
decompose Σt = αI + A, where A is matrix with easy-to-find eigenvalues.
2. Build a stationary process having Σt as auto-covariance matrix for all t.

4 Covariance, spectral measure and spectral density

Exercise 4.1 (Functions of weak white noise). Let {Zt , t ∈ Z} be a weak white noise, centered, with variance
σ 2 . Let a, b, c ∈ R. Are the following processes s.o.2? If yes, compute the autocovariance function and the
spectral measure.

1. Xt = a + bZ0
2. Xt = Z0 cos(ct)
3. Xt = a + bZt + cZt−1

4. Xt = Z1 cos(ct) + Z2 sin(ct)
5. Xt = Zt cos(ct) + Zt−1 sin(ct)
Reminders:
Z π
γ(h) = eihλ ν(dλ)
−π
Z π
γ(h) = eihλ f (λ)dλ if the density f (·) exists
−π
1 X
f (λ) = γ(h)e−ihλ if γ ∈ L1 (Z)
2π
h∈Z

5
Exercise 4.2 (Autocovariance function characterization). Let us introduce the following sequence on the
integers: 
1 if h = 0

γ : h ∈ Z → γ(h) = ρ if |h| = 1

0 if |h| > 1


We want to show that such a function is an autocovariance function if and only if |ρ| ≤ 12 .
1. Let Γk be a k × k matrix such that ∀i, j ∈ {1, 2, . . . , k}, Γk (i, j) = γ(i − j).
 
1 ρ 0 0 ... 0 0
ρ 1 ρ 0 ... 0 0
 
0
Γk =  ρ 1 ρ ... 0 0 
. . . ... ... ... ... ... . . .
0 0 0 0 ... ρ 1

Find the recurrence equation among the determinants of matrices Γk

2. Show that if |ρ| is not greater than a given value, Γk is positive definite for all k. Use or the previous
point or the Herglotz theorem.
3. Build a s.o.2 process having γ(h) as autocovariance function. [Hint: use Question 3 of Exercise 4.2.]
Exercise 4.3 (Band-limited stationary process). Let S(f ) = 1(−f0 ,f0 ) (f ), with f0 ∈ (0, π) be the spectral
density of a stationary process.
1. Compute the autocovariance function.
2. Is it ℓ1 ?
Exercise 4.4 (Process generated by linear combination). Let γ be the autocovariance function of a stationary,
zero-mean process. Let us suppose that it exist a finite subset of this process such that the corresponding
autocovariance matrix is not invertible, i.e., it is not full rank.

1. Show that either γ(0) = 0, or it exists k ≥ 1 such that:

- Xk+1 ∈ Vect(X1 , . . . , Xk ); and
P 
k
- (X1 , . . . , Xk ) is a set of linearly independent vectors: ∀a ∈ Rk − {0}, Var i=1 ai Xi > 0.

2. Let Γk be the autocovariance matrix of X1 , . . . , Xk . Find a property of its minimum eigenvalue.

3. Show that the process {Xt , t ∈ Z} is linearly predictable, i.e., for all p ≥ 1, there exists a set of k
scalars ϕp,1 , ϕp,2 , . . . , ϕp,k such that:
k
X
Xk+p = ϕp,ℓ Xℓ . (1)
ℓ=1

Pk
4. Show that supp≥1 ℓ=1 |ϕp,ℓ |2 < ∞.
5. Deduce that, if in addition lim|t|→∞ γ(t) = 0, then γ(0) = 0.

5 Linear filtering, ARMA processes

Exercise 5.1 (Linear filtering and stationarity). Let β ∈ R, {St , t ∈ Z} a w.s., periodical (period = 4) real
process, and {Xt , t ∈ Z} a w.s. real process, uncorrelated with St .
Let us consider the process {Yt = βt + St + Xt , t ∈ Z}.
1. Is {Yt , t ∈ Z} w.s.?

6


2. Let us refer to the back-shift operator as B, and let us consider the process {S̄t = 1 + B + B 2 + B 3 ◦
St , t ∈ Z}. Show that γ is periodic and that S̄t = S0 + S1 + S2 + S3


3. Let us consider the process {Zt = (1 − B) ◦ 1 + B + B 2 + B 3 ◦ St , t ∈ Z}. Show that {Zt , t ∈ Z} is
w.s. and compute γZ as a function of γX (autocovariance functions).
4. Find the shape of the spectral measure µ of {St , t ∈ Z}.
5. Find the spectral measure of (1 − B 4 ) ◦ Yt as a function of the spectral measure of {Xt , t ∈ Z}.
Exercise 5.2 (Characterization of MA(q)). Let q ∈ Z and q > 0. Let {Xt , t ∈ Z} be a centered w.s.
real process and let γ be its autocovariance function. Let us suppose that γ has a compact support, i.e.
∀t > q, γ(t) = 0.
We also introduce

Ht = Vect(Xs , s ≤ t)
et = Proj(Xt |Ht−1 )
X

1. Recall why Zt = Xt − X
et is a white noise.

2. Show that Xt ⊥Ht−q−1 .

3. Deduce that Xt ∈ Vect(Zs , s ∈ {t, t − 1, . . . t − q}).

4. Show that {Xt , t ∈ Z} is a MA(q) process.
Exercise 5.3 (Sum of MA processes). Let {Xt , t ∈ Z} and {Yt , t ∈ Z} be two real uncorrelated MA processes
of order q and p respectively:
q
X p
X
Xt = ϵt + θn ϵt−n Yt = ηt + ρn ηt−n
n=1 n=1

where ∀n ∈ {1, . . . , q}, θn ∈ R, ∀n ∈ {1, . . . , p}, ρn ∈ R, {ϵt , t ∈ Z} and {ηt , t ∈ Z} are white noises whose
variances are respectively noted as σϵ2 and ση2 . Let us also introduce {Zt = Xt + Yt , t ∈ Z}.
1. Which kind of process is {Zt , t ∈ Z}?

2. Let us consider the case p=1 , q=1, 0<θ1 <1 and 0<ρ1 <1. Show that {ϵt , t ∈ Z} and {ηt , t ∈ Z} are
uncorrelated.
3. For p=1 , q=1, θ1 =ρ1 =θ and 0<θ<1, what is the innovation process for {Zt , t ∈ Z}?
4. For p=1 , q=1, 0<θ1 <1 and 0<ρ1 <1, compute the variance of the innovation of {Zt , t ∈ Z}.

Exercise 5.4 (Sum of AR processes). Let {Xt , t ∈ Z} and {Yt , t ∈ Z} be two real uncorrelated AR(1)
processes such that

Xt = aXt−1 + ϵt
Yt = bYt−1 + ηt

where a ∈]0, 1[, b ∈]0, 1[. Moreover, {ϵt , t ∈ Z} and {ηt , t ∈ Z} are white noises whose variances are
respectively noted as σϵ2 and ση2 . Let us also introduce {Zt = Xt + Yt , t ∈ Z}.
1. Show that there exists a white noise {ξt , t ∈ Z} and a real number θ ∈] − 1, 1[ such that:

Zt − (a + b)Zt−1 + abZt−2 = ξt − θξt−1 .

7
 2. Show that:
∞
X ∞
X
ξt = ϵt + (θ − b) θk ϵt−1−k + ηt + (θ − a) θh ηt−1−h .
k=0 h=0

3. Compute the prediction of Zt+1 when (Xs , s ≤ t) and (Ys , s ≤ t) are all known.
4. Compute the prediction of Zt+1 when (Zs , s ≤ t) are all known.
5. Compare the variances of the prediction errors in the two previous cases.
Exercise 5.5 (Forward/backward prediction of a MA(1) process). Let {Xt = Zt + θZt−1 , t ∈ Z} be a real
w.s. process, with {Zt , t ∈ Z} centered white noise and θ ∈] − 1, 1[.
1. Find the best (in terms of MSE) linear prediction of X3 as a function of X1 and X2 .
2. Find the best linear prediction of X3 as a function of X4 and X5 .
3. Find the best linear prediction of X3 as a function of X1 , X2 , X4 and X5 .
Exercise 5.6 (Canonical representation of an ARMA process). Let {Xt , t ∈ Z} be a centered, s.o.2 process
satisfying the recurrence equation
Xt − 2Xt−1 = ϵt + 4ϵt−1
where {ϵt , t ∈ Z} is a white noise with variance σ 2 .
1. Compute the spectral density of {Xt , t ∈ Z}.
2. Compute the canonical representation of {Xt , t ∈ Z}.
3. What is the variance of the innovation of {Xt , t ∈ Z}?
4. Find a representation of Xt as a function of (ϵs , s ≤ t).

σ 2 8 cos λ+17
Figure 1: fZ (λ) = 2π 5−4 cos λ

Exercise 5.7 (ACF of an AR(1) process). Let {Xt , t ∈ Z} be a w.s. process defined by:
Xt − ϕXt−1 = ϵt
where ϕ ∈] − 1, 1[ and {ϵt , t ∈ Z} is a centered WN with variance σϵ2 .
1. Compute the weights ψi of the representation
X
Xt = ψk ϵt−k
k∈Z

2. Deduce the autocovariance function of {Xt , t ∈ Z}.

8
6 Solutions
(
X if |X| < a
Solution of Exercise 2.1 1. The r.v. Y satisfies the following equation: Y =
−X if |X| > a

Figure 2: Y = g(X)

y2
e− 2
If |y| < a pY (y) = pX (y) = √
2π
(−y)2
e− 2
If |y| > a pY (y) = pX (−y) = √
2π

Thus, Y ∼ N (0, 1)
2. Let us compute the covariance of X and Y a :
Cov (X, Y a ) = E [XY a ] = E X 2 1{|X|<a} − X 2 1{|X|≥a}
 

= E X 2 1{|X|<a} − 1{|X|≥a } = E X 2 21{|X|<a} − 1


 

r Z a
2 x2
x2 e− 2 dx − 1 = h(a)
 2   2
= 2E X 1{|X|<a} − E X =
π −a

 function h : a → h(a) is continuous and strictly increasing. Moreover h(0) = −1 and lima→+∞ h(a) =
 The
E X 2 = 1. Therefore, ∃a0 ∈]0, +∞[: h(a0 ) = 0. For such a value a0 , X and Y a0 are uncorrelated but they
are not independent, since Y |X is deterministic. Another way to show that X and Y a0 are not independent
is the following. Since they are both Gaussian, if they were independent, the vector (X, Y a0 ) would be a
Gaussian Vector, therefore X + Y a0 would be Gaussian. But this is impossible, since X + Y a0 = 2X1|X|<a0
cannot be larger than 2a0 . This also answers to points 3. As for point 4, the since X + Y a is not a Gaussian
r.v. for any real positive a, the vector (X, Y a ) cannot be a Gaussian vector.

9
 Figure 3: Function h(a) = Cov (X, Y a )

Solution of Exercise 3.1 First, since {Xt , t ∈ Z} and {Yt , t ∈ Z} are w.s.,
E [Xt ] = µX E [Yt ] = µY
Cov (Xt , Xs ) = γX (t − s) Cov (Yt , Ys ) = γY (t − s)

Moreover, {Xt , t ∈ Z} and {Yt , t ∈ Z} are uncorrelated, meaning that ∀t, s, Cov (Xt , Ys ) = 0, therefore we
find:
E [Zt ] = E [Xt + Yt ] = µX + µY
Cov (Zt , Zs ) = Cov (Xt + Yt , Xs + Ys ) = Cov (Xt , Xs ) + Cov (Xt , Ys ) + Cov (Yt , Xs ) + Cov (Yt , Ys )
= γX (t − s) + γY (t − s)

Therefore {Zt , t ∈ Z} is w.s. with E [Zt ] = µX + µY and γZ (h) = γX (h) + γY (h). From the previous point
we deduce that the spectral measure of {Zt , t ∈ Z} is the sum of those of {Xt , t ∈ Z} and {Yt , t ∈ Z}.
Solution of Exercise 3.2 We remind that if g is a measurable moving transformation, it preserves the
strict stationarity, meaning that, since {ϵt , t ∈ Z} is strictly stationary, so g(ϵ) is.
1. and 2. We are in the case of a moving transformation. In both cases g is measurable, so {Wt , t ∈ Z}
and {Xt , t ∈ Z} are strictly stationary.
3. This is not a moving transformation. Actually, Yt is alternatively equal to ϵt and −ϵt . Since {ϵt , t ∈ Z}
is iid, the pdf of Yt is (
pϵ (y) if t is even
pY (y) =
pϵ (−y) if t is odd
Therefore, if the pdf of ϵt is symmetric, {Yt , t ∈ Z} is iid; otherwise, it is not strictly stationary.
As for weak stationarity, it is achieved if E [ϵt ] = 0. This actually implies that E [Yt ] = 0. Moreover,
(  
E ϵ20 if t = s
Cov (Yt , Ys ) =
Cov (±ϵt , ±ϵs ) = 0 otherwise

Thus, Yt is w.s. if E [ϵt ] = 0.

4. In that case,Zt = 2ϵt if t is even, and Zt = 0 if t is odd ,implying that:
( (
0 if t is even 4σϵ2 if t is even
E [Zt ] = Var (Zt ) =
0 if t is odd 0 if t is odd

10
Therefore {Zt , t ∈ Z} is s.o.1, but it is s.o.2 if and only if σϵ2 = 0: in that case, ϵt = Zt = 0 for all t.

Solution of Exercise 3.3 1. A covariance matrix is an Hermitian, non-negative matrix. Since ρ is real,
matrices Σt are Hermitian. As for non-negativity, it is equivalent to the fact that the eigenvalues of Σt , let
them be {λ1 , λ2 , . . . , λt }, are all non-negative.
Let us define A as a t × t matrix such that Ai,j = ρ for all i and j. Then we have Σt = (1 − ρ)It + A.
Now, λi = (1 − ρ) + ωi , where ωi is the i-th eigenvalue of A. Since the rank of A is 1, t − 1 of its
Pteigenvalues
are equal to 0. Let us say that ωt is the remaining, non null eigenvalue. Moreover, Tr(A) = i=1 ωi = ωt ,
but also Tr(A) = tρ, thus ωt = tρ. In conclusions we have

∀i ∈ {1, 2, . . . , t − 1}, λi = 1 − ρ
λt = 1 − ρ + tρ = 1 + (t − 1)ρ

The non-negativity conditions are:

1−ρ≥0 1 + (t − 1)ρ ≥ 0
1
ρ≤1 ρ≥− →t→+∞ 0−
t−1

In conclusion, 0 ≤ ρ ≤ 1.
2. Let us consider a process {Xt = αϵt + βZ, t ∈ Z}, with {ϵt , t ∈ Z} being a real-valued, zero-mean,
unitary-variance strong white noise, Z a real-valued, zero-mean, unitary-variance r.v. independent from any
ϵt , and α, β ∈ R. We would have:

Cov (Xt , Xt+h ) = E [(αϵt + βZ)(αϵt+h + βZ)] = α2 E [ϵt ϵt+h ] + β 2 E Z 2 = α2 δh + β2

 
 2
α + β2 β2 β2 β2

...
 β2 α2 + β 2 β2 ... β2 
 2 2 2 2

Σt =  β β α +β ... β2  
 ... ... ... ... ... 
β2 β2 β2 . . . α2 + β 2

α2 + β 2 = 1 β2 = ρ
α2 = 1 − ρ β2 = ρ
p √
α= 1−ρ β= ρ

Since ρ ∈ [0, 1], then also α, β ∈ [0, 1].

Solution of Exercise 4.1 1. Xt = a + bZ0 is a constant with respect to t, thus strictly stationary.

E [Xt ] = a Cov (Xt , Xt+h ) = Cov (a + bZ0 , a + bZ0 ) = b2 σ 2 < +∞

Since the acf is a constant, the spectral measure is ν(dλ) = b2 σ 2 δ(dλ).

2. Xt = Z0 cos(ct)

Cov (Xt , Xt+h ) = E |Z0 |2 cos(ct) cos(ch + ct)

 
E [Xt ] = 0
σ2
= [cos(ch) + cos(c(2t + h))]
2

The covariance of Xt and Xt+h depends on t, thus the process is not s.o.2.

11
 3. Xt = a + bZt + cZt−1

E [Xt ] = a Cov (Xt , Xt+h ) = Cov (bZt + cZt−1 , bZt+h + cZt+h−1 )

= (c2 + b2 )γZ (h) + bcγZ (h − 1) + bcγZ (h + 1)
= (c2 + b2 )δh + bcδh−1 + bcδh+1

Thus, Cov (Xt , Xt+h ) does not depend on t and Var (Xt ) = γX (0) = c2 + b2 < +∞. Therefore, it is a w.s.
process. Finally,
1 X 1 2
γ(h)e−ihλ = b + c2 + 2bc cos λ


f (λ) =
2π 2π
h∈Z

4. Xt = Z1 cos(ct) + Z2 sin(ct)

E [Xt ] = 0 Cov (Xt , Xt+h ) = Cov (Z1 cos(ct) + Z2 sin(ct), Z1 cos(c(t + h)) + Z2 sin(c(t + h)))
= σ 2 [cos(ct) cos(c(t + h)) + sin(ct) sin(c(t + h))]
1
= σ 2 [cos(2ct + 2ch) + cos(ch) + cos(ch) − cos(2ct + 2ch)] = σ 2 cos(ch)
2
Thus, Cov (Xt , Xt+h ) does not depend on t and Var (Xt ) = σ 2 < +∞. Therefore, it is a w.s. process. Finally,

σ2
ν(dλ) = [δ(dλ − c) + δ(dλ + c)]
2
5. Xt = Zt cos(ct) + Zt−1 sin(ct) ⇒ E [Xt ] = 0

Cov (Xt , Xt+h ) = Cov (Zt cos(ct) + Zt−1 sin(ct), Zt+h cos(c(t + h)) + Zt+h−1 sin(c(t + h)))
= σ 2 [δh cos(ct) cos(c(t + h)) + δh−1 cos(ct) sin(c(t + h)) + δh+1 sin(ct) cos(c(t + h))+
+ δh sin(ct) sin(c(t + h))]
 
1 1
= σ 2 δh cos(ch) + δh−1 (sin(c(2t + h)) + sin(ch)) + δh+1 (sin(c(2t + h)) − sin(ch))
2 2

Thus, Cov (Xt , Xt+h ) depends on t, the process is not s.o.2.

Solution of Exercise 4.2 Let us define the sequence d : k ∈ N0 → det(Γk+1 ). We have the following:

k=0 Γ1 = [1] d0 = det(Γ1 ) = 1

 
1 ρ
k=1 Γ2 = d1 = det(Γ2 ) = 1 − ρ2
ρ 1

For k ≥ 2, we can write Γk+1 as a block matrix:

   
1 ρ 0 0 ... 0 0 1 ρ 0 0 ... 0 0
 ρ   ρ 1 ρ 0 ... 0 0 
   
Γk+1 =  = 0
 0 ρ 
 ... Γ k
 
  ...

... Γk−1 
0 0 0

Therefore we have:
 
ρ ρ 0 0 ... 0 0
 0 
 = dk−1 − ρ2 dk−2
 
dk = det(Γk+1 ) = det(Γk ) − ρdet  0
 Γk−1 
 ... 
0

12
Thus we have that the sequence d is the solution of the following recurrent equation:

 dk =dk−1 − ρ2 dk−2
d0 =0 (2)
d1 =1 − ρ2


The characteristic equation is x2 − x + ρ2 = 0, with solutions

p p
1 − 1 − 4ρ2 1+ 1 − 4ρ2
x0 = x1 =
2 2
Therefore, the sequence dk has the following form:
(
1
αxk0 + βxk1 if x0 = ̸ x1 ⇔ |ρ| =
̸ 2
dk = k 1 1
(α + βk)x0 if x0 = x1 ⇔ |ρ| = 2 ⇒ x0 = x1 = 2

where α and β are defined by the initial conditions.

2. We have now to show that the matrices Γk are positive definite given some condition on ρ. Using
the expression Eq. (2) for the sequence of determinants, we have to find under which conditions on ρ, the
determinants are all positive: dk > 0∀k ∈ N0 .
We have to consider three cases, with respect to the discriminant of the characteristic equation x2 − x +
ρ2 = 0: positive, null and negative discriminant. Since ∆ = 1 − 4ρ2 , these conditions correspond respectively
to |ρ| < 21 , |ρ| = 12 , and |ρ| > 12 .
If ρ = |1/2|, by applying the initial condition, one can easily find that α = 1 and β = 1/2. In that case

k
dk = (1 + k2 ) 12 > 0∀k. Then the Γk matrices are all definite positive, thus they can be autocovariance
matrices. 2 √  1 ρ2  √  1 ρ2 
x1√−ρ2
If |ρ| ̸= 12 , one can find that α = ρ √−x
∆
0
= 1
2 − ∆ 2 + ∆ and β = ∆
= 1
2 + ∆ 2 + ∆ . Now,
if |ρ| < 12 then ∆ > 0 and both α and β are real. It can also be proven that β > 1 , α < 0 and |β| − |α > |1.
Since 0 < x0 < x1 , |β||x1 |n > |α||x0 |n , proving that ∀k ∈ N0 , dk > 0, q.d.e..
Finally, if |ρ| > 12 , it can be shown that dk has sinusoidal terms, hence it can be negative, which prevents
Γk from being an autocovariance matrix.
As alternative method, we can use the Herglotz R πtheorem, stating that γ(h) is positive if and only
if it exists a positive measure ν such that γ(h) = −π eihλ ν(dλ). Here we can use the density: γ(h) =
R π ihλ
−π
e f (λ)dλ where
1 X 1
f (λ) = γ(h)e−ihλ (1 + 2ρ cos λ)
2π 2π
h∈Z

The density is non-negative for all λ if and only if |ρ|leq 21 , q.d.e..

3. Let us consider a weak white noise {ϵt , t ∈ Z} and a process {Xt = aϵt + bϵt−1 , t ∈ Z}, with a, b ∈ R.
Then, the new process is real-valued and centered: E [Xt ] = 0. Moreover,

Cov (Xt , Xt+h ) = E [Xt Xt+h ] = E a2 ϵt ϵt+h + b2 ϵt−1 ϵt−1+h + abϵt+h ϵt−1 + abϵt ϵt−1+h
 

= a2 + b2 δh + ab (δh−1 + δh+1 )

Finally, we find a and b by setting:

a 2 + b2 = 1

ab = ρ

13
 Figure 4: Example of autocovariance function for a band-limited stationary process, Exercise 4.3.

√
implying a2 + b2 + 2ab = 1 + 2ρ and thus a + b = 1 + 2ρ. Then we have:
p
b= 1 + 2ρ − a
p
b2 = a2 + 1 + 2ρ − 2a 1 + 2ρ
p
a2 + b2 = 2a2 + 1 + 2ρ − 2a 1 + 2ρ
p
1 = 2a2 + 1 + 2ρ − 2a 1 + 2ρ
p
2a2 + 2ρ − 2a 1 + 2ρ = 0
√ √
1 + 2ρ ± 1 − 2ρ
a=
√ 2√
1 + 2ρ ∓ 1 − 2ρ
b=
2
Note that, since |ρ| ≤ 12 , a, b ∈ R.

Solution of Exercise 4.3

Z π
γ(h) = eihλ f (λ)dλ
−π
Zf0
= eihλ dλ
−f0
1 ihf0
− e−ihf0


= e
ih
sin(hf0 )
=2 = 2f0 Sinc(f0 h)
h
An example of this function is given in Fig. 4. It is not L1 since in that case its density would have been
continuous.

Solution of Exercise 4.4 1. Let W = {ℓ ∈ Z+ |(X1 , . . . , Xℓ ) is a set of linearly independent vectors}. If

this set is empty, this means that even (X1 ) is not a set of linearly independent vectors, thus ∃a ∈ R+ such
that Var (aX1 ) = 0. Since a ̸= 0, γ(0) = Var (X1 ) = 0/a = 0.

14
 If W is not empty, we define k as the maximum value in W . Since the elements of W are drawn from
Z+ , we have k ≥ 1. Then, by our choice of k, (X1 , . . . , Xk+1 ) is a not set of linearly independent vectors,
while (X1 , . . . , Xk ) is such. This imply Xk+1 ∈ Vect(X1 , . . . , Xk ).
2. Since the autocovariance matrix is invertible, its smallest eigenvalue is positive
3. We have to show that, ∀p ≥ 1, Xk+p ∈ Vect(X1 , . . . , Xk ). If γ(0) = 0 this is trivial. Otherwise, we
will prove it by recurrence.
3.1. The basis of the recurrence is already proved: Xk+1 ∈ Vect(X1 , . . . , Xk )
3.2. We have to prove that, if ∀ℓ < p, Xk+ℓ ∈ Vect(X1 , . . . , Xk ), then, also Xk+p ∈ Vect(X1 , . . . , Xk ).
By stationarity, Xk+1 ∈ Vect(X1 , . . . , Xk ) ⇒ Xk+p ∈ Vect(Xp , . . . , Xp+k−1 ).
By recurrence hypothesis, each of (Xp , . . . , Xp+k−1 ) is in Vect(X1 , . . . , Xk ). Therefore, the same for
Xk+p , q.e.d.
4. We rewrite Eq.(1) as Xk+p = φTp X = XT φp , where φp is the vector of the scalars ϕp,1 , . . . , ϕp,k and
T
X is the random vector [X1 , . . . , Xk ] . We have

γ(0)
γ(0) = E |Xk+p |2 = E φH H H 2 2
   
p X X φp = φp Γk φp ≥ λmin ∥φp ∥ ⇔ ∥φp ∥ ≤ < +∞
λmin
5.
k
! k
X X
γ(0) = Cov (Xk+p , Xk+p ) = Cov Xk+p , ϕp,ℓ Xℓ = Cov (Xk+p , ϕp,ℓ Xℓ )
ℓ=1 ℓ=1
s
k k
X X γ(0)
= ϕp,ℓ γ(p + k − ℓ) ≤ γ(p + k − ℓ)
λmin
ℓ=1 ℓ=1

By passing to the limit for p → +∞, we obtain γ(0) for the left-hand term and 0 for the right-hand term.
Solution of Exercise 5.1 We know that, ∀t, k ∈ Z, St+4k = St
1. E [Yt ] = E [βt + St + Xt ] = βt + µS + µX . Therefore {Yt , t ∈ Z} is not w.s. unless β = 0.
2.1.
∀k ∈ Z, γS (h) = Cov (St , St+h ) = Cov (St , St+h+4k ) = γS (h + 4k)
Therefore γS is periodic with period equal to 4.

2.2. By applying the operator 1 + B + B 2 + B 3 on S, we obtain:
∀t ∈ Z, S̄t = St + St−1 + St−2 + St−3 ⇒
∀t ∈ Z, S̄t − S̄t−1 = St − St−4 = 0 ⇒
∀t ∈ Z, S̄t = S̄0 = S0 + S1 + S2 + S3

3. First, we observe that, given a process {Wt , t ∈ Z}, (1 − B) ◦ 1 + B + B 2 + B 3 ◦ Wt = (1 − B 4 ) ◦ Wt .
Therefore,
Zt = (1 − B 4 ) ◦ (βt + St + Xt ) = βt + St + Xt − β(t − 4) − St−4 − Xt−4 = 4β + Xt − Xt−4
Then, E [Zt ] = 4β and:
Cov (Zt , Zt+h ) = Cov (Xt − Xt−4 , Xt+h − Xt+h−4 ) = 2γX (h) − γX (h − 4) − γX (h + 4)
Therefore {Zt , t ∈ Z} is w.s. and γZ (h) = 2γX (h) − γX (h − 4) − γX (h + 4).
4. As an autocovariance function, γS is Hermitian, but since {St , t ∈ Z} is real, it is symmetric:
γS (−h) = γS (h). Moreover, we have shown that γS is periodic, thus defined by the values of its period. We
set:
γS (0) = γ0
γS (1) = γ1
γS (2) = γ2
γS (3) = γS (−1) = γS (1) = γ1

15
Thus γS has three degrees of freedom. Let us now show that a function
π 
η(h) = a + b cos h + c cos (πh)
2
satisfies all the constraint of γS . First we observe that η is real, periodical of period 4 and symmetric.
Moreover,
η(0) = a + b + c
η(1) = a − c
η(2) = a − b + c
Finally, the parameters a, b, c are found by solving
a = γ40 + γ21 + γ42
     
1 1 1 a γ0
1 0 −1 ·  b  = γ1  ⇒ b = γ0 − γ2
2 2
1 −1 1 c γ2 c = γ40 − γ21 + γ42
As for the spectral measure, from γS (h) = a + b cos π2 h + c cos (πh), we have that νS (dλ) = aδ0 (dλ) +


b π b
2 δ 2 (dλ) + 2 δ− 2 (dλ) + cδπ (dλ).
π

5.
γZ (h) = 2γX (h) − γX (h − 4) − γX (h + 4) ⇒
fZ (λ) = 2fX (λ) − fX (λ)e−i4λ − fX (λ)ei4λ
ei4λ + e−i4λ
 
= 2fX (λ) 1 − = 2fX (λ) [1 − cos(4λ)] = 4fX (λ) sin2 (2λ)
2

Solution of Exercise 5.2 1. For a centered w.s. process {Xt , t ∈ Z}, the innovation process is defined at
each t as the difference between Xt and its projection on the linear past of the process. Thus, {Zt , t ∈ Z}
is the innovation process of {Xt , t ∈ Z}, and as such, it is a white noise (Corollary 2.4.1 in the text book).
Let us prove that in this special case.
It is easy to see that E [Zt ] = 0. We also have that Zt ∈ Ht , since both Xt and Xet are in Ht .

Proj(Zt |Ht−1 ) = Proj(Xt − X

et |Ht−1 ) = Xet − Xet = 0 ⇒ Zt ⊥Ht−1 ⇒ Zt ⊥X et ⇒
h i h i h i
E |Xt |2 = E |Zt |2 + |Xet |2 = E |Zt |2 + E |X et |2 ⇒ E |Zt |2 = E |Xt |2 − E |X
et |2
       
(3)
∀s < t,Zs ∈ Hs ⊆ Ht−1 ⇒ Zt ⊥Zs ⇔ Cov (Zt , Zs ) = 0 (4)
Eq. (3) shows that Cov (Zt , Zt ) does not depend on t and Eq. (3) shows that Cov (Zt , Zt+h ) does not depend
on t neither, and is null. Therefore, {Zt , t ∈ Z} is a weak white noise.
2. ∀s ≤ t − q − 1, Cov (Xt , Xs ) = γX (t − s) = 0 since t − s > q. This means that ∀s ≤ t − q − 1, Xt ⊥Xs ,
q.e.d.
3. We know that Xt ⊥Ht−q−1 and Xt ∈ Ht , i.e., Xt is in the orthogonal complement of Ht−q−1 in Ht ,
which is a space with q + 1 dimensions. In this space, the set (Zs , s ∈ {t, t − 1, . . . , t − q}) is made up of
orthogonal vectors, so it is a basis, implying Xt ∈PVect(Zs , s ∈ {t, t − 1, . . . , t − q}).
q
4. From the previous, we can write Xt = p=0 θt,p Zt−p . The coefficients of the projection on the
orthogonal basis are found as:
 
θt,p = Cov (Xt , Zt−p ) = Cov Xt , Xt−p − X et−p
 
= γX (p) − Cov Xt , X et−p
 
By stationarity, Cov Xt , Xet−p does not depend on t, thus θt,p also only depends on p, and can be referred
to as θp . In conclusion, we can write:
q
X
∀t ∈ Z Xt = θp Zt−p ,
p=0

16
with {Zt , t ∈ Z} a white noise: this is the definition of MA(q) process.

Solution of Exercise 5.3 1. Let us compute the average and the covariance for the sum of the MA
processes:

E [Zt ] = E [Xt ] + E [Yt ] = 0

Cov (Zt+h , Zt ) = Cov (Xt+h + Yt+h , Xt + Yt ) = γX (h) + γ( h)

Thus, {Zt , t ∈ Z} is a w.s. process. Moreover, since γZ (h) = γX (h) + γ( h), the support of γZ (h) is
s = max{p, q}. As shown in Exercise 5.2, this implies that {Zt , t ∈ Z} is an MA(s) process.
2. Let us use the shortcuts θ = θ1 and ρ = ρ1 . The process X can be seen as the filtering of the WN ϵ
with an FIR filter with impulse response a : n ∈ Z → δn + θδn−1 . This means that ϵ can be recovered from
X by applying the inverse filter with impulse response
(
n
(−θ) if n ≥ 0
b:n∈Z→
0 otherwise

Similarly, we can recover η from Y . We have

+∞
X +∞
X
k k
ϵt = (−θ) Xt−k ηt = (−ρ) Yt−k
k=0 k=0
"+∞ +∞
# +∞ X
+∞
X k
X ℓ
X k ℓ
E [ϵt , ηs ] = E (−θ) Xt−k (−ρ) Ys−ℓ = (−θ) (−ρ) E [Xt−k Ys−ℓ ] = 0 q.e.d.
k=0 ℓ=0 k=0 ℓ=0

3. In this case, introducing ξt = ϵt + ηt , we have Zt = ϵt + ηt + θ (ϵt−1 + ηt−1 ) = ξt + θξt−1 . Since |θ| < 1,
we know that this is a canonical MA representation and thus ξ is the innovation process.
4. In this case we have:

Xt = ϵt + θϵt−1 Yt = ηt + ρηt−1 ⇒
γX (h) = σϵ2 (1 + θ2 )δh + θδh−1 + θδh+1 γY (h) = ση2 (1 + ρ2 )δh + ρδh−1 + ρδh+1
   

In Question 1 we have shown that Z must be MA(1). This means that it must exist a WN ϕ and a real
number α such that ϕ is the innovation of Z and

Zt = ϕt + αϕt−1
γZ (h) = σϕ2 (1 + α2 )δh + δh−1 + δh+1
 

The unknown α and σϕ2 can be found by the identity γZ (h) = γX (h) + γY (h):

σϕ2 (1 + α2 )δh + δh−1 + δh+1 = σϵ2 (1 + θ2 )δh + θδh−1 + θδh+1 + ση2 (1 + ρ2 )δh + ρδh−1 + ρδh+1
     

σϕ2 (1 + α2 )=σϵ2 (1 + θ2 ) + ση2 (1 + ρ2 )


σϕ2 α=σϵ2 θ + ση2 ρ
b
Let us first set a = σϵ2 (1 + θ2 ) + ση2 (1 + ρ2 ) and b = σϵ2 θ + ση2 ρ. We find that α = σϕ2 and then:
!
b2 b2
σϕ2 1+ 4 =a σϕ2 + −a=0
σϕ σϕ2
1 p 
σϕ4 − aσϕ2 + b2 = 0 σϕ2 = a ± a2 − 4b2
2
1h 2 q i
σϕ2 = σϵ (1 + θ2 ) + ση2 (1 + ρ2 ) ± σϵ4 (1 − θ2 )2 + ση4 (1 − ρ2 )2 + 2σϵ2 ση2 (1 + θ2 )(1 + ρ2 )
2

17
Solution of Exercise 5.4 Let us observe that ϵt = Xt − aXt−1 and ηt = Yt − bYt−1 . We can write the
following:
Zt − (a + b)Zt−1 + abZt−2 = Xt + Yt − aXt−1 − aYt−1 − bXt−1 − bYt−1 + abXt−2 + abYt−2
= Xt − aXt−1 − b(Xt−1 − bXt−2 ) + Yt − bYt−1 − a(Yt−1 − bYt−2 )
= ϵt − bϵt−1 + ηt − aηt−1 = Wt + Vt
Now, both {Wt = ϵt − bϵt−1 , t ∈ Z} and {Vt = ηt − aηt−1 , t ∈ Z} are MA(1) processes, and thus their
sum is also a MA(1) process, meaning that it exists a WN ξ and a real number θ ∈] − 1, 1[ such that
Zt − (a + b)Zt−1 + abZt−2 = ξt − θξt−1 , q.e.d.
2. From the previous point, we can write
ξt − θξt−1 = ϵt − bϵt−1 + ηt − aηt−1 (5)
(1 − θB) ◦ ξt = (1 − bB) ◦ ϵt + (1 − aB) ◦ ηt (6)
where we use the back-shift operator B. The left-hand term of this equation can be read as the filtering of
ξ with a FIR with impulse response hk = δk − θδk−1 . As shown in Exercise 5.3, this filter can be inversed
by applying a filter with impulse response
(
n
(θ) if n ≥ 0
g:n∈Z→
0 otherwise

Apply the inverse filter to both members of Eq. (6), we get:

X X
ξt = (1 − bB) θn ϵt−n + (1 − aB) θn ηt−n
n≥0 n≥0
   
X X
= (1 − bB) ϵt + θn ϵt−n  + (1 − aB) ηt + θn ηt−n 
n≥1 n≥1
X X X X
= ϵt + θn ϵt−n − b θn ϵt−1−n + ηt + θn ηt−n − a θn ηt−1−n
n≥1 n≥0 n≥1 n≥0
X X X X
m+1 n m+1
= ϵt + θ ϵt−1−m − b θ ϵt−1−n + ηt + θ ηt−1−m − a θn ηt−1−n
m≥0 n≥0 m≥0 n≥0
X X
k h
= ϵt + (θ − b) θ ϵt−1−k + ηt + (θ − a) θ ηt−1−h q.e.d.
k≥0 h≥0

3. We write the following:

Zt+1 = (a + b)Zt − abZt−1 + ξt+1 − θξt
X X
= (a + b)Zt − abZt−1 + ϵt+1 + (θ − b) θk ϵt−k + ηt+1 + (θ − a) θh ηt−h − θξt
k≥0 h≥=
 
X X
= [ϵt+1 + ηt+1 ] + (a + b)Zt − abZt−1 + (θ − b) θk ϵt−k + (θ − a) θh ηt−h − θξt  (7)
k≥0 h≥0

If we know (Xs ∀s ≤ t) and (Ys ∀s ≤ t), we also know Zt , Zt−1 . Moreover, by applying an inverse filtering,
we know also ϵt−k ∀k ≥ 0 and ηt−h ∀h ≥ 0. On the contrary, we do not know ϵt+1 nor ηt+1 , and both are
uncorrelated with (Xs ∀s ≤ t) and (Ys ∀s ≤ t) Therefor the first term in the right-hand part of Eq. (7) is the
innovation, while the second term is the prediction.
4. In this case we do not know separately (Xs ∀s ≤ t) and (Ys ∀s ≤ t), but only their sum. We write
therefore:
Zt+1 = (a + b)Zt − abZt−1 + ξt+1 − θξt
= ξt+1 + (a + b)Zt − abZt−1 − θξt
= ξt+1 + Zet

18
Thus ξt+1 is the innovation and Zet = (a + b)Zt − abZt−1 − θξt is the prediction. Again, ξt is obtained by
inverse filtering of Zt − (a + b)Zt−1 + abZt−2 .
5. In the first case, h i
2
E |ηt+1 + ϵt+1 | = ση2 + σϵ2 .
In the second we have:
X X
ξt = ϵt + (θ − b) θk ϵt−1−k + ηt + (θ − a) θh ηt−1−h
k≥0 h≥0

= ϵt + (θ − b)αt + ηt + (θ − a)βt
with:
X X
αt = θk ϵt−1−k βt = θh ηt−1−h
k≥0 h≥0

Therefore ξ is expressed as the sum of four uncorrelated processes. We can then compute its variance,
referred to as σ 2 , as the sum of the four variances:
σ 2 = Var (ξt ) = σϵ2 + (θ − b)2 Var (αt ) + ση2 + (θ − a)2 Var (βt )
We have:
 
X X XX
Var (αt ) = E  θk ϵt−1−k θℓ ϵt−1−ℓ  = θk θℓ γϵ (k − ℓ)
k≥0 ℓ≥0 k≥0 ℓ≥0
XX X σϵ2
= θk θℓ σϵ2 δk−ℓ = σϵ2 θ2 k =
1 − θ2
k≥0 ℓ≥0 k≥0

ση2
and, likewise, Var (βt ) = 1−θ 2 . In conclusion,

σϵ2 ση2
σ 2 = Var (ξt ) = σϵ2 + (θ − b)2 + σ 2
η + (θ − a) 2
1 − θ2 1 − θ2
2 2
   
(θ − b) (θ − a)
= σϵ2 1 + + σ 2
η 1 +
1 − θ2 1 − θ2
Thus we see that the variance of the innovation in the second case is always larger than that of the first
case, unless θ = a = b.
Solution of Exercise 5.5 We observe that {Xt , t ∈ Z} is a MA(1) process, thus, if γ be the autocovariance
function of {Xt , t ∈ Z}, its support is {−1, 0, +1}. In facts, we have:
γ(h) = E [(Zt + θZt−1 )(Zt+h + θZt+h−1 )] = σ 2 (1 + θ2 )δh + θδh−1 + θδh+1
 

1. The linear prediction of X3 is written as:

Xb3 = αX1 + βX2 .
 2 
Our problem consists in minimizing the mean square error E X3 − X3
b . The optimal solution is found

the the error (X3 − X

b3 ) is orthogonal to data (X1 , X2 ). Thus we have:
   
Cov X3 − X b3 , X1 = 0 Cov X3 − X
b3 , X2 = 0

Cov (X3 − αX1 − βX2 , X1 ) = 0 Cov (X3 − αX1 − βX2 , X2 ) = 0

γ(2) − αγ(0) − βγ(1) = 0 γ(−1) − αγ(1) − βγ(0) = 0
2 2 2
−ασ (1 + θ ) − βσ θ = 0 σ θ − ασ 2 θ − βσ 2 (1 + θ2 ) = 0
2

19
This is a linear system, and we can actually get rid of σ 2 :

(1 + θ2 )
    
θ α 0
=
θ (1 + θ2 ) β θ

We find:
   " −θ2 #
(1 + θ2 )
  
α 1 −θ 0 4 +θ 2 +1
= 4 = θ θ+θ 3
β θ + θ2 + 1 −θ (1 + θ2 ) θ 4 2 θ +θ +1

2. If we now set
X
b3 = αX5 + βX4 .

and we look for α, β minimizing the MSE, we end up exactly with the same equation as before, since for real
processes, γ(h) = γ(−h). Therefore, the same optimal values of the coefficients are found:

−θ2 θ + θ3
α= β=
θ4 + θ2 + 1 θ4 + θ2 + 1
3. Let us define the spaces V1 = Vect(X1 , X2 ) and V2 = Vect(X4 , X5 ). Any element of V1 is uncorrelated
to any element of V2 (i.e., they are orthogonal):

Cov (aX1 + bX2 , cX4 + dX5 )

=acCov (X1 , X4 ) + adCov (X1 , X5 ) + bcCov (X2 , X4 ) + bdCov (X2 , X5 )
=acγ(−3) + adγ(−4) + bcγ(−2) + bdγ(−3) = 0

Thus, Vect(X1 , X2 , X4 , X5 ) = V1 ⊕ V2 , which implies that

b3 = Proj(X3 |V1 ⊕ V2 ) = Proj(X3 |V1 ) + Proj(X3 |V2 ) = X

X b3,1 + X
b3,2
 
Since Xb3,1 and X b3,2 are orthogonal, when we impose Cov X3 − X b3 , Xi = 0, with i ∈ {1, 2, 4, 5}, only
one between X b3,1 and X

b3,2 gives a non-zero covariance (depending on i). Therefore, we end up with
 
Cov X3 − X b3,1 , Xi = 0 or Cov X3 − X b3,2 , Xi = 0, i.e., the same equations as in Questions 1 and 2.
Therefore we find the same partial solutions. In conclusion:

−θ2 θ + θ3 θ + θ3 −θ2
X
b3 = X 1 + X 2 + X4 + X5
θ4 + θ2 + 1 θ4 + θ2 + 1 θ4 + θ2 + 1 θ4 + θ2 + 1
θ + θ3 θ2
= 4 (X 2 + X 4 ) − (X1 + X5 )
θ + θ2 + 1 θ4 + θ2 + 1
Solution of Exercise 1 1. Let us first rewrite the equation defining X as an ARMA(p, q) equation:
p
X p
X
Xt − ϕk Xt−k = ϵt + θk ϵt−k (8)
k=1 k=1

Let us introduce the polynomials Φ(z), Θ(z):

p
X p
X
Φ(z) = 1 − ϕk z k Θ(z) = 1 + θk z k
k=1 k=1

Introducing the backshift operator B, the ARMA equation (Eq. (14)) can be written as:

Φ(B)X = Θ(B)ϵ (9)

20
 Now we have just to check that a) Φ(z) and Θ(z) do not have common roots and that b) Φ(z) does not
vanish on the unit circle of C. This is straighforward since the only root of Φ is 1/2 while the only root of
Θ is −1/4. We can then apply theorem 3.3.2: X is the unique w.s. solution of Eq. (14), and it admits a
spectral density function given by:
2
σ 2 Θ(e−iλ )
fZ (λ) =
2π |Φ(e−iλ )|2
In our case we have the following function, shown in Fig. 1:
2
σ 2 1 + 4e−iλ σ 2 8 cos λ + 17
fZ (λ) = = .
2π |1 − 2e−iλ |2 2π 5 − 4 cos λ

2. We remind that a canonical representation of an ARMA process is characterized by the fact that X
is a causal and invertible filtering of weak noise. This is equivalent to say that neither Φ nor Θ vanish on
the closed unit disk ∆1 = {z ∈ C : |z| ≤ 1}.
A given representation of an ARMA process is not necessarily canonical but it is possible to get a
canonical representation by using an all-pass filter. We recall that, given ψ ∈ ℓ1 , the filter Fψ is an all-pass
filter if and only if:
X
∀z ∈ Γ1 , ψk z k = c,
k∈Z

where Γ1 = {{z ∈ C : |z| = 1} is the complex unit circle and c > 0 is a constant.
A key property of all-pass filters is that they transform a WN process At into another WN process Bt .
To prove this, let us first recall that, since ψ ∈ ℓ1 , then theorem 3.1.2 and corollary 3.1.3 apply. Thus
B = Fψ (A) is a w.s. centered process, with spectral density function
2
σ2 X 2
σA
fB (λ) = A ψk e−ikλ = c2 ,
2π 2π
k∈Z

where we applied the definition of all-pass filter for z = e−iλ ∈ Γ1 . We also have that:
1 X
fB (λ) = γB (h)e−ikλ
2π
k∈Z

Comparing the two last equations and remembering that the Discrete-Time Fourier Transform is injective
for ℓ1 sequences, we get γB (h) = c2 σA
2
δh , .
A second, crucial property of all-pass filters is that they can be used toQinvert the moduli of the roots of
q
a polynomial (example 3.2.2): let Q be a polynomial defined by Q(z) = k=1 (1 − νk z), such that none of
the νk have neither unitary nor zero modulus. We observe that Q(0) = 1 and that the q roots of Q are νk−1
for k = 1, . . . , q.
Qn  
Now we define the polinomial Q(z)
e = k=1 1 − νk−1 z and the function Ξ : z = Q Qe (z). Ξ is a rational
function with poles νk ̸= Γ1 . Then we know that it exists a unique ℓ1 sequence ξk such that Ξ(z) =
k
P
k∈Z ξk z . Let us now prove that the filter Fξ is then an all-pass. First, we have:

n
X Y |1 − νk z|
ξk z k = . (10)
k∈Z k=1 1 − νk−1 z

Now, since z ∈ Γ1 ⇒ z = z −1 , for any k = 1, . . . , n and for z ∈ Γ1 we have:

1 − νk−1 z = −νk−1 z −νk z −1 + 1 = νk−1 |z| 1 − νk z −1 = νk−1 |z| 1 − νk z

= νk−1 1 − νk z = νk−1 |1 − νk z| .

21
Replacing in Eq. (10), we get:
n n
X Y |1 − νk z| Y
ξk z k = = |νk | = c > 0
k∈Z k=1 νk−1 |1 − νk z| k=1

Equipped with the all-pass filter properties, we can rewrite an ARMA filter in canonical form. Let us
−1
consider an all-pass filter in the form Ξ = a Q e . The roots νk
Q
and the constant a will be defined later on.
If Φ has no roots on Γ1 we know that Fϕ is invertible. Likewise, by construction Ξ is invertible. Using a
fraction notation to refer to inverse filters, we can formally rewrite Eq. (9) as:
 
Θ Θ Ξ Θ Θ
e
X= ◦ϵ= ◦ ◦ϵ= ◦ (Ξ ◦ ϵ) = ◦η
Φ Φ Ξ ΦΞ Φ
e
with:
Θ
e Θ
= η = Ξ ◦ ϵ.
Φ
e ΦΞ
We already know that η is a WN process, since it is an all-pass filtering of a WN. We have to show that we
Q
can build such a Ξ(B) = a Q e (B) that Θ and Φ do not have roots in the closed unit disk ∆1 . This is always
e e
possible since we can write:
Qq (θ) n
Θ(z)
e Θ(z) 1 (1 − νk z) 1 Y 1 − νk−1 z
= = Qpk=1 (ϕ)
Φ(z) Φ(z) Ξ(z) a 1 − νk z
k=1 (1 − νk z)
e
k=1

(ϕ) (θ)
where νk (resp. νk ) are the inverse of the roots of Φ (resp. of Θ). Now we build Ξ such that we cancel
(θ)
out the roots of Φ and of Θ in ∆1 . More precisely, to cancel out a given νk we introduce as a root of Q the
 −1
(θ) (ϕ) (θ)
number νk = νk and to cancel out a given νk we introduce as a root of Q the number νk = νk .
Θ(z) 1+4z
In our case, we have: Φ(z) = 1−2z with roots − 41 and 12 . To cancel out these roots, we set:

Θ(z)
e Θ(z) 1 1 + 4z 1 1 + 14 z 1 − 2z 1 1 + 14 z
= = · =
Φ(z)
e Φ(z) Ξ(z) 1 − 2z a 1 + 4z 1 − 12 z a 1 − 12 z
1 + 4z 1 − 12 z
Ξ(z) = a
1 + 41 z 1 − 2z
5 1/2
Since ∀z ∈ Γ1 , |Ξ(z)| = c, given that Ξ(1) = a 5/4 −1 = −2a, choosing a = −1/2 we get ∀z ∈ Γ1 |Ξ(z)| =
|Ξ(1)| = 1. This also implies fη (λ) = fϵ (λ) and thus Var (η) = Var (ϵ). In conclusion,

Θ(z)
e −2 − 12 z
=
Φ(z)
e 1 − 21 z
1 1 + 4z 1 − 21 z
η=− ϵ
2 1 + 14 z 1 − 2z
1 1
Xt − Xt−1 = −2ηt − ηt−1
2 2
3. Let us recall here the results of theorem 3.5.1. The canonical representation of an ARMA process is
desirable since it express the former as an causal and inversible filtering of WN:
Xt = ϕe1 Xt−1 + . . . + ϕep Xt−p + θe0 ηt + θe1 ηt−1 + . . . + θeq ηt−q

This means that there exist two causal ℓ1 sequences, ξ and ξ,

e such that:

X = Fξ (η) (11)
η = Fξe(X) (12)

22
 t t
From Eq. (11), since ξ is causal, we deduce that HX ⊆ HZ . From Eq. (12), since ξe is causal, we deduce that
t t t t
HZ ⊆ HX . In conclusion, HX = HZ . If we set:

X
bt = ϕe1 Xt−1 + . . . + ϕep Xt−p + +θe1 ηt−1 + . . . + θeq ηt−q

we see that Xt − X bt ⊥Hηt−1 but then Xt − X

bt = θe0 ηt . Since η is WN, Xt − X bt ⊥Ht−1 . This means that X
bt
X
is the projection of Xt onto its linear past, and therefore θe0 ηt is the innovation process of X.
The canonical form gives therefore a direct access to the innovation of an ARMA process.
Now we can answer immediately to the question. The variance of the innovation is:

Var (−2ηt ) = 4Var (ηt ) = 4Var (ϵt ) .

4. From the definition of X we can write: (1 − 2B)Xt = (1 + 4B)ϵt . Setting the AR process Wt such
that (1 − 2B)Wt = ϵt , we have Xt = (1 + 4B)Wt .
Now,

   k
1 1 1 1 −1 X 1 −1
Wt = ϵt = − ϵ t = − B B ϵt
1 − 2B 2B 1 − 12 B −1 2 2
k≥0
X 1 k X  1 k
−1
=− B ϵt = − ϵt+k
2 2
k≥1 k≥1
   
X  1 k X  1 n
Xt = Wt + 4Wt−1 = −  ϵt+k  − 4  ϵt+n−1  set ℓ = n − 1
2 2
k≥1 n≥1
   
X 1  k X 1   ℓ
1
= − ϵt+k  − 4  ϵt+ℓ 
2 2 2
k≥1 ℓ≥0
   
X  1 k 1 X  ℓ
1 1
= − ϵt+k  − 4  ϵt + ϵt+ℓ 
2 2 2 2
k≥1 ℓ≥1
   
X  1 k X  1 ℓ
= −2ϵt −  ϵt+k  − 2  ϵt+ℓ 
2 2
k≥1 ℓ≥1
X 3
= −2ϵt − ϵt+k
2k
k≥1

Solution of Exercise 5.7 We have to compute the impulse response of a recursive filter. Since |ϕ| < 1, a
stable, causal solution exists. The weights ψk are such that:
(
X
k 1 X
k k ϕk if k ≥ 0
ψk z = = ϕ z ⇒ ψk =
1 − ϕz 0 if k < 0
k∈Z k≥

Therefore, Xt = k≥0 ϕk ϵt−k

P
2. We can apply Corollary 3.1.3 on the linear filtering of WN. Therefore, observing that ψk is real,
X
γX (h) = σϵ2 ψk+h ψk =
k∈Z
σϵ2 ϕh
(
σϵ2 ϕh k≥0 ϕ2k = 1−ϕ
P
2 if h ≥ 0
= σϵ2 ϕ−h
σϵ2 k≥−h ϕk+h ϕk = σϵ2 n≥0 ϕn ϕn−h = σϵ2 ϕ−h k≥0 ϕ2k =
P P P
1−ϕ2 if h < 0
σϵ2 ϕ|h|
=
1 − ϕ2

23
A Annals
A.1 Exam of 2021
Documents autorisés: polycopié, notes de cours/TD.
Durée: 1 heure 30.
Authorized Documents : lecture notes, tutorial notes.
Duration: 1 hour and 30 minutes.

Exercise A.1. Let us consider {Xt , t ∈ Z} , {Yt , t ∈ Z} two L2 , stationary and independent stochastic pro-
cesses with means µX , µY and autocovariance functions γX , γY , respectively.
1. Show that St := Xt + Yt is weakly stationary and compute its mean µS and autocovariance function
γS .
2. Assuming that X and Y have spectral densities fX and fY , show that S admits a spectral density fS
and express it using fX and fY .
3. Show that the process Zt := Xt Yt is weakly stationnary and compute its mean µZ and its autocovari-
ance function γZ , first in the case where µX = µY = 0, then in the general case.
4. Show that Z admits a spectral density fZ and compute it first in the case where µX = µY = 0, then
in the general case. Use the convolution of two functions with a period of 2π defined by
Z π
f ⋆ g (x) := f (u) g (x − u) du
−π

Exercise A.2. Consider a random process X = (Xt )t∈Z satisfying the following recurrence equation:
Xt = ρXt−1 + Zt − (a + 1/a)Zt−1 + Zt−2 (13)
where Zt is a zero-mean weak white noise with variance σ 2 and both ρ and a are numbers in (−1, 1) such
that a ̸= ρ and a ̸= 0.
5. Justify that this equation admits a weakly stationary solution X and find the expression of the power
spectral density f (λ) of this solution.
6. Is Eq. (14) an ARMA equation in canonical form?
7. Express X in its MA(∞) form, that is, compute (ϕk ) such that
X
Xt = ϕk Zt−k .
k∈Z

8. Find b and c such that, for all z ∈ C \ {a, 1/a},

b c 1
+ = .
1 − az 1 − z/a 1 − (a + 1/a)z + z 2
Compute (ψk ) using a, b, c and ρ such that
X
Zt = ψk Xt−k .
k∈Z

9. Determine the variance of the innovation process W of X.

10. Compute (αk ) such that X
Wt = αk Zt−k .
k∈Z

X
11. Express proj(Xt |Ht−1 ) using X and W .

24
A.2 Exam of 2020

Documents autorisés: polycopié, notes de cours/TD.

Durée: 1 heure 30.
Authorized Documents : lecture notes, tutorial notes.
Duration: 1 hour and 30 minutes.

Exercise A.3 (Representations of an ARMA(2,1) process). We consider a random process (Xt )t∈Z satisfying
the following recurrence equation:
1
Xt = 6Xt−1 − 9Xt−2 + εt + εt−1 , (14)
2
where (ϵt ) is a zero-mean weak white noise with variance σ 2 .

1. Why does Eq. (14) admit a unique weakly stationary solution ? What is the nature of this solution
(Xt )?
2. Find the expression of the power spectral density f (λ) of the process X.
3. Find a canonical representation of X by using a suitable all-pass filter.
4. What is the innovation process of X? What is its variance?
5. Compute the coefficients (ϕk )k≥1 of the AR(∞) representation
X
Xt = ϕk Xt−k + Zt ,
k≥1

where (Zt ) is the innovation process of (Xt ).

Exercise A.4 (Linear prediction). Let {Xt , t ∈ Z} be a weakly stationary, zero-mean, real random process
satisfying the equation
Xt = θXt−1 + Zt ,
where θ ∈] − 1, 1[, and {Zt , t ∈ Z} is weak noise with Var(Zt ) = σ 2 . Let X̂t be a linear predictor of Xt of
the form
P
X
X̂t = αk Xt−k ,
k=1

with P ∈ N being the order of the predictor. Finally, we define

Yt = Xt − X̂t ,

as the prediction error. We want to compare the variance (power) and the autocorrelation function of the
prediction error with those of the original process X. In several applications (e.g., signal compression) it is
desirable to have a prediction error with a smaller power than the original process. Also, achieving a white
prediction error is desirable.
1. The input signal
(a) Is X a causal filtering of Z?
(b) Find the autocorrelation function (ACF) of X, ρX (h)
(c) Find the variance of Xt
2. Simple first-order predictor
(a) Let us consider the simplest predictor: X̂t = Xt−1 . Find the variance of the prediction error.

25
 (b) In which case the variance of Y is smaller than the variance of X?
(c) Find the ACF of Y
3. Optimal first-order predictor

(a) The optimal first-order predictor is: X̂t = αXt−1 with α ∈ R such that the variance of Y is
minimized. Find the optimal value of α.
Hint: recall that the optimal α is such that Cov (Yt , Xt−1 ) = 0
(b) Find the variance of Y : is it smaller than that of X?
(c) Find the ACF of Y and justify the name “whitening filter”
4. Optimal second-order predictor
(a) A second-order predictor is in the form X̂t = αXt−1 + βXt−2 . Show that for the optimal second-
order predictor, β = 0, and conclude.

26
B Solutions of annals
Solution of Exercise A.1 1. E [Xt + Yt ] = E (Xt )+E (Yt ) = µX +µY , γX+Y (t, t + h) = γX (h)+γY (h)
(does not depends on t)
Theorem (2.3.1) + Corollary (2.3.2) : γZ (h) = γX (h) + γY (h) = eihλ νX (dλ) +
R
2. RBy using Herglotz
eihλ νY (dλ) ihλ
R
1
P= e (νX−ihλ (dλ) + νY (dλ)).
1
P Then νZ = ν
X + νY P
−ihλ 1
.
−ihλ


fZ (λ) = 2π h∈Z γZ (h) e = 2π γ
h∈Z X (h) e + 2π h∈Z γY (h) e = fX (λ) + fY (λ)
3. Using the independence, we get: E [Xt Yt ] = E (Xt ) E (Yt ) = µX µY . Moreover:

γXY (t, t + h) = E (Xt Xt+h ) E (Yt Yt+h ) − µX µY µX µY

= γX (h) + µ2X γY (h) + µ2Y − µ2X µ2Y

= γX (h) γY (h) + µ2X γY (h) + µ2Y γX (h)

4. Let fix t ∈ [−π, π], by using Fubini-Tonelli and Hertglotz theorem, we get:
Z π
(fX ⋆ fY ) (t) = fX (u)fY (t − u)du
−π
Z π
1 X 1 X
= γX (h)e−ihu γY (k)e−ik(t−u) du
−π 2π h∈Z 2π
k∈Z
Z π
1 X 1
= γX (h)γY (k)e−ikt e−iu(h−k) du
2π 2π −π
h,k∈Z | {z }
1h=k
1 X
= γX (h)γY (h)e−iht
2π
h∈Z
1 X −iht
γXY (h) − µ2Y γX (h) − µ2X γY (h)


= e
2π
h∈Z

= fXY (t) − µ2Y fX (t) − µ2X fY (t)

Meaning that finally:

fZ (t) = (fX ⋆ fY ) (t) + µ2Y fX (t) + µ2X fY (t)

Solution of Exercise A.2 5. Let Φ(z) = 1 − ρz and Θ(z) = 1 − (a + 1/a)z + z 2 . Then Φ have no
roots on the unit disk of C. In this case the ARMA equation Φ(B)X = Θ(B)Z has a unique weakly
stationary solution X and its spectral density is
2
σ 2 1 − (a + 1/a)e−iλ + 1
f (λ) = .
2π 1 − ρe−iλ

6. Since Θ has roots a and a−1 it vanishes on the unit disk. Hence (14) is not a canonical ARMA equation.
7. Inverting the filter Φ(B) leads to
∞
X
Xt = ρk (Zt−k − (a + 1/a)Zt−k−1 + Zt−k−2 )
k=0
X
ρj−2 ρ2 − ρ(a + 1/a) + 1 Zt−j .


= Zt + (ρ − a − 1/a)Zt−1 +
j≥2

It is then easy to identify ϕj for all j ≥ 0.

27
 8. We must have b + c = 1 and b/a + ca = 0. We find b = −ca2 , so
c = 1/(1 − a2 ) ,
b = −a2 /(1 − a2 ) .
Since |a| < 1 and a ̸= 0, we have for z ∈ C with |z| = 1,
∞
X
(1 − az)−1 = ak z k
k=0
∞
X
(1 − z/a)−1 = −a/z(1 − a/z)−1 = − ak z −k .
k=1

Hence we get
∞
1 X X
2
= b ak z k − c a−k z k .
1 − (a + 1/a)z + z
k=0 −∞<k<0
This provides the inverse linear filter of Θ(B) and we obtain
∞
X X
Zt = b ak (Xt−k − ρXt−k−1 ) − c a−k (Xt−k − ρXt−k−1 )
k=0 −∞<k<0
X∞ ∞
X X X
= b ak Xt−k − b ak−1 ρXt−k − c a−k Xt−k + c a−k+1 ρXt−k
k=0 k=1 −∞<k<0 −∞<k≤0
X ∞
X
= c a−k (a ρ − 1)Xt−k + (b + c a ρ)Xt + b ak (1 − a−1 ρ)Xt−k .
−∞<k<0 k=1

It is then easy to identify ψj for all j ∈ Z.

9. Let W = Fα (Z) with
X 1 − z/a
αk z k = , z ∈ C, |z| = 1 ,
1 − az
k∈Z

Then if |z| = 1 we have | 1−z/a

1−az | = 1/|a| so that W ∼ WN(0, σ 2 /|a|2 ) and, moreover, Z = Fβ (W ) with
X 1 − az
βk z k = ,
1 − z/a
k∈Z

which gives that

Φ(B)X = Θ(B) ◦ Fβ (W ) = (1 − aB)2 W .
The latter equation is canonical, so W is the innovation of X. It has variance σ 2 /|a|2 .
10. Moreover W can be written as X
Wt = αk Zt−k .
k∈Z
The coefficients (αk ) are identified by the equation, for z ∈ C, |z| = 1 ,
X 1 − z/a
αk z k =
1 − az
k∈Z
X
= (1 − z/a) (az)k
k≥0
X X
k
= (az) − ak−1 z k+1
k≥0 k≥0
X
=1+ a (1 − a−2 ) z k .
k

k≥0

It is then easy to identify αj for all j ∈ Z.

28
 11. Since Φ(B)X = (1 − aB)2 W is canonical, we have Ht−1
X W
= Ht−1 ⊥Wt and
Xt = ρXt−1 + Wt − 2aWt−1 + a2 Wt−2
gives that
X
proj(Xt |Ht−1 ) = ρXt−1 − 2aWt−1 + a2 Wt−2 .
Solution of Exercise A.3 1. We have that Φ(z) := 1 − 6z + 9z 2 = (1 − 3z)2 dos not vanish on the
unit circle, which ensures existence and uniqueness of the solution, which is then called an ARMA(2,1)
process.
2. The spectral density is given by
2
σ 2 1 + e−iλ /2
f (λ) = .
2π |1 − 3e−iλ |4

3. Let Fβ denote the all-pass filter with coefficients (βk ) ∈ ℓ1 defined by the equation
X 1 − z −1 /3
βk z k = , z ∈ C , |z| = 1 .
1 − 3z
k∈Z

We apply this filter twice on both sides of (14) and obtain that X is solution of
(1 − B/3)2 X = (1 + B/2) Z , (15)
where Z = Fβ (ϵ) has spectral density
4
σ2 1 − eiλ /3 σ2
f Z (λ) = =
2π 1 − 3e−iλ 34 .2π
Hence Z is a white noise with variance σ 2 /34 . The representation (15) is a canonical representation of
X.
4. From the previous question, we deduce that Z is the innovation process of X. It has variance σ 2 /34 .
5. From (15), we have
Z = Fα (X) ,
1
where (α)k is the ℓ sequence satisfying
X (1 − z/3)2
αk z k = , z ∈ C , |z| = 1 .
1 + z/2
k∈Z

Now, we have, for all z ∈ C with |z| = 1,

(1 − z/3)2 X
= (1 − z/3)2 (−1/2)k z k
1 + z/2
k≥0
 
X
(−1/2)k − (−1/2)k−1 /3 z k 


= (1 − z/3) 1 +
k≥1
 
5 X
= (1 − z/3) 1 + (−1/2)k z k 
3
k≥1
7 5X
(−1/2)k − (−1/2)k−1 /3 z k


=1− z+
6 3
k≥2
 2 X
7 5
=1− z+ (−1/2)k z k .
6 3
k≥2

We conclude that ϕ1 = −α1 = 7/6 and, for all k ≥ 2, ϕk = −αk = −(5/3)2 (−1/2)k .

29
Solution of Exercise A.4 1. The input signal
1
P of the polynomial Θ(z) = 1 − θz is
(a) X is a causal filtering of Z because the only root θ, outside
the unit circle. Therefore, one can write Xt = ℓ≥0 θℓ Zt−ℓ
(b) For h ⩾ 0, the autocovariance function of X, γX (h) is
 
X X XX
γX (h) = E  θℓ Zn−ℓ θk Zn+h−k  = θℓ+k E [Zn−ℓ Zn+h−k ]
ℓ≥0 k≥0 ℓ≥0 k≥0
XX X
= θℓ+k σ 2 δk−(ℓ+h) = σ 2 θ2ℓ+h
ℓ≥0 k≥0 ℓ≥0

σ2
= θh
1 − θ2
2
By symmetry, we have γX (h) = θ|h| 1−θ
σ
2 . Thus the ACF reads

ρX (h) = γX (h)/γX (0) = θ|h| , h∈Z.

(c) The variance of Xt is

2 σ2
σX = γX (0) = .
1 − θ2
2. Simple first order predictor

(a) The variance of the prediction error is:

h i
2
Var (Yt ) = E Yt2 = E (Xt − Xt−1 ) = E Xt2 + Xt−1
2
   
− 2Xt Xt−1
2σ 2
= 2γX (0) − 2γX (1) = (1 − θ)
1 − θ2
2
= σX 2(1 − θ)

(b) From the previous, the variance of Y is smaller than the variance of X if and only if 2(1 − θ) < 1,
implying θ > 12 . Also, remember that θ < 1 by hypothesis. In conclusion the simple predictor is
effective only if consecutive samples of X are correlated enough.
(c) The autocovariance function of Y is computed as follows for h > 0:

γY (h) = E [Yt Yt+h ] = E [(Xt − Xt−1 ) (Xt+h − Xt−1+h )]

σ2
2θh − θh−1 − θh+1


= 2γX (h) − γX (h − 1) − γX (h + 1) = 2
1−θ
−σ 2 2 −σ 2 −(1 − θ)θh−1 2
= (1 − θ) θh−1 = (1 − θ) θh−1 = σY
1 − θ2 1+θ 2
For h = 0, γY (h) = Var (Yt ) and for h < 0, γY (h) = γY (−h).
3. Optimal first order predictor
(a) The optimal first order predictor is found by setting Cov (αXt−1 − Xt , Xt−1 ) = 0

0 = Cov (αXt−1 − Xt , Xt−1 ) = αγX (0) − γX (1)

γX (1)
α= =θ
γX (0)
X̂t = θXt−1
Yt = Xt − θXt−1 = Zt

30
 2
σ
(b) Since Yt = Zt , its variance is σ 2 , which is smaller than σX
2
= 1−θ 2 for any θ ∈] − 1, 1[. The
h i
2
variance of Yt can also be found explicitly as E (Xt − θXt−1 ) .
(c) The ACF of Y is the one of Z: ρY (h) = δh . Therefore Y is white noise. Again, γY can be found
by calculating E [(Xt − θXt−1 ) (Xt+h − θXt−1+h )]
4. Optimal second order predictor

(a) The optimal second order predictor is such that:

Cov (αXt−1 + βXt−2 − Xt , Xt−1 ) = 0 αγX (0) + βγX (1) = γX (1)

Cov (αXt−1 + βXt−2 − Xt , Xt−2 ) = 0 αγX (1) + βγX (0) = γX (2)

    
γX (0) γX (1) α γX (1)
=
γX (1) γX (0) β γX (2)
    
α 1 γX (0) −γX (1) γX (1)
= 2 2 (1) −γ (1)
β γX (0) − γX X γX (0) γX (2)
2
γX (0)γX (2) − γX (1)
β= 2 2
γX (0) − γX (1)
2 4 2 4 2
But γX (0)γX (2) − γX (1) = σX θ − σX θ = 0, thus β = 0.
Conclusion: since X is an AR(1) process, there is no advantage in considering linear predictors
of order greater than 1.

31