Introduction to Random Processes

Non-stationary process

Example
Consider the random signal x[k] = A cos(!k + ), where is a random variable with
uniform distribution in [0, ⇡]. The mean of this signal (process) is
Z ⇡ Z ⇡
A
E(x[k]) = E(A cos(!k + )) = A cos(!k + )f ( ) d = cos(!k + ) d
0 ⇡ 0
2A
= sin(!k)
⇡
which is a function of time. Thus, the process is non-stationary.

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 14

Introduction to Random Processes

Order stationarity in distribution

A stochastic process is said to be N th -order stationary (in distribution) if the joint

distribution of N observations is invariant to shifts in time,

FXk1 ,··· ,XkN (x1 , · · · , xN ) = FXT +k1 ,··· ,XT +kN (x1 , · · · , xN ) 8T, k1 , · · · , kN 2 Z + (2)

where Xk1 , · · · , XkN are the RVs associated with the observations at k = k1 , · · · , kN ,
respectively and x1 , · · · , xN are any real numbers.

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 15

Introduction to Random Processes

Special cases

1. First-order stationarity in distribution:

FXk1 (x1 ) = FXk1 +T (x1 ) 8T, k1 2 Z + (3)

Every observation should fall out of the same distribution.

2. Second-order stationarity in distribution:

FXk1 ,Xk2 (x1 , x2 ) = FXk1 +T ,Xk2 +T (x1 , x2 ) 8T, k1 , k2 2 Z + (4)

The distribution depends only on the time-di↵erence k2 k1 .

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 16

Introduction to Random Processes

Relaxation of strict stationarity

The requirement of strict stationarity is similar to the strict requirement of time-invariance

or linearity (in deterministic processes), which are also academically convenient assump-
tions, but rarely satisfied in practice.

In reality, rarely will we find a process that satisfies the strict requirements of stationarity
defined above.

A weaker requirement is that certain key statistical properties of interest such as mean,
variance and a few others at least, remain invariant with time

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 17

Introduction to Random Processes

Weak stationarity
A common relaxation, is to require invariance up to second-order moments.

Weak or wide-sense or second-order stationarity

A process is said to be weakly or wide-sense or second-order stationary if:
i. The mean of the process is independent of time, i.e., invariant w.r.t. time.
ii. It has finite variance.
iii. The auto-covariance function of the process

xx [k1 , k2 ] = cov(Xk1 , Xk2 ) = E((Xk1 µ1 )(Xk2 µ2 )) (5)

is only a function of the “time-di↵erence” (lag l = k2 k1 ) but not the time.

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 18

Introduction to Random Processes

On wide-sense stationarity (WSS)

Q: Under what conditions is the weak stationarity assumption justified?
Where linear models are concerned, the optimal parameter estimates are fully
determined by the first- and second-order properties of the joint p.d.f.

Example
For a stationary process, suppose a linear predictor of the form

x̂[k] = d1 x[k 1] d2 x[k 2]

is considered. Determine the optimal (in the m.s. prediction error sense) estimates.
Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 19
Introduction to Random Processes

Gaussian WSS process

A WSS multivariate Gaussian process is also strictly stationary.

Why?

I A joint Gaussian distribution is completely characterized by the first two moments.

Therefore, when the first two moments remain invariant, the joint pdf also remains
invariant.

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 20

Introduction to Random Processes

Non-stationarities

Just as with non-linearities, there are di↵erent types of non-stationarities, for e.g., mean
non-stationarity, variance non-stationarity, and so on. It is useful to categorize them, for
working purposes, into two classes:

1. Deterministic: Polynomial trend, variance non-stationarity, periodic, etc.

2. Stochastic: Integrating type, i.e., random walk, heteroskedastic processes, etc.

We shall be particularly interested in two types of non-stationarities, namely, trend-type

and random walk or integrating type non-stationarities.

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 21

Introduction to Random Processes

Trend-type non-stationarity

A suitable mathematical model for such a process is,

x[k] = µk + w[k] (6)

where µk is a polynomial function of time and w[k] is a stationary process.

For example, a linear trend is modeled as µk = a + bk.

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 22

Introduction to Random Processes

Trend type non-stationarity . . . contd.

I When the removal of a trend (e.g., linear ,quadratic) results in stationary residuals,
the process is said to be trend non-stationary.
I Trends may be removed by either fitting a suitable polynomial (parametric approach)
or by applying an appropriate smoothing filter (non-parametric approach).

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 23

Introduction to Random Processes

Example: Trend-type non-stationarity

I A linear trend is fit to the series. Airline passenger series
800

Airline passengers
I Residuals show variance type 600

400

nonstationarity. This is typical of a 200

growth series. 0
1949 1951 1953 1955 1957 1959 1961
200
I Two approaches: 100

Residuals
1. Trend fit + transformation OR 0

−100
2. Advanced models known as GARCH −200
1949 1951 1953 1955 1957 1959 1961
(generalized auto-regressive Year

conditional heteroskedastic) models. Monthly airline passenger series.

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 24

Introduction to Random Processes

Integrating type non-stationarity

One of the most commonly encountered non-stationary processes is the random walk
process (special case of Brownian motion).
The simplest random walk process is an integrating process,

k
X
x[k] = e[k] (7)
n=0

where e[k] is the (unpredictable) white-noise a↵ecting the process at the k th instant.

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 25

Introduction to Random Processes

Integrating processes

I At any instant the signal is the accumulation of all shock-wave like changes from
the beginning. Hence the name integrating.
I It is also known as a di↵erence stationary process because

x[k] x[k 1] = e[k] (8)

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 26

Introduction to Random Processes

Example
Original Series

I Non-stationarity can be easily 10

Amplitude
discerned by a visual inspection. −10

−20

I The di↵erenced series appears to be −30

0 50 100 150 200

Differenced Series

stationary. 4

Amplitude
I In general, a single degree of 0

−2

di↵erencing is capable of removing a −4

0 50 100
Time
150 200

linear trend, two degrees removes

Top panel: N = 200 samples of a random walk
quadratic trends and so on.
series. Bottom plot: di↵erenced series.

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 27

Introduction to Random Processes

Caution

Despite its capability in handling a wide range of stationarities, the di↵erencing

approach also has potentially a few detrimental e↵ects.
I Excessive or unnecessary di↵erencing can lead to spurious correlations in the
series.
I Amplification of noise, i.e., decrease in SNR in system identification.

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 28

Introduction to Random Processes

Ergodicity

When the process is stationary, the second requirement on the time-series stems from
the fact that in practice we work with only a single record of data.

Estimates computed from averages of time (or other domain) samples should serve as suit-
able representatives of the theoretical statistical properties, which are defined as averages
in the outcome space (ensemble).

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 29

Introduction to Random Processes

Ergodicity: Formal statement

Ergodicity
A process is said to be ergodic if the (time averaged) estimate converges to the true
value (statistical average) when the number of observations N ! 1.

Examples
1. A stationary i.i.d. random process is ergodic (by the strong LLN)

N
1 X a.s.
x[k] ! E(Xk ) as N ! 1 (9)
N k=1

2. A process such that x[k] = A, 8k, s.t.E(A) = 0. Is it ergodic?

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 30

Introduction to Random Processes

Remarks
I We can speak of ergodicity only when the process is stationary!
I Loose interpretation: given sufficient time, the process would have
unravelled nearly all possibilities that exist at any time instant (regardless
of the starting point),
I Ergodicity is not necessarily a characteristic of the process, but can also be of the
experiment that is carried out to obtain the time-series.
I Ergodicity is difficult to verify in practice; however, can be ensured by a careful
experimentation, particularly through a proper selection and configuration of
sensors and instrumentation.

Arun K. Tangirala (IIT Madras) Applied Time-Series Analysis 31