Bivariate Distributions

Aria Nosratinia Probability and Statistics 5-1

Why Bivariate Distributions?
There are times we must deal with two random quantities.
Furthermore, it makes more sense to consider them together
rather than separately. .
EXAMPLE: radio signal transmitted and corresponding radio
signal received
Joint analysis of random variables gives rise to new and interesting
concepts that we will explore.
We start with discrete probabilities, then consider continuous
probabilities.
Aria Nosratinia Probability and Statistics 5-2
Joint PMF
For a discrete bivariate distribution, we have a joint PMF.
Example:
Simple bivariate distribution with two prob-
abilities:
P
X,Y
(x, y) =
_

_
1/2 x = 1, y = 0
1/2 x = 0, y = 1
0 otherwise
1/2
1/2
x
y
f(x,y)
Denition: The joint probability mass function of discrete random
variables Y and Y is:
P
X,Y
(x, y) = P[X = x, Y = y]
Aria Nosratinia Probability and Statistics 5-3
Joint CDF
Denition: The joint CDF of random variables X and Y is:
F
X,Y
(x, y) = P(X x , Y y)
The CDF at (x,y): probability of the innite quarter-plane whose
corner is at (x,y):
CDF calculated
at this point...
...includes proabilities
of all these points
Y
X
Aria Nosratinia Probability and Statistics 5-4
Example
Question: Draw the joint CDF of the r.v. taking equal probabilities
on (0, 1) and (1, 0).
Aria Nosratinia Probability and Statistics 5-5
Example
Question: Draw the joint CDF of the r.v. taking equal probabilities
on (0, 1) and (1, 0).
x
F(x,y)
Aria Nosratinia Probability and Statistics 5-5
Properties of CDF
0 F
X,Y
(x, y) 1 and it is an increasing function, i.e.,
x
2
x
1
, y
2
y
1
F
X,Y
(x
2
, y
2
) F
X,Y
(x
1
, y
1
)

F
X,Y
(, y) = F
X,Y
(x, ) = 0
F
X,Y
(, ) = 1

F
X,Y
(x, ) = F
X
(x)
F
X,Y
(, y) = F
Y
(y)
Aria Nosratinia Probability and Statistics 5-6
Probability of a Rectangle
We can show the probability of a rectangle is:
P(x
1
< X x
2
, y
1
< Y y
2
) = F
X,Y
(x
2
, y
2
)F
X,Y
(x
1
, y
2
) F
X,Y
(x
2
, y
1
)
+F
X,Y
(x
1
, y
1
)
x
1
x
2
y
1
y
2
Note that we used the CDF at four corners.
Aria Nosratinia Probability and Statistics 5-7
Events in Two Dimensions
Recall that for one random variable, events could be considered as a
union of intervals. For bivariate distributions, it is not that easy.
Aria Nosratinia Probability and Statistics 5-8
Event Probabilities
For bivariate discrete random variable (X, Y ), the probability of the
event {(X, Y ) B} is:
P(B) =

(x,y)B
P
X,Y
(x, y)
So we just add up the probability atoms inside the region B.
Example: Consider random pair X, Y taking integer values that
satisfy all the following: 1 X 8, 1 Y 8 and X +Y 10. All
(x, y) pairs in this region have equal probability. Find the probability
of three events X +Y = 5, X = Y , and X
2
+Y
2
3.
Aria Nosratinia Probability and Statistics 5-9
Marginal PMF
Question: In a bivariate distribution, what is the behavior of one
variable if we want to ignore the other?
Denition: Marginal distributions:
P
X
(x) =

y
P
X,Y
(x, y) P
Y
(y) =

x
P
X,Y
(x, y)
Example 1: What are the marginal distributions for the simple
two-valued distribution seen earlier?
Aria Nosratinia Probability and Statistics 5-10
Example 2
What are the marginal PMFs for the following distribution:
P(x,y) y=1 y=10 y=100
x=-3 0 0.02 0.05
x=0 0.02 0.05 0.1
x=5 0.7 0.06 0
Aria Nosratinia Probability and Statistics 5-11
Joint PDF
We now move on to continuous probabilities.
The PDF is the derivative of CDF, i.e.,
f
X,Y
(x, y) =

2
F
X,Y
(x, y)
x y
F
X,Y
(x, y) =
_
x

_
y

f
X,Y
(, ) d d
Aria Nosratinia Probability and Statistics 5-12
Event Probabilities
The probability of any event is calculated:
P(B) =
__
B
f
X,Y
(x, y) dx dy
Therefore:
_

f
X,Y
(x, y) dxdy = 1
Aria Nosratinia Probability and Statistics 5-13
Example
Consider the continuous bivariate uniformly distributed over
[0, 3] [0, 2]. What is the pdf? Find the probability of the event
X +Y 1.
Aria Nosratinia Probability and Statistics 5-14
Example
Consider the continuous bivariate uniformly distributed over
[0, 3] [0, 2]. What is the pdf? Find the probability of the event
X +Y 1.
y+x=1
x
y
f
X,Y
(x, y) =
_
_
_
1
6
(x, y) [0, 3] [0, 2]
0 else
P(X +Y 1) =
0.5
6
=
1
12
Aria Nosratinia Probability and Statistics 5-14
Example
Find the joint CDF for the following pdf:
f
X,Y
(x, y) =
_
_
_
2 0 y x < 1
0 else
x
y
We have to nd P(X x , Y y) for all (x, y).
Aria Nosratinia Probability and Statistics 5-15
Case 1: x or y negative
X
Y
(X<x , Y<y)
x < 0 or y < 0 F
X,Y
(x, y) = 0
Aria Nosratinia Probability and Statistics 5-16
Case 2: 0 < y < x < 1
X
Y
0 < y < x < 1 F
X,Y
(x, y) = 2xy y
2
Aria Nosratinia Probability and Statistics 5-17
Case 3: 0 < x < 1, y > x
X
Y
0 < x < 1 , y > x F
X,Y
(x, y) = x
2
Aria Nosratinia Probability and Statistics 5-18
Case 4: 0 < y < 1 < x
X
Y
0 < y < x < 1 F
X,Y
(x, y) = 2y y
2
Aria Nosratinia Probability and Statistics 5-19
Case 5: x, y > 1
X
Y
0 < y < x < 1 F
X,Y
(x, y) = 1
Aria Nosratinia Probability and Statistics 5-20
Overall Answer
0.5
0
0.5
1
1.5
2
0.5
0
0.5
1
1.5
2
0
0.2
0.4
0.6
0.8
1
F
X,Y
(x, y) =

0 x or y negative
2xy y
2
0 y x 1
x
2
x y , 0 x 1
2y y
2
0 y 1 , x > 1
1 otherwise
Aria Nosratinia Probability and Statistics 5-21
Marginal PDF
f
X
(x) =
_

f
X,Y
(x, y) dy
f
Y
(y) =
_

f
X,Y
(x, y) dx
IDEA: To get the marginal, sum over unwanted variable.
PROOF: Use the CDF.
F
X
(x
0
) = P(X x
0
) = P(X x
0
, < y < )
=

x
0

f
X,Y
(x, y)dydx
Aria Nosratinia Probability and Statistics 5-22
Example
Find the marginals of a bivariate uniformly distributed over a unit
circle.
Aria Nosratinia Probability and Statistics 5-23
Example
Find the marginals of a bivariate uniformly distributed over a unit
circle.
Solution:
1-x
2
x
f
X
(x) =
_

f
X,Y
(x, y)dy
=
_

1x
2

1x
2
1

dy
=
2

1 x
2

Similarly
f
Y
(y) =
2
_
1 y
2

Aria Nosratinia Probability and Statistics 5-23

Functions of Two Random Variables
Often we need to calculate a distribution that depends on two r.v.
Examples:
Observation of signal X that is contaminated by noise N
Y = X +N
Choosing the maximum of several (random) quantities
X = max(X
1
, X
2
)
This type of problem leads to an area known as order statistics
We will rst see a few specic cases, then learn a general way to
deal with all such problems.
Aria Nosratinia Probability and Statistics 5-24
Functions of Discrete Bivariate Distributions
If (X, Y ) are discrete and W = g(X, Y ), then:
P
W
(w) =

g(x,y)=w
P
X,Y
(x, y)
Example 1: Consider a uniform distribution over four possibilities
x, y = 0, 1 and nd the distribution of W = XY .
Example 2: Consider P
XY
(x, y) =
1
2
x
2
y
and nd the distribution of
W = X +Y .
Aria Nosratinia Probability and Statistics 5-25
Functions of Continuous Bivariate Distributions
If W = g(X, Y ), then
F
W
(w) = P(W w)
= P(g(X, Y ) w)
=
__
g(x,y)w
f
X,Y
(x, y) dxdy
We then dierentiate to get f
W
(w).
Aria Nosratinia Probability and Statistics 5-26
Examples
Example 1: g(X, Y ) = max(X, Y ),
F
W
(w) =
_
w

_
w

f
X,Y
(x, y) dxdy
Example 2: g(X, Y ) = X +Y
F
W
(w) =
_

_ _
wy

f
X,Y
(x, y) dx
_
dy
Aria Nosratinia Probability and Statistics 5-27
General Method
We now present a powerful method that solves the problem
without integration.
Consider random variables (X, Y ) and the following functions.
v = g
1
(x, y) w = g
2
(x, y)
which we can also write compactly as a vector function:
(v, w) = g(x, y)
What is the distribution of (V, W)?
First consider the case where g() is one-to-one.
Aria Nosratinia Probability and Statistics 5-28
Change of Variable
Same event in two coordinate systems must have same probability.
f
V W
(v, w) A
1
= f
XY
(x, y) A
2
where A
1
= dwdv. Area of parallelogram A
2
is given by cross-product
A
2
= |V
2
V
1
| =

x
v
y
w
dvdw
x
w
y
v
dvdw

v v+dv
w+dw
w
v=const
w=const
A
1
A
2
x
y
w+dw=const
v+dv=const
Aria Nosratinia Probability and Statistics 5-29
Inverting the dierentiation, we have:
f
V W
(v, w) =
f
XY
(x, y)
|J(x, y)|
where |J| is the absolute value Jacobian determinant:
J =
_
_
v
x
v
y
w
x
w
y
_
_
Note: it is often easier to use the Jacobian approach than to
custom-solve the problem via CDF.
If the function is not one-to-one, make following adjustment:
f
V W
(v, w) =

g(x
i
,y
i
)=(v,w)
f
XY
(x
i
, y
i
)
|J(x
i
, y
i
)|
Aria Nosratinia Probability and Statistics 5-30
Example
Consider a bivariate distribution (X, Y ) and functions:
V = X +Y
W = X Y
Using the method explained above:
f
V W
(v, w) =
f
XY
(x, y)
2
=
1
2
f
XY
(
v +w
2
,
v w
2
)
Aria Nosratinia Probability and Statistics 5-31
Expected Values
For W = g(X, Y ), if (X, Y ) are discrete:
E[W] =

y
g(x, y) P
XY
(x, y)
and if they are continuous:
E[W] =
_

g(x, y) f
XY
(x, y) dxdy
This tells us:
E[X +Y ] = E[X] +E[Y ]
Aria Nosratinia Probability and Statistics 5-32
Covariance, Correlation
Denition: Correlation of two random variables
r
XY
= E[XY ]
Denition: Covariance of two random variables:
Cov(X, Y ) = E
_
(X
X
)(Y
Y
)

Denition: Correlation coecient

=
Cov(X, Y )
_
V ar(X) V ar(Y )
Aria Nosratinia Probability and Statistics 5-33
More on Correlation and Covariance
V ar(X +Y ) = V ar(X) +V ar(Y ) + 2Cov(X, Y )
Cov(X, Y ) = r
XY

X

Y
1 1
If X = Y , then = 1.
Denition: If Cov(X, Y ) = 0 (equivalently, = 0) we say the
random variables are Uncorrelated.
Denition: If r
XY
= 0, we say the random variables are Orthogonal.
For zero-mean random variables, uncorrelated orthogonal.
Aria Nosratinia Probability and Statistics 5-34
Example
Consider the following bivariate distribution:
P
XY
X = 0 X = 1
Y = 0 0.1 0.25
Y = 1 0.6 0.05
1. Find the covariance and correlation coecient.
2. Are X, Y uncorrelated? Orthogonal?
Aria Nosratinia Probability and Statistics 5-35
Conditioning by an Event
P
XY |B
(x, y) =
_
_
_
P
XY
(x,y)
P(B)
(x, y) B
0 otherwise
f
XY |B
(x, y) =
_
_
_
f
XY
(x,y)
P(B)
(x, y) B
0 otherwise
Aria Nosratinia Probability and Statistics 5-36
Example
Consider the following bivariate distribution:
P
XY
X = 0 X = 1
Y = 0 0.1 0.25
Y = 1 0.6 0.05
Find the distribution conditioned on the event B = {X +Y = 1}.
Aria Nosratinia Probability and Statistics 5-37
Conditional Mean and Variance
Assuming W = g(X, Y )
E[W|B] =

y
g(x, y)P
XY |B
(x, y)
or
E[X|B] =
_

g(x, y) f
XY |B
(x, y)dxdy
V ar(W|B) = E
_
(W
W|B
)
2

B
_
= E[W
2
|B]
2
W|B
Aria Nosratinia Probability and Statistics 5-38
Conditioning by a Random Variable
Denition:
P
X|Y
(x|y) =
P
XY
(x, y)
P
Y
(y)
f
X|Y
(x|y) =
f
XY
(x, y)
f
Y
(y)
F
X|Y
(x|y) =
F
XY
(x, y)
F
Y
(y)
The denitions are symmetric with respect to x and y, therefore:
f
XY
(x, y) = f
X
(x)f
Y |X
(y|x)
= f
Y
(y)f
X|Y
(x|y)
The same is true for P
XY
and F
XY
.
Aria Nosratinia Probability and Statistics 5-39
Example
Consider the following bivariate distribution:
P
XY
X = 0 X = 1
Y = 0 0.1 0.25
Y = 1 0.6 0.05
Find the conditional distribution P
Y |X
(y|x).
Aria Nosratinia Probability and Statistics 5-40
Conditional Expectation
Denition:
E[g(X, Y )|Y = y
0
] =
_
g(x, y
0
)f
X|Y
(x|y
0
) dx
E[X|Y = y
0
] =
_
x f
X|Y
(x|y
0
) dx
Clearly the result depends on y
0
. Therefore, E[X|Y ] = (Y ) is a
function of random variable Y .
Since E[X|Y ] is random, we can take its average. The result is known
as the smoothing property of expectation.
Theorem:
E
_
E[X|Y ]
_
= E[X]
Aria Nosratinia Probability and Statistics 5-41
Independence
Two random variables are independent if
P
XY
(x, y) = P
X
(x) P
Y
(y)
for continuous distributions
f
XY
(x, y) = f
X
(x) f
Y
(y)
Note 1: When two variable are independent, the conditional
distribution is the same as unconditional distribution, e.g.,
f
X|Y
(x|y) = f
X
(x).
Note 2: Both conditions above are equivalent to:
F
XY
(x, y) = F
X
(x) F
Y
(y)
Aria Nosratinia Probability and Statistics 5-42
Example
Consider the following bivariate distribution:
P
XY
X = 0 X = 1
Y = 0 0.1 0.25
Y = 1 0.6 0.05
Are X and Y independent? Why?
Aria Nosratinia Probability and Statistics 5-43
Consequences of Independence
If two random variables X and Y are independent:
E
_
g(X) h(Y )
_
=E[g(X)] E[h(Y )] g(), h()
E[XY ] = E[X]E[Y ]
Cov(X, Y ) = 0

XY
= 0
V ar(X +Y ) = V ar(X) +V ar(Y )
E[X|Y ] = E[X]
E[Y |X] = E[Y ]
Aria Nosratinia Probability and Statistics 5-44
Bivariate Gaussian Distribution
Consider the two random variables X, Y , which we now show in a
vector Z = [X Y ]
t
. Then the bivariate Gaussian PDF has the form:
f
XY
(x, y) =
1
2||
1
2
exp
_

1
2
(Z )
t

1
(Z )
_
Where we have:
=
_
_

Y
_
_
=
_
_

2
X

X

Y

2
Y
_
_
Equivalently
f
XY
(x, y) =
1
2
X

Y
_
1
2
exp
_

_
x
X

X
_
2

2(x
X
)(y
Y
)

Y
+
_
y
Y

Y
_
2
2(1
2
)
_
Aria Nosratinia Probability and Statistics 5-45
2D Gaussian Examples
5
0
5
5
0
5
0
0.05
0.1
0.15
0.2
Small variance
5
0
5
5
0
5
0
0.05
0.1
0.15
0.2
Large variance
Aria Nosratinia Probability and Statistics 5-46
5
0
5
5
0
5
0
0.02
0.04
0.06
0.08
0.1
Correlated
Aria Nosratinia Probability and Statistics 5-47
More about the Gaussian
The marginals of a (jointly) Gaussian r.v. are Gaussian:
f
X
(x) =
1

2
X
e

(x
X
)
2
2
2
X
f
Y
(y) =
1

2
Y
e

(y
Y
)
2
2
2
Y
The conditional Gaussian is also a Gaussian:
f
Y |X
(y|x) =
1

2
2
exp
_

(y )
2
2
2
_
where
=
Y
+

X
(x
X
) =
Y
_
1
2
NOTE: the conditional average of Y depends on x, but the
conditional variance does not.
Sum of two Gaussian random variables is also a Gaussian.
If two Gaussian r.v. are uncorrelated, they are independent.
Aria Nosratinia Probability and Statistics 5-48