Probability and Stochastic Processes
Chapter 4 Pairs of Random Variables
0-0
A Pair of Random Variables
• An experiment produces both X and Y
• Joint range:
SX,Y = {(x, y)|PX,Y (x, y) > 0} ⊂ SX × SY
1
Joint Cumulative Distribution Function (CDF)
• The joint CDF of random variables X and Y is
FX,Y (x, y) = P [X ≤ x, Y ≤ y]
2
Joint CDF Area
Y
{X<x, Y<y} (x,y)
3
Joint CDF Properties
For any pair of random variable X and Y ,
• 0 ≤ FX,Y (x, y) ≤ 1
• FX (x) = FX,Y (x, ∞)
• FY (y) = FX,Y (∞, y)
• FX,Y (−∞, y) = FX,Y (x, −∞) = 0
• FX,Y (∞, ∞) = 1
• If x ≤ x1 and y ≤ y1 , then
FX,Y (x, y) ≤ FX,Y (x1 , y1 )
4
Proof Example
FX (x) = P [X ≤ x]
= P [X ≤ x, Y < ∞]
= lim FX,Y (x, y) = FX,Y (x, ∞)
y→∞
5
Joint Probability Mass Function (PMF)
• Definition 4.2: The joint probability mass function of X and Y is
PX,Y (x, y) = P [X = x, Y = y]
6
Example
• Flip a coin three times.
• X = no. of H
• Y = no. of T after the first H
• Write all 8 outcomes: {HHH, . . . , T T T }.
• For each outcome, record X and Y
7
Probability of Event {(X, Y ) ∈ B}
• Theorem 4.2: For discrete random variables X and Y and any set B in
the X, Y plane, the probability of the event {(X, Y ) ∈ B} is
P [B] = PX,Y (x, y)
(x,y)∈B
8
Marginal PMF
• Theorem 4.3: For discrete random variables X and Y with joint PMF
PX,Y (x, y),
PX (x) = PX,Y (x, y) , PY (y) = PX,Y (x, y) .
y∈SY x∈SX
• Calculate by summing across rows, write PMF in margin
9
Joint Probability Density Function (PDF)
• Definition 4.3: The joint PDF of the continuous random variables X and
Y is a function fX,Y (x, y) satisfying
x y
FX,Y (x, y) = fX,Y (u, v) dv du
−∞ −∞
∂ 2 FX,Y (x,y)
• Theorem 4.4: fX,Y (x, y) = ∂x ∂y
10
Joint PDF Properties
• fX,Y (x, y) ≥ 0 for all (x, y)
∞ ∞
• −∞ −∞ fX,Y (x, y) dx dy = 1
• The probability that the continuous random variables (X, Y ) are in A
(region of X, Y plane) is
P [A] = fX,Y (x, y) dx dy
A
• Example 4.5 (MATLAB Demo)
11
Marginal PDF
• If X and Y are random variables with joint PDF fX,Y (x, y),
∞
fX (x) = fX,Y (x, y) dy
−∞
∞
fY (y) = fX,Y (x, y) dx
−∞
• Proof: Write FX (x) as an integral, take the derivative with respect to x.
• Example 4.7 (MATLAB Demo)
12
Functions of Two Random Variables
• Theorem 4.9: For discrete random variables X and Y , the derived
random variable W = g(X, Y ) has PMF
PW (w) = P [g(X, Y ) = w] = PX,Y (x, y)
(x,y):g(x,y)=w
• Theorem 4.10: For continuous random variables X and Y , the CDF of
W = g(X, Y ) is
FW (w) = P [W ≤ w] = P [g(X, Y ) ≤ w] = fX,Y (x, y) dx dy
g(x,y)≤w
.
13
W = g(X, Y ): Examples
• Examples:
W1 = X + Y
W2 = max(X, Y )
W3 = XY
W4 = X/Y
• Example 4.10 (MATLAB Demo)
• Quiz 4.6(B) (MATLAB Demo)
14
Expected Values
• Theorem 4.12: For random variables X and Y , the expected value of
W = g(X, Y ) is
– Discrete:
E[W ] = E[g(X, Y )] = g(x, y)PX,Y (x, y)
x∈SX y∈SY
– Continuous:
∞ ∞
E[W ] = E[g(X, Y )] = g(x, y)fX,Y (x, y) dx dy
−∞ −∞
15
Expectations of Functions
• Theorem 4.13:
E[g1 (X, Y ) + g2 (X, Y ) + · · · + gn (X, Y )] = E[g1 (X, Y )]+· · ·+E[gn (X, Y )]
16
Expectations of Sums
For any two random variables X and Y ,
• E[X + Y ] = E[X] + E[Y ]
• Var [X + Y ] = Var [X] + Var [Y ] + 2E[(X − μX )(Y − μY )]
17
Covariance
• Definition 4.4: The covariance of two random variables X and Y is
Cov [X, Y ] = E[(X − μX ) (Y − μY )]
• Covariance is also Cov [X, Y ] = E[XY ] − μX μY
• Cov [X, Y ] > 0 says, X > E[X] implies Y > E[Y ] is likely (X goes
up, Y goes up)
• Var [X + Y ] = Var [X] + Var [Y ] + 2 Cov [X, Y ]
18
Correlation
• Definition 4.5: The correlation of X and Y is rX,Y = E[XY ]
• Cov [X, Y ] = rX,Y − μX μY
• Correlation = Covariance if E[X] = 0 or E[Y ] = 0
• If X = Y, Cov [X, Y ] = Var [X] = Var [Y ] and
2 2
rX,Y = E X = E Y
• E[XY ] > 0 suggests that X > 0 increases chance Y > 0
19
Orthogonality and Uncorrelateness
• Definition 4.6: X and Y are orthogonal if E[XY ] = 0
• Definition 4.7: X and Y are uncorrelated if Cov [X, Y ] = 0
20
Correlation Coefficient
• Definition 4.8: The correlation coefficient of two random variables X
and Y is
Cov [X, Y ]
ρX,Y =
Var [X] Var [Y ]
• Theorem 4.17: −1 ≤ ρX,Y ≤ 1
• Theorem 4.18: If Y = aX + b, then
⎧
⎪
⎨ −1
⎪ a<0
ρX,Y = 0 a=0
⎪
⎪
⎩
1 a>0
• Quiz 4.7(B) (MATLAB Demo)
21
Conditioning by an Event
• Definition 4.9: For discrete random variables X and Y and an event B
with P [B] > 0, the conditional joint PMF of X and Y given B is
1
PX,Y |B (x, y) = P [X = x, Y = y|B] = P [(X = x, Y = y) ∩ B]
P [B]
• Theorem 4.19: For any event B, a region of the X, Y plane with
P [B] > 0,
⎧
⎨ PX,Y (x,y) (x, y) ∈ B
P [B]
PX,Y |B (x, y) =
⎩ 0 otherwise
22
Conditional Joint PDF
• Definition 4.10: Given an event B with P [B] > 0, the conditional joint
PDF of X and Y is
⎧
⎨ fX,Y (x,y) (x, y) ∈ B
P [B]
fX,Y |B (x, y) =
⎩ 0 otherwise
23
Conditional Expected Value
• Theorem 4.20: For random variables X and Y and an event B with
P [B] > 0, the conditional expected value of W = g(X, Y ) given B is
– Discrete:
E[W |B] = E[g(X, Y )|B] = g(x, y)PX,Y |B (x, y)
x∈SX y∈SY
– Continuous:
∞ ∞
E[W |B] = E[g(X, Y )|B] = g(x, y)fX,Y |B (x, y) dx dy
−∞ −∞
24
Conditional Variance
• Definition 4.11: The conditional variance of the random variable
W = g(X, Y ) is
Var [W |B] = E (W − E[W |B])2 |B
• Theorem 4.21: Var [W |B] = E W |B − (E[W |B])2
2
25
Conditioning by a Random Variable
• Definition 4.12: For any event Y = y such that PY (y) > 0, the
conditional PMF of X given Y = y is
P [X = x, Y = y]
PX|Y (x|y) = P [X = x|Y = y] =
P [Y = y]
• Theorem 4.22: For random variables X and Y with joint PMF
PX,Y (x, y), and x and y such that PX (x) > 0 and PY (y) > 0,
PX,Y (x, y) = PX|Y (x|y) PY (y) = PY |X (y|x) PX (x) .
26
Conditional Expected Value
• Theorem 4.23: X and Y are discrete random variables. For any y ∈ SY ,
the conditional expected value of g(X, Y ) given Y = y is
E[g(X, Y )|Y = y] = g(x, y)PX|Y (x|y)
x∈SX
• Special cases
– E[X|Y = y]
– Cond. Variance:
2
Var [X|Y = y] = E (X − E[X|Y = y]) |Y = y
27
Conditional PDF
• Definition 4.13: For y such that fY (y) > 0, the conditional PDF of X
given Y = y is
fX,Y (x, y)
fX|Y (x|y) = .
fY (y)
• Theorem 4.24: fX,Y (x, y) = fY |X (y|x) fX (x) = fX|Y (x|y) fY (y)
• fX|Y (x|y):
fX|Y (x|y) dx = P [x < X ≤ x + dx|y < Y ≤ y + dy]
• Example 4.22 (MATLAB Demo)
28
Conditional Expected Value
• Definition 4.14: For continuous random variables X and Y and any y
such that fY (y) > 0, the conditional expected value of g(X, Y ) given
Y = y is
∞
E[g(X, Y )|Y = y] = g(x, y)fX|Y (x|y) dx.
−∞
• The conditional expected value of X given Y = y is a special case of
Definition 4.14:
∞
E[X|Y = y] = xfX|Y (x|y) dx
−∞
29
Conditional Expected Value
• Definition 4.15: The conditional expected value E[X|Y ] is a function of
random variable Y such that if Y = y then E[X|Y ] = E[X|Y = y].
• Let h(y) = E[X|Y = y]. When Y = y, E[X|Y ] takes on the value
h(y). That is, E[X|Y ] = h(Y ).
• Theorem 4.25: EY [EX|Y [X|Y ]] = EX [X].
• Theorem 4.26:
+∞
EY [EX|Y [g(X, Y )|Y ]] = EX|Y [g(X, Y )|Y = y]fY (y)dy
−∞
= EX,Y [g(X, Y )]
30
Independent Random Variables
• Definition 4.16: X and Y are independent if and only if for all values of
x and y
– Discrete: PX,Y (x, y) = PX (x) PY (y)
– Continuous: fX,Y (x, y) = fX (x) fY (y)
• Theorem 4.22 implies that if X and Y are independent discrete random
variables, then
PX|Y (x|y) = PX (x) , PY |X (y|x) = PY (y) .
• Theorem 4.24 implies that if X and Y are independent continuous
random variables, then
fX|Y (x|y) = fX (x) , fY |X (y|x) = fY (y) .
31
Independence: Example 4.23
• Are X and Y independent?
⎧
⎨ 4xy 0 ≤ x ≤ 1, 0 ≤ y ≤ 1
fX,Y (x, y) =
⎩ 0 otherwise
..................................................................
• The marginal PDFs of X and Y are
⎧
⎨ 2x 0≤x≤1
fX (x) =
⎩ 0 otherwise
⎧
⎨ 2y 0≤y≤1
fY (y) =
⎩ 0 otherwise
• Is fX,Y (x, y) = fX (x) fY (y)? Yes.
32
Independence: Example 4.24
• Are U and V independent?
⎧
⎨ 24uv u ≥ 0, v ≥ 0, u + v ≤ 1
fU,V (u, v) =
⎩ 0 otherwise
33
Example 4.24 (continued)
• Region of nonzero density is triangular and
⎧
⎨ 12u(1 − u)2 0≤u≤1
fU (u) =
⎩ 0 otherwise
⎧
⎨ 12v(1 − v)2 0≤v≤1
fV (v) =
⎩ 0 otherwise
• U and V are not independent!
34
Properties of Independent RVs
• Theorem 4.27: For independent random variables X and Y ,
1. E[g(X)h(Y )] = E[g(X)]E[h(Y )]
2. rX,Y = E[XY ] = E[X]E[Y ]
3. Cov [X, Y ] = 0, ρX,Y = 0
4. Var [X + Y ] = Var [X] + Var [Y ]
5. E[X|Y = y] = E[X] for all y ∈ SY
6. E[Y |X = x] = E[Y ] for all x ∈ SX
35
Bivariate Gaussian Random Variables
• Definition 4.17: X and Y have a bivariate Gaussian PDF with
parameters μ1 , σ1 , μ2 , σ2 , and ρ if
⎡ 2 2 ⎤
x−μ1 2ρ(x−μ1 )(y−μ2 ) y−μ2
⎢ σ1 − σ1 σ2 + σ2 ⎥
exp ⎣− ⎦
2(1 − ρ2 )
fX,Y (x, y) =
2πσ1 σ2 1 − ρ2
where μ1 and μ2 can be any real numbers, σ1 > 0, σ2 > 0, and
−1 ≤ ρ ≤ 1.
36
ρ=0
fX,Y (x, y)
0.35
0.3
0.25
0.2 -3
0.15 -2
0.1 -1
0.05
0
0
-3 1
y
-2
-1
0 2
1
2
x 33
37
ρ = 0.9
fX,Y (x, y)
0.35
0.3
0.25
0.2 -3
0.15 -2
0.1 -1
0.05
0
0 y
-3 1
-2
-1
0 2
x 1
2
33
• Three-dimensional plot of a bivariate Gaussian PDF (MATLAB Demo)
38
ρ = −0.9
fX,Y (x, y)
0.35
0.3
0.25
0.2 -3
0.15 -2
0.1 -1
0.05
0
0 y
-3 1
-2
-1
0 2
x 1
2
33
39
Bivariate Gaussian PDF
• Theorem 4.28: If X and Y are the bivariate Gaussian random variables
in Definition 4.17, X is the Gaussian (μ1 , σ1 ) random variable and Y is
the Gaussian (μ2 , σ2 ) random variable:
1 2 2
fX (x) = 2
exp [−(x − μ 1 ) /2σ1 ]
2πσ1
and
1 2 2
fY (y) = 2
exp [−(y − μ 2 ) /2σ2 ].
2πσ2
40
Rewriting the Bivariate Gaussian PDF
Complete the square of the exponent to write
fX,Y (x, y) = fX (x) fY |X (y|x)
where
1 −(x−μ1 )2 /2σ12
fX (x) = √ e
σ1 2π
1 −(y−μ̃2 (x))2 /2σ̃22
fY |X (y|x) = √ e
σ̃2 2π
σ2
μ̃2 (x) = μ2 + ρ (x − μ1 ) = E[Y |X = x]
σ1
σ̃2 = σ2 1 − ρ = Var [Y |X = x]
2
41
Conditional PDF of the Bivariate Gaussian RV
• Theorem 4.29: If X and Y are the bivariate Gaussian random variables
in Definition 4.17, the conditional PDF of Y given X is
1 2 2
fY |X (y|x) = √ e−(y−μ̃2 (x)) /2σ̃2
σ̃2 2π
where
σ2
μ̃2 (x) = μ2 + ρ (x − μ1 ) = E[Y |X = x]
σ1
2
σ̃2 = σ2 1 − ρ = Var [Y |X = x]
42
Gaussian PDF
43
Bivariate Gaussian Properties
• E[X] = μ1 , Var [X] = σ12
• Given X = x, Y is Gaussian
• Conditional mean of Y given X = x:
σ2
μ̃2 (x) = E[Y |X = x] = μ2 + ρ (x − μ1 )
σ1
σ̃2 = Var [Y |X = x] = σ2 1 − ρ2
44
Bivariate Gaussian Properties
• Theorem 4.31: Bivariate Gaussian random variables X and Y in
Definition 4.17 have correlation coefficient ρX,Y = ρ.
• Theorem 4.32: Bivariate Gaussian random variables X and Y are
uncorrelated if and only if they are independent.
• Bivariate Normal Experiment (Virtual Laboratories)
45
Virtual Laboratories
• Bivariate Normal Experiment
• Bivariate Uniform Experiment
46
Chapter Summary
• Joint CDF FX,Y (x, y), joint PMF PX,Y (x, y), and joint PDF
fX,Y (x, y)
• Marginal PMF and marginal PDF
• Expected value E[g(X, Y )] and covariance Cov [X, Y ]
• Conditional joint PMF PX,Y |A (x, y) and conditional PMF PX|Y (x|y)
as well as conditional joint PDF fX,Y |A (x, y) and conditional PDF
fX|Y (x|y)
• Independent random variables
• Bivariate Gaussian random variables
47
