Introduction to Bayesian Statistics
Richard Yi Da Xu
School of Computing & Communication, UTS
January 31, 2017
Richard Yi Da Xu Introduction to Bayesian Statistics
Random variables
I Pre-university: A number is just a fixed value.
When we talk about probabilities:
I When X is a continuous random variable, it has a probability density function (pdf)
I When X is a discrete random variable, it has a probability mass function (pmf)
p(x) = p(X = x) means that:
The probability when a random variable X is equal to a fixed number x, i.e.,
the probablity that number of machine learning participants = 20
Richard Yi Da Xu Introduction to Bayesian Statistics
Mean or Expectation
I discrete case:
N
1 X
µ = E(X ) = xi
N
i=1
I continous case:
Z
µ = E(X ) = xp(x)dx
x∈S
I can also measure the expecation of a function:
Z
E(f (X )) = f (x)p(x)dx
x∈S
For example,
Z Z
E(cos(X )) = cos(x)p(x)dx E(X 2 ) = x 2 p(x)dx
x∈S x∈S
I What about f (E(X )): Discuss later when we discuss Jensens Equality in
Expecation-Maximization
Richard Yi Da Xu Introduction to Bayesian Statistics
Variances an intuitive explanation
I You have data X = {2, 3, 3, 2, 1, 4}, i.e., x1 = 2, x2 = 3, . . . x6 = 4
I You have the mean:
2+3+3+2+1+4
µ= = 2.5
6
I The variance is then:
N
2 1 X 2
VAR(data) = σ = (xi − µ)
N i=1
I Division by N is intuitive. Otherwise, more data implies more variance
I Also think about what kind of values can VAR and σ take? - we will look at what kind of
distribuiton is required for them.
Richard Yi Da Xu Introduction to Bayesian Statistics
Two alternative expression:
People sometimes use: Other times, people use:
I You have data X = {2, 3, 3, 2, 1, 4}, I You have data X = {1, 2, 3, 4}, and
i.e., x1 = 2, x2 = 3, . . . x6 = 4 P(X = 1) = 62 , P(X = 2) = 26 , P(X = 3) = 1
6
and P(X = 4) = 16 .
N
2 1 X 2
VAR(X ) = σ = (xi − µ) 2
X 2
N i=1 Discrete :VAR(X ) = σ = (x − µ) p(x)
x∈X
N N N
1 X 2 1 X 1 X 2 Z
= xi − 2xi µ + µ Continous :VAR(X ) = σ =
2
(x − µ) p(x)
2
N i=1 N i=1 N i=1
x∈X
N N
1 X 2 1 X 2
= x − 2µ xi +µ
N i=1 i N i=1 X
2 2
VAR(X ) = x − 2µx + µ p(x)
| {z } x∈X
µ
2 2
X X X
N
! = x p(x) − 2 µxp(x) + µ p(x)
1 X 2 2
= x −µ x∈X x∈X x∈X
N i=1 i X 2
X 2
X
= x p(x) − 2µ xp(x) +µ p(x)
x∈X x∈X x∈X
| {z } | {z }
µ 1
2 2
X
= x p(x) − µ
x∈X
It’s easy to verify that both sides are the same
Richard Yi Da Xu Introduction to Bayesian Statistics
Numerical example
First version Second version
I X = {2, 3, 3, 2, 1, 4}, i.e., I X = {1, 2, 3, 4}, and P(X = 1) = 62 ,
x1 = 2, x2 = 3, . . . x6 = 4 P(X = 2) = 26 , P(X = 3) = 16 and
P(X = 4) = 16 .
N
!
1 X 2 2
VAR(X ) = x −µ
N i=1 i 2 2
X
Discrete :VAR(X ) = σ = (x − µ) p(x)
x∈X
1 2 2 2
= (2 − 2.5) + (3 − 2.5) + (3 − 2.5) 2
Z
2
6 Continous :VAR(X ) = σ = (x − µ) p(x)
2 2 2 x∈X
+ (2 − 2.5) + (1 − 2.5) + (4 − 2.5)
| {z }
f (x)
≈ 0.917
2 2
X
VAR(X ) = x p(x) − µ
x∈X
1 2 22 22
(1 − 2.5) + (2 − 2.5) + (3 − 2.5) +
6 6 6
21
(4 − 2.5)
6
≈ 0.917
Both sides are the same
Richard Yi Da Xu Introduction to Bayesian Statistics
Important fact of the Variances
Z
VAR(X ) = E[(X − E(X ))2 ] = (x − µ)2 p(x)dx
x∈S
Z Z Z
= x 2 p(x)dx − 2µ xp(x)dx + µ2 xp(x)dx
x∈S x∈S x∈S
= E(X ) − (E(X))2
2
Think VAR(X ) as “mean-subtracted” second order moment of random variable X .
Richard Yi Da Xu Introduction to Bayesian Statistics
Joint distributions
I The following is a tablet form of joint density Pr(X , Y ):
Y =0 Y =1 Y =2 Total
3 3 6
X =0 0 15 15 15
2 6 8
X =1 15 15
0 15
1 1
X =2 15
0 0 15
3 9 3
Total 15 15 15
1
I This table shows Pr(X , Y ) or Pr(X = x, Y = y).
6
I For example, p(X = 1, Y = 1) = 15
:
I exercise what is the probablity that X = 2, Y = 1?
I exercise what is the probablity that X = 3, Y = 2?
I exercise what is the value of:
2 X
X 2
Pr(X = i, Y = j)?
i=0 j=0
Richard Yi Da Xu Introduction to Bayesian Statistics
Marginal distributions
Y =0 Y =1 Y =2 Total
3 3 6
X =0 0 15 15 15
2 6 8
X =1 15 15
0 15
1 1
X =2 15
0 0 15
3 9 3
Total 15 15 15
1
I Using sum rule, the marginal distribution tells us that:
X Z
Pr(X ) = Pr(x, y) or p(X ) = p(x, y)dy
y ∈Sy y∈Sy
I For example:
2 X
2
X 3 6 0 9
Pr(Y = 1) = p(x = i, y = 1) = + + =
15 15 15 15
i=0 j=0
I exercise what is Pr(X = 2) and Pr(X = 1)?
Richard Yi Da Xu Introduction to Bayesian Statistics
Conditional distributions
Y =0 Y =1 Y =2 Total
3 3 6
X =0 0 15 15 15
2 6 8
X =1 15 15
0 15
1 1
X =2 15
0 0 15
3 9 3
Total 15 15 15
1
I Conditional density:
p(X , Y ) p(Y |X )p(X ) p(Y |X )p(X )
p(X |Y ) = = = P
p(Y ) p(Y ) X p(Y |X )p(X )
I What about p(X |Y = y)? Pick an example:
p(X = 1, Y = 1) 6/15 2
p(X = 1|Y = 1) = = =
p(Y = 1) 9/15 3
Richard Yi Da Xu Introduction to Bayesian Statistics
Conditional distributions: Exercise
Y =0 Y =1 Y =2 Total
3 3 6
X =0 0 15 15 15
2 6 8
X =1 15 15
0 15
1 1
X =2 15
0 0 15
3 9 3
Total 15 15 15
1
I The formulation for conditional density:
p(X , Y ) p(Y |X )p(X ) p(Y |X )p(X )
p(X |Y ) = = = P
p(Y ) p(Y ) X p(Y |X )p(X )
I exercise: What is p(X = 2|Y = 1)?
I exercise: What is p(X = 1|Y = 2)?
Richard Yi Da Xu Introduction to Bayesian Statistics
Independence
If X and Y are independent:
I p(X |Y ) = p(X )
I p(X , Y ) = P(X )P(Y )
I Both factors are related when A and B are independent:
p(X , Y ) p(X )p(Y )
p(X |Y ) = = = p(X )
p(Y ) p(Y )
Y =0 Y =1 Y =2 Total Y =0 Y =1 Y =2 Total
3 3 6 18 54 18 6
X =0 0 15 15 15 X =0 225 225 225 15
2 6 8 24 72 24 8
X =1 15 15 0 15 X =1 225 225 225 15
1 1 3 9 3 1
X =2 15 0 0 15 X =2 225 225 225 15
3 9 3 3 9 3
Total 15 15 15 1 Total 15 15 15 1
X and Y are NOT independent X and Y are independent
Richard Yi Da Xu Introduction to Bayesian Statistics
Conditional Independence
I Imagine we have three random variables: X , Y and Z :
I Once we know Z , then knowing Y does NOT tell us any additional information
about X
I Therefore:
Pr(X |Y , Z ) = Pr(X |Z )
I This means that X is conditionally independent of Y given Z .
I If Pr(X |Y , Z ) = Pr(X |Z ), then what about Pr(X , Y |Z )?
Pr(X , Y , Z ) Pr(X |Y , Z ) Pr(Y , Z )
Pr(X , Y |Z ) = =
Pr(Z ) Pr(Z )
= Pr(X |Y , Z ) Pr(Y |Z )
= Pr(X |Z ) Pr(Y |Z )
Richard Yi Da Xu Introduction to Bayesian Statistics
An example of Conditional Independence
We will study Dynamic model later.
xt−1 xt xt+1
yt−1 yt yt+1
From this model, we can see:
p(xt |x1 , . . . , xt−1 , y1 , . . . , yt−1 ) = p(xt |xt−1 )
p(yt |x1 , . . . , xt−1 , xt , y1 , . . . , yt−1 ) = p(yt |xt )
Right now, think of if a given variable is the only item that “blocks” the path between two
(or more) variables.
Richard Yi Da Xu Introduction to Bayesian Statistics
Another Example: Bayesian Linear Regression
We have data pairs:
I Input: X = x1 , . . . xN
I Output: Y = y1 , . . . yN
Each pair, xi and yi are related through model equation:
yi = f (xi |w) + N (0, σ 2 )
I Input alone isn’t going to tell you model parameter: p(w|X ) = p(w)
I Output alone isn’t going to tell you model parameter: p(w|Y ) = p(w)
I Obviously: p(w|X , Y ) 6= p(w)
Posterior over parameter w:
p(y|w, x)p(w|x)p(x) p(y|w, x)p(w) p(y |w, x)p(w)
p(w|x, y) = = = R
p(y|x)p(x) p(y|x) w p(y|x, w)p(w)
Richard Yi Da Xu Introduction to Bayesian Statistics
Expectation of Joint probabilities
Given that X , Y is a two-dimensional random variable:
I Continous case:
Z Z
E[f (X , Y )] = f (x, y)p(x, y)dxdy
y ∈Sy x∈Sx
I Discrete case:
Ni jN
X X
E[f (X , Y )] = f (X = i, Y = j)p(X = i, Y = j)
i=1 j=1
Richard Yi Da Xu Introduction to Bayesian Statistics
Numerical Example:
Y =1 Y =2 Y =3 Y =1 Y =2 Y =3
3 3 X =1 6 7 8
X =1 0 15 15
X =2 2 6
0 X =2 3 6 2
15 15 X =3 1 8 6
1
X =3 15 0 0
f (X,Y)
p (X,Y)
Ni jN
X X
E[f (X , Y )] = f (X = i, Y = j)p(X = i, Y = j)
i=1 j=1
3 3 2 6
=6×0+7× +8× +3× +6×
15 15 15 15
1
+2×0+1× +8×0+6×0
15
Richard Yi Da Xu Introduction to Bayesian Statistics
Conditional Expectation
It’s a useful property for later
Z
E(Y ) = E(Y |X )p(X )dx
ZX Z Z Z
= yp(Y |X )dy p(X )dx = yp(Y , X )dy dx
X Y X Y
| {z }
Z Z
= y p(Y , X )dx dy
Y X
Z
= yp(Y )dy = E(Y )
Y
Richard Yi Da Xu Introduction to Bayesian Statistics
Bayesian Predictive distribution
Put marginal distribution and Conditional Independence into a test:
I Very often, in machine learning, you want to compute the probability of new data
y ∗ given training data Y , i.e., p(y ∗ |Y ). You assume there are some model
explains both Y and y ∗ . The model parameter is θ:
Z
p(y ∗ |Y ) = p(y ∗ |θ)p(θ|Y )dθ
θ
I Excercise, tell me why the above works?
Richard Yi Da Xu Introduction to Bayesian Statistics
Revisit Bayes Theorem
Instead of using arbitrary random variable symbols, we now use:
I θ for model parameter
I and X = x1 , . . . xn for dataset:
p(X |θ) p(θ)
| {z } |{z}
likelihood prior p(X |θ)p(θ)
p(θ|X ) = = R
| {z } p(X ) θ p(X |θ)p(θ)
posterior | {z }
normalization constant
Richard Yi Da Xu Introduction to Bayesian Statistics
An Intrusion Detection System (IDS) Example
The setting: Imagine out of all the TCP connections (say millons), 1% of which are
intrusions:
I When there is an intrusion, the probability of system sends alarm is 87%.
I When there is no intrusion, the probability of system sends alarm is 6%.
I Prior probability:
1% of which are intrusions
=⇒ p(θ = intrusion) = 0.01 p(θ = no intrusion) = 0.99
I Likelihood probability:
I given intrusion occur, probability of system sends alarm is 87%
p(X = alarm|θ = intrusion) = 0.87 p(X = no alarm|θ = intrusion) = 0.13
I given there is no intrusion, the probability of system sends alarm is 6%:
p(X = alarm|θ = no intrusion) = 0.06 p(X = no alarm|θ = no intrusion) = 0.94
Richard Yi Da Xu Introduction to Bayesian Statistics
Posterior
I We are interested in posterior probability: Pr(θ|X ):
I There 2 two possible values for parameter θ and 2 possible observation X
I Therefore, there are 4 rates we need to compute:
I True Positive When system sends alarm, probability of an intrusion occurs:
Pr(θ = intrusion|X = alarm)
I False Positive When system sends alarm, probability that there is no intrusion:
Pr(θ = no intrusion|X = alarm)
I True Negative When system sends no alarm, probability that there is no intrusion:
Pr(θ = no intrusion|X = no alarm)
I False Negative When system sends no alarm, probability that an intrusion occurs:
Pr(θ = intrusion|X = no alarm)
I Question which are the two probabilities you’d like to maximise?
Richard Yi Da Xu Introduction to Bayesian Statistics
Apply Bayes Theorem in this setting
Pr(X |θ) Pr(θ)
Pr(θ|X ) = P
θ Pr(X |θ) Pr(θ)
Pr(X |θ) Pr(θ)
=
Pr(X |θ = Intrusion) Pr(θ = Intrusion) + Pr(X |θ = no intrusion) Pr(θ = no intrusion)
Richard Yi Da Xu Introduction to Bayesian Statistics
Apply Bayes Theorem in this setting
True Positive rate When system sends alarm, what is the probability of an intrusion
occurs:
Pr(θ = intrusion|X = alarm)
Pr(X = alarm|θ = intrusion) Pr(θ = intrusion)
=
Pr(X = alarm|θ = Intrusion) Pr(θ = Intrusion) + Pr(X = alarm|θ = no intrusion) Pr(θ = Intrusion)
0.87 × 0.01
= = 0.1278
0.87 × 0.01 + 0.06 × 0.99
False Positive rate When system sends alarm, what is the probability that there is no
intrusion:
Pr(θ = no intrusion|X = alarm)
Pr(X = alarm|θ = no intrusion) Pr(θ = no intrusion)
=
Pr(X = alarm|θ = no intrusion) Pr(θ = no intrusion) + Pr(X = alarm|θ = no intrusion) Pr(θ = no intrusion)
0.06 × 0.99
= = 0.8722
0.87 × 0.01 + 0.06 × 0.99
Richard Yi Da Xu Introduction to Bayesian Statistics
Apply Bayes Theorem in this setting
False Negative When system sends no alarm, what is the probability that an intrusion
occurs?
Pr(θ = intrusion|X = no alarm)
Pr(X = no alarm|θ = intrusion)p(θ = intrusion)
=
Pr(X = no alarm|θ = Intrusion) Pr(θ = Intrusion) + Pr(X = no alarm|θ = no intrusion) Pr(θ = no Intrusion)
0.13 × 0.01
= = 0.0014
0.13 × 0.01 + 0.94 × 0.99
True Negative When system sends no alarm, what is the probability that there is no
intrusion?
Pr(θ = no intrusion|X = no alarm)
Pr(X = no alarm|θ = no intrusion) Pr(θ = no intrusion)
=
Pr(X = no alarm|θ = no intrusion) Pr(θ = no intrusion) + Pr(X = no alarm|θ = no intrusion) Pr(θ = no intrusion)
0.94 × 0.99
= = 0.9986
0.87 × 0.001 + 0.06 × 0.99
Richard Yi Da Xu Introduction to Bayesian Statistics
Statistics way to think about Posterior Inference
The posterior inference is to find the best q(θ) to approximate p(θ|X ), such that:
infq(θ)∈Q KL(q(θ)kp(θ)) − Eθ∼q(θ) ln(p(X |θ)
Z
q(θ)
Z
=infq(θ)∈Q ln q(θ) − ln(p(X |θ)q(θ)
θ p(θ) θ
Z
=infq(θ)∈Q [ln q(θ) − (ln p(θ) + ln p(X |θ))] q(θ)
θ
Z
q(θ)
=infq(θ)∈Q ln q(θ)
θ p(θ)p(X |θ)
Z
1 q(θ)
= infq(θ)∈Q ln q(θ)
p(X ) θ p(θ|X )
=infq(θ)∈Q {KL(q(θ)kp(θ|X ))}
Richard Yi Da Xu Introduction to Bayesian Statistics
