Chapter 2 BayesianStatistics1
Uploaded by
mintia766Chapter 2 BayesianStatistics1
Uploaded by
mintia766II.
Bayesian Statistics [ From 1945, “Bayesian” ]
* Targets of Statistical Inference :
1. Bayesian Paradigm
- Let be the parameter of interest. What is the nature of ?
⋅Frequentist(Fisherian) :
⋅Bayesian :
- Bayesian analysis is performed by combining
- Note that and have joint density
- Then, has marginal pdf and . That is, .
- We call is the posterior pdf of and it is the conditional pdf of given .
- 1 -
- Just as the prior distribution reflects so reflects
- In other words, the posterior distribution combines the prior beliefs about with the information
about contained in the sample to give a composite picture of the final beliefs about .
- Since ∝ , the choice of the prior pdf is very important for Bayesian inference.
* Predictive Inference : inference to the future obs.
⋅
: the posterior predictive pdf of
given
⋅ If
and are conditionally independent when is given
- 2 -
2. Conjugate Priors
If the prior pdf and the posterior pdf are in the same class of pdf’s,
then is said to be conjugate prior.
-----------------------------------------------------
Sample Distribution(Likelihood) Conjugate prior
-----------------------------------------------------
∼
Poisson( ) ∼Gamma(a,b)
is known ∼
is known ∼Gamma(a,b)
Gamma ∼Gamma(a,b)
-----------------------------------------------------
- 3 -
◎ Beta Distribution : ∼
,
Example 2.1 Female birth rate in Europe (Laplace)
▪ : the number of girls in recorded births
▪ : the probability of a female birth
▪ ,
▪ (conjugate) Prior of :
- 4 -
▪ The posterior distribution:
Thus, ≡Beta(1,1) is a conjugate prior.
▪ The marginal distribution of :
⇒ Discrete Uniform Distribution
- 5 -
▪ The predictive probability: If we are interested in the probability of the new female birth ( ),
then the predictive probability is as following and we call it the Law of Succession.
❈ Laplace collected data in Paris from 1745~1770.
▪ Result: 241,945 girls, 251,527 boys ⇒
Posterior Mean:
(Posterior mean is the Bayes estimator under the squared error loss)
: In this case, the MLE and posterior mean are very close
- 6 -
Example 2.2 Binomial Distribution
▪ ∼ ≤ ≤
▪ ∼
⇔ Bayesian notation:
▪ Likelihood function :
▪ (Conjugate) Prior : ∼Beta i.e.
▪ Posterior :
▪ Posterior mean : Since the mean of Beta is ,
- 7 -
▪ The predictive probability of
:
& are independent
- 8 -
Example 2.3 Placenta previa (전치태반)
▪ Placenta previa : unusual condition in pregnancy
▪ Study in Germany : births, female
▪ : the probability of a female birth of placenta previa
▪ Is there evidence to think ?
▪
▪ Conjugate Prior of :
▪ The posterior distribution: from Example 2.1,
∼
⇒ posterior mean 0.446 , posterior s.d 0.016
95% posterior interval by normal approximation
⇒ Conclude that since ▦
- 9 -
Example 2.4 Normal Distribution with known variance
▪ ∼ , ( is known)
⇔ ∼ , ( is known)
▪ Likelihood function :
▪ (Conjugate) Prior distribution ∼ ( are known) :
▪ The posterior distribution:
where
- 10 -
also, ← free from
i.e.
- 11 -
▪ The predictive probability of
:
& are independent
∞
∞
←
and ′
∞
∞
∞
∝
∞
exp
∞
∝ exp
∞
where
.
Therefore,
- 12 -
∞
∝ exp
exp
∞
∝ exp
⇒ The predictive probability of
⇒ Predictive mean = posterior mean of
Predictive Variance + (posterior variance of ) ▦
Example 2.5 IQ-test [Jim Berger’s Example]
A child is given an intelligent test. Assume that the test result ∼ , where is the true IQ
level of the child, as measured by the test. In other words, if the child were to take a large number of
independent similar tests, his average score would be about .
Assume also that ∼ . Then,
- 13 -
∼
Thus, if a child scores 115 on this test, the posterior distribution is as following;
≡ . ▦
[NOTE] Inverse Gamma Distribution
▪ ∼ ⇒ ∼
▪ pdf: ∞
▪ Mean and Variance:
- 14 -
Example 2.6 Normal Distribution with known mean
▪ ∼ , ( is known)
↔ ∼ , ( is known)
▪ Likelihood function :
▪(Conjugate) Prior :
∼
( are known) :
▪Posterior : ∝
- 15 -
Thus, the posterior distribution is
∼
,
and the posterior mean is given by
,
. ▦
Example 2.7 Poisson Distribution
▪ ∼
↔ ∼
▪ Likelihood function :
▪(Conjugate) Prior : ∼ Gamma( ) ( are known)
- 16 -
▪Posterior :
Thus, the posterior distribution is
∼
and the posterior mean is given by
- 17 -
3. Noninformative Priors (Objective Priors)
When no (or minimal) prior information is available, we need a prior distribution which is
relatively “flat” to the sample information (likelihood function). It is called noninformative prior.
It has minimal impact on the posterior distribution.
- Laplace Prior
- Jeffreys’ Prior
- Reference Prior
- Probability Matching Prior
(e.g) If ≤ ≤ , then ≤ ≤ is a noninformative prior of (Laplace prior)
(e.g) If ∞ ∞, for ∼ , then let → ∞ and we can use it as a noninformative prior
of since ∼ will be very flatted distribution.
- 18 -
3.1 Improper prior
∞
: A prior is said to be improper if ∞
∞ .
- (e.g) A uniform prior distribution on the real line, ∝ for ∞ ∞, is an improper prior.
- Improper priors are often used in Bayesian inference since they usually yield noninformative priors
and proper posterior distributions.
- Improper prior distributions can lead to posterior impropriety (improper posterior distribution).
- To determine whether a posterior distribution is proper, you need to make sure that the propriety of
posterior pdf.
- If an improper prior distribution leads to an improper posterior distribution, inference based on
the improper posterior distribution is invalid.
Example 3.1 Normal Distribution with known variance
▪ ∼ , ( is known)
▪ Noninformative Prior : ∝ ( is a constant)
∞
∞
∞ ⇒ improper prior
- 19 -
▪ Posterior :
∝
⇒ and it is a proper posterior distribution. ▦
Example 3.2 Normal Distribution with known mean
▪ ∼
▪ Noninformative Prior :
▪ Posterior :
∝
⇒
and it is a proper posterior distribution.
Also, we know that ∝ is a conjugate prior. ▦
- 20 -
Example 3.3 Binomial Distribution
▪ ∼ ≤ ≤
▪ ∼
↔ ∼
▪ Likelihood :
▪ Noninformative Prior (Haldane) : ∝
∞ ⇒ improper prior
▪ Posterior :
∝ ⇒ ∼
It is a proper posterior distribution excepts and . ▦
Example 3.4 Poisson Distribution
▪ ∼
▪ Likelihood :
▪ Noninformative Prior :
- 21 -
▪ Posterior :
∝
⇒ ∼
Example 3.5 Exponential Distribution
: improper prior leads improper posterior
▪ ∼ Exp ⇒ Likelihood : ∝
▪ Noninformative Prior : ∝ [improper prior]
▪ Posterior : ∝
Let , then . Thus, and
∞ ∞ ∞
Let → for the first term
- 22 -
Thus,
≈ ∞
Also, ≥ ⇔ ≤ , so for the second term
∞ ∞ ∞
≤
∞
∞
⇒
∞ [improper posterior!!]
3.2 Jeffreys’ Prior for one-parameter
▪ Laplace(1812) Flat Prior ∝ ⇒ Lack of invariance
(e.g.) Let ∝ and .
Then ∝ since and ∝ is not a flat prior.
▪ Jeffreys’ Prior
- It is proportional to the square root of the determinant of the Fisher information.
- For one-parameter case, it has the invariant property under one-to-one transformation.
log log
⇒ ∝
where
- 23 -
Example 3.6 Normal Distribution with known variance
▪ ∼ , ( is known)
exp
log ⇒ [free from ],
▪ Jeffreys' Prior :
Example 3.7 Normal Distribution with known mean
▪ ∼ , ( is known)
exp
log ⇒
▪ Jeffreys' Prior : ∝ ∝ . Also,
- 24 -
Example 3.8 Binomial Distribution
▪ ∼ ≤ ≤ ,
▪ ∼ , ∝
log
▪ Jeffreys' Prior :
Example 3.9 Poisson Distribution
▪ ∼ ,
log ⇒
▪ Jeffreys' Prior :
- 25 -
Example 3.10 Exponential Distribution
▪ ∼ Exp ,
exp
log
⇒
▪ Jeffreys' Prior : ∝
▪ Posterior :
∝ exp
. Let , then
and
∞
∞ [proper]
3.3 Jeffreys’ Prior for multi-parameter
▪ In general the statistical model has several parameters.
▪ multi-parameter = parameter(s) of interest + nuisance parameter(s)
▪ We should find the marginal posterior pdf for the parameter(s) of interest.
- 26 -
(i) find the joint posterior pdf for all parameters
(ii) after integrating out for the nuisance, one can get the marginal posterior pdf
for the parameter(s) of interest
▦ multi-parameter :
: (scalar or vector) is the parameter(s) of interest and
(scalar or vector) is the nuisance parameter(s)
▪ : the posterior pdf of
▪ : the likelihood function, : the prior pdf
⇒ ∝
⇒
(e.g) normal with both unknown and
▪ ∼
▪ the parameter vector:
▪ the likelihood function: ∝
exp
- 27 -
▪ Fisher Information Matrix :
log log
,
log
Thus,
log
log
E E
I and
log log
E E
Therefore, the Jeffreys’ prior for is
∝
- 28 -
▦ Jeffreys’ Independent Prior
▪ Jeffreys considered the prior ∝ for and the prior ∝ for , respectively
▪ Jeffreys considered and are independent
i.e.
▪ Joint posterior pdf of and using Jeffreys' independent prior
∝
exp
▪ If is the parameter of interest, one can get the marginal posterior pdf of as following;
∝
∞
∝
exp
Useful Fact
∞
exp
∝
- 29 -
- Using the fact, since ,
in this case
∝
×
∝ where
- as a result, the marginal posterior distribution of is the Student's t-distribution
with as the location parameter and
as the scale parameter and is the d.f.
- 30 -
[Remark: Another way to find the marginal posterior pdf of ]
Let , then . And using the property of Gamma distribution, one can get
the same result.
▪ If is the parameter of interest, one can get the marginal posterior pdf of as following;
- rewrite the above joint posterior as a function of and :
∝
exp
- then
∞
∝
∞
∞
∝ exp
∞
∝
exp
⇒ the marginal posterior distribution of :
- 31 -
You might also like
- Lecture Material 2.5 - Bayesian Estimation & ConceptsNo ratings yetLecture Material 2.5 - Bayesian Estimation & Concepts12 pages
- Introduction To Bayesian Methods With An ExampleNo ratings yetIntroduction To Bayesian Methods With An Example25 pages
- Bayesian Statistics: MA501, Statistics For InsuranceNo ratings yetBayesian Statistics: MA501, Statistics For Insurance28 pages
- A Beginner's Notes On Bayesian Econometrics (Art)No ratings yetA Beginner's Notes On Bayesian Econometrics (Art)21 pages
- Introduction To Bayesian Methods: Jessi Cisewski Department of Statistics Yale UniversityNo ratings yetIntroduction To Bayesian Methods: Jessi Cisewski Department of Statistics Yale University53 pages
- Nonparametric Inference Techniques For High-Dimensional Data: Challenges and SolutionsNo ratings yetNonparametric Inference Techniques For High-Dimensional Data: Challenges and Solutions16 pages
- Bayesian Data Analysis - Reading Instructions 2: Chapter 2 - OutlineNo ratings yetBayesian Data Analysis - Reading Instructions 2: Chapter 2 - Outline36 pages
- Bayesian Theory-Priors, Part 2: 3.1 Specifying HyperparametersNo ratings yetBayesian Theory-Priors, Part 2: 3.1 Specifying Hyperparameters13 pages
- Course On Bayesian Methods in Environmental Valuation: Basics (Continued) : Models For Proportions and MeansNo ratings yetCourse On Bayesian Methods in Environmental Valuation: Basics (Continued) : Models For Proportions and Means34 pages
- Bayesian Theory-Priors, Part 1: Other ReadingNo ratings yetBayesian Theory-Priors, Part 1: Other Reading14 pages
- Lec12 13 BayesianInferenceForTheGaussianNo ratings yetLec12 13 BayesianInferenceForTheGaussian57 pages
- 2 Statistical Definitions: 2.1 Probability Density FunctionNo ratings yet2 Statistical Definitions: 2.1 Probability Density Function9 pages
- Ch3 - 2009 Conjugate Families of DistributionsNo ratings yetCh3 - 2009 Conjugate Families of Distributions67 pages
- Bayesian Inference: by Hoai Nam Nguyen September 9, 2017No ratings yetBayesian Inference: by Hoai Nam Nguyen September 9, 20177 pages
- Hashem Pesaran, Lung-Fei Lee - Analysis of Panels and Limited Dependent Variable Models (1999)No ratings yetHashem Pesaran, Lung-Fei Lee - Analysis of Panels and Limited Dependent Variable Models (1999)350 pages
- The - Impact - of - Oil - Price - On - South - AfricanNo ratings yetThe - Impact - of - Oil - Price - On - South - African28 pages
- Model Based Fault Diagnosis Methods, Theory, and Automotive Engine ApplicationsNo ratings yetModel Based Fault Diagnosis Methods, Theory, and Automotive Engine Applications285 pages
- The Effect of Firm Size, Leverage, and Liquidity On Hedging Decisions of Consumer Goods Industryon The Indonesia Stock ExchangeNo ratings yetThe Effect of Firm Size, Leverage, and Liquidity On Hedging Decisions of Consumer Goods Industryon The Indonesia Stock Exchange8 pages
- Gábor Békés, Gábor Kézdi - Data Analysis For Business, Economics, and Policy-Cambridge University Press (2021)100% (9)Gábor Békés, Gábor Kézdi - Data Analysis For Business, Economics, and Policy-Cambridge University Press (2021)742 pages
- Short-Term Actuarial Mathematics Exam-October 2018No ratings yetShort-Term Actuarial Mathematics Exam-October 20188 pages
- Pinheiro y Bates 2000 - Mixed Effects Models in S and S-Plus100% (2)Pinheiro y Bates 2000 - Mixed Effects Models in S and S-Plus537 pages
- Introduction To Computational Linguistics: Eugene Charniak and Mark JohnsonNo ratings yetIntroduction To Computational Linguistics: Eugene Charniak and Mark Johnson148 pages
- (Ebook) Generative Deep Learning, 2nd Edition (Third Early Release) by David Foster ISBN 9781098134174, 1098134176No ratings yet(Ebook) Generative Deep Learning, 2nd Edition (Third Early Release) by David Foster ISBN 9781098134174, 109813417681 pages
- M.Tech in Quality & Operations ResearchNo ratings yetM.Tech in Quality & Operations Research33 pages
- Auto-Encoding Variational Bayes: Diederik P. Kingma Max WellingNo ratings yetAuto-Encoding Variational Bayes: Diederik P. Kingma Max Welling9 pages
