12/6/2022
REGRESSION
ANALYSIS
WITH
LINEAR PROBABILITY MODEL, LOGIT AND PROBIT MODELS
HELLO!
I am Elijah Appiah from
Ghana.
I am an Economist by
profession.
I love everything about R!
You can reach me:
secret behind the smile! eappiah.uew@gmail.com
1
12/6/2022
Lesson Goals
▸ Understand how to estimate
qualitative response models.
▸ Advance your knowledge of
statistical inference for limited
dependent variables.
Lecture Series
2
12/6/2022
Limited Dependent Variables
Sometimes, the dependent variable may be
categorical.
Two categories – binary or dichotomous
Three categories – trichotomous
Four or more categories - polychotomous
Binary Response Variable
The dependent (or response) variable is binary.
Independent variable(s) may be quantitative,
qualitative, or mixed as usual.
Examples of binary variables:
Yes or No responses (0 - No, 1 - Yes)
Patient Satisfaction(0 - Not Satisfied vs. 1- Satisfied)
Health Outcome (0 - Not Cured vs. 1- Cured)
Votes (0 - Democrat vs. 1 - Republican)
3
12/6/2022
Binary Response Variable
Three approaches to measuring binary response
models:
Linear Probability Model (LPM)
Logistic Regression (or Logit) Model
Probit (or Normit) Model
LINEAR
PROBABILITY
MODEL
4
12/6/2022
Linear Probability Model
LPM is estimated using Ordinary Least Squares
(OLS).
Coded values, 0 and 1, of binary response
variables are treated as numbers, not categories.
10
Linear Probability Model
Consider the regression model:
𝒊 𝟎 𝟏 𝒊
where: 𝑿𝒊 = family income, 𝒀𝒊 = 𝟏 if the family owns
a house and 𝟎 if it does not own a house.
Conditional expectation - 𝑬(𝒀𝒊 | 𝑿𝒊 ) – is the
conditional probability that the event will occur
given 𝑿𝒊 , that is, 𝑷𝒓(𝒀𝒊 = 𝟏|𝑿𝒊 ).
𝑷𝒓(𝒀𝒊 = 𝟏|𝑿) = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊
Hence, Linear Probability Model.
5
12/6/2022
11
Linear Probability Model
If 𝑷𝒊 is probability that 𝒀𝒊 = 𝟏, and (𝟏 − 𝑷𝒊 ) is
probability that 𝒀𝒊 = 𝟎, then the variable 𝒀𝒊 will
have the probability distribution as:
𝒀𝒊 Probability
Bernoulli 0 𝟏 − 𝑷𝒊
Probability
Distribution 1 𝑷𝒊
Total 𝟏
12
Linear Probability Model - Problems
Recall: Rule of Probability ~ 𝒊 𝒊 .
Real problem with OLS estimation of LPM is
nonfulfillment of 𝟎 ≤ 𝑬(𝒀𝒊 |𝑿𝒊 ) ≤ 𝟏. That is,
predicted probabilities can be negative, or greater
than 1.
Heteroscedasticity (can be solved using robust
standard errors).
Low R-squared value.
6
12/6/2022
13
Now, let’s practice
14
Linear Probability Model
𝐏𝐫(𝒀 > 𝟏)
𝐏𝐫(𝒀 < 𝟎)
7
12/6/2022
15
Linear Probability Model - Problems
Probability Problem (How do we get around this problem?)
Estimate the LPM by the usual OLS
method. If estimated Y has some values
less than 0 [i.e. negative], Y is assumed Constrained LPM
to be zero for those cases; if they are
greater than 1, they are assumed to be 1.
Devise an estimating technique that will
Logit and Probit Models guarantee that the estimated conditional
probabilities Y will lie between 0 and 1.
16
Now, let’s practice
8
12/6/2022
17
MODEL
18
Logistic Regression or Logit Model
Cumulative plot.ecdf(rnorm(500))
distribution functions
(CDF) are sigmoid, or
S-shaped curves.
Some CDFs are the
logistic CDF (logit) and
standard normal CDF
(probit).
9
12/6/2022
19
Logit Model
match match frequency What can we
Win
Win 4 do with this
Win
Lose 6 count data?
Win
Win
Lose
Lose What is the probability that a
Lose team wins the match?
Lose
Lose What are the odds of winning?
Lose
20
Logit Model
Probability is the number of events divided by total
outcome.
match frequency 𝟒
𝒑𝒓𝒐𝒃 = = 𝟎. 𝟒𝟎
Win 4 𝟏𝟎
Lose 6
Odds are the chances of something happening to
something not happening.
𝟒
𝒐𝒅𝒅𝒔 = = 𝟎. 𝟔𝟕
𝟔
10
12/6/2022
21
Logit Model
Probability and odds can be expressed in terms of each other.
𝒐𝒅𝒅𝒔 𝒑𝒓𝒐𝒃
𝒑𝒓𝒐𝒃 = 𝒐𝒅𝒅𝒔 =
𝟏 + 𝒐𝒅𝒅𝒔 𝟏 − 𝒑𝒓𝒐𝒃
𝟒 𝟒
𝒑𝒓𝒐𝒃 = = 𝟎. 𝟒𝟎 𝒐𝒅𝒅𝒔 = = 𝟎. 𝟔𝟕
𝟏𝟎 𝟔
𝟎. 𝟔𝟕 𝟎. 𝟒𝟎
𝒑𝒓𝒐𝒃 = = 𝟎. 𝟒𝟎 𝒐𝒅𝒅𝒔 = = 𝟎. 𝟔𝟕
𝟏 + 𝟎. 𝟔𝟕 𝟏 − 𝟎. 𝟒𝟎
22
Logit Model
Logistic regression (logit) model calculates the
log odds given known values of the predictors.
𝐥𝐨𝐠 𝒐𝒅𝒅𝒔 𝑬 𝒀𝒊 𝑿𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊
11
12/6/2022
23
Logit Model
𝒐𝒅𝒅𝒔
Recall: 𝒑𝒓𝒐𝒃 =
𝟏 + 𝒐𝒅𝒅𝒔
𝐥𝐨𝐠 𝒐𝒅𝒅𝒔 𝑬 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊
𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = Not Correct Yet!
𝟏 + 𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)
Exponential function? ----> 𝒇(𝒙)
𝒍𝒏[𝒇(𝒙)] 𝒍𝒏 𝒙 𝒍𝒏 𝟐
24
Logit Model
𝒐𝒅𝒅𝒔
Recall: 𝒑𝒓𝒐𝒃 =
𝟏 + 𝒐𝒅𝒅𝒔
𝐥𝐨𝐠 𝒐𝒅𝒅𝒔 𝑬 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊
𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = Not Correct Yet!
𝟏 + 𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)
𝒆𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = Correct
𝟏 + 𝒆𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)
12
12/6/2022
25
Logit Model
𝒆𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = Correct
𝟏 + 𝒆𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)
𝐥𝐨𝐠 𝒐𝒅𝒅𝒔 𝑬 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊
𝒆 𝜷 𝟎 𝜷 𝟏 𝑿𝒊
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 =
𝟏 + 𝒆 𝜷 𝟎 𝜷 𝟏 𝑿𝒊
𝟏
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = (𝜷𝟎 𝜷𝟏 𝑿𝒊 )
𝟏+𝒆
26
Logit Model
𝒆 𝜷 𝟎 𝜷 𝟏 𝑿𝒊
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 =
𝟏 + 𝒆 𝜷 𝟎 𝜷 𝟏 𝑿𝒊
𝜷𝟎 𝜷𝟏 𝑿𝒊
𝜷𝟎 𝜷𝟏 𝑿𝒊
𝒊 𝒊 𝜷𝟎 𝜷𝟏 𝑿𝒊
𝜷𝟎 𝜷𝟏 𝑿𝒊 𝜷𝟎 𝜷𝟏 𝑿𝒊
𝟏
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = (𝜷𝟎 𝜷𝟏 𝑿𝒊 )
𝟏+𝒆
13
12/6/2022
27
Logit Model
𝐥𝐨𝐠 𝒐𝒅𝒅𝒔 𝑬 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊
Probability is not
𝒁𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊 linear in 𝑿𝒊 or the
parameters in 𝒁𝒊 .
𝒆𝒁𝒊
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = Logistic
𝟏 + 𝒆𝒁𝒊
Distribution
𝟏 Function
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝒁𝒊
𝟏+𝒆
28
Now, let’s practice
14
12/6/2022
29
Logit Model
𝟏
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝒁𝒊
𝟏+𝒆
𝟏 𝟏 𝟏
𝑷𝒊 = 𝒁𝒊 𝟏 − 𝑷𝒊 = 𝟏 − 𝒁𝒊 =
𝟏+𝒆 𝟏+𝒆 𝟏 + 𝒆𝒁𝒊
𝒑𝒓𝒐𝒃
𝒐𝒅𝒅𝒔 =
𝟏 − 𝒑𝒓𝒐𝒃
𝒊 𝒁𝒊
𝒁𝒊
𝒊
𝒁𝒊 Odds Ratio
30
Logit Model
𝒑𝒓𝒐𝒃
𝒐𝒅𝒅𝒔 =
𝟏 − 𝒑𝒓𝒐𝒃
𝒊 𝒁𝒊
𝒁𝒊
𝒊
𝒁𝒊 Odds Ratio
If ,
The odds that the family
will own a house is 4 to 1.
15
12/6/2022
31
Logit Model
𝑷𝒊
𝑶𝒅𝒅𝒔 = = 𝒆𝒁𝒊
𝒙 𝟏 − 𝑷𝒊
𝑷𝒊 𝑷𝒊 𝒁
𝑶𝒅𝒅𝒔 = 𝒍𝒏
𝑳 = 𝒍𝒏(𝑶𝒅𝒅𝒔) = 𝒆 𝒊 = 𝒍𝒏 𝒆𝒁𝒊 = 𝒁𝒊
𝟏 − 𝑷𝒊 𝑷𝒊
𝟏 −
𝑳 = 𝒁𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊
where: 𝑳, is the log of the odds ratio, is not only
linear in 𝑿𝒊 but also the parameters. L is called
the Logit, hence the name Logit model.
32
Logit Model
Key Takeaways:
The logits are not bounded by 0 and 1, but the
probabilities will (so we need to calculate
probabilities for the logit model – MARGINAL
EFFECTS).
Although L is linear in X, but the probabilities are
not.
You can include many predictors.
16
12/6/2022
33
Logit Model
Key Takeaways:
If L, the logit, is positive, it means that when the
value of the regressor(s) increases, the odds that
the regressand equals 1 (meaning some event of
interest happens) increases. If L is negative, the
odds that the regressand equals 1 decreases as
the value of X increases.
34
MODEL
17
12/6/2022
35
Probit Model
The probit model uses a standardized normal CDF.
If a variable 𝑿 follows the normal distribution with
mean 𝝁 and standard deviation 𝝈, then the PDF is:
𝟏 𝟐 𝟐
𝒇 𝑿 = 𝒆 𝑿 𝝁 /𝟐𝝈
𝟐𝝈𝟐 𝝅
Its CDF becomes:
𝑿𝟎
𝟏 𝑿 𝝁 𝟐 /𝟐𝝈𝟐
𝑭 𝑿 = 𝒆
𝟐𝝈𝟐 𝝅
36
Probit Model
The probit model becomes:
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝑭(𝜷𝟎 + 𝜷𝟏 𝑿𝒊 )
Thus, if 𝒁 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊 , then the CDF is:
𝜷𝟎 𝜷𝟏 𝑿𝒊
𝟏 𝒁𝟐 /𝟐
𝑭 𝑰 = 𝒆 𝒅𝒁
𝟐𝝅
18
12/6/2022
37
Logit and Probit Models
Key Notes:
Both models estimate the log odds, but logit
follows a logistic CDF whiles probit follows a
standard normal CDF.
To express coefficients in probabilities, we
calculate the marginal effects.
Both models have predicted probabilities lying
between 0 and 1 inclusive.
38
Logit and Probit Models
Key Notes:
Both models are estimated using an iterative
algorithm to calculate the log odds of the
regression coefficients.
Maximum Likelihood Estimation
19
12/6/2022
39
Logit and Probit Models - R
Logit and Probit estimates in log odds:
stat::glm()
Logit and Probit estimates in probabilities:
mfx:logitmfx()
mfx::probitmfx()
40
Logit and Probit Models - R
Goodness of Fit:
𝒅𝒆𝒗𝒊𝒂𝒏𝒄𝒆
𝑷𝒔𝒆𝒖𝒅𝒐 𝑹𝟐 = 𝟏 −
𝒏𝒖𝒍𝒍. 𝒅𝒆𝒗𝒊𝒂𝒏𝒄𝒆
deviance and null.deviance are attributes of
both models.
𝑵𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒄𝒐𝒓𝒓𝒆𝒄𝒕 𝒑𝒓𝒆𝒅𝒊𝒄𝒕𝒊𝒐𝒏𝒔
𝑪𝒐𝒖𝒏𝒕 𝑹𝟐 =
𝑻𝒐𝒕𝒂𝒍 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒐𝒃𝒔𝒆𝒓𝒗𝒂𝒕𝒊𝒐𝒏𝒔
20
12/6/2022
41
Now, let’s practice
42
THANKS!
Any questions?
Reach me anytime!
Email
eappiah.uew@gmail.com
LinkedIn
https://www.linkedin.com/in/appiah-elijah-383231123/
21
