IIMT 2641 Introduction to Business Analytics
Module 3: Linear Regression
Topic 1: Simple Linear Regression
1
Bordeaux wine
§ Large differences in price and quality between years, although wine is
produced in a similar way
§ Meant to be aged, so hard to tell if wine will be good when it is on the
market
§ Expert tasters predict which ones will be good
§ Can analytics be used to come up with a different system for judging wine?
3
Predicting the quality of wine
§ March 1990 - Orley Ashenfelter, a Princeton economics professor, claims
he can predict wine quality without tasting the wine
4
Building a model
§ Ashenfelter used a method called linear regression
– Predicts an outcome variable, or dependent variable
– Predicts using a set of independent variables
5
Building a model
§ Dependent variable:
– Typical price in 1990-1991 wine auctions (approximates quality)
– Conduct logarithmic transformation
q A better linear fit
§ Independent variables:
– Age of wine (in 1990)
q Older wines are more expensive
– Weather
q Average Growing Season Temperature (AGST)
q Harvest Rain
q Winter Rain
– Population of France
6
The wine data (1952 - 1978)
8
The wine data (1952 - 1978)
Quick Question: What is the relationship between harvest rain, average
growing season temperature, and wine prices?
9
Baseline model (?)
10
Baseline model (Take the mean)
ne
y .....
In
&
11
One-Variable Linear Regression
me
e ⑧
12
Simple Regression Model
The population model of y with one predictor variable x is:
-
-
-
! =# +# %+ε
! "
I
-
↑
-
↑
-
§ y is the dependent variable (DV) Bl
§ x is the independent variable (IV) Pr
§ Regression Function
§ E[Y|x] = -
-
!
- ! + !" # is the mean of Y given x
-
§ !! is the y-intercept (value of E[Y|0] when x=0)
e n
§ !" is the slope for x, which is the change in E[Y|x] for a unit increase&
in x
-
§ Random errors e (not required)
-
§ Random errors are a random sample from $ 0, '#
-
Random samples are i.i.d. or
independent and identically
-
Each observation has a random error
-
§ The output does not show these, but it does estimate se distributed random variables
§ The random errors e and IV (X) are uncorrelated
§ These assumptions are important for effective business analytics
-
13
Estimated Regression Function
§ Estimates the regression model with n observations (xi,yi) for i = 1, …, n
-
§ The estimated or predicted value of y given x is:
&
"! = $ ! + $" &
44
&
§ '! is the sample estimate of the population intercept #!-
§ '" is the sample estimate of the population slope #"
-
'! and '" are sample statistics
)
(similar to *)
and have sampling distributions
14
One-Variable Linear Regression
brtb i)
,
15
Data and Predicted Values
§ What is the observed y when x = 1?
YG =
§ What is the predicted y when x = 1?
4 =
§ What is the observed y when x = 4?
O
§ What is the predicted y when x = 4?
16
Data and Predicted Values
§ What is the observed y when x = 1?
y=6
§ What is the predicted y when x = 1?
!=1+(2)(1)
, =3
§ What is the observed y when x = 4?
y=4
§ What is the predicted y when x = 4?
!=1+(2)(4)
, =9
17
Estimated Model and-
Residuals
e
§ Residuals are the difference between the observed values of ! and
-
predicted values of !,
-
– y - $# 8
! =-=
– Each observation has one observed y, one predicted $,
# and one residual r.
§ The residuals are errors between the observed and predicted values.
y3
"!$
## = "# − "!#
y1
"!" #$ = "$ − "!$
"!#
#! = "! − "!!
y4
"!! y2
#" = "" − "!"
18
Computing Residuals
r3
↓ r4
r1 r2
j
§ What is the residual r2 at x = 2? 1
y Y
=
-
§ What is the residual r3 at x = 3?
19
Computing Residuals
r3
r4
r1 r2
r -
§ What is the residual r2 at x = 2?
-# = !# − !,# = 3 − 1 + 2 ∗ 2 = 3 − 5 = −2
§ What is the residual r3 at x = 3?
-$ = !$ − !,$ = 11 − 1 + 2 ∗ 3 = 11 − 7 = 4
20
Ordinary Least Squares Criterion or (OLS)
The least squares line finds the estimates '! and '" of the coefficients to
minimize the sums-of-squares error for a sample {(xi,yi)} with n observations:
W I
-
667 = !% − !,% # ∑'%&"
!,% = '! + '" %% for < = 1, … , ? SSE
Why squared?
↑ min
The sum of residuals could be zero.
-> '
667('! , '" ) = A !% − '! − '" %% #
∑'%&" %% − %̅ !% − !)
D
-
%&" '" =
B667('! , '" ) ∑'%&" %% − %̅ #
3
= 0
B'! C
'! = !) − '" %̅
B667('! , '" ) %:̅ sample average of independent variable
-
= 0 -
B'" ) sample average of dependent variable
!:
- -
- -
Do not need to memorize.
21
Estimate a linear model H0: AGST Coefficient = 0 versus HA: AGST
(One Variable ) Coefficient ≠ 0
-
O0
Estimated Standard Errors t-score = (Estimated Coefficient – 0)/(Standard Error)
intercept and for estimated
slope coefficients
Two-Tail Test: p-value = 2*P(T<-|t-score|)
V
-
Coefficient of Determination: R-Squared
23
One-Variable Linear Regression
, -3.4178 + 0.6351*AGST
!=
24
Estimate a linear model
(One Variable )
• Estimated model for price:
D
, -3.4178 + 0.6351*AGST
!=
• The predicted LogPrice increases by
-
-
0.6351 for every 1 degree increase in
-
average growing season temperature.
• If AGST = 15, then !=, ?
• If AGST = 18, then !=
, ?
O • If AGST = 20, then !=
, ?
i
25
T-Tests for the Coefficients: H0: bj = 0 versus HA: bj ≠ 0
& -
Two-Tail Test for the Slope
(Very important. Can you predict Y from X?)
H0: b1 = 0 versus HA: b1 ≠ 0
• t-score = (coefficient – 0)/(std.error)
-
• t-score = (0.6351-0)/0.1509 = 4.208
• p_value = 2*P(T < -|4.208|)
↓- =2*t.dist(-4.208, 23, 1) < .001
• df = n-1-#IV = df Error under Sum of squares
-
O ⑧
e⑪
men
-
df = 23
⑰
-
& A
n
-
1 -
#IF
I
st ↑
I
25 1 1 23
=
- -
=
4 208
. ,
of =
23 .
0 .
001 <P rate -
<0 .
01 **
P([s> 200) xP(ts -3 76) 0 001
=
4 2 <
.
<
2 x
-
.
.
26
How well the model fits data
§ The simplest commonly used measure of fit is R# (the coefficient of
determination): R# = 1 − SSE/SST
-
-
– SSE = ∑&$%" y$ − y. $ ' : sum of squared errors - *
q Variation of Y that cannot be explained by the regression R 1
– SST = ∑&$%" y$ − y0 ' : total sum of squares v
1-
e
q Total amount of variation of Y around its mean
q “Error” generated by a baseline model without any inputs
-
– Decomposition of variation of Y:
SSF
&
SSE =
∑'%&" !% − !) # = ∑'%&" !% − !,% # + ∑'%&" !,% − !) #
7
q
-
-> Total variation Unexplained variation Explained variation
1- SE
s =
-
SSE *
SS7
R# = Proportion of the variance in DV is explained by the
regression model.
27
Coefficient of Determination: R-Squared
• R-Squared is a measure of fit
• Bigger R-Squared indicates better fit all
else being equal
• 43.5% of the variation of prices is
explained by the simple regression on
AGST
-
• 0 < R-Squared < 1
28
Use each variable on its own
§ R# =0.44 using Average growing season temperature (Variable
-
Significant, 0.001)
R# =0.32 using Harvest rain (Variable Significant, 0.01)
↓
§
§ R# =0.22 using France Population (Variable Significant, 0.05)
§ R# =0.20 using Age (Variable Significant, 0.05)
§ R# =0.02 using Winter rain (Not Significant)
§ Multivariate linear regression allows us to use more than one
variable to potentially improve our predictive ability.
29
