Predictive Analytics
Regression and Classification
Module 7
Sourish Das
Chennai Mathematical Institute
Linear Regression
mpg = β0 + β1 wt +
10 15 20 25 30
mpg
2 3 4 5
wt
Linear Regression
I mpg = β0 + β1 wt +
I We write the model in terms of linear models
y = Xβ +
where y = (mpg1 , mpg2 , . . . , mpgn )T ;
1 wt1
1 wt2
X = .
..
.. .
1 wtn
β = (β0 , β1 )T and = (1 , 2 , . . . , n )T
Linear Regression
I Normal Equations:
β̂ = (β̂0 β̂1 )T = (X T X )−1 X T y
Pn −1 Pn
n i=1 wti i=1 mpg i
= Pn P 2
Pn
i=1 wti i=1 wti i=1 wti .mpgi
Regression Plane
mpg=β0 +β1 wt+β2 disp+
35
30
25
mpg
disp
500
20
400
300
15
200
100
10
0
1 2 3 4 5 6
wt
Regression Plane
I mpg=β0 +β1 wt+β2 disp+
I We write the model in terms of linear models
y = Xβ +
where y = (mpg1 , mpg2 , . . . , mpgn )T ;
1 wt1 disp1
1 wt2 disp2
X = .
.. ..
.. . .
1 wtn dispn
β = (β0 , β1 , β2 )T and = (1 , 2 , . . . , n )T
Linear Plane
I mpg=β0 +β1 wt+β2 disp+
I Normal Equations:
β̂ = (β̂0 β̂1 β̂2 )T
= (X T X )−1 X T y
I Ask yourself.
Linear Plane
I mpg=β0 +β1 wt+β2 disp+
I Normal Equations:
β̂ = (β̂0 β̂1 β̂2 )T
= (X T X )−1 X T y
Pn Pn −1
n wt dispi
Pn P i=1 2i P i=1
= wt wt i=1 wti dispi
Pn i=1 i
P i=1 i P 2
i=1 disp i i=1 wt i dispi i=1 dispi
Pn
i=1 mpg i
× P ni=1 wti .mpgi
P
n
i=1 dispi .mpgi
Quadratic Regression
mpg = β0 + β1 hp + β2 hp2 +
10 15 20 25 30
mpg
50 150 250
hp
Feature Engineering
mpg=β0 +β1 hp+β2 hp2 +
35
30
25
hp2
mpg
120000
100000
20
80000
60000
15
40000
20000
10
0
50 100 150 200 250 300 350
hp
Quadratic Regression
I mpg = β0 + β1 hp + β2 hp2 +
I We write the model in terms of linear models
y = Xβ +
where y = (mpg1 , mpg2 , . . . , mpgn )T ;
1 hp1 hp21
1 hp2 hp2
2
X = . .. ,
. ..
. . .
1 hpn hp2n
β = (β0 , β1 , β2 )T and = (1 , 2 , . . . , n )T
I The linear model is linear in parameter.
Quadratic Regression
I Normal Equations:
β̂ = (β̂0 β̂1 β̂2 )T
= (X T X )−1 X T y
Pn Pn −1 Pn
hp2i
n hp i=1 mpgi
P i=1 2i Pi=1
ni=1 hpi .mpgi
Pn n 3
P
= i=1 hpi i=1 hpi i=1 hpi Pn
Pn 2
P n 3
Pn 4 2
i=1 hpi i=1 hpi i=1 hpi i=1 hpi .mpgi
Feature Engineering
mpg=β0 +β1 hp+β2 hp2 +
35
30
25
mpg
hp^2
500
20
400
300
15
200
100
10
0
1 2 3 4 5 6
hp
Feature Engineering/ Variable Transformation
I we put the original data into a higher dimension and
I hope that we will find a good fit for linear hyper-plane in a
higher dimension,
I which will explain the non-linear relationship between the
feature space and target variable.
Non-linear Regression Basis Functions
I Consider i th record
yi = f (x i ) + i , i = 1, 2, · · · , n
represents f (x) as
K
X
f (x) = βj φj (x) = φβ
j=1
we say φ is a basis system for f (x).
Representing Functions with Basis Functions
I mpg = β0 + β1 hp + β2 hp2 +
I Generic terms for curvature in linear regression
y = β1 + β2 x + β3 x 2 + · · · + i
implies
f (x) = β1 + β2 x + β3 x 2 + · · ·
I Sometimes in ML φ is known as ‘engineered features’
and the process is known as ‘feature engineering’
Fourier Basis
I sine cosine functions of incresing frequencies
y = β1 +β2 sin(ωx)+β3 cos(ωx)+β4 sin(2ωx)+β5 cos(2ωx) · · ·+i
I constant ω = 2π/P defines the period P of oscillation of
the first sine/cosine pair. P is known.
I φ = {1, sin(ωx), cos(ωx), sin(2ωx), cos(2ωx)...}
I β T = {β1 , β2 , β3 , · · · }
y = φβ +
I Again in ML φ is known as ‘engineered features’
I mpg = β0 + β1 sin(ω hp ) +
Functional Estimation/Learning
I We are writing the function with its basis expansion
y = φβ +
I Lets assume basis (or engineered features) φ are fully
known.
I Problem is β is unknown - hence we estimate β.
Functional Estimation/Learning
I We are writing the function with its basis expansion
y = φβ +
I Lets assume basis (or engineered features) φ are fully
known.
I OLS Estimator:
β̂ = (φT φ)−1 φT y
Uncertainty associated with the OLS estimator
I How do we estimate the uncertainty (i.e., margin of error)
associate with OLS estimator β̂?
I If x0 is a test point, then
ŷ = φ(x0 )β̂
is the predicted value of true but unknown y0 .
I What is the margin of error of ŷ ?
Next ...
I We will discuss sampling distributions and inference of
regression coefficients!
