Deep Learning

CS F425
BITS Pilani Dr. Bharat Richhariya
Department of CSIS
Pilani Campus
 Lecture 3: Approximating a function (Regression)
BITS Pilani
Pilani Campus
Linear Regression
• Regression refers to a set of methods for modeling the relationship
between one or more independent variables and a dependent variable.
o The purpose of regression is most often to characterize the relationship
between the inputs and outputs.
o Machine learning, on the other hand, is most often concerned with prediction.

o We can use regression whenever we want to predict a numerical value.

o Predicting prices (of homes, stocks, etc.)
o Predicting length of stay (for patients in the hospital)
o Demand forecasting (for retail sales)
Linear Regression
• Linear regression flows from a few simple assumptions:
• The relationship between the independent variables 𝒙 and the
dependent variable 𝑦 is linear, i.e., 𝑦 can be expressed as a weighted
sum of the elements in 𝒙, given some noise on the observations.
• We will use 𝑛 to denote the number of examples. We index the data
examples by 𝑖.

• and the corresponding label as 𝑦(𝑖)

Basic Elements of Linear Regression

Linear Model Loss Function Analytic Solution

Making Predictions
Gradient Descent with the Learned
Model
Basic Elements of Linear Regression

Linear Model Loss Function Analytic Solution

Making Predictions
Gradient Descent with the Learned
Model
Linear regression: linear model
• The linearity assumption says that the target (price) can be expressed as a
weighted sum of the features (area and age):
𝑝𝑟𝑖𝑐𝑒 = 𝑤𝑎𝑟𝑒𝑎 ⋅ 𝑎𝑟𝑒𝑎 + 𝑤𝑎𝑔𝑒 ⋅ 𝑎𝑔𝑒 + 𝑏
• 𝑤𝑎𝑟𝑒𝑎 and 𝑤𝑎𝑔𝑒 are called weights, and 𝑏 is called a bias(/offset/intercept).
• The weights determine the influence of each feature on our prediction.
• The bias just says what value the predicted price should take when all of the
features take value 0.
• The equation above is an affine transformation of input features, which is
characterized by a linear transformation of features via weighted sum, combined
with a translation via the added bias.
Linear regression: linear model
• Given a dataset, our goal is to choose the weights 𝒘 and the bias 𝑏 such that, on
average, the predictions made by our model best fit the true observations.

• Models whose output prediction is determined by the affine transformation of

input features are linear models.
 Linear regression: linear model
• In machine learning, we usually work with high-dimensional datasets.
• When our inputs consist of 𝑑 features, we express our prediction 𝑦ො as
𝑦ො = 𝑤1 𝑥1 + ⋯ + 𝑤𝑑 𝑥𝑑 + 𝑏
• Collecting all features into a vector 𝒙 ∈ 𝑅𝑑 and all weights into a vector 𝒘 ∈ 𝑅𝑑 ,
we can express our model using a dot product:
𝑦ො = 𝒘⊤ 𝒙 + 𝑏
the vector 𝒙 corresponds to features of a single data example.
• We refer to features of our entire dataset of 𝒏 examples via the matrix 𝑿 ∈ 𝑅𝑛×𝑑 ,
where, 𝑿 contains one row for every example and one column for every feature.
• For a collection of features 𝑿, the predictions 𝑦ො ∈ 𝑅𝑛 can be expressed via the
matrix-vector product:
𝑦ො = 𝑿𝒘 + 𝑏
Linear regression: linear model
• Given the features of a training dataset 𝑿 and corresponding (known) labels 𝑦, the goal
of linear regression is to find the weight vector 𝒘 and the bias term 𝑏 such that, given
features of a new data example sampled from the same distribution as 𝑿, the new
example’s label will (in expectation) be predicted with the lowest error.

• We would not expect to find a real-world dataset of 𝑛 examples where 𝑦 𝑖 exactly

equals 𝒘⊤ 𝒙 𝑖 + 𝑏 for all 1 ≤ 𝑖 ≤ 𝑛.
o Thus, even when we are confident that the underlying relationship is linear, we will
incorporate a noise term to account for such errors.
• Before searching for the best parameters (or model parameters) 𝒘 and 𝑏, we need two
more things:
1. A quality measure for some given model.
2. A procedure for updating the model to improve its quality.
Basic Elements of Linear Regression

Linear Model Loss Function Analytic Solution

Making Predictions
Gradient Descent with the Learned
Model
Linear regression: Loss function
• To think about how to fit data with our model, we need to determine a measure
of fitness.
• The loss function quantifies the distance between the real and predicted value of
the target.
o The loss will be a non-negative number where smaller values are better.
o Perfect predictions incur a loss of 0.
• The most popular loss function in regression problems is the squared error:
𝑖 1 𝑖 𝑖 2
𝑙 (𝒘, 𝑏) = 𝑦ො − 𝑦
2
1
• The constant makes no difference but will prove to be convenient, cancelling out when we take
2
the derivative of the loss.
Linear regression: Loss function
• The empirical error is only a function of the model parameters.

• Consider the example below where we plot a regression problem for

a one-dimensional case.

• Large differences between estimates

𝑦ො 𝑖 and observations 𝑦 𝑖 lead to
even larger contributions to the loss,
due to the quadratic dependence.
Linear regression: Loss function
• To measure the quality of a model on the entire dataset of 𝑛 examples, we
average (or equivalently, sum) the losses on the training set:

1 𝑛 1 𝑛 2
𝐿(𝒘, 𝑏) = σ𝑖=1 𝑙 𝑖 𝒘, 𝑏 = σ𝑖=1 𝒘⊤ 𝑥 𝑖 +𝑏−𝑦 𝑖
𝑛 𝑛

• When training the model, we want to find parameters (𝒘∗ , 𝑏 ∗ ) that

minimize the total loss across all training examples:

𝒘∗ , 𝑏 ∗ = 𝑎𝑟𝑔𝑚𝑖𝑛𝒘,𝑏 𝐿(𝒘, 𝑏)
Basic Elements of Linear Regression

Linear Model Loss Function Analytic Solution

Making Predictions
Gradient Descent with the Learned
Model
Linear regression: analytic solution
• Linear regression can be solved analytically by applying a simple formula:
• Subsume the bias 𝑏 into the parameter 𝒘 by appending a column to the design matrix
(𝑿) consisting of all ones.
• Then our prediction problem is to minimize ∥ 𝑦 − 𝑿𝒘 ∥2 .
• Take the loss surface to be the minimum of the loss over the entire domain.
• Taking the derivative of the loss with respect to 𝒘 and setting it equal to zero yields the
analytic solution:

𝒘 ∗ = 𝑿⊤ 𝑿 −1
𝑿⊤ 𝑦
• The requirement of an analytic solution is so restrictive that it would exclude all exciting
aspects of deep learning.
• Simple problems like linear regression may admit analytic solutions but, we should not
get used to such good fortune!
Basic Elements of Linear Regression

Linear Model Loss Function Analytic Solution

Making Predictions
Gradient Descent with the Learned
Model
Linear regression: gradient descent
• In cases where we cannot solve the models analytically, we can still train models
effectively in practice.
• The key technique for optimizing nearly any deep learning model is called
gradient descent.
• Gradient descent iteratively reduces the error by updating the parameters in the
direction that incrementally lowers the loss function.
Starting
point

loss

Value of weight Point of convergence

Linear regression: gradient descent
• The most naïve application of gradient descent consists of taking the derivative
of the loss function, which is an average of the losses computed on every single
example in the dataset.
o This is extremely slow: we must pass over the entire dataset before making a single update.
o Thus, we will sample a random minibatch of examples every time we need to compute the
update, this variant called minibatch stochastic gradient descent.
• In each iteration:
1. We first randomly sample a minibatch B consisting of a fixed number of training examples.
2. We then compute the derivative (gradient) of the average loss on the minibatch with
regard to the model parameters.
3. Finally, we multiply the gradient by a predetermined positive value η and subtract the
resulting term from the current parameter values.
Linear regression: gradient descent
• We can express the update mathematically:
𝜂
𝒘, 𝑏 ← 𝒘, 𝑏 − σ𝑖∈𝐵 𝜕 𝒘,𝑏 𝑙 𝑖 𝒘, 𝑏
𝐵
• 𝒘 is the weights vector, 𝑏 is the bias, η is predetermined positive
value (learning rate), cardinality |𝑩| represents the number of
examples in each minibatch (the batch size), and “𝜕 𝒘,𝑏 𝑙 𝑖 𝒘, 𝑏 ”
means the partial derivative of the loss of 𝑖th element.

Randomly Iteratively sample Update the

initialize the random parameters in the
values of the minibatches from direction of the
model parameters the data negative gradient
Linear regression: gradient descent
• We can write this out explicitly as follows:

𝜂 𝑖 𝜂 𝑖
𝒘←𝒘− σ𝑖∈𝐵 𝜕𝒘 𝑙 𝒘, 𝑏 = 𝒘 − σ𝑖∈𝐵 𝒙 𝒘⊤ 𝒙 𝑖 +𝑏−𝑦 𝑖
𝐵 𝐵
𝜂 𝑖 𝜂
𝑏←𝑏− σ𝑖∈𝐵 𝜕𝑏 𝑙 𝒘, 𝑏 = 𝑏 − σ𝑖∈𝐵(𝒘⊤ 𝒙 𝑖
+𝑏−𝑦 𝑖 )
𝐵 𝐵

• The values of the batch size and learning rate are manually pre-
specified and not typically learned through model training.
o These parameters that are tunable but not updated in the training loop are
called hyperparameters.
 Linear regression: gradient descent
• Linear regression happens to be a learning problem where there is only one minimum
over the entire domain.
• For more complicated models, like deep networks, the loss surfaces contain many
minima.
• Deep learning practitioners seldom struggle to find parameters that minimize the loss on
training sets.
• The more formidable task is to find parameters that will achieve
low loss on data that we have not seen before.
• A challenge called generalization.
Basic Elements of Linear Regression

Linear Model Loss Function Analytic Solution

Making Predictions
Gradient Descent with the Learned
Model
Making Predictions with the Learned Model
• Given the learned linear regression model 𝒘 ෝ ⊤ 𝒙 + 𝑏෠ , we can estimate
the price of a new house given its area 𝑥1 and age 𝑥2 .
o Estimating targets given features is commonly called prediction or
inference.
 BITS Pilani
Pilani Campus

Logistic Regression
Regression vs. Classification
• Regression estimates a continuous value
• Classification predicts a discrete category
Logistic regression
• This is just like linear regression, except that the values y we want to
predict takes on only a small number of discrete values.
• For now, we will focus on the binary classification problem in which y
can take on only two values: 0 and 1.
• For instance, if we are trying to build a spam classifier for email, then
x (i) may be some features of a piece of email, and y may be 1 if it is a
piece of spam mail, and 0 otherwise.
Logistic regression
• We could approach the classification problem ignoring the fact that y
is discrete-valued, and use our old linear regression algorithm to try
to predict y given x.
• However, this method performs very poorly.
• Intuitively, it also doesn’t make sense to consider 𝑦ො values larger than
1 or smaller than 0 when we know that y ∈ {0, 1}.
Logistic regression
• Let’s change the form for our hypotheses for the prediction 𝑦.
ො
• We will choose
ෝ=σ(θT)
𝒚
• where

• is called the logistic function or the sigmoid function

The logistic/sigmoid function
Linear vs logistic regression
Logistic regression
Sources
• https://www.cs.bu.edu/fac/snyder/cs237/Lectures%20and%20Materi
als/Lecture%2023%20--%20Logistic%20Regression.pdf