Machine Learning Question Bank
Unit 2
1. What do you mean by a linear regression? Which applications are best modelled by
linear regression? [5]
2. Write a short note on: Types of regression
3. Write short notes on: a) Linearly and non- linearly separable data b) ROC curve
4. Explain the Lasso, and ElasticNet types of regression.
5. Explain isotonic regression and write the applications in brief
6. Explain in detail the Ridge regression and the Lasso regression.
Solution
What do you mean by a linear regression?
Linear regression is one of the most basic types of regression in machine learning.
Linear regression may be defined as the statistical model that analyses the linear
relationship between a dependent variable with given set of independent variables.
The linear regression model consists of a predictor variable and a dependent
variable related linearly to each other.
Mathematically the relationship can be represented with the help of following
equation − Y = mX + b
Here, Y is the dependent variable we are trying to predict
X is the dependent variable we are using to make predictions.
m is the slop of the regression line which represents the effect X has on Y
b is a constant, known as the Y-intercept. If X = 0,Y would be equal to b.
Furthermore, the linear relationship can be positive or negative in nature
Positive Linear Relationship
A linear relationship will be called positive if both independent and dependent
variable increases. It can be understood with the help of following graph −
Negative Linear relationship
A linear relationship will be called positive if independent increases and dependent
variable decreases. It can be understood with the help of following graph −
Which applications are best modelled by linear regression?
1. Businesses often use linear regression to understand the relationship between
advertising spending and revenue.
2. Medical researchers often use linear regression to understand the relationship
between drug dosage and blood pressure of patients.
3. Agricultural scientists often use linear regression to measure the effect of
fertilizer and water on crop yields.
4. Data scientists for professional sports teams often use linear regression to
measure the effect that different training regimens have on player
performance.
Write a short note on: Types of regression
Types of Linear Regression
Linear regression is of the following two types −
Simple Linear Regression
Multiple Linear Regression
Simple Linear Regression (SLR)
It is the most basic version of linear regression which predicts a response using a
single feature. The assumption in SLR is that the two variables are linearly related.
Multiple Linear Regression (MLR)
It is the extension of simple linear regression that predicts a response using two or
more features.
9 Types of Regression Analysis
The types of regression analysis that we are going to study here are:
1. Simple Linear Regression
2. Multiple Linear Regression
3. Polynomial Regression: same as Multiple Linear Regression with a little
modification. In Polynomial Regression, the relationship between independent and
dependent variables, that is X and Y, is denoted by the n-th degree.
4. Logistic Regression: Logistic regression is one of the types of regression analysis
technique, which gets used when the dependent variable is discrete.
5. Ridge Regression: This is another one of the types of regression in machine
learning which is usually used when there is a high correlation between the
independent variables.
6. Lasso Regression: Lasso Regression is one of the types of regression in machine
learning that performs regularization along with feature selection. It prohibits the
absolute size of the regression coefficient.
7. Bayesian Linear Regression: uses the Bayes theorem to find out the value of
regression coefficients. In this method of regression, the posterior distribution of the
features is determined instead of finding the least-squares.
There are some algorithms we use to train a regression model to create predictions with
continuous values.
8. Decision Tree Regression: The decision tree as the name suggests works on the
principle of conditions. It is efficient and has strong algorithms used for predictive
analysis. It has mainly attributed that include internal nodes, branches, and a terminal
node.
9. Random Forest Regression: Random forest, as its name suggests, comprises an
enormous amount of individual decision trees that work as a group or as they say, an
ensemble. Every individual decision tree in the random forest lets out a class
prediction and the class with the most votes is considered as the model's prediction.
Write short notes on: a) Linearly and non- linearly separable data
There is a whole class of problems which are termed as linearly separable. This name
is given to them, because if we were to represent them in the input space, we could
classify them using a straight line. The simplest examples are the logical AND or OR.
If you can draw a line or hyper plane that can separate those points into two classes,
then the data is separable. If not, then it may be separated by a hyper plane in higher
dimensions. Still if any of the hyper planes could not separate them, then the data is
termed as non-linearly separable data.
A set of input vectors (or a training set) will be said to be linearly non-separable if no
hyperplane exists such that each vector lies on the pre-assigned side of the hyperplane.
b) ROC curve
In machine learning, the Receiver Operating Characteristic curve(ROC) is nothing
but a graph displaying the performance of a classification model. It is a very popular
method to measure the accuracy of a classification model. ROC curve is an
evaluation metric that measures the performance of a machine learning model by
visualizing, especially when data is skewed. It gives us the trade-off between
the True Positive Rate (TPR) and the False Positive Rate (FPR) at different
classification thresholds.
True Positive Rate:
True Positive Rate is the proportion of observations that are correctly predicted to be
positive.
False Positive Rate:
False Positive Rate is the proportion of observations that are incorrectly predicted to
be positive.
For different threshold values we will get different TPR and FPR. So, in order to
visualise which threshold is best suited for the classifier we plot the ROC curve.
Explain the Lasso, and ElasticNet types of regression.
LASSO (L1 regularization) : If a regression model uses the L1 Regularization technique,
then it is called Lasso Regression.
Lasso regression uses the L1 penalty term and stands for Least
Absolute Shrinkage and Selection Operator. The penalty
applied for L2 is equal to the absolute value of the magnitude of
the coefficients:
Similar to ridge regression, a lambda value of zero spits out the
basic OLS equation, however given a suitable lambda value
lasso regression can drive some coefficients to zero. The larger
the value of lambda the more features are shrunk to zero. This
can eliminate some features entirely and give us a subset of
predictors that helps mitigate multi-collinearity and model
complexity. Predictors not shrunk towards zero signify that they
are important and thus L1 regularization allows for feature
selection (sparse selection).
regularization term penalizes absolute value of the coefficients
sets irrelevant values to 0
might remove too many features in your model
Elastic Net
A third commonly used model of regression is the Elastic Net which
incorporates penalties from both L1 and L2 regularization:
In addition to setting and choosing a lambda value elastic net also
allows us to tune the alpha parameter where 𝞪 = 0 corresponds to
ridge and 𝞪 = 1 to lasso. Simply put, if you plug in 0 for alpha, the
penalty function reduces to the L1 (ridge) term and if we set alpha
to 1 we get the L2 (lasso) term. Therefore we can choose an alpha
value between 0 and 1 to optimize the elastic net. Effectively this
will shrink some coefficients and set some to 0 for sparse selection.
Explain isotonic regression and write the applications in brief
The word ‘isotonic’ has Greek root words origins, made of two parts, ‘iso’ and ‘tonic.’
Here, ‘iso’ means equal and ‘tonic’ means stretching. In terms of machine learning
algorithms, isotonic regression can, therefore, be understood as equal stretching
along the linear regression line. There are various types of regression models
(algorithms) that are used to train machine learning programs, such as linear, logistics,
ridge, and lasso regression. Of these, the linear regression model is the most basic and
most widely used regression model. Isotonic regression in machine learning is
based on linear regression. Isotonic regression is highly helpful if you have multiple
input variables. We can inspect each and every dimension as each and every function
and interpolate it in a linear way. This allows for easy multidimensional scaling.
A commonly used algorithm to obtain the isotonic regression solution is the pool-
adjacent-violators algorithm (PAVA). It runs in linear time and linear memory.
Explain in detail the Ridge regression and the Lasso regression.
There are mainly two types of regularization techniques, which are given below :
1. Ridge Regression (L-2 norm)
2. Lasso regression (L-1 norm)
Ridge Regression :
In this technique, the cost function of the linear regression is altered by adding the penalty
term (shrinkage term), which multiplies the lambda (hyperparameter) with the squared weight
of each feature. The cost function for ridge regression becomes:
Cost function for Ridge Regression (Image by author)
According to the above equation, the penalty term regularizes the coefficients or weights of the
model. Hence ridge regression reduces the magnitudes of the coefficients that will help in
decreasing the complexity of the model.
Lasso Regression :
Lasso stands for Least absolute and Selection Operator. It is another type of regularization
technique used to reduce the complexity of the model. It is similar to ridge regression except
that the penalty term includes the absolute weight instead of the square of weights. The cost
function lasso regression is given below :
Cost function for Lasso Regression (Image by author)
When lambda equals zero, the cost function of ridge or lasso regression becomes equal to RSS.
As we increase the value of lambda, the variance decreases, and bias increases. The slope of
the best fit line will get reduced and the line becomes horizontal. As this term is increased, the
model becomes less responsive to the independent variables. So, we have to choose the value
of lambda wisely by iterating it through a range of values and take one which gives the lowest
error.
The differences between Ridge and Lasso Regression:
In ridge regression, the complexity of the model is reduced by decreasing the magnitude
of coefficients, but it never sets the value of coefficients to absolute zero. Whereas, lasso
regression tends to make coefficients to absolute zero.
Limitations of Ridge and Lasso Regressions :
▹ Ridge regression does not help in feature selection.
▹ Ridge regression use to shrink the coefficients, but never sets their values as absolute zero.
The model will retain all the features and will remain complex, which may lead to poor model
performance.
▹ When we apply Lasso regression to a model which has highly correlated variables, then it
will retain only a few variables and sets other variables to be zero. That will lead to some loss
of information as well as lower accuracy of the model.
