Dimensionality reduction technique can be defined as, 

"It is a way of
converting the higher dimensions dataset into lesser dimensions
dataset ensuring that it provides similar information." These techniques
are widely used in machine learning for obtaining a better fit predictive
model while solving the classification and regression problems.

It is commonly used in the fields that deal with high-dimensional data,

such as speech recognition, signal processing, bioinformatics, etc. It
can also be used for data visualization, noise reduction, cluster
analysis, etc.

Benefits of applying Dimensionality Reduction

Some benefits of applying dimensionality reduction technique to the

given dataset are given below:

o By reducing the dimensions of the features, the space required to

store the dataset also gets reduced.
o Less Computation training time is required for reduced
dimensions of features.
o Reduced dimensions of features of the dataset help in visualizing
the data quickly.
o It removes the redundant features (if present) by taking care of
multicollinearity.

Disadvantages of dimensionality Reduction

There are also some disadvantages of applying the dimensionality

reduction, which are given below:

o Some data may be lost due to dimensionality reduction.

o In the PCA dimensionality reduction technique, sometimes the
principal components required to consider are unknown.

Approaches of Dimension Reduction

There are two ways to apply the dimension reduction technique, which
are given below:

Feature Selection

Feature selection is the process of selecting the subset of the relevant

features and leaving out the irrelevant features present in a dataset to
build a model of high accuracy. In other words, it is a way of selecting
the optimal features from the input dataset.

Three methods are used for the feature selection:

1. Filters Methods

In this method, the dataset is filtered, and a subset that contains only
the relevant features is taken. Some common techniques of filters
method are:

o Correlation
o Chi-Square Test
o ANOVA
o Information Gain, etc.

2. Wrappers Methods

The wrapper method has the same goal as the filter method, but it takes
a machine learning model for its evaluation. In this method, some
features are fed to the ML model, and evaluate the performance. The
performance decides whether to add those features or remove to
increase the accuracy of the model. This method is more accurate than
the filtering method but complex to work. Some common techniques of
wrapper methods are:

o Forward Selection
o Backward Selection
o Bi-directional Elimination
3. Embedded Methods: Embedded methods check the different training
iterations of the machine learning model and evaluate the importance of
each feature. Some common techniques of embedded methods are:

o LASSO
o Elastic Net
o Ridge Regression, etc.

Feature Extraction:

Feature extraction is the process of transforming the space containing

many dimensions into space with fewer dimensions. This approach is
useful when we want to keep the whole information but use fewer
resources while processing the information.

Some common feature extraction techniques are:

a. Component Analysis
b. Linear Discriminant Analysis
c. Kernel PCA
d. Quadratic Discriminant Analysis

Common techniques of Dimensionality Reduction

a. Principal Component Analysis

b. Backward Elimination
c. Forward Selection
d. Score comparison
e. Missing Value Ratio
f. Low Variance Filter
g. High Correlation Filter
h. Random Forest
i. Factor Analysis
j. Auto-Encoder

Principal Component Analysis (PCA)

Principal Component Analysis is a statistical process that converts the

observations of correlated features into a set of linearly uncorrelated
features with the help of orthogonal transformation. These new
transformed features are called the Principal Components. It is one of
the popular tools that is used for exploratory data analysis and
predictive modeling.

PCA works by considering the variance of each attribute because the

high attribute shows the good split between the classes, and hence it
reduces the dimensionality. Some real-world applications of PCA
are image processing, movie recommendation system, optimizing the
power allocation in various communication channels.

PCA Algorithm-
 
The steps involved in PCA Algorithm are as follows-
 
Step-01: Get data.
Step-02: Compute the mean vector (µ).
Step-03: Subtract mean from the given data.
Step-04: Calculate the covariance matrix.
Step-05: Calculate the eigen vectors and eigen values of the covariance
matrix.
Step-06: Choosing components and forming a feature vector.
Step-07: Deriving the new data set.

Given data = { 2, 3, 4, 5, 6, 7 ; 1, 5, 3, 6, 7, 8 }.
Compute the principal component using PCA Algorithm.

OR
 
Consider the two dimensional patterns (2, 1), (3, 5), (4, 3), (5, 6), (6, 7), (7,
8).
Compute the principal component using PCA Algorithm.
 
 OR
 
Compute the principal component of following data-
CLASS 1
X = 2, 3, 4
Y = 1, 5, 3
CLASS 2
X = 5, 6, 7
Y = 6, 7, 8

Solution-
 
We use the above discussed PCA Algorithm-
 
Step-01:
 
Get data.
The given feature vectors are-
 x1 = (2, 1)
 x2 = (3, 5)
 x3 = (4, 3)
 x4 = (5, 6)
 x5 = (6, 7)
 x6 = (7, 8)
 
  
Step-02:
 
Calculate the mean vector (µ).
 
Mean vector (µ)
= ((2 + 3 + 4 + 5 + 6 + 7) / 6, (1 + 5 + 3 + 6 + 7 + 8) / 6)
= (4.5, 5)
 
Thus,

 
Step-03:
 
Subtract mean vector (µ) from the given feature vectors.
 x1 – µ = (2 – 4.5, 1 – 5) = (-2.5, -4)
 x2 – µ = (3 – 4.5, 5 – 5) = (-1.5, 0)
 x3 – µ = (4 – 4.5, 3 – 5) = (-0.5, -2)
 x4 – µ = (5 – 4.5, 6 – 5) = (0.5, 1)
 x5 – µ = (6 – 4.5, 7 – 5) = (1.5, 2)
 x6 – µ = (7 – 4.5, 8 – 5) = (2.5, 3)
 
Feature vectors (xi) after subtracting mean vector (µ) are-
 
 
Step-04:
 
Calculate the covariance matrix.
Covariance matrix is given by-
 

 
Now,
 
 
Now,
Covariance matrix
= (m1 + m2 + m3 + m4 + m5 + m6) / 6
 
On adding the above matrices and dividing by 6, we get-
 
 
Step-05:
 
Calculate the eigen values and eigen vectors of the covariance matrix.
λ is an eigen value for a matrix M if it is a solution of the characteristic
equation |M – λI| = 0.
So, we have-
 

 
From here,
(2.92 – λ)(5.67 – λ) – (3.67 x 3.67) = 0
16.56 – 2.92λ – 5.67λ + λ2 – 13.47 = 0
λ2 – 8.59λ + 3.09 = 0
 
Solving this quadratic equation, we get λ = 8.22, 0.38
 Thus, two eigen values are λ1 = 8.22 and λ2 = 0.38.
 
Clearly, the second Eigen value is very small compared to the first Eigen
value.
So, the second Eigen vector can be left out.
 
Eigen vector corresponding to the greatest eigen value is the principal
component for the given data set.
So. we find the eigen vector corresponding to eigen value λ1.
 
We use the following equation to find the eigen vector-
MX = λX
where-
 M = Covariance Matrix
 X = Eigen vector
 λ = Eigen value
 
Substituting the values in the above equation, we get-
 

 
Solving these, we get-
2.92X1 + 3.67X2 = 8.22X1
3.67X1 + 5.67X2 = 8.22X2
 
On simplification, we get-
5.3X1 = 3.67X2 ……… (1)
3.67X1 = 2.55X2 ……… (2)
 
From (1) and (2), X1 = 0.69X2
From (2), the Eigen vector is-
 
Refer class notes for further calculation
 

 
Lastly, we project the data points onto the new subspace as-
 
Backward Feature Elimination

The backward feature elimination technique is mainly used while

developing Linear Regression or Logistic Regression model. Below
steps are performed in this technique to reduce the dimensionality or in
feature selection:

o In this technique, firstly, all the n variables of the given dataset are
taken to train the model.
o The performance of the model is checked.
o Now we will remove one feature each time and train the model on
n-1 features for n times, and will compute the performance of the
model.
o We will check the variable that has made the smallest or no
change in the performance of the model, and then we will drop
that variable or features; after that, we will be left with n-1 features.
o Repeat the complete process until no feature can be dropped.

In this technique, by selecting the optimum performance of the model

and maximum tolerable error rate, we can define the optimal number of
features require for the machine learning algorithms.

Forward Feature Selection

Forward feature selection follows the inverse process of the backward

elimination process. It means, in this technique, we don't eliminate the
feature; instead, we will find the best features that can produce the
highest increase in the performance of the model. Below steps are
performed in this technique:

o We start with a single feature only, and progressively we will add

each feature at a time.
o Here we will train the model on each feature separately.
o The feature with the best performance is selected.
o The process will be repeated until we get a significant increase in
the performance of the model.
Missing Value Ratio

If a dataset has too many missing values, then we drop those variables
as they do not carry much useful information. To perform this, we can
set a threshold level, and if a variable has missing values more than that
threshold, we will drop that variable. The higher the threshold value, the
more efficient the reduction.

Low Variance Filter

As same as missing value ratio technique, data columns with some

changes in the data have less information. Therefore, we need to
calculate the variance of each variable, and all data columns with
variance lower than a given threshold are dropped because low variance
features will not affect the target variable.

High Correlation Filter

High Correlation refers to the case when two variables carry

approximately similar information. Due to this factor, the performance of
the model can be degraded. This correlation between the independent
numerical variable gives the calculated value of the correlation
coefficient. If this value is higher than the threshold value, we can
remove one of the variables from the dataset. We can consider those
variables or features that show a high correlation with the target variable.

Random Forest

Random Forest is a popular and very useful feature selection algorithm

in machine learning. This algorithm contains an in-built feature
importance package, so we do not need to program it separately. In this
technique, we need to generate a large set of trees against the target
variable, and with the help of usage statistics of each attribute, we need
to find the subset of features.

Random forest algorithm takes only numerical variables, so we need to

convert the input data into numeric data using hot encoding.

Factor Analysis

Factor analysis is a technique in which each variable is kept within a

group according to the correlation with other variables, it means
variables within a group can have a high correlation between themselves,
but they have a low correlation with variables of other groups.
Auto-encoders

One of the popular methods of dimensionality reduction is auto-encoder,

which is a type of ANN or artificial neural network, and its main aim is to
copy the inputs to their outputs. In this, the input is compressed into
latent-space representation, and output is occurred using this
representation. It has mainly two parts:

o Encoder: The function of the encoder is to compress the input to

form the latent-space representation.
o Decoder: The function of the decoder is to recreate the output
from the latent-space representation.