UNIT-4: FEATURE
ENGINEERING
Prof. Atmiya Patel
Feature Transformation
■ It conforms the assumption of a model.
■ Important tool for dimensionality reduction.
■ Two goals of feature transformation:
– Achieving best reconstruction in the original features.
– Achieving highest efficiency in the learning task.
■ It can apply to numeric features or non-numeric. (Like: text and
images)
Feature Construction
■ The process discovers missing information about the relationships
between features and expands the feature space by creating
additional features.
■ Added more features.
■ Techniques:
– Quantization or Binning
– Log Transform
– Feature Scaling or Normalization etc...
Quantization or Binning
■ The original data values which fall into a given small interval, a bin,
are replaced by a value representative of that interval, often the
central value. It is a form of quantization.
■ Statistical data binning is a way to group numbers of more or less
continuous values into a smaller number of "bins"
Log Transform
■ Power tool for dealing with large positive number with a heavy-tailed
distribution.
■ It compress the long tall in the heigh end of the distribution into a
shorter tail and expands the low end into a longer head.
Feature Scaling / Normalization
■ Some feature are bounded in value and other numeric features
increase without bound are affected by the scale of the input.
■ If the model is sensitive to the scale of input feature, feature scaling
could help.
■ It also called feature normalization.
■ It done individually to each feature.
Min-Max Scaling
Variance Scaling
■ Standardization (or Z-Score normalization or Variance scaling) scales
the values taking standard deviation of the features into account.
■ The mean of the feature is subtracted and divided by the standard
deviation of the feature.
■ The resulting feature has a mean 0 and a standard deviation of 1.
■ If the original feature is a normal distribution then the scaled feature
is also normal distribution.
l2 Normalization
■ It normalize the original feature value by l2 norm. It also known as the
Euclidean norm.
■ The l2 norm measures the length of the vector in coordinate space.
Encoding Categorical Variables
■ A categorical variable is used to represent categories or labels.
■ Large categorical variable are particularly common in transactional
records. Like IP address.
■ Even though user IDs and IP address are numeric, Their magnitude is
not relevant to the task.
■ The IP address might be relevant when doing fraud detection on
individual transaction.
■ The categories of a categorical variable are usually not numeric. So an
encoding method is needed to turn these non- numeric categories in
to numbers.
One-Hot Encoding
■ It creates new(binary) columns, indicating the presence of each
possible value from the original data.
■ Each bit represents a possible category.
■ One-Hot Encoding is simple but uses more bit than it strictly
necessary.
■ The sum of all the bits must be equal to 1.
Dummy Coding
■ The Problem with One-Hot encoding is that it allows for k degrees of
freedom, while the variable itself needs only k-1.
■ Dummy coding removes the extra degree of freedom by using only k-
1 features in the representation.
■ One feature is disregarded and is represented by the vector of all
zeros. This is known as the reference category.
■ The column “Blue” is deleted as it contained 0 for the first two rows.
■ The last row has both “Black” and “Brown” as 0 meaning that “Blue”
must be 1.
Feature Hashing
■ Large categorical features, such as user ID, website URL, IP address
etc., pose computation challenges in terms of memory efficiency and
storage.
■ To overcome this problem, Feature Hashing is used that makes
working with large categorical variables less computation intensive
and yet produces accurate models that fast to train.
■ Hashing, in general, is the process of taking any length of input
information and finding a unique fixed length representation of that
input information.
■ It is the process of finding a unique message digest (or hash value)
that corresponds to the input information.
■ It can be used in several different domains such as information
security, cryptocurrency, high-performance programming and for
creating quick lookup tables.
■ In machine learning, hash functions can be constructed for any object
that can be represented numerically. Like numbers, strings, complex
structures, etc.
Handling Textual Features
■ Need to apply machine learning on textural features such as product
reviews, comments, story line, news reports, etc.
■ List of techniques:
– Bag-of-Words
– Bag-of-n-Grams
Feature Extraction
■ It is the process of extracting or creating a new set of features from the
current dataset using some functional mapping.
■ It use for dimensionality reduction.
■ This can be having supervised and unsupervised.
■ Popular methods for the feature extractions are:
– Principal Components Analysis (PCA)
– Singular Value Decomposition (SVD)
– Linear Discriminant Analysis (LDA)
■ Both are linear projection methods. PCA is unsupervised and LDA is
supervised method.
Feature Subset Selection
■ Feature selection technique discards unnecessary features to reduce
the complexity of the resulting model.
■ Similar activity as dimensionality reduction.
■ The goal is to prudent model that is fast to compute, with little or no
degradation in predictive accuracy.
Key Drivers of Feature Selection
■ Which feature is going to be select?
■ Which feature to exclude?
■ Two key drivers for selecting features.
– Feature Relevance
– Feature Redundancy
Feature Relevance
■ Any feature, which is irrelevant in the context of machine learning
task on hand, is a potential candidate for rejection when selecting
subset of features.
■ Done by case-by-case basis.
■ In this “Name” feature is the most
irrelevant feature for age prediction.
Feature Redundancy
■ A feature may contribute information which is similar to the
information contribution by one or more features in the same data
set.
■ All features having potential redundancy are candidates for rejection
in the final feature subset.
■ The “Site length”, “Site Breadth” and “Site Area”
Reveal the dimensions of the site and can be
removed.
Overall Feature Selection Process
■ Generation of possible subsets.
■ Subset evaluation
■ Stop searching based on some stopping criterion
■ Validation of the result with respect to the chosen subsets.
Feature Selection Approaches
1. Filter:- Features are pre-processed to remove the ones that are
unlikely to be useful for the model.
2. Wrapper:- Allow to try out subsets of features.
3. Hybrid:- Takes the advantages of both.
4. Embedded:-Performs feature selection as part of the model training
process.
Thank you…
