Natural Language Processing

Feature Engineering on Text Data

 Learning Objectives

By the end of this lesson, you will be able to:

Explain N-gram

Demonstrate the different word embedding models

Perform operations on word analogies

Demonstrate the working of Bag-of-Words

Feature Extraction
 What Is Feature Extraction?

Clean Data Model

Computers do not have any standard representation of words

Once the text is cleaned and normalized, it needs to be

transformed into features which can be used for modeling
Feature
Extraction
 Feature Extraction Techniques

Feature extraction technique depends on what kind of model is

intended to be used.

TF-IDF
N-Gram
Girl
Male
Female
Boy

Bag-of-Words N-Gram Document-Term Matrix TF-IDF

N-Gram
 N-Gram: Introduction

N-grams are combinations of adjacent words or letters of length n in the source text.

Trigram
Group (contiguous sequence) of n
words or characters

Unigram Bigram

P(w | h) probability of N-Gram Probabilistic model n >= 1

word w given per of word sequence
history h n =1 Unigram

n =2 Bigram

Assigns probabilities to n =3 Trigram

the sequences of words . . .
. . .
n =n N-Gram
 N-Gram: Example

Example: This is a sentence

n = 1 (Unigram) This is a sentence

n = 2 (Bigram) This is a sentence

n = 3 (Trigram) This is a sentence

 N-Gram: Applications

Text Comparison

Spelling Error Detection

Information Retrieval

Applications

Automatic Text
Categorization

Spelling Error Correction

Autocomplete
Bag-of-Words
 Bag-of-Words

Used to perform document-level task

1

Is a vectorization technique to represent text data

2

Has no effect of grammar and order of words in sentence

3

Example Usage:

Sentiment Analysis Spam Detection

 Bag-of-Words

Male Girl
Boy Female

Processed Data
• Document
• Tweet Unordered
• Review comments collection of words
 Bag-of-Words

Bag-of-Words model is the way of extracting features from text and representing the
text data, while modeling the text with a machine learning algorithm.

Tokenization:
While creating the bag of
words, tokenized word of each
01 Tokenization observation is used.

Process:
• Collect data
• Create a vocabulary by listing all
Process 02 unique words
• Create document vectors after
scoring

Scoring mechanism:
Scoring Mechanism 03 • Word hashing
• TF-IDF
• Boolean value
 Bag-of-Words: Example

Apply Text Processing

I have a little daughter “littl”,”, ”daughter”
Inefficient

Mary had a little lamb “mari”,”littl,”lamb”

Difficult to compare
Twinkle twinkle little star “twinkl”,”littl,”star”

Multiple occurrences
The silence of lambs “silence”,”lamb” of word: difficult to
handle

Corpus (D): Set of

Documents
 Bag-of-Words: Example

I have a little daughter Vocabulary (V)

Mary had a little lamb

“littl”, “daughter”,“mari”,”lamb”,”twinkl”, “star”, ”silenc”
Twinkle twinkle little star

Collect unique words

The silence of lambs

Corpus (D): Set of

Documents
 Bag-of-Words: Vector Representation Example

Term or Word
daughter lamb littl mari star silenc twinkl
1 0 1 0 0 0 0
I have a little daughter
0 1 1 1 0 0 0
Mary had a little lamb
0 0 1 0 1 0 2
Twinkle twinkle little star
0 1 0 0 0 1 0
The silence of lambs

Term Frequency Document-Term Matrix

Corpus (D): Set of
Documents Frequency of a term or
word-occurrence in a document
 Bag-of-Words: Recap of Terms Used

Term Term Frequency

Each processed word is Frequency of a

called term term-occurrence in a
document

Term Matrix

Matrix showing frequency of

each term-occurrence in
documents
TF-IDF
 TF-IDF

The Term Frequency-Inverse

Document Frequency is
abbreviated as TF-IDF

• Bag-of-Words assumes that each word is equally important

• In real-world scenario, each word has its own weight based on the context

Example:
• Cost occurs more frequently in an economy related document. To overcome
this limitation TF-IDF is used which assigns weights to the words based on
their relevance in the document.
 TF-IDF

It has two parts:

It represents the numerical
• Term Frequency (TF)
statistics
• Inverse Document Frequency
(IDF)

Applications of TF-IDF are:

• Text Mining
• User Modeling
 TF-IDF: Example

Random Forest is an ensemble learning method machine technique application of ai

Doc1 1 1 1 1 1 1 1 0 0 0 0 0

Doc2 0 0 1 0 1 1 1 1 1 0 0 0

Doc3 0 0 1 1 0 1 0 1 0 1 1 1

Document 1 1 3 2 2 3 2 2 1 1 1 1
Frequency

Sum of occurrence of a word across documents

 TF-IDF: Example

Random Forest is an ensemble learning method machine technique application of ai

Doc1 1/1 1/1 1/3 1/2 1/2 1/3 1/2 0/2 0/1 0/1 0/1 0/1

Doc2 0/1 0/1 1/3 0/2 1/2 1/3 1/2 1/2 1/1 0/1 0/1 0/1

Doc3 0/1 0/1 1/3 1/2 0/2 1/3 0/2 1/2 0/1 1/1 1/1 1/1

1 1 3 2 2 3 2 2 1 1 1 1

Document Frequency

Term Frequency

Sum of occurrence of a word across documents

 TF-IDF: Example

Random Forest is an ensemble learning method machine technique application of ai

Doc1 1 1 1/3 1/2 1/2 1/3 1/2 0 0 0 0 0

Doc2 0 0 1/3 0 1/2 1/3 1/2 1/2 1 0 0 0

Doc3 0 0 1/3 1/2 0 1/3 0 1/2 0 1 1 1

Term Frequency • Is proportional to frequency of occurrence of a word or term in a document

• Is inversely proportional to the number of documents in which a word or
term occurs
 TF-IDF: Example

Random Forest is an ensemble learning method machine technique application of ai

Doc1 1 1 1/3 1/2 1/2 1/3 1/2 0 0 0 0 0

Doc2 0 0 1/3 0 1/2 1/3 1/2 1/2 1 0 0 0

Doc3 0 0 1/3 1/2 0 1/3 0 1/2 0 1 1 1

Term Frequency • Highlights the words or terms which are unique to the document
• These words are better for characterizing
 TF-IDF

TF-IDF = TF(t,d) * IDF(t,D)

t is terms
d is document

TF = Term Frequency
IDF = Inverse Document Frequency
TF= count(t,d)
Count of term ‘t’ in document ‘d’
--------------
|d| Total number of terms in document ‘d’

IDF = log(|D|) Log of total number of documents in collection ‘D’

-------------
|{d ⊂ D} : {t ⊂ d}
Number of documents where ‘t’ is present
 TF-IDF

Term Frequency (TF)

TF Frequent occurrence of a term in a document is measured by term frequency.
TF (t, d) = Number of times t appears in document d / Total number of terms in
the document d

Inverse Document Frequency (IDF)

IDF IDF measures how important a term is.
IDF (t) = Log_e (Total number of documents / Number of documents
with term t in it)

TF-IDF = TF (t,d) * IDF (t)

t is term
d is document
One-Hot Encoding
 One-Hot Encoding

Used for deeper analysis of text

1

Performs numerical representation of each word

2

Used for categorical data

3

Higher the distinct categorical value, higher the sparsity

4
 One-Hot Encoding

Assigns vector value 1

Treats each word
where the particular word
as class
is present and 0 at other
places

How does it work?

 One-Hot Encoding: Example

daughter lamb littl mari star silenc twinkl

lamb 0 1 0 0 0 0 0

littl 0 0 1 0 0 0 0

silenc 0 0 0 0 0 1 0

twinkl 0 0 0 0 0 0 1
Word2vec
 Word2vec

Word2vec is one of
Word2vec is a
the most popular
two-layer neural
techniques of word
network.
embedding.
MALE

Word2vec

FEMALE

Two flavors of
Input is text corpus
algorithm:
and output is set of
• Continuous
vectors.
Bag-of-Words
(CBOW)
• Skip-Gram
 Word2vec

The core concept of Word2vec approach is to predict a word with the given neighboring word
or predict a neighboring word with the given word which is likely to capture the contextual
meaning of the word.

The quick brown fox jumps over the lazy dog

Context Context

Focus Word
 Word2vec Algorithms

Word2vec Algorithms

Continuous
Skip-Gram
Bag-of-Words (CBOW)

Predict a “neighboring word” Predict a “given word” given the

given the “given word” “neighboring word”
 Skip-Gram Model

It is used to predict the source context words given in a target word.

w(t-2)

w(t-1)

w(t)

w(t+1)

w(t+2)
 Skip-Gram Model: Example

0
brown

0 w(t-2)
0

0
0 fox
0
1 Neural Network
Jumps (or any other 1 w(t-1)
0
probabilistic model)
0
0
0
over
w(t) 1

One-hot encoded 0 w(t+1)

vector
0

1
the
0

0 w(t+2)
 CBOW Model

Common Bag-of-Words (CBOW) algorithm is used to predict the target word in the given context.

w(t-2)

w(t-1)
sum

w(t)

w(t+1)

w(t+2)
 Word2vec: Advantages

Ready to be used in deep Meaning of word is

learning-ready architecture distributed in vector

Train vectors are Vector size does not

reused grow with vocabulary
Word2vec Model Creation

Problem Statement: In vector space model, the entities are transformed into vector
representation. Based on the co-ordinate points, we can apply the techniques to find the most
similar points in vector space. Create a word-to-vector model which gives you the similar word for
happy.

Access: Click on the Practice Labs tab on the left side panel of the LMS. Copy or note the
username and password that is generated. Click on the Launch Lab button. On the page that
appears, enter the username and password in the respective fields, and click Login.
Doc2vec Model
 Doc2vec Model

The following are the uses of Doc2vec model:

• Creates numeric representation of a document

• Uses unsupervised algorithm
• Finds similarity between sentences, paragraphs, and documents

Classifier on

Average or Concatenate

Paragraph Matrix W W W

Paragraph id the cat sat

 Doc2vec Model

• It is an extension of CBOW model.

• It is called distributed memory version of paragraph vector.

• This algorithm may not be the ideal choice for the corpus with lots
of misspellings like tweets.
Principal Component Analysis (PCA)
 Principal Component Analysis (PCA)

It is a dimensionality reduction method that reduces the number of variables.

Standardization: Recasting:
05
Standardize the range of Recast the data along with
continuous variables the principal component
axes
01 04

Feature vector:
Find the principal
Covariance matrix components in the order
computation: of significance
Understand how variables
vary from mean
02 03
Eigenvector and values
computation:
Determine principal
components of the data
 Principal Component Analysis: Steps

Feature Vector Standardization

Eigenvectors and Covariance Matrix

Eigenvalues Computation Computation
 Step 1: Standardization

Standardize the range of continuous variables for their equal contribution

1

Higher range will dominate, which will create a bias

2

After standardization is done, all the variables will be on the same scale
3

It can be achieved by z = (value - mean) / std deviation

4
 Step 2: Covariance Matrix Computation

It is used to identify the relationship between the variables

1

Variables should not be highly correlated

2

Covariance matrix (n x n) is calculated where n is number of dimensions

3
Step 3: Eigenvectors and Eigenvalues Computation

It is used to determine the principal components

1

New variables are constructed as linear combinations of initial variables

and are called principal components
2

New variables will have less correlated data

3
 Step 4: Feature Vector

Decision is taken to keep all components or remove lesser significant

variables
1

Remaining components will form the matrix of vectors

2
 Principal Component Analysis

Two-dimensional data transformation after applying PCA:

 Principal Component Analysis

It is the process to automatically identify

topics present in text object.

Topic 1
It is an unsupervised approach Topic 2
that involves techniques such as: Topic 3
• TF-IDF Topic 4
Word • Non-negative matrix
Analogies factorization
• Latent Dirichlet Allocation
• LSA

Applications include:
• Document clustering
• Information retrieval
Latent Dirichlet Allocation (LDA)
 Latent Dirichlet Allocation (LDA)

LDA is a matrix
factorization technique.

For each word w of each

doc d, word assignment is Documents will be
updated till the represented as
convergence point. document-term matrix.

LDA converts
M2 is a topic-term document-term matrix into
matrix. two lower-dimensional
matrix, M1and M2.
M1 is a document-topic
matrix.
 Latent Dirichlet Allocation: Example

Term/Word Bag of Word Model

daughter lamb littl mari star silenc twinkl

D1 1 0 1 0 0 0 0
I have a little daughter

D2 0 1 1 1 0 0 0 Document
Mary had a little lamb
Term
Matrix
D3 0 0 1 0 1 0 2
Twinkle Twinkle little star

The silence of lambs 0 1 0 0 0 1 0

Dn

For D3 P(t|d)=1/4 ¼ 2/4

Corpus (D) : Set of Probability of word occurring in Document No. of parameters: 3

Documents
For 1 document and 3 words, number of parameters are = 1*3=3
 Latent Dirichlet Allocation: Example

1000 documents(d) …………………………………..

T1 t2 t3 t4…………………………………………………………….t1000
5000 terms/words (t)

For 1000 documents and 5000 words, number of parameters

Parameters P(t|d) are = 1000*5000=5000000 (50 Lakhs)

Problem:
There are so many parameters to extract information and so, the
task is to reduce number of parameters without losing
information
 Latent Dirichlet Allocation: Example

Solution: Topic is a mix of terms that is likely to generate the

Introduce a layer of topics called the Latent Variable term.
Example: Finance, Science, Sport, etc.
LDA Model

1000
documents(d) …………………………………..

Probability of
P(z|d) topic z given
document d

Topics/Latent
Z1 z2 z3
Variable (z)

Probability
P(t|z)
of term t
given topic z

5000 T1 t2 t3 t4…………………………………………………………….t1000

Terms/Words (t)

For 1000 documents, 5000 words, 10 topics, the number of parameters

are = 1000*10+10*5000=60000
 Latent Dirichlet Allocation: Example

LDA Model M1

Z1 Z2 .. zn
1000 d1
documents (d) ………………………
d2
…………..
Probability of d3
P(z
topic z given d4
|d)
document d
…

dn
Topics/Latent Z1 z2
Variable (z) z3……………………………zn

P(t Probability of M2
|z) term t given
topic z
t1 t2 t3 t4 t5 …. tn
5000 T1 t2 t3 t4……………………………tn
z1
terms/words (t)
z2

..

zn
 Latent Dirichlet Allocation: Example

LDA Model
Bag-of-Word Model

1000 …………………………
documents(d) ………………………… ………..
………..

Z1 z2 z3
Topics

5000 T1 t2 t3 t4………………….t1000
terms/words (t) T1 t2 t3 t4………………t1000

Parameters
50 Lakhs 60 Thousand
P(t|d)
Topic Modeling
 Topic Modeling

It is a type of statistical model and has the following advantages:

Discovering the abstract Document

topics in a collection of clustering
documents

Information retrieval from Organizing large blocks of

unstructured text and textual data
feature selection
 Topic Modeling: Industry Use Cases

HR Search Engine

News Companies E-Commerce

Document Sorting
Gensim
 Gensim: Introduction

Gensim is a free python library which is platform-independent.

1

It is open-source.
2

It is robust and scalable.

3

It analyzes plain-text documents for semantic structure.

4

It is used to retrieve semantically similar documents.

5
 Gensim: Syntax and Library

System Requirement:

Operating system:
macOS / OS X · Linux ·
Windows

Python version:
Python >=2.7 >> import gensim

Dependency:
• NumPy >= 1.11.3
• SciPy >= 0.18.1
• Six >= 1.5.0
• smart_open >= 1.2.1
 Gensim: Vectorization

#Gensim Library
#Load Gensim
from gensim import corpora

#documents for building vocabulary

documents = ["Simplilearn is an ed-tech company",
"We provide multiple e-learning courses"]

#text processing
texts = [[word
for word in document.lower().split()]
for document in documents
]
#convert into dictionary
dictionary = corpora.Dictionary(texts)
#document to convert in vector
new_doc = "ed-tech company for e-learning courses"
#document to bag of words conversion
new_vec = dictionary.doc2bow(new_doc.lower().split())
print(new_vec)
 Gensim: Vectorization

Output: [(1, 1), (2, 1), (5, 1), (6, 1)]

 Gensim: Topic Modeling

#Gensim library
#Loading gensim
from gensim.test.utils import common_texts
from gensim.corpora.dictionary import Dictionary
from gensim.models.ldamodel import LdaModel
#create a corpus from a list of text
common_dictionary = Dictionary(common_texts)
common_corpus = [common_dictionary.doc2bow(text) for text in common_texts]
#Train the model
lda = LdaModel(common_corpus, num_topics=10)
#new corpus of unseen documents
other_texts = [
['data', 'unstructured', 'time'],
['bigdata', 'intelligence', 'natural'],
['language', 'machine', 'computer']
]
other_corpus = [common_dictionary.doc2bow(text) for text in other_texts]
unseen_doc = other_corpus[0]
#get topic probability distribution for a document
vector = lda[unseen_doc]
print(vector)
 Gensim: Topic Modeling

Output:
[(0, 0.050000038), (1, 0.5499996), (2, 0.050000038), (3, 0.05000004), (4, 0.050000038),
(5, 0.050000038), (6, 0.05000004), (7, 0.05000004), (8, 0.05000004), (9, 0.050000038)]
Word Embedding
 Word Embedding

Use the following while working with individual words or phrases:

Text Generation Machine Translation

Large Vocabulary
 Word Embedding

It represents text in the

N-dimensional space, in the Vectors are called
form of vectors embeddings

It is the distributed Each word is mapped to

representation one real-valued vector

Word embedding
techniques: Applications of word embedding:
• Word2vec • Music or video
• Glove recommendation system
• Analyzing survey responses
 Word Embedding: Overview

Child
kid

• Word embedding represents word in vector form

Lion
• Some properties must be exhibited while representing a
word in vector form:
o Similar meaning words should be closer to each other School

when compared to the words which don’t have similar

meaning
o Words having difference in meaning should be kept at
the same distance from each other
Queen

• This kind of representation helps in finding:

King
o Analogy word
o Synonym Woman
o Classification of the word: Positive, negative or neutral Man
Identify Topics from News Items

Problem Statement: Identification of document for a domain or keyword is a tough task. Write a script
which will provide the important topics from the news data.
Access: Click on the Practice Labs tab on the left side panel of the LMS. Copy or note the
username and password that is generated. Click on the Launch Lab button. On the page that
appears, enter the username and password in the respective fields, and click Login.
Working of Word Analogies

Problem Statement: Apply word analogies technique using word2vec for identification of new next
word.
Access: Click on the Practice Labs tab on the left side panel of the LMS. Copy or note the
username and password that is generated. Click on the Launch Lab button. On the page that
appears, enter the username and password in the respective fields, and click Login.
Build Your Own News Search Engine

Objective: Use text feature engineering (TF-IDF) and some rules to make
our first search engine for news articles. For any input query, we’ll
present the five most relevant news articles.

Problem Statement: Reuters Ltd. is an international news agency

headquartered in London and is a division of Thomson Reuters. The
data was originally collected and labeled by Carnegie Group Inc. and
Reuters Ltd. in the course of developing the construe text
categorization system. An important step before assessing similarity
between documents, or between documents and a search query, is the
right representation i.e., correct feature engineering. We’ll make a
process that provides the most similar news articles to a given text
string (search query).
 Key Takeaways

You are now able to:

Explain N-gram

Demonstrate the different word embedding models

Perform operations on word analogies

Demonstrate the working of Bag-of-Words

Demonstrate the working of top modeling technique

Knowledge Check
Knowledge
Check How many bigrams can be generated from the given sentence?
“Simplilearn is a great source to learn machine learning”
1

a. 7

b. 8

c. 9

d. 10
 Knowledge
Check How many bigrams can be generated from given sentence?
“Simplilearn is a great source to learn machine learning”
1

a. 7

b. 8

c. 9

d. 10

The correct answer is b

Bigrams: Simplilearn is, is a, a great, great source, source to, to learn, learn machine, machine learning
Knowledge
Check The main advantages of document-term matrix are:

a. Feature engineering

b. Understanding the frequency of word

c. Converting text into vectors

d. All of the above

 Knowledge
Check The main advantages of document-term matrix are:

a. Feature engineering

b. Understanding the frequency of word

c. Converting text into vectors

d. All of the above

The correct answer is d

Document-term matrix converts sentences into vectors, and it is achieved by creating matrix of unique
words of sentences.
Knowledge
Check Highest distance in the Levenshtein approach depicts:

a. More similar words

b. More dissimilar words

c. Cannot decide the distance

d. Depends on the length of words

 Knowledge
Check Highest distance in the Levenshtein approach depicts:

a. More similar words

b. More dissimilar words

c. Cannot decide the distance

d. Depends on the length of words

The correct answer is b

Highest distance in the Levenshtein approach depicts more dissimilar words.
Knowledge
Check What is the purpose of topic modeling?

a. Clustering the documents

b. Converting text into vectors

c. Understanding the frequency of word

d. Vectorization
 Knowledge
Check What is the purpose of topic modeling?

a. Clustering the documents

b. Converting text into vectors

c. Understanding the frequency of word

d. Vectorization

The correct answer is a

Topic modeling provides the topic which is used to map the documents.
Knowledge
Check Which techniques are used to find the similarity between text?

a. Cosine, Levenshtein, Document-Term Matrix

b. Cosine, Word2vec, Document-Term Matrix

c. POS, Document-Term Matrix, Levenshtein

d. Cosine, Levenshtein, Word2vec, POS

 Knowledge
Check Which techniques are used to find the similarity between text?

a. Cosine, Levenshtein, Document-Term Matrix

b. Cosine, Word2vec, Document-Term Matrix

c. POS, Document-Term Matrix, Levenshtein

d. Cosine, Levenshtein, Word2vec, POS

The correct answer is d

Cosine, Levenshtein, Word2vec, and POS are the techniques used to find the similarity between text.
Thank You