Naïve Bayes
Naïve Bayes is a supervised machine learning algorithm which is used to classify data
into predefined classes. It uses the concept of conditional probability to classify the test
data.
Conditional Probability: - It help us to find the probability that something will happen
given that something else is happened.
Consider two events namely A & B. then,
[ P(A/B) = P(A)*P(B/A) ]
Here, P(B/A) is the probability that “B” happens given that “A” is already happens.
The fundamental assumptions of Naïve Bayes are that each feature makes an
independent & equal contribution to the outcome.
Theorem: - Bayes theorem finds and helps us to know how often “A” happens given
that “B” has already happened i.e. P(A/B) is only when we know how often “B” happens
given that “A” is already happened i.e. P(B/A).
In other words, we can say that, it basically finds the probability of an event occurring
based on the probability of an event that has already occurred.
Mathematically we can state the theorem as: -
𝐵
𝐴 𝑃 ∗ 𝑃(𝐴)
𝐴
𝑃 =
𝐵 𝑃(𝐵)
P(B) != 0
B is called as an evidence.
P(A) is known as priori of A. i.e. probability of an event before evidence is seen.
P(A/B) is known as posteriori probability of B. i.e. probability of an event after evidence
is seen.
In simple terms, we can say that Naïve Bayes finds the probability of every feature then
it selects the outcome with highest probability.
The dataset we keep for Naïve Bayes algorithm is divided into two parts namely:
1. Feature Matrix: - It contains all the vectors (rows) of dataset in which each
vector consists of the value of dependent features.
2. Response Vector: - It contains the value of class variable (prediction or output)
for each row of feature matrix.
Now, let’s discuss the working of Naïve Bayes algorithm mathematically with the help
of an example: -
S.No. Colour Type Origin Stolen (Class)
1. Red Sports Domestic Yes
2. Yellow Sports Domestic No
3. Red SUV Imported No
4. Red Sports Imported Yes
5. Yellow SUV Imported Yes
6. Red Sports Imported Yes
In the above dataset, we have {“Colour”, “Type”, “Origin”} are features and {“Stolen”} is
class variable.
Now, let’s find the output result of Naïve Bayes algorithm on above dataset by using the
following steps: -
Step – 1: - Probability of class variable: -
4 2 2 1
𝑃(𝑦𝑒𝑠) = = 𝑎𝑛𝑑 𝑃(𝑁𝑜) = =
6 3 6 3
Step – 2: - Probability of every features w.r.t each class variable
1. For Colour feature
𝑃 = 𝑃 =
𝑃 = 𝑃 =
2. For Type feature
𝑃 = 𝑃 =
𝑃 = 𝑃 =
3. For Origin feature
𝑃 = 𝑃 =
𝑃 = 𝑃 =
Step – 3: - Classify the unknown data based on conditional probability formula of Bayes
theorem.
X = <Yellow, SUV, Domestic>
For this we have to create a classifier model and we will find the probability of given
unknown dataset for all possible values of the class variable and pick up the output with
maximum probability. i.e.
𝑥
𝑌 = 𝑎𝑟𝑔𝑚𝑎𝑥 𝑃(𝑦) ∗ 𝑃
𝑦
Here, P(y) is class probability and 𝑃 is conditional probability.
Now final calculation will be:
𝑋 𝑌𝑒𝑙𝑙𝑜𝑤 𝑆𝑈𝑉 𝐷𝑜𝑚𝑒𝑠𝑡𝑖𝑐 1 1 1 1
𝑃 =𝑃 ∗𝑃 ∗𝑃 = ∗ ∗ =
𝑌𝑒𝑠 𝑌𝑒𝑠 𝑌𝑒𝑠 𝑌𝑒𝑠 4 4 4 64
𝑋 𝑌𝑒𝑙𝑙𝑜𝑤 𝑆𝑈𝑉 𝐷𝑜𝑚𝑒𝑠𝑡𝑖𝑐 1 1 1 1
𝑃 =𝑃 ∗𝑃 ∗𝑃 = ∗ ∗ =
𝑁𝑜 𝑁𝑜 𝑁𝑜 𝑁𝑜 2 2 2 8
Clearly, we can say that 𝑃 <𝑃
Thus, we can say that the SUV car of yellow colour having domestic origin has not been
stolen.
Gaussian Naïve Bayes Classifier
In this classifier, continuous values are associated with each feature are assumed
to be distributed according to a Gaussian distribution.
Q. What is Gaussian Distribution?
Ans: - It is a distribution plot which when plotted gives a bell-shaped curve which is
symmetric about the mean of the feature values.
The likelihood of the features in Gaussian distribution is assumed to be Gaussian in
nature, hence the conditional probability is given by:
⎡ ⎤
𝑥 1
⎢𝑃 = 𝑒 ⎥
⎢ 𝑦 ⎥
2𝜋𝜎
⎣ ⎦
There are another popular Naïve Bayes Classifiers are available in Machine Learning
fraternity. Some of these are
1. Multinomial Naïve Bayes Classifier
2. Bernoulli Naïve Bayes Classifier
Pro: -
1. Requires a small amount of training data
2. Easy and fast to predict the class variable
Cons: -
1. Does not work well if we have a correlated data
2. It requires some evidence to work on
3. Not suitable for larger set
Applications: - Document Classification, Spam Filtration, etc.
