NATURAL LANGUAGE PROCESSING

(IT3EA06)
- Dr. Kush Bhushanwar
UNIT - III
 Speech Processing: Speech and Phonetics,
 Vocal Organ,
 Phonological Rules and Transducer,
 Probabilistic Models,
 Spelling Error,
 Bayesian Method to Spelling,
 Minimum Edit Distance,
 Bayesian Method of Pronunciation Variation
SPEECH PROCESSING
 Speech processing and language technology contains lots of
special concepts and terminology.
 To understand how different speech synthesis and analysis
methods work we must have some knowledge of speech
production, articulatory phonetics, and some other related
terminology.
 The study of the pronunciation of words is part of the field
of phonetics, the study of the speech sounds used in the
languages of the world.
 Phonetics focuses on the production and classification of
the world’s speech sounds.
TYPES OF PHONETICS
 The production of speech looks at the interaction of
different vocal organs, for example, the lips, tongue
and teeth, to produce particular sounds.
 Articulatory phonetics: The study of how phones are
produced, as the various organs in the mouth, throat, and
nose modify the airflow from the lungs.
 Acoustic phonetics: The study of the physical properties
of the soundwaves created by the vocal organ (duration,
frequency, etc.).
 Auditory phonetics: The examination of how speech
sounds are perceived and identified by the hearer’s ear and
brain.
THE VOCAL ORGANS
 Most sounds in human languages are produced by
expelling air from the lungs through the windpipe
(technically the trachea) and then out the mouth or
nose.
 As it passes through the trachea, the air passes
through the larynx, commonly known as the Adam’s
apple or voicebox.
 The larynx contains two small folds of muscle, the
vocal folds (often referred to non-technically as the
vocal cords) which can be moved together or apart.
 The space between these two folds is called the
glottis.
 Sounds made with the vocal folds together and
vibrating are called voiced; sounds made without
this vocal cord vibration are called unvoiced or
voiceless.
 Voiced sounds include [b], [d], [g], [v], [z], and all the
English vowels, among others.
 Unvoiced sounds include [p], [t], [k], [f],[z], and
others.
PHONEME
 A phoneme is the smallest/indivisible and contrastive or
significant unit of the sound of a language, which, when
replaced with another sound, results in a change in
meaning.
 Therefore, a phoneme can not be subdivided into a smaller
unit.
 “Can” can be broken to /k/+/æ/+/n/ but can not further be
broken into smaller units.
ALLOPHONE
 An allophone is a phoneme that shares similarities of
pronunciation.
 More technically, allophones are phonemes, which share a
variety of pronunciations determined by phonetic
circumstances such as types of words, morphemes or
positions.
 Ashby and Maidment (2005:189) define an allophone as a
positional (initial, middle, final) variant of a phoneme due
to the position it occupies in a word, which calls for a slight
difference in pronunciation.
EXAMPLE
 the phoneme, /t/ and some of its variants
(allophones).
 /th/ aspirated as in teach
 /t/ unaspirated as in water
 /t/ /tn/ nasalize as in tin
 /tw/ labialised as in twice
 /t-/ unreleased as in great
BASIC TERMINOLOGIES
 We can represent the pronunciation of words in terms of units
called phones.
 The standard system for representing phones is the
International Phonetic Alphabet or IPA.
 An alternative English-only transcription system that uses ASCII
letters is the ARPAbet.
 A phoneme is a generalization or abstraction over different
phonetic realizations. Allophonic rules express how a phoneme
is realized in a given context.
PHONOLOGICAL RULE
 The relationship between a phoneme and its allophones is often captured
by writing a phonological rule.
 Here is the phonological rule for dentalization in the traditional notation
of Chomsky and Halle (1968):

 In this notation, the surface allophone appears to the right of the arrow,
and the phonetic environment is indicated by the symbols surrounding the
underbar ( _ ).
 These rules resemble the rules of two-level morphology but since they
don’t use multiple types of rewrite arrows, this rule is ambiguous between
an obligatory or optional rule. Here is a version of the flapping rule:
PHONOLOGICAL RULES AND TRANSDUCERS
 There are several different models of computational phonology that use
finite automata in various ways to realize phonological rules.
 As a first example, Figure shows a transducer that models the application
of the simplified flapping rule in (3.1):
(3.1)

Transducer for English Flapping:

ARPAbet ‘dx’ indicates a
flap, and the ‘other’ symbol means ‘any
feasible pair not used elsewhere in the
transducer’.
‘@’ means ‘any symbol not used
elsewhere on any arc’.
PROBABILISTIC MODELS
 Probabilistic models are an essential component of machine
learning, which aims to learn patterns from data and make
predictions on new, unseen data.
 They are statistical models that capture the inherent
uncertainty in data and incorporate it into their
predictions.
 Probabilistic models are used in various applications such
as image and speech recognition, natural language
processing, and recommendation systems.
 Examples: Bayesian networks, Gaussian mixture models,
hidden Markov models, and probabilistic graphical models.
TYPES OF PROBABILISTIC MODELS
o Generative models
o Discriminative models.
o Graphical models
GENERATIVE MODELS:
 Generative models aim to model the joint distribution of
the input and output variables.
 These models generate new data based on the probability
distribution of the original dataset.
 Generative models are powerful because they can generate
new data that resembles the training data.
 They can be used for tasks such as image and speech
synthesis, language translation, and text generation.
DISCRIMINATIVE MODELS:
 The discriminative model aims to model the conditional
distribution of the output variable given the input variable.
 They learn a decision boundary that separates the different
classes of the output variable.
 Discriminative models are useful when the focus is on
making accurate predictions rather than generating new
data.
 They can be used for tasks such as image recognition,
speech recognition, and sentiment analysis.
GRAPHICAL MODELS:
 These models use graphical representations to show the
conditional dependence between variables.
 They are commonly used for tasks such as image
recognition, natural language processing, and causal
inference.
NAIVE BAYES ALGORITHM IN PROBABILISTIC
MODELS
 Naive Bayes is a probabilistic algorithm that is used for
classification problems.
 It is based on the Bayes theorem of probability and
assumes that the features are conditionally independent of
each other given the class.
 It is also known as a probabilistic classifier. The Naive
Bayes Algorithm comes under supervised learning and is
mainly used to solve classification problems.
 For example, you cannot identify a bird based on its
features and color as there are many birds with similar
attributes.
NAIVE BAYES ALGORITHM
BAYESIAN PROBABILITY
 Bayesian Probability allows to calculate the conditional
probabilities.
 It enables to use of partial knowledge for calculating the
probability of the occurrence of a specific event.
 This algorithm is used for developing models for
prediction and classification problems like Naive Bayes.
 The Bayesian Rule is used in probability theory for
computing - conditional probabilities.
 What is important is that you cannot discover just how the
evidence will impact the probability of an event occurring,
but you can find the exact probability.
SPELLING ERROR
 A spellchecker points to spelling errors and
possibly suggests alternatives.
 An auto-corrector usually goes a step further and
automatically picks the most likely word.
 In case of the correct word already having been
typed, the same is retained.
 So, in practice, an autocorrect is a bit more
aggressive than a spellchecker, but this is more
of an implementation detail — tools allow you to
configure the behavior.
TYPES OF SPELLING ERRORS
 Non-word Errors: These are the most common type
of errors. You either miss a few keystrokes or let your
fingers hurtle a bit longer.
 E.g., typing langage when you meant language;
or hurryu when you meant hurry

 Real Word Errors: If you have fat fingers,

sometimes instead of creating a non-word, you end up
creating a real word, but one you didn’t intend.
 E.g, typing buckled when you meant bucked. Or your
fingers are a tad wonky, and you type in three when you
meant there.
TYPES OF SPELLING ERRORS CONT.
 Cognitive Errors: The previous two types of
errors result not from ignorance of a word or its
correct spelling.
 Cognitive errors can occur due to those factors.

 The words piece and peace are homophones

(sound the same). So you are not sure which one
is which.
 Sometimes your damn sure about your spellings
despite a few grammar nazis claim you’re not.
TYPES OF SPELLING ERRORS CONT.
 Short forms/Slang/Lingo: These are possibly
not even spelling errors. May be u r just being
kewl (cool). Or you are trying hard to fit in
everything within a text message or a tweet and
must commit a spelling sin.
 Intentional Typos: Well, because you are
clever. You type in teh and pwned and zomg
carefully and frown if they get autocorrected.
BAYESIAN METHOD TO SPELLING
 In this approach Bayes’ theorem is used to compute the
probability of the intended word being ‘w’ when the typist has
typed ‘x’:

x: input word the typist has typed (observed data)

w: the word they meant to type (parameter to estimate)
 This is called the posterior probability of ‘w’ being the intended
word.
 Then, the word in dictionary with the highest posterior
probability is chosen as the intended word:
BAYESIAN METHOD TO SPELLING
 However, it will be computationally inefficient and too expensive
to check w’s over all the dictionary.
 So, we should generate a set of candidates for any input word (x),
which we call C, and do the maximization over the C set:

 And since P(x) is a normalizing constant (independent from w) we

can drop it from the maximized function without any harm.
 We will also rank candidates according to log-posterior instead of
posterior probability:
DAMERAU-LEVENSHTEIN EDIT DISTANCE
MINIMUM EDIT DISTANCE
 Minimal edit distance between two strings,
where edits are:
 Insertion
 Deletion
 Replace
 Transposition of two adjacent letters