Roll No.
Name of the student
10271023026 Satyanand Thapa
10271023009 Ayush Das
10271023004 Ajay Gupta
10271023028 Sima Mondal
LET’s GET STARTED
S ign Language R ecognition:
1 Introduction 2 Recognition Techniques
An overview of sign language recognition and its Exploring various approaches for recognizing signs.
importance.
3 Architectural Design 4 Gesture Recognition Algorithms
A UML-based design of a sign language recognition Deep dive into algorithms used for gesture detection.
system.
5 Evaluation and Metrics Challenges and Future Directions
6
Evaluating the system's performance with metrics. Addressing current challenges and exploring future
advancements.
7 Cost Analysis
8 References
Estimate the cost
.
Problem Statement
Sign language consists of 3 major components
1. Finger Spelling 2. Word level sign vocabulary
Used to spell words letter by Used for the majority of
letter. Communication
3. Non-manual feature
Facial expressions and tongue,
mouth and body position,
Problem Statement: Sign Language and its Importance
Communication Barrier Empowering Communication
1. Sign language is a complex visual 1. More than 70 million deaf people around
language, often inaccessible to those the world use sign languages to
who don't understand it. communicate.
2. Sign language allows them to learn, work,
2. This presents a communication barrier
access services, and be included in the
for the Deaf and hard of hearing
communities.
community.
Sign Language Recognition
using Machine Learning
This presentation explores the exciting potential of
using Convolutional Neural Networks (CNNs) to
empower communication for the Deaf and hard of
hearing communities.
Methodology
How we generated data set
and did Data Preprocessing ?
Data Acquisition
• We used the OpenCV (Open Computer
Vision) library to produce our dataset.
• We captured approximately 180 images
of each symbol in American Sign
Language (ASL)
• Around 50 images per symbol for testing.
Capturing Raw Image Gray Scale Image Image Post Gaussian
BLUR
MediaPipe LandMark Region of Interest (ROI) Gauss filter
Mediapipe Landmark System:
Overview of
Convolutional Neural Networks (CNNs)
Feature Extraction Image Classification
CNNs are adept at extracting complex These features are then used for image
features from images, such as shapes, classification, enabling the network to
edges, and textures. distinguish between different classes of
images.
Convolutional Neural Networks
● CNNs consist of multiple convolutional
layers each layer containing numerous
“filters” which perform feature
extraction.
● Initially these “filters” are random and by
training, the feature extraction gets better
by better.
● It’s primarily used for image classification.
CNN Architecture Design for Sign Language
Recognition
Convolutional Layers
1 Extract spatial features from the input image.
Pooling Layers
2
Reduce the spatial dimensions, making the model
more efficient.
Fully Connected Layers
3 Learn complex relationships between
features for classification.
Gesture Classification
Layer 1 Layer 2
Classify Classify
between 27 between Similar
Symbols Symbols
Algorithm Layer 1:
1. Apply gaussian blur filter and threshold to the frame taken with opencv to get
the processed image after feature extraction.
2. This processed image is passed to the CNN model for prediction and if a letter
is detected for more than 50 frames then the letter is printed and taken into
consideration for forming the word.
3. Space between the words are considered using the blank symbol.
Algorithm Layer 2:
● We detect various sets of symbols which show similar results on getting
detected.
● We then classify between those sets using classifiers made for those sets only.
● In our testing we found that following symbols were not showing properly and
were giving other symbols also :
1. For D : R and U
2. For U : D and R
3. For I : T, D, K and I
4. For S : M and N
Layer 2:
• We got a very bad accuracy in 26 different classes.
• We divided all 26 alphabets into 8 classes in which every class contains similar
alphabets.
Figure 2
Figure 1 Figure 3
[b,d,f,I,u,v,k,r,w] [y,j] [a,e,m,n,s,t]
Finger Spelling Sentence
Formation
Reading The Finger Gestures Using Mediapipe
Getting the finger points right
Implementation of Finger Spelling
1. Whenever the count of a letter detected exceeds a specific value and no other letter
is close to it by a threshold we print the letter and add it to the current string(In our
code we kept the value as 50 and difference threshold as 20).
2. Otherwise we clear the current dictionary which has the count of detections
of present symbol to avoid the probability of a wrong letter getting predicted.
3. Whenever the count of a blank(plain background) detected exceeds a
specific value and if the current buffer is empty no spaces are detected.
4. In other case it predicts the end of word by printing a space and the
current gets appended to the sentence below.
Autocorrect feature
A python library Enchant is used to suggest correct alternatives for each
(incorrect) input word and we display a set of words matching the current word
in which the user can select a word to append it to the current sentence.This
helps in reducing mistakes committed in spellings and assists in predicting
complex words.
Real-time Sign Language Recognition using the
Trained Model
Capture Hand Sign
The model captures the hand sign
image from a live video stream.
Process Image
The image is preprocessed and fed
into the trained CNN model.
Predict Sign
The model predicts the corresponding
sign based on its learned features.
Display Results
The predicted sign is displayed in real-time,
enabling seamless communication.
Results Under Different Conditions
Challenges and Limitations in Sign Language
Recognition
Diversity of Signs Lighting and Background
Different sign languages have variations in Fluctuations in lighting and complex
handshapes, movements, and expressions, backgrounds can affect the model's
making it challenging to develop a universal accuracy.
model.
Future Advancements and Applications
of Sign Language Recognition
1 2
Deployment as A web App. Improve Accuracy under
low light condition and
complex background
3 4
Support multiple languages Incorporate dynamic gesture
to form full sentences
Conclusion
● In this report, a functional real time vision based american sign language
recognition for D&M people have been developed for asl alphabets.
● We achieved an accuracy of 98.00% on our dataset.
● Prediction has been improved after implementing two layers of algorithms in
which we verify and predict symbols which are more similar to each other.
Thank You !
