A Project on

Hand gesture recognition for specially challenged people.

Submitted by
Ishika Mishra(20650)
Saurabh Shukla(20742)

Under the Guidance of

Mrs. Vijayalaxmi
Submitted in partially fulfillment of requirement for
Qualifying BSc-(IT) Semester V Examination
Department of Information Technology
Academic Year:
2020-2021
 Certificate
This is to certify that the Project entitled
Hand gesture recognition for specially challenged people.

Submitted by
Ms. Ishika Mishra (20650)
In partial fulfillment for BSc.IT Degree (Semester V) Examination for the
academic year 2019-2020 and has not been submitted for any other examination
and does not form a part of any other course undergone by the candidate. It is
further certified that he/she has completed all the required phases of the project.

_______________ _______________

External Examiner Internal Examiner

_______________ _______________

Project Guide Head of Department

 Certificate
This is to certify that the Project entitled
Hand gesture recognition for specially challenged people.

Submitted by
Ms. Saurabh Shukla (20742)
In partial fulfillment for BSc.IT Degree (Semester V) Examination for the
academic year 2019-2020 and has not been submitted for any other examination
and does not form a part of any other course undergone by the candidate. It is
further certified that he/she has completed all the required phases of the project.

_______________ _______________

External Examiner Internal Examiner

_______________ _______________

Project Guide Head of Department

 INDEX
CONTENTS

CHAPTER 1: INTRODUCTION
1.1 Introduction……………………………………….………………2
1.2 Scope……………………………………………….……………..2
1.3 Problem Statement……..………....................................................3

CHAPTER 2 SYSTEM SPECIFICATION

2.1 System Requirement………………………………….……………5
2.2 System Features………………………………………….……...…5

CHAPTER 3: SYSTEM DESIGN

3.1 System Architecture…………...…………………………………..7
3.2 Modules in the System………...…………………………………..7
3.3 Use Case Diagram ………...………………………………………8
3.4 Activity Diagram………….………......…………….......................9

CHAPTER 4: IMPLEMENTATION
4.1 Code Snippets………..…...……….…….………….......................11
4.2 Screen Shots…...………………………………………………….
4.3 Detailed life cycle of the project …………………………………
4.4 Cost Estimation ……………………………………………………

CHAPTER 5: CONCLUSION
5.1 Conclusion….…………………………………………………….
5.2 Future Scope.......…………………………………………………

REFERENCES...…………………………………………………………..................
 CHAPTER 1

INTRODUCTION
1.1 INTRODUCTION

Some of the major problems faced by a person who are unable to speak is they cannot
express their emotion as freely in this world. Utilize that voice recognition and voice
search systems in smartphone(s).[1] Audio results cannot be retrieved. They are not able
to utilize (Artificial Intelligence/personal Butler) like google assistance, or Apple's SIRI
etc.[2] because all those apps are based on voice controlling.

There is a need for such platforms for such kind of people. American Sign Language
(ASL) is a complete, complex language that employs signs made by moving the hands
combined with facial expressions and postures of the body. It is the go-to language of
many North Americans who are not able to talk and is one of various communication
alternatives used by people who are deaf or hard-of-hearing.[3]

While sign language is very essential for deaf-mute people, to communicate both with
normal people and with themselves, is still getting less attention from the normal
people.[4] The importance of sign language has been tending to ignored, unless there are
areas of concern with individuals who are deaf-mute. One of the solutions to talk with
the deaf-mute people is by using the mechanisms of sign language.

Hand gesture is one of the methods used in sign language for non-verbal communication.
It is most commonly used by deaf & dumb people who have hearing or talking disorders
to communicate among themselves or with normal people.[3] Various sign language
systems have been developed by many manufacturers around the world but they are
neither flexible nor cost-effective for the end users.[6]

1.2 SCOPE

One of the solutions to communicate with the deaf-mute people is by using the services
of sign language interpreter. But the usage of sign language interpreters could be
expensive.[3] Cost-effective solution is required so that the deaf-mute and normal
people can communicate normally and easily. [3]

Our strategy involves implementing such an application which detects pre-defined

American sign language (ASL) through hand gestures. For the detection of movement
of gesture, we would use basic level of hardware component like camera and
interfacing is required. Our application would be a comprehensive User-friendly Based
system built on PyQt5 module.

Instead of using technology like gloves or kinect, we are trying to solve this problem
using state of the art computer vision and machine learning algorithms. [6]

This application will comprise of two core module one is that simply detects the gesture
and displays appropriate alphabet. The second is after a certain amount of interval period
the scanned frame would be stored into buffer so that a string of character could be
generated forming a meaningful word.

Additionally, an-addon facility for the user would be available where a user can build
their own custom-based gesture for a special character like period (.) or any delimiter so
that a user could form a whole bunch of sentences enhancing this into paragraph and
likewise. Whatever the predicted outcome was, it would be stored into a .txt file.

1.3 PROBLEM STATEMENT

Given a hand gesture, implementing such an application which detects pre-defined

American sign language (ASL) in a real time through hand gestures and providing
facility for the user to be able to store the result of the character detected in a txt file, also
allowing such users to build their customized gesture so that the problems faced by
persons who aren’t able to talk vocally can be accommodated with technological
assistance and the barrier of expressing can be overshadowed.
 CHAPTER 2

SYSTEM
SPECIFICATION
2.1 SYSTEM REQUIREMENT

2.1.1 HARDWARE REQUIREMENTS

Intel Core i3 3rd gen processor or later.

512 MB disk space.
512 MB RAM.
Any external or inbuild camera with minimum pixel resolution 200 x 200
(300ppi or 150lpi) 4-megapixel cameras and up.

2.1.2 SOFTWARE REQUIREMENTS

Microsoft Windows XP or later / Ubuntu 12.0 LTS or later /MAC OS 10.1

or later.
Python Interpreter (3.6).
TensorFlow framework, Keras API.
PyQT5, Tkinter module.
Python OpenCV2, scipy, qimage2ndarray, winGuiAuto, pypiwin32, sys,
keyboard, pyttsx3, pillow libraries.

2.2 SYSTEM FEATURES

User-friendly based GUI built using industrial standard PyQT5.
Real time American standard character detection based on gesture made by
user.
Customized gesture generation.
Forming a stream of sentences based on the gesture made after a certain
interval of time.
 CHAPTER 3

SYSTEM DESIGN
3.1 SYSTEM ARCHITECTURE

Fig 1: System Architecture for Sign Language Recognition Using Hand Gestures.

3.2 MODULES IN THE SYSTEM

Data Pre-Processing – In this module, based on the object detected in front

of the camera its binary images is being populated. Meaning the object will
be filled with solid white and background will be filled with solid black.
Based on the pixel’s regions, their numerical value in range of either 0 or 1 is
being given to next process for modules.
  Scan Single Gesture – A gesture scanner will be available in front of the end
user where the user will have to do a hand gesture. Based on Pre-Processed
module output, a user shall be able to see associated label assigned for each
hand gestures, based on the predefined American Sign Language (ASL)
standard inside the output window screen.
 Create gesture –A user will give a desired hand gesture as an input to the
system with the text box available at the bottom of the screen where the user
needs to type whatever he/she desires to associate that gesture with. This
customize gesture will then be stored for future purposes and will be detected
in the upcoming time.

3.3 USE CASE DIAGRAM

Fig 2: Use Case Diagram for Sign Language Recognition Using Hand Gestures.
3.4 ACTIVITY DIAGRAM

Fig 3: Activity Diagram for Sign Language Recognition Using Hand Gestures.
 CHAPTER 4

IMPLEMENTATION
 4.1 CODE SNIPPETS

Recognise.py
import cv2
import numpy as np
def nothing(x):
pass
image_x, image_y = 64,64
from keras.models import load_model
classifier = load_model('Trained_model.h5')
def predictor():
import numpy as np
from keras.preprocessing import image
test_image = image.load_img('1.png', target_size=(64, 64))
test_image = image.img_to_array(test_image)
test_image = np.expand_dims(test_image, axis = 0)
result = classifier.predict(test_image)
if result[0][0] == 1:
return 'A'
elif result[0][1] == 1:
return 'B'
elif result[0][2] == 1:
return 'C'
elif result[0][3] == 1:
return 'D'
elif result[0][4] == 1:
return 'E'
elif result[0][5] == 1:
return 'F'
elif result[0][6] == 1:
return 'G'
elif result[0][7] == 1:
return 'H'
elif result[0][8] == 1:
return 'I'
elif result[0][9] == 1:
return 'J'
elif result[0][10] == 1:
return 'K'
elif result[0][11] == 1:
return 'L'
elif result[0][12] == 1:
return 'M'
elif result[0][13] == 1:
return 'N'
elif result[0][14] == 1:
return 'O'
elif result[0][15] == 1:
return 'P'
elif result[0][16] == 1:
return 'Q'
elif result[0][17] == 1:
return 'R'
elif result[0][18] == 1:
return 'S'
elif result[0][19] == 1:
return 'T'
elif result[0][20] == 1:
return 'U'
elif result[0][21] == 1:
return 'V'
elif result[0][22] == 1:
return 'W'
elif result[0][23] == 1:
return 'X'
elif result[0][24] == 1:
return 'Y'
elif result[0][25] == 1:
return 'Z'

cam = cv2.VideoCapture(0)
cv2.namedWindow("Trackbars")
cv2.createTrackbar("L - H", "Trackbars", 0, 179, nothing)
cv2.createTrackbar("L - S", "Trackbars", 0, 255, nothing)
cv2.createTrackbar("L - V", "Trackbars", 0, 255, nothing)
cv2.createTrackbar("U - H", "Trackbars", 179, 179, nothing)
cv2.createTrackbar("U - S", "Trackbars", 255, 255, nothing)
cv2.createTrackbar("U - V", "Trackbars", 255, 255, nothing)
cv2.namedWindow("test")
img_counter = 0
img_text = ''
while True:
ret, frame = cam.read()
frame = cv2.flip(frame,1)
l_h = cv2.getTrackbarPos("L - H", "Trackbars")
l_s = cv2.getTrackbarPos("L - S", "Trackbars")
l_v = cv2.getTrackbarPos("L - V", "Trackbars")
u_h = cv2.getTrackbarPos("U - H", "Trackbars")
u_s = cv2.getTrackbarPos("U - S", "Trackbars")
u_v = cv2.getTrackbarPos("U - V", "Trackbars")
img = cv2.rectangle(frame, (425,100),(625,300), (0,255,0), thickness=2, lineType=8,
shift=0)
 lower_blue = np.array([l_h, l_s, l_v])
upper_blue = np.array([u_h, u_s, u_v])
imcrop = img[102:298, 427:623]
hsv = cv2.cvtColor(imcrop, cv2.COLOR_BGR2HSV)
mask = cv2.inRange(hsv, lower_blue, upper_blue)
cv2.putText(frame, img_text, (30, 400), cv2.FONT_HERSHEY_TRIPLEX, 1.5, (0, 255,
0))
cv2.imshow("test", frame)
cv2.imshow("mask", mask)
#if cv2.waitKey(1) == ord('c'):
img_name = "1.png"
save_img = cv2.resize(mask, (image_x, image_y))
cv2.imwrite(img_name, save_img)
print("{} written!".format(img_name))
img_text = predictor()
if cv2.waitKey(1) == 27:
break
cam.release()
cv2.destroyAllWindows()
4.2 SCREENSHOTS
DETAILED LIFECYCLE OF PROJECT:

i) MODEL: Incremental model

Incremental Model is a process of software development where requirements divided into

multiple standalone modules of the software development cycle. In this model, each module
goes through the requirements, design, implementation and testing phases. Every subsequent
release of the module adds function to the previous release. The process continues until the
complete system achieved.

THE VARIOUS PHASES OF INCREMENTAL MODEL ARE AS FOLLOWS:

1. Requirement analysis: In the first phase of the incremental model, the product analysis
expertise identifies the requirements. And the system functional requirements are understood
by the requirement analysis team. To develop the software under the incremental model, this
phase performs a crucial role.
2. Design & Development: In this phase of the Incremental model of SDLC, the design of
the system functionality and the development method are finished with success. When
software develops new practicality, the incremental model uses style and development phase.
We have designed various diagrams like context diagram, data flow diagram, entity
relationship diagram, class diagram, use case diagram and architecture diagram.
3. Testing: Partially Implemented.
4.Implementation: Partially done.
PERT CHART
Cost Estimation of the Project:

Function Points Analysis

The FPA is a reliable method for measuring the size of computer software. It essentially
measures functionality that the user requests and receives. It also measures the software
development and maintenance cost and size independently of the technology used for
implementation
The general approach that FPA follows is
· Count the number of inputs, outputs, inquiries, master files, and interfaces required, then
calculate the Unadjusted Function Points (UFP)
Calculate the adjusted function point (AFP) by multiplying these counts by an adjustment
factor; the UFP and the product complexity adjustment.
· Calculate the Source Lines of Code (SLOC) with the help of the AFP and the Language
Factor (LF)

1. Calculation of the unadjusted function points (UFP):

The FPA measures functionality that the user requires. The specific user functionality is a
measurement of the functionality delivered by the application as for user request. The 5
function types identified are:

external input which receives information from outside the application boundary,
external output which presents information of the information system,
external enquiries which is special kind of an external output.
An external inquiry presents information of the information system based on a uniquely
identifying search criterion, without applying additional processing (such as calculations).
internal logical files contains permanent data that is relevant to the user The information
system references and maintains the data and
external interface files also contains permanent data that is relevant to the user. The
information system references the data, but the data is maintained by another information
system
For each function identified above the function is further classified as simple, average or
complex and a weight is given to each. The sum of the weights quantifies the size of
information processing and is referred to as the Unadjusted Function points.

The table below shows the function types and the weighting factors for the varying
complexities.
From the above table we get:

Weighting Count
Factor

Simple Average Complex

Inputs Gesture 4 4
Outputs User Gesture 5 10

User Gesture 5
in text

Inquiries Validate 4 8
gesture
Gesture is in 4
Box
Files User details 10 30

Gesture 10
details
Receiver 10
details

Interfaces User to 10 10
application
server

Total UFP 62

2. Calculate Adjusted Function Point

To calculate the Complexity adjustment value, several factors have to be considered, such as
Backup and recovery, code design for reuse, etc.
Considering the 14 Questions ,
Complexity Adjustment Value=56

To Compute function points(FP),the following relationship is used:

FP = total UFP * (0.65 + (0.01 * Total complexity adjustment value)

=62*(0.65+(0.01*56)
= 75.02

3. Calculate the Source Lines of Code (SLOC) and the formulas used

· Total Unadjusted Function Points (UFP) = 62

· Product Complexity Adjustment (PC) = 0.65 + (0.01 *56) = 1.21

· Total Adjusted Function Points (FP) = UFP * PC = 75.02

· Language Factor (LF) for Python assumed as = 38

· Source Lines of Code (SLOC) = FP * LF = 2,850

COCOMO MODEL:
Cocomo (Constructive Cost Model) is a regression model based on LOC, i.e number
of Lines of Code. It is a procedural cost estimate model for software projects and often used as
a process of reliably predicting the various parameters associated with making a project such
as size, effort, cost, time and quality.
The model that we going to use is Basic COCOMO Model.
Basic COCOMO can be used for quick and slightly rough calculations of Software Costs. Its
accuracy is somewhat restricted due to the absence of sufficient factor considerations.

Effort= a(KLOC)^b
time= c(Effort)^d
Person required = Effort/ time
The above formula is used for the cost estimation of for the basic COCOMO model, and also
is used in the subsequent models.And the values of a,b,c,d are constant.

Software Projects. a. b. c. d.
Organic. 2.4 1.05 2.5 0.38
Semi Detached 3.0 1.12 2.5 0.35
Embedded 3.6 1.20 2.5 0.32
The above
while the above mention
calculation
be conclude in values
the are the values
formulaes,based on of the
which constants
the cost that are going
estimation of thetoproject
be usedwill
 CHAPTER 5

CONCLUSION
5.1 CONCLUSION

From this project/application we have tried to overshadow some of the major problems
faced by the disabled persons in terms of talking. We found out the root cause of why
they can’t express more freely. The result that we got was the other side of the audience
are not able to interpret what these persons are trying to say or what is the message that
they want to convey.

Thereby this application serves the person who wants to learn and talk in sign languages.
With this application a person will quickly adapt various gestures and their meaning as per
ASL standards. They can quickly learn what alphabet is assigned to which gesture. Add-on
to this custom gesture facility is also provided along with sentence formation. A user need
not be a literate person if they know the action of the gesture, they can quickly form the
gesture and appropriate assigned character will be shown onto the screen.

Concerning to the implementation, we have used TensorFlow framework, with keras

API. And for the user feasibility complete front-end is designed using PyQT5.
Appropriate user-friendly messages are prompted as per the user actions along with what
gesture means which character window. Additionally, an export to file module is also
provided with TTS(Text-To-Speech) assistance meaning whatever the sentence was
formed a user will be able to listen to it and then quickly export along with observing
what gesture he/she made during the sentence formation.

5.2 FUTURE SCOPE

➢
It can be integrated with various search engines and texting application such
as google, WhatsApp. So that even the illiterate people could be able to chat
with other persons, or query something from web just with the help of gesture.

➢
This project is working on image currently, further development can lead to
detecting the motion of video sequence and assigning it to a meaningful
sentence with TTS assistance.
REFERENCES
REFERENCES

[1] Shobhit Agarwal, “What are some problems faced by deaf and dumb people while
using todays common tech like phones and PCs”, 2017 [Online]. Available:
https://www.quora.com/What-are-some-problems-faced-by-deaf-and-dumb-people-while-
using-todays-common-tech-like-phones-and-PCs, [Accessed April 06, 2019].

[2] NIDCD, “american sign language”, 2017 [Online]. Available:

https://www.nidcd.nih.gov/health/american-sign-language, [Accessed April 06, 2019].

[3] Suharjito MT, “Sign Language Recognition Application Systems for Deaf-Mute
People A Review Based on Input-Process-Output”, 2017 [Online]. Available:
https://www.academia.edu/35314119/Sign_Language_Recognition_Application_Syste
ms_for_Deaf-Mute_People_A_Review_Based_on_Input-Process-Output [Accessed April
06, 2019].

[4] M. Ibrahim, “Sign Language Translation via Image Processing”, [Online]. Available:
https://www.kics.edu.pk/project/startup/203 [Accessed April 06, 2019].

[5] NAD, “American sign language-community and culture frequently asked questions”,
2017 [Online]. Available:
https://www.nad.org/resources/american-sign-language/community-and-culture-
frequently-asked-questions/ [Accessed April 06, 2019].

[6] Sanil Jain and K.V.Sameer Raja, “Indian Sign Language Character Recognition” , [Online].
Available: https://cse.iitk.ac.in/users/cs365/2015/_submissions/vinsam/report.pdf
[Accessed April 06, 2019].