A Project Report on

Sign language Recognition using

Machine Learning

Submitted by

Amisha B. Sawant SS20CO005

Megha M. Pawar SS20CO006

Sanika .S. Khade SS20CO038

Arati. M.Prajapati SS20CO047

Yukta R. More SS20CO065

In partial fulfillment of the requirements for

Diploma in Computer Engineering
2022-2023

Under the guidance of

Prof. P.S. Sadafule

Department of Computer Engineering

Government Polytechnic, Mumbai
Mumbai-400051
 Department of Computer Engineering
Government Polytechnic, Mumbai
Mumbai-400051

CERTIFICATE

This is to certify that following students have successfully and satisfactorily completed
the project on Sign Language Reorganization using Machine Learning presented their
report in the partial fulfillment of requirement for Diploma in Computer Engineering from
Government Polytechnic, Mumbai under the guidance of Prof. P.S. Sadafule in the academic
year 2020-2022.

Arati M. Prajapati SS20CO047

Yukta R. More SS20CO065
Sanika S. Khade SS20CO038
Amisha B. Sawant SS20CO005
Megha M. Pawar SS20CO006

Project Guide External Examiner

 Head Of the Department
Principal

ACKNOWLEDGEMENT

It is always a difficult job to acknowledge all those who have been of tremendous help
in the development of any work. In spite of that, we honestly thank all those who had put in
tremendous efforts in guiding and helping us to successfully build this project named “Sign
Language Reorganization using ML”. We take this opportunity to thank each and every
one for their co-operation and support.

First and firmly, we are grateful to (principal, Government Polytechnic Mumbai) for
allowing us to do this project collectively and with advanced techniques.

We express our sincere gratitude towards Prof. Neha. H. Vachani (Head of Computer
Department).

We heartly thank Prof. P. S. Sadafule our project guide for their valuable guidance
throughout, without which the project would not have stood its ground as it has now. We
would even like to thank people who have directly or indirectly helped us with the making of
the project and their support.

We are thankful to and fortunate enough to get constant encouragement, support, and
guidance from all Teaching staffs of Computer department, GPM which helped us in
successfully progressing with our project work.

ARATI M. PRAJAPATI -SS20CO047

YUKTA R. MORE-SS20CO065

SANIKA S. KHADE -SS20CO038

AMISHA B. SAWANT-SS20CO005

MEGHA M. PAWAR-SS20CO006
 Index
SR.NO. CONTENTS PAGE NO.

1. INTRODUCTION
1.1 Objectives
1.2 Problem Statement
1.3 Literature survey

2. Feasibility Study
2.1 project Scope

2.2 Approach

3. Project Planning
3.1 Scope Schedule and Cost

3.2 Communication, Resources And Risks

3.3 Change in Plan

4. Requirement Analysis
4.1 Hardware Requirements
4.2 Software Requirements

5. System Design

5.1 System Architecture

6. Coding
6.1 dataCollection.py

6.2 test.py

7. Implementation

8. Testing

9. Output/Screenshots

10 Bibliography
 ABSTRACT
Sign Language is mainly used by deaf (hard hearing) and dumb people to
exchange information between their own community and with other people. It is a
language where people use their hand gestures to communicate as they can’t speak or
hear. Sign Language Recognition (SLR) deals with recognizing the hand gestures and
continues till text or speech is generated for corresponding hand gestures. Here hand
gestures for sign language can be classified as static and dynamic. However, static
hand gesture recognition is simpler than dynamic hand gesture recognition, but both
recognition is important to the human community. We can use Deep Learning
Computer Vision to recognize the hand gestures.

Where the model will learn to recognize the hand gestures images. Once the model
Successfully recognizes the gesture the corresponding English text is generated and
then text can be converted to speech. This model will be more efficient and hence
communicate for the

deaf (hard hearing) and dump people will be easier. In this, we will discuss how Sign
Language Recognition is done using Machine Learning.
 Chapter – 1
INTRODUCTION
 INTRODUCTION
Deaf (hard hearing) and dumb people use Sign Language (SL) as their primary
means To express their ideas and thoughts with their own community and with other
people with hand and body gestures. It has its own vocabulary, meaning, and syntax
which is different from the spoken language or written language. Spoken language is
a language produced by articulate sounds mapped against specific words and
grammatical combinations to convey meaningful messages. Sign language uses visual
hand and body gestures to convey meaningful messages. There are somewhere
between 138 and 300 different types of Sign Language used around globally today. In
India, there are only about 250 certified sign language interpreters for a deaf
population of around 7 million. This would be a problem to teach sign language to the
deaf and dumb people as there is a limited number of sign language interpreters exits
today. Sign Language Recognition is an attempt to recognize these hand gestures and
convert them to the corresponding text or speech. Today Computer Vision and Deep
Learning have gained a lot of popularity. Using Deep Learning algorithms and Image
Processing we can be able to classify these hand gestures and able to produce
corresponding text.

1.1 Objectives
The main objectives of this project are to contribute to the field of automatic sign
language recognition and translation to text or speech. In our project, we focus on static
sign language hand gestures. This work focused on recognizing the hand gestures which
includes 26

English alphabets (A-Z) and 10 digits (0-9). By using ‘PyCharm’ and using ML with
the help oh python libraires we executed the code to show the hand gesture.
1.2 PROBLEM STATEMENT
Speech impaired people use hand signs and gestures to communicate. Normal
people face difficulty in understanding their language. Hence there is a need of a system
which recognizes the different signs, gestures and conveys the information to the
normal people. It bridges the gap between physically challenged people and normal
people.

1.3 LITERATURE SURVEY

This works focuses on static fingerspelling in American Sign Language A

method for implementing a sign language to text/voice conversion system without
using handheld gloves and sensors, by capturing the gesture continuously and
converting them to voice. In this method images were captured for recognition. The
design of a communication aid for the physically challenged.

TensorFlow:
TensorFlow is a free and open-source software library for dataflow and
differentiable programming across a range of tasks. It is a symbolic math l library and
is also used for machine learning applications such as neural networks. It is used for
both research and production at Google.

Features:

 TensorFlow provides stable Python (for version 3.7 across all platforms) and C
APIs; and without API backwards compatibility guarantee: C++, Go, Java,
JavaScript and Swift (early release).
  Third-party packages are available for C#,. "New language support should be
built on top of the C API. However, not all functionality is available in C yet."
Some more functionality is provided by the Python API.
 Application among the applications for which TensorFlow is the foundation, are
automated image-captioning software, such as Deep Dream.

OpenCV:
OpenCV (Open-Source Computer Vision Library) is a library of programming
functions mainly aimed at real-time computer vision.[1] Originally developed by Intel,
it was later supported by Willow Garage then Itrez (which was later acquired by Intel
[2]). The library is cross-platform and free for use under the open-source BSD license.

OpenCV's application areas include:

 2D and 3D feature toolkits
 Ego motion estimation
 Facial recognition system
 Gesture recognition
 Human–computer interaction (HCI)
 Mobile robotics
 Motion understanding
 Object identification
 Segmentation and recognition

List of free and open-source software packages

 OpenCV Functionality
 Image/video I/O, processing, display (core, improv, high-up)
 Object/feature detection (obj detect, features2d, nonfee)
 Geometry-based monocular or stereo computer vision (calib3d, stitching,
 video stab)
 Computational photography (photo, video, supers)
 Machine learning & clustering (ml, Flann)
 CUDA acceleration (gnu)

Image-Processing:
Image processing is a method to perform some operations on an image, in order to get
an enhanced image and or to extract some useful information from it. If we talk about
the basic definition of image processing, then “Image processing is the analysis and
manipulation of a digitized image, especially in order to improve its quality”.

keras:
Keras is an open-source neural-network library written in Python. It can run on top of
TensorFlow, Microsoft Cognitive Toolkit, R, Theano, or Plaid ml. Designed to enable
fast experimentation with deep neural networks, it focuses on being user-friendly,
modular, and extensible.

Features:

1)Large Dataset:

Keras contains a large pre-defined dataset. It provides you a variety of datasets. You
can use this dataset to be directly importing and loading it.

2) Pre-processing of Data:
Keras provides you several functions for the pre-processing of data.
3)Modularity:

Keras is modular, the model you train and use this model later by loading it.

NumPy:
It is a library for the Python programming language, adding support for large,
multidimensional arrays and matrices, along with a large collection of high-level
mathematical functions to operate on these arrays.

Features:

 NumPy targets the Python reference implementation of Python, which is a

non-optimizing bytecode interpreter. Mathematical algorithms written for this version

of Python often run much slower than compiled equivalents.

 NumPy addresses the slowness problem partly by providing multidimensional
arrays and functions and operators that operate efficiently on arrays, requiring
rewriting some code, mostly inner loops using Num

CHAPTER 2
Feasibility Study
 Feasibility Study

A feasibility study is an analysis that takes all a project’s relevant factors into account—
including economic, technical, legal, and scheduling considerations—to ascertain the
likelihood of completing the project successfully. Project managers use feasibility studies to
discern the pros an cons of undertaking a project before they invest a lot of time into it.

2.1 Project Scope

This project base on the Machine Learning to help the people who are deaf and dump. By
approaching this application in future or in current life people can easily communicate with
other peoples.

By doing this project we opened the door for the people who can’t communicate properly or
adjust with world. Using this application people will enjoy and learn something new
language.

2.2 Approach
This application will be using for native approach of development. The application is created
using Machine learning &Python, recognition is achieved by TensorFlow, OpenCV, Cvzone.
Recognize the user which show the sign and display on screen. Mostly usable for all people
who can interested to learn sign language, detect sign language and help deaf and dump
people communicating with each other.
 CHAPTER 3
Project Planning
 project planning

Project planning is a discipline addressing how to complete a project in a certain time period,
usually with defined stages and designated resources. One view of project planning divides
the activity into these steps:

 setting measurable objectives

 identifying deliverable
 scheduling
 planning tasks

Scheduling of the project given below:

Sr. Date Day Present Subjects

No Members

1. 20/07/2022 Wednesday All Introduction to Third Year Project

2. 28/07/2022 Thursday All Group Selected

3. 29/07/2022 Friday All Abstract Submitted

4.. 05/08/2022 Friday All Guide Allocated

5. 12/08/2022 Friday 4 members Abstract Rejected

6. 26/08/2022 Friday All Project Topic Selected

7. 17/09/2022 Saturday All Project Detail Research

8. 29/09/2022 Thursday 3 members Synopsis Submitted

9. 01/10/2022 All members Coding Started

10. 05/10/2022 All members Working on Capture Images Frame

11. 06/10/2022 All members Data Collection started

12. 10/10/2022 All members Data Collection Halfway

13. 13/10/2022 All members Data Collection Completed

14. 15/10/2022 All members Show output until now completed

project (45% project completed)

15. 21/10/2022 All members Addition in data

16. 28/10/2022 New features added

17

18.
 CHAPTER 4
Requirement Analysis
 Requirement Analysis

We present a sign language recognition that can be helpful for the deaf and dumb
people while communicating with others. In the sign language recognition, we
create sign detector which detects numbers from 0 to 9 and hand gesture
including the alphabets and words.

Requirement Specification:
 Hardware Requirements
 Software Requirements

4.1 Hardware Requirements:

The hardware used for the development of the project:

 Processor: AMD Ryzen 5 4600H with Radeon Graphics 3.00GHz

 RAM: 8.00 GB
 Devices: Laptop, Laptop Webcam

4.2 Software Requirements:

The software used for the development of the project:
 IDE such as PyCharm (Community Version 2022.2.3) will be used for coding.

Libraries/ Version Description

Packages:
OpenCV python 4.5.4.60  It is used for image processing and
performing computer vision task.

TensorFlow 2.9.1  Used to create deep learning model.

Cvzone 1.5.6  It makes easy to run image processing and AI
function.

Mediapipe 0.8.10.1  media pipe is a pipeline framework use to

build custom machine learning solutions for
live and streaming media

CHAPTER 5
System Design
 System designing
System designing is the process of design/draw the architecture, modules and components,
the data goes through that system. Though the design process it becomes easier to
understands the software/ products flow to working.

Let’s draw the System Architecture of our Sign Language Recognition to understand
clearly:
As shown in Figure 1, the project will be the entire system is divided into 3 parts: -

1) Data Processing

2) Training

3) Classify Gesture

Let’s discuss in detail: -

1)Data Processing:
The DataCollection.py script contains functions to load the images and save it on data as
NumPy arrays into file storage. in short Data Collection script is use for webcam through
which we can
capture the Fig.1: Block Diagram of System images and
save it on “Data”
Folder. After that it will go for training.

2)Training:

The training loop for the model is contained in train model. The model is trained with n
number of saved image and that images will be save on Models folder in which two file will
be generated “keras_models.h5” and “labels.txt” in which all the input value predefined.

3) Classify Gesture:
As we have used supervised machine learning algorithm in our programs. After training, the
users give the input, and the system will process it or recognize it and gives the output on the
screen.

let’s take it more simple-

through this diagram you can clearly understand how the system is worked flow.
CHAPTER 6
Coding
 CHAPTER 7
Implementation
TensorFlow is a Python-friendly open-source library for numerical
computation that makes machine learning and developing neural networks
faster and easier.

Working of TensorFlow

TensorFlow allows you to create dataflow graphs that describe how data moves through a
graph. The graph consists of nodes that represent a mathematical operation. A connection or
edge between nodes is a multidimensional data array. It takes inputs as a multi-dimensional
array where you can construct a flowchart of operations that can be performed on these
inputs.

Architecture of TensorFlow:
  Input Layer

The input layer accepts large volumes of data as input to build the network. The data can be
in the form of text, image, audio, etc. 

 Hidden Layer:

This layer processes data by performing complex computations and carries out feature
extraction. As part of the training, these layers have weights and biases that are
continuously updated until the training process is complete.

 Output Layer

The output layer generates predicted output by applying suitable activation functions. The
output can be in the form of numeric or categorical values.

For example, if it is an image classification application, it tells us which class a particular

image may belong to. The input can be multiple images, such as cats and dogs. The output

Flowchart of Sign Language Recognition

can be in the form of binary classification like the number zero for the dog and the number
one for the cat

Explanation: -

Step1: - In this phase we run the program and it will open the “Webcam”.

Step2: -It invokes the OpenCV, Keras, and TensorFlow such library for processing the
images.

Step3: - This step consists of digitize the information from the user to make it usable to
classification algorithm.

Step4: - It will re-process on the datasets.

Step5: - It is the process of transforming raw data into numerical features that can be
processed while preserving the information in the original datasets.

Step6: - It compare two different sets of data and matching them with input data. If data is
matched, it will display the outputs otherwise it will repeat the process until the data cannot
be matched.
CHAPTER 8
Testing
 TESTING

Testing is the process of checking that software product or application are

matches to the expected requirement or not and identifying the correctness of
software by considering its all attributes (Reliability, Scalability, Portability, Re-
usability, Usability).

The purpose of testing is to discover the error and every conceivable fault or
weakness in a work product.

Software testing is an important element of the software quality assurance

and evaluate work products such as user requirements, design, and code.

Testing Objectives:
Testing objectives can serve as a several rules they are:

 To find failures and defects.

 To provide sufficient information to the stakeholder to make decisions regarding
the level of quality of the test objects.

Types of testing:
 Unit Testing:
Unit testing is done on individual models as they completed and becomes
executable. It helps to detect flaw in code which is more difficult to find in
later in testing process. Unit testing is different from and should be
preceded by other techniques including:
 Inform debugging
  Code Inspection

 Black Box Testing:

This type of testing approach focusses on the input that goes into the software and
that produced output.
This testing is used to find the error of following categories:
 Interface and performance errors
 Initialization and termination of errors
 The logical flow of data is not checked
 In this testing the only Output is checked for correctness

 White Box Testing:

This testing approach allow tester to verify the inner working of software
systems (code, infrastructure, integration with external system).
It has been used to generate test cases in the following cases:
 Execute internal data structure to ensure their validity
 Guarantee that are all independent paths have been executed

 Integration Testing:

Integration testing is the second level of the software testing process comes
after unit testing.
In this testing, units or individual components of the software are tested in a
group. The software is developed with a few software modules that are
coded by different coders or programmers.

 System Testing:

It is a level of testing that validates the complete and fully integrated software product.
The purpose of a system test is to evaluate the end-to-end system specifications.
Usually, the software is only one element of a larger computer -based system.
 Ultimately, the software is interfaced with other software/hardware system.

Test Cases:
Test Test Case Input Data Expected Actual Result Remar
Case Objectives /Description Result k/
Status
ID

TC 1 Loading Initializing the Loaded the The Datasets was

Datasets trained Datasets Datasets without loaded
Pass
and load it into error successfully

TC 2 Convert Capture the Image words The Conversion

image to image and and alphabets of of image to
alphabets convert it into the captured hand alphabets and
and words words and gesture words Pass
alphabets successfully

TC 3 Recognize Image Words and Label Recognition of

the hand Alphabets that the hand gesture
Pass
gesture contain hand successfully
objects
 CHAPTER 9
Output/Screenshots
 CHAPTER 10
References /Bibliography
References /Bibliography

1.Thakur, Pujan Budhathoki, Sarmila Upreti, Shirish Shrestha, Subarna Shakya, “Real Time Sign Language
Recognition and Speech Generation,” Journal of Innovative Image Processing(Vol.02/No.02) Published on
01-02-2020.

Aman Pathak, Avinash Kumar, Priyam, Priyanshu Gupta and Gunjan Chugh, “Real Time Sign Language
Detection,” International Journal for Modern Trends in Science and Technology 2022, 8 pp. 32-37.

Reddygari Sandhya Rani , R Rumana , R. Prema,” Sign Language Recognition for The Deaf and Dumb,”
International Journal of Engineering Research & Technology (IJERT), Volume 10, Issue 10 (October 2021).

S. M. Prajapati, V. V. Tambe and D. D. Bendre, "SMART GLOVE THAT TRANSLATE SIGN

LANGUAGE INTO TEXT AND SPEECH," Accepted and presented by Electronics Engineering, Feb 2022.
[hardwere]
Anbarasi Rajamohan, Hemavathy R., Dhanalakshmi M.,” Deaf-Mute Communication Interpreter,”
International Journal of Scientific Engineering and Technology Volume 2 Issue 5, pp : 336-341 1 May 2013.
[hardware]

Other references:

https://www.eecg.utoronto.ca/~jayar/mie324/asl.pdf

https://ieeexplore.ieee.org/document/7916786

https://babysignlanguage.com/dictionary/

https://data-flair.training/blogs/sign-language-recognition-python-ml-opencv/

http://cs231n.stanford.edu/reports/2016/pdfs/214Report.pdf