A

Project report
on
AI BASED DEAF ASSISTANCE SYSTEM
submitted in partial fulfillment of the Requirements for the Award of the Degree of
BACHELOR OF TECHNOLOGY
In
COMPUTER SCIENCE AND ENGINEERING
(Artificial Intelligence and Machine Learning)
By
GAJULA MANI SRINIVAS 21VD1A6626

Under the guidance of

Mr. G. SRIDHAR
Assistant Professor(c),
Department of Computer Science and Engineering.

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

(Artificial Intelligence and Machine Learning)
JAWAHARLAL NEHRU TECHNOLOGICAL UNIVERSITY HYDERABAD
UNIVERSITY COLLEGE OF ENGINEERING MANTHANI
Pannur (Vill), Ramagiri (Mdl), Peddapally-505212, Telangana (India).
2024-2025

AI BASED DEAF ASSISTANCE SYSTEM i

JAWAHARLAL NEHRU TECHNOLOGICAL UNIVERSITY HYDERABAD
UNIVERSITY COLLEGE OF ENGINEERING MANTHANI
Pannur (Vill), Ramagiri (Mdl), Peddapally-505212, Telangana (India).
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING
(Artificial Intelligence and Machine Learning)

DECLARATION BY THE CANDIDATE

I,
GAJULA MANI SRINIVAS 21VD1A6626

hereby declare that the Major project report entitled “AI BASED DEAF ASSISTANCE
SYSTEM” under the guidance of Mr. G. SRIDHAR, Assistant Professor(c), Department of
CSE, JNTUH University College of Engineering Manthani submitted in partial fulfillment for
the award of the Degree of Bachelor of Technology in Computer Science and
Engineering(AIML) during the Academic year 2024-2025.
This is a record of bonafide work carried out by us and the results embodied in this project
report have not been reproduced or copied from any source. The results embodied in this
project have not been submitted to any other University or Institute for the award of any degree
or diploma.
GAJULA MANI SRINIVAS 21VD1A6626

AI BASED DEAF ASSISTANCE SYSTEM ii

 JAWAHARLAL NEHRU TECHNOLOGICAL UNIVERSITY HYDERABAD
UNIVERSITY COLLEGE OF ENGINEERING MANTHANI
Pannur (Vill), Ramagiri (Mdl), Peddapally-505212, Telangana (India).
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING
(Artificial Intelligence and Machine Learning)

CERTIFICATE
This is to certify that the Major project report entitled “AI BASED DEAF ASSISTANCE
SYSTEM” being submitted by
GAJULA MANI SRINIVAS 21VD1A6637

in the partial fulfillment of the requirements for the award of the Degree of Bachelor of Technology
in Computer Science and Engineering(AIML) by the Jawaharlal Nehru Technological
University Hyderabad University College of Engineering Manthani during the academic
year 2024-25.
The results of investigation enclosed in this report have been verified and found satisfactory.
The results embodied in this project report have not been submitted to any other University or
Institute for the award of any degree or diploma.

PROJECT GUIDE HEAD OF THE DEPARTMENT

Mr. G. SRIDHAR,
(Assistant Professor, CSE)
EXTERNAL EXAMINAR
DATE:

AI BASED DEAF ASSISTANCE SYSTEM iii

 ACKNOWLEDGMENT
We express our sincere gratitude to Dr. B. Vishnu Vardhan, Professor of Computer Science
and Engineering, Principal and Head of the Department of Computer Science and
Engineering, JNTUH University College of Engineering Manthani, for encouraging us and
granting permission to successfully accomplish our project.
We feel honored and privileged to place our warm salutation to our Vice-Principal Mr. M.
UDAY KUMAR ,Associate Professor of Computer Science and Engineering, JNTUH
University College of Engineering Manthani, who gave us the opportunity to have experience in
engineering and profound technical knowledge.
We extend our profound gratitude to our project guide, Mr. G. SRIDHAR, Assistant Professor(c),
Department of Computer Science and Engineering, JNTUH University College of Engineering
Manthani, for his invaluable assistance, expert guidance, and consistent encouragement throughout
this project, enabling us to complete it successfully and on time.
We also take this opportunity to thank the Department of CSE faculty members for their kind
cooperation and support.
We are deeply grateful to our parents, whose unwavering support and encouragement have been the
cornerstone of our project’s success.
Finally, we express our heartfelt thanks to everyone who extended their helping hands, directly or
indirectly, in the completion of this project.

GAJULA MANI SRINIVAS 21VD1A6626

AI BASED DEAF ASSISTANCE SYSTEM iv

 TABLE OF CONTENTS

S.NO NAME PAGE NO

i. TITLE PAGE i

ii. DECLARATION ii

iii. CERTIFICATION iii

i. ACKNOLEDGEMENT iv

ii. ABSTRACT v
iii. TABLE OF CONTENTS vi
1 INTRODUCTION 1
1.1 Problem Statement 2
1.2 Objective 3
2 LITERATURE SURVEY 5
2.1 Deaf Culture and Identity 6
2.2 Challenges in Sign Language Processing 6
2.3 Reviewed progress in Sign Language Processing 7
2.4 Explanation of Key Functions 8
3 SYSTEM ANALYSIS 9
3.1 Existing System 10
3.2 Proposed System 11
3.3 Feasibility Study 12
4 SYSTEM REQUIREMENT SPECIFICATION 13
4.1 Functional Requirements 14
4.2 Non-Functional Requirements 14
5 SYSTEM DESIGN 16
5.1 Use Case Diagram 17
5.2 Activity Diagram 18
5.3 Class Diagram 19
5.4 Sequence Diagram 20
6 IMPLEMENTATION 21
6.1 Working Description 22
6.2 Methodology 23

AI BASED DEAF ASSISTANCE SYSTEM v

 6.3 Sample Code 26
6.4 Web Application 27
7 TESTING 38
7.1 Unit Testing 39
7.2 Integration Testing 39
7.3 Performance Testing 39
8 RESULTS 40
9 CONCLUSION 44
10 BIBLOGRAPHY 46

AI BASED DEAF ASSISTANCE SYSTEM vi

 LIST OF FIGURES

FIGURE_NO NAME PAGE_NO

Fig 5.1.1 Use Case Diagram 17
Fig 5.2.1 Activity Diagram 18
Fig 5.3.1 Class Diagram 19
Fig 5.4.1 Sequence Diagram 20
Fig 5.5.1 System Architecture 21

AI BASED DEAF ASSISTANCE SYSTEM vii

 INTRODUCTION

AI BASED DEAF ASSISTANCE SYSTEM 1

 1.INTRODUCTION
The integration of Artificial Intelligence (AI) in healthcare has transformed patient care by
enabling real-time health monitoring, automated diagnostics, and personalized treatment
recommendations. AI-driven healthcare systems leverage cutting-edge technologies such as machine
learning, natural language processing (NLP), and computer vision to enhance the accuracy and
efficiency of medical services. These advancements have improved disease detection, reduced response
times in emergencies, and facilitated better patient management. However, a critical challenge in
healthcare accessibility remains—communication barriers faced by individuals with hearing
impairments, speech disabilities, or language constraints.
To address this challenge, AI-based health assistance systems are being designed to incorporate
multimodal interaction capabilities, allowing patients with diverse communication needs to access
healthcare services seamlessly. One key feature of such systems is sign language recognition and
translation, which plays a pivotal role in bridging the communication gap between Deaf and hard-of-
hearing (DHH) individuals and healthcare providers. Sign language processing requires expertise from
multiple disciplines, including computer vision, linguistics, and human-computer interaction (HCI). By
integrating these fields, AI-powered health assistants can interpret sign language in real time, translate
medical queries into sign language videos, and facilitate doctor-patient interactions without reliance on
human interpreters.
Moreover, gesture-based recognition, voice-to-text conversion, and avatar-based sign language
generation can further enhance the accessibility of healthcare systems. AI-powered avatars can generate
natural, expressive sign language representations, making medical information more understandable for
sign language users. For example, animated avatars can provide step-by-step instructions for taking
medications, explain symptoms, or guide patients through a medical procedure. Gesture recognition,
combined with speech-to-text technologies, allows patients with speech disabilities to communicate
their symptoms effectively.

AI BASED DEAF ASSISTANCE SYSTEM 2

 1.1 PROBLEM STATEMENT
Individuals with hearing and speaking impairments face significant challenges in communicating
with healthcare professionals who do not understand sign language. Miscommunication in medical
settings can lead to misdiagnoses, improper treatment, and delays in emergency response, ultimately
affecting patient safety and well-being.
Existing solutions, such as word-for-word sign language translations or human interpreters, have
limitations. Direct translations fail to capture the grammar and context of sign languages, while human
interpreters may not always be available due to cost, privacy concerns, and accessibility issues.Here are
few challenges:
1. Misinterpretation of Medical Information
Existing AI-based translation systems often provide word-for-word translations, failing to capture
the nuances of sign language grammar. This can result in ambiguous medical instructions, making it
harder for patients to follow treatment plans accurately.
2. Social and Psychological Barriers
Many deaf individuals feel isolated and hesitant to seek medical care due to past experiences of
miscommunication and frustration. The lack of accessible and inclusive healthcare services contributes
to health disparities among the deaf community.
Therefore, there is an urgent need for an AI-powered health assistance system that integrates sign
language recognition, real-time translation, and interactive health guidance. Leveraging computer
vision, natural language processing (NLP), and deep learning, this system can bridge the
communication gap and ensure inclusive, accurate, and efficient healthcare interactions for individuals
with communication disabilities.
1.2 OBJECTIVE
The goal of developing an AI-based deaf assistance system that uses voice recognition to display
corresponding sign language videos involves several key objectives. The system will convert speech
into text using a speech-to-text model, map the text to a sign language sign, retrieve the appropriate
video from a dataset, and display it for the user. The dataset should contain labeled sign language videos
that correspond to specific words or phrases, and an AI model will be trained to predict the correct sign
based on the transcribed text. The system must be designed to provide real-time performance with
smooth interactions, ensuring low-latency between speech input, text transcription, and video display.
An intuitive user interface will make the system accessible to all users, and the performance must be
tested and optimized for accuracy and responsiveness.
This approach is designed to achieve the following objectives:
AI BASED DEAF ASSISTANCE SYSTEM 3
 1. Implement Voice Input Recognition:
o Convert spoken language into text using a speech-to-text model.
2. Develop Text-to-Sign Mapping:
o Map transcribed text to corresponding sign language signs from the dataset.
3. Collect and Preprocess Sign Language Dataset:
o Gather or utilize labeled sign language video datasets.
4. Train AI Model for Sign Prediction:
o Use NLP or sequence-to-sequence models to predict signs from text.
5. Retrieve and Display Sign Language Video:
o Retrieve the correct sign video from the dataset based on recognized text.
6. Design User Interface (UI):
o Develop an accessible and intuitive UI for the deaf assistance system.
7. Ensure Real-Time Interaction:
o Synchronize voice recognition, text processing, and sign video display with minimal delay.
8. Optimize Performance for Real-Time Use:
o Ensure fast processing and accurate response in real-time.
9. Evaluate System Performance:
o Test the system using datasets and real-world scenarios for accuracy and user experience.
10. Ensure Scalability and Adaptability:
o Enable the system to be expanded with more signs or languages in the future.

AI BASED DEAF ASSISTANCE SYSTEM 4

 LITERATURE SURVEY

AI BASED DEAF ASSISTANCE SYSTEM 5

 2.LITERATURE SURVEY
This literature survey delves into the evolving field of sign language processing, highlighting its
importance in bridging communication gaps for the Deaf community. It examines key aspects such as
Deaf culture, sign language linguistics, and the challenges faced in developing recognition systems. The
survey also explores existing reviews on the topic, identifying gaps in the research and suggesting areas
for improvement. The interdisciplinary nature of the field is emphasized, showing the need for
collaboration across computer science, linguistics, and Deaf studies. Through this survey, we aim to
understand the current state of sign language technologies and the necessary steps for advancing their
effectiveness.
2.1 Deaf Culture and Identity

1. Cultural Identity vs. Audiological Condition

o The term "Deaf" (with a capital "D") refers to a cultural identity, distinct from the audiological
condition of being "deaf."
o Deaf culture is built around the shared use of sign language, community bonds, and experiences
of socialization in a visual and gestural world.
2. Historical Suppression of Sign Language
o Historically, sign language communication has faced suppression, particularly in the late 19th
century when oralism was enforced in educational institutions.
o Oralism promoted the idea that Deaf individuals should use speech and lip-reading over sign
language, leading to the prohibition of sign language in classrooms.
o As a result, Deaf individuals were socially stigmatized and often punished for using sign
language, which hindered their cultural and linguistic development.
3. Validation of Sign Languages as Natural Languages
o Efforts have been made to establish sign languages as legitimate, full languages with their own
grammar, syntax, and vocabulary.
o Linguists conducted research demonstrating that sign languages have the same linguistic
properties as spoken languages, including complex grammatical structures.
o These studies challenged misconceptions and recognized sign languages as valid modes of
communication, rather than mere gestures.

4. Legislative Support for Deaf Rights

AI BASED DEAF ASSISTANCE SYSTEM 6

 o In the 20th century, legal frameworks began to support the rights of the Deaf community.

o Laws like the Americans with Disabilities Act (ADA) in the U.S. have affirmed the right of
Deaf individuals to access services in their native language.

o Legal recognition helped empower the Deaf community and increased the visibility and respect
for their language and culture.
5. Impact on Sign Language Processing Systems
o When developing sign language processing technologies, it is crucial to respect the cultural
identity and communication preferences of the Deaf community.
o Sign language recognition systems should not undermine the cultural significance of sign
language but rather aim to enhance communication and accessibility for Deaf individuals.
o Collaboration with the Deaf community is essential to ensure that these systems are
culturally respectful and useful to the users.
2.2 Challenges in Sign Language Processing
Sign language processing faces several significant challenges due to the inherent complexity of
sign languages. These challenges include:
1. Diversity in Variations: Sign languages differ regionally, socially, and individually. Variations in
signs across different geographic locations, social contexts, and individual preferences make it
difficult for recognition systems to generalize and accurately process sign language.
2. Lack of Standardized Written Form: The absence of a universally accepted written form for sign
languages makes it hard to document and annotate sign language data consistently. This lack of
standardization complicates the creation of annotated datasets needed for training machine learning
models, leading to inconsistent annotations and lower-quality datasets.
3. Data Annotation Issues: Inconsistent annotation systems and the difficulty of labeling complex
signs or continuous signing further hinder the development of robust recognition systems. Limited
and non-diverse datasets also restrict the effectiveness of models, making them less applicable
across different regions, dialects, and user groups.
2.3 Reviewed progress in Sign Language Processing
2.3.1 Limitations of Previous Reviews:
Early reviews focused mainly on technical issues like recognition algorithms and datasets,
neglecting interdisciplinary perspectives. Many were written before deep learning became
central to the field, and they often ignored the cultural, social, and linguistic contexts needed
to address the real-world needs of the Deaf community.

AI BASED DEAF ASSISTANCE SYSTEM 7

 2.3.2 Workshop and Collaborative Efforts:
Experts from various fields gathered for a two-day workshop to assess the current state of
sign language technologies. The event emphasized the need for cross-disciplinary
collaboration and led to a call for more focused efforts in addressing existing challenges in
sign language processing.
2.4 EXPLANATION OF KEY LIBRARIES
The key libraries used in this code are:

Backend (Python/Django):
• Django (core web framework)
• django.core.management – for command-line tools like manage.py
• django.contrib.auth – for login, signup, and authentication
• django.forms – for form handling (Login/Signup)
• django.shortcuts – likely used to render pages and redirect users
• django.urls – for URL routing
• CSRF protection via {% csrf_token %}

Frontend (HTML/CSS/JS):
• HTML5 – for webpage structure
• CSS – for layout, color themes, responsiveness
• JavaScript – for interactivity (e.g., mic button)
• Web Speech API (webkitSpeechRecognition) – to convert voice input to text
• HTML5 <video> element – to play sign language videos

Static & Media Assets:

• mic3.png – microphone icon
• Hello.mp4, word.mp4, etc. – sign language animation videos
• CSS styles – for split screen, buttons, form elements, etc.
Optional Additional Tools:
• SQLite/PostgreSQL – Default or custom database backend
• Django Admin – For admin panel management
• Pillow – If handling image files

AI BASED DEAF ASSISTANCE SYSTEM 8

 SYSTEM ANALYSIS

AI BASED DEAF ASSISTANCE SYSTEM 9

 3.SYSTEM ANALYSIS
System analysis for the AI-based Health Assistance System focuses on addressing the unique needs
of elderly patients, a group often facing challenges such as cognitive decline, limited mobility, and managing
complex health conditions. Traditional healthcare solutions like telehealth platforms and basic reminder apps
are inadequate for providing a comprehensive and personalized approach for this demographic. This analysis
aims to design an AI-driven system that not only enhances medication adherence through automated reminders
and dosage tracking but also supports proactive health monitoring by continuously assessing vital signs,
predicting potential risks, and adapting to changing health conditions.
The system will utilize AI algorithms to analyze real-time health data, detect anomalies, and provide timely
recommendations to both patients and caregivers. The AI system will empower elderly individuals to manage
their health with greater autonomy by offering personalized health insights, medication schedules, and
reminders. It will also facilitate seamless communication between patients, caregivers, and healthcare
providers through AI-enabled tools, ensuring that health updates are shared efficiently.
3.1 EXISTING SYSTEM
Existing systems and technologies to support people with hearing impairments or deafness include:
1. Hearing Aids:

o Function: Amplify sounds to help people with mild to moderate hearing loss.
o Types: Behind-the-ear (BTE), in-the-ear (ITE), and in-the-canal (ITC) hearing aids.

2. Cochlear Implants:

o Function: A surgically implanted device that bypasses damaged parts of the ear and stimulates the auditory
nerve directly. It's typically used for severe hearing loss.
o Process: Includes an external microphone and processor that captures sounds and transmits them to an internal
device surgically placed in the cochlea.

3. Bone-Anchored Hearing Aids (BAHA):

o Function: These devices are surgically implanted behind the ear and use bone conduction to transmit sound
vibrations to the inner ear, bypassing the middle ear.
o Usage: Ideal for people with conductive hearing loss or single-sided deafness.

4. Sign Language:
AI BASED DEAF ASSISTANCE SYSTEM 10
 o Function: A visual language used by the deaf community, with distinct hand shapes, facial
expressions, and movements to convey meaning.
o Systems: American Sign Language (ASL), British Sign Language (BSL), etc.
5. Captioning and Subtitling:
o Function: Provides real-time transcription of spoken dialogue, making media (like television,
movies, and videos) accessible to the deaf community.
o Tools: Closed captions (CC), real-time captioning software, and subtitling features.
3.2 PROPOSED SYSTEM
The proposed AI-based Deaf Assistance System leverages voice and text input to deliver real-time sign
language translation through video output, enhancing communication between deaf and hearing
individuals. This system is designed to offer a seamless, intuitive solution that automatically translates
spoken language into sign language, making communication natural and effective without requiring
constant human intervention.
Key features include:
1. Voice and Text Input: The system allows users to speak or type their messages, which are then
processed using advanced speech recognition and natural language processing (NLP) algorithms.
2. Real-time Sign Language Translation: Using AI-driven models, the system translates voice or
text into sign language in real-time, ensuring accurate representation of words and phrases through
video output.
3. Video Output: The translated content is displayed as video clips, featuring sign language gestures
performed by a digital avatar, or using pre-recorded sign language from a video dictionary to ensure
proper representation.
4. Offline Functionality: The system operates independently without the need for constant
connectivity. It stores a library of common sign language gestures and phrases, ensuring that the
translation can function effectively in areas with limited or no internet access.
5. User-Friendly Interface: Designed to be easy for both hearing and deaf individuals, the system
provides an accessible interface where users can input voice or text and instantly receive sign
language translations in video format.
3.3 Feasibility Study
A feasibility study assesses the practicality and viability of implementing the proposed system across
several dimensions:
Technical Feasibility:
• Compatible with mobile devices, leveraging frameworks like TensorFlow Lite or PyTorch Mobile.
• Real-time processing enabled with edge computing for low-latency responses.

AI BASED DEAF ASSISTANCE SYSTEM 11

 • Integrates with voice-to-text technologies for seamless communication.
Operational Feasibility:
• Enhances communication for the deaf and hard-of-hearing community in real-world settings.
• User-friendly interface with minimal training required for end-users.
• Offers real-time feedback, aiding in everyday interactions like shopping, education, and services.
• Easily integrates with existing assistive technologies like hearing aids or captioning systems.
Economic Feasibility:
• Uses open-source AI tools and frameworks to minimize development costs.

• Reduces dependence on specialized human interpreters, cutting costs for long-term assistance.

• Potential for partnership with healthcare providers, governments, or NGOs for funding.

• Low cost of deployment due to cloud-based solutions and device compatibility.

AI BASED DEAF ASSISTANCE SYSTEM 12

 SYSTEM REQUIREMENTS SPECIFICATION

AI BASED DEAF ASSISTANCE SYSTEM 13

 4.SYSTEM REQUIREMENTS SPECIFICATION
Software Requirements Specification plays an important role in creating quality software
solutions. Specification is basically a representation process. Requirements are represented in a manner
that ultimately leads to successful software implementation. Requirements may be specified in a variety
of ways. However, there are some guidelines worth following:
• Representation format and content should be relevant to the problem.
• Information contained within the specification should be nested
• Diagrams and other notational forms should be restricted in number and consistent in use.
4.1 Functional Requirements
• Django:
o Backend processing and displaying image
• Speech Recognition:
o Recognize and convert live speech to text.
o Detect ambient noise and adjust recognition sensitivity.
• ISL GIF and Image Display:
o Match recognized text with ISL GIFs or alphabet images.
o Display the appropriate ISL representation.
• User Interaction:
o GUI with buttons for "Live Voice" and "Exit".
o Display confirmation messages.
• Error Handling:
o Detect unrecognized or invalid speech.
4.2. Non-Functional Requirements
• Performance:
o Speech recognition should occur in real-time with minimal delay.

o GIF rendering must be smooth with no lag.

• Reliability:

o The system should handle noisy environments by adjusting microphone sensitivity.

• Usability:

o Simple GUI with accessible buttons.

o Display clear ISL representations.
AI BASED DEAF ASSISTANCE SYSTEM 14
 • Scalability:
o Expandable to include more ISL symbols and regional variations.
• Portability:
o Should run on Windows OS with minimal hardware dependencies.
4.3 Hardware Requirements
• Processor: Intel Core i3 or higher
• RAM: 4 GB (Minimum)
• Storage: 1 GB free space
• Microphone: For live speech input
• Display: Standard monitor with 1366x768 resolution or higher
4.4 Software Requirements
• Operating System: Windows 10 or higher
• Programming Language: Python 3.x
• Python Libraries:
o speech_recognition (for speech-to-text)
o numpy (for image processing)
o matplotlib (for displaying images)
o Django

AI BASED DEAF ASSISTANCE SYSTEM 15

 SYSTEM DESIGN

AI BASED DEAF ASSISTANCE SYSTEM 16

 5.SYSTEM DESIGN
System design is the process of defining the elements of a system such as the
architecture, modules and components, the different interfaces of those components and the data that
goes through that system. It is meant to satisfy specific needs and requirements through the
engineering of a coherent and well-running System.
Concept of UML: UML is a standard language for specifying, visualizing, constructing, and
documenting the artifacts of software systems.ML stands for Unified Modeling Language. UML is
different from the other common programming languages such as C++, Java, COBOL, etc. UML is
a pictorial language used to make software blueprints. There are a number of goals for developing
UML but the most important is to define some general-purpose modeling language, which all
modelers can use and it also needs to be made simple to understand and use.
5.1 USE CASE DIAGRAM

Fig 5.1.1 Use case Diagram

A use case diagram in the Unified Modeling language (UML) is a type of behavioral diagram defined
by and created from a Use-case analysis. Its purpose is to present a graphical overview of the
functionality provided by a system in terms of actors, their goals (represented as use case), and any
dependencies between those cases. The main purpose of a use case diagram is to show system functions
are performed for which actor. Roles of the actors in the system can be depicted.
The figure 5.1 illustrates the Use Case Diagram for the "Audio to Sign Language Tool". It identifies
two primary actors: the "User" and the "Admin". The "User" can perform actions such as converting
text/voice to sign language, viewing sign language animations, registering, logging in, contacting
support, and viewing information. The "Admin" is responsible for managing users,
uploading/updating sign videos, viewing user queries, and logging into the system. This diagram
provides a high-level overview of the system's behavioral aspects, detailing what functions are performed
for each type of user.
AI BASED DEAF ASSISTANCE SYSTEM 17
 5.2 ACTIVITY DIAGRAM

Fig 5.2.1 Activity Diagram

An activity diagram is a type of Unified Modeling Language (UML) diagram used to model workflows
and business processes. It visually represents the sequential and parallel activities within a system,
providing a high-level view of the dynamic aspects of the system. Here are key points about activity
diagrams.
The figure 5.2.1 visualizes the step-by-step workflow of the core system functionality. It outlines
the sequence of actions, starting with the user opening the converter page and providing input (text or
voice). The process continues through converting voice to text, submitting input to the Django backend,
extracting keywords, and finding matching sign videos. The diagram concludes with the preparation
and display of the video sequence, which plays the corresponding sign videos sequentially upon user
AI BASED DEAF ASSISTANCE SYSTEM 18
 interaction. This diagram helps in understanding the dynamic flow and control within this specific
process.
5.3 CLASS DIAGRAM

5.3.1Class Diagram
The figure 5.3.1 illustrates the structural organization of the system by showing its classes, attributes,
operations, and the relationships between them. Key components include "Views" (e.g., ContactView,
AnimationView), which manage different user interfaces and interactions. The "Models" section defines
the core data entities like User (storing username, email, password) and SignVideo (containing keyword
and video_path). Additionally, "Forms" such as LoginForm and SignupForm are depicted for user
authentication. This diagram provides a blueprint of the system's architecture, demonstrating how data
and functionalities are structured.

AI BASED DEAF ASSISTANCE SYSTEM 19

 5.4 SEQUNCE DIAGRAM

Fig 5.4.1 Sequence Diagram

The Sequence Diagram focuses on the Login process within the "Audio to Sign Language Tool",
illustrating the time-ordered interactions between various objects. It begins with the user providing
their username and password. These credentials are then sent to the LoginView for validation. The
LoginView interacts with the User Model to authenticate the user. Upon successful authentication, the
LoginView redirects the user to the HomePage. This diagram provides a detailed view of the message
flow and the sequence of operations involved in a user logging into the system.
5.5 SYSTEM ARCHITECTURE
The AI-based Deaf Assistance System processes voice or text input, converts it into Indian Sign Language
using NLP models, and displays the output through a 3D animated avatar for effective communication with deaf
users.

AI BASED DEAF ASSISTANCE SYSTEM 20

 5.5.1 SYSTEM ARCHITECTURE

AI BASED DEAF ASSISTANCE SYSTEM 21

 IMPLEMENTATION

AI BASED DEAF ASSISTANCE SYSTEM 22

 6.IMPLEMENTATION
6.1 WORKING DESCRIPTION
The Hearing Impairment Assistant system is designed to convert live speech into Indian Sign
Language (ISL) representations using images and GIFs. The application follows a structured process
to achieve this. It begins with Speech Recognition, where the system uses the speech_recognition
library to capture live audio from the microphone. The recognized speech is then converted into text
using the Google Speech Recognition API. Next, the Text Processing stage refines the captured text by
removing punctuation and converting it to lowercase, ensuring better matching with the ISL
vocabulary. During the ISL Representation phase, if the spoken word matches a pre-defined ISL GIF
keyword, the corresponding GIF is displayed. However, if no matching GIF is found, the system
displays ISL alphabet images letter by letter. The User Interface offers two options: "Live Voice,"
which initiates real-time speech recognition and displays the ISL translation, and "All Done!" which
exits the application. For Error Handling, in cases of unrecognized speech or background noise, the
system displays a message prompting the user to try again, ensuring smooth and user-friendly
operation.

6.2 METHODOLOGY
The methodology follows a combination of speech processing and image-based representation
to convert live speech into Indian Sign Language (ISL) symbols.
Speech Recognition:
from django.http import HttpResponse
from django.shortcuts import render, redirect
from django.contrib.auth.forms import UserCreationForm, AuthenticationForm
from django.contrib.auth import login,logout
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
import nltk
from django.contrib.staticfiles import finders
from django.contrib.auth.decorators import login_required
def home_view(request):
return render(request,'home.html')
def about_view(request):
return render(request,'about.html')
def contact_view(request):
AI BASED DEAF ASSISTANCE SYSTEM 23
 return render(request,'contact.html')
@login_required(login_url="login")
def animation_view(request):
if request.method == 'POST':
text = request.POST.get('sen')
#tokenizing the sentence
text.lower()
#tokenizing the sentence
words = word_tokenize(text)
tagged = nltk.pos_tag(words)
tense = {}
tense["future"] = len([word for word in tagged if word[1] == "MD"])
tense["present"] = len([word for word in tagged if word[1] in ["VBP", "VBZ","VBG"]])
tense["past"] = len([word for word in tagged if word[1] in ["VBD", "VBN"]])
tense["present_continuous"] = len([word for word in tagged if word[1] in ["VBG"]])
#stopwords that will be removed
stop_words = set(["mightn't", 're', 'wasn', 'wouldn', 'be', 'has', 'that', 'does', 'shouldn', 'do',
"you've",'off', 'for', "didn't", 'm', 'ain', 'haven', "weren't", 'are', "she's", "wasn't", 'its', "haven't",
"wouldn't", 'don', 'weren', 's', "you'd", "don't", 'doesn', "hadn't", 'is', 'was', "that'll", "should've", 'a', 'then',
'the', 'mustn', 'i', 'nor', 'as', "it's", "needn't", 'd', 'am', 'have', 'hasn', 'o', "aren't", "you'll", "couldn't",
"you're", "mustn't", 'didn', "doesn't", 'll', 'an', 'hadn', 'whom', 'y', "hasn't", 'itself', 'couldn', 'needn',
"shan't", 'isn', 'been', 'such', 'shan', "shouldn't", 'aren', 'being', 'were', 'did', 'ma', 't', 'having', 'mightn', 've',
"isn't", "won't"])
#removing stopwords and applying lemmatizing nlp process to words
lr = WordNetLemmatizer()
filtered_text = []
for w,p in zip(words,tagged):
if w not in stop_words:
if p[1]=='VBG' or p[1]=='VBD' or p[1]=='VBZ' or p[1]=='VBN' or p[1]=='NN':
filtered_text.append(lr.lemmatize(w,pos='v'))
elif p[1]=='JJ' or p[1]=='JJR' or p[1]=='JJS'or p[1]=='RBR' or p[1]=='RBS':
filtered_text.append(lr.lemmatize(w,pos='a'))
else:
filtered_text.append(lr.lemmatize(w))
#adding the specific word to specify tense
AI BASED DEAF ASSISTANCE SYSTEM 24
 words = filtered_text
temp=[]
for w in words:
if w=='I':
temp.append('Me')
else:
temp.append(w)
words = temp
probable_tense = max(tense,key=tense.get)
if probable_tense == "past" and tense["past"]>=1:
temp = ["Before"]
temp = temp + words
words = temp
elif probable_tense == "future" and tense["future"]>=1:
if "Will" not in words:
temp = ["Will"]
temp = temp + words
words = temp
else:
pass
elif probable_tense == "present":
if tense["present_continuous"]>=1:
temp = ["Now"]
temp = temp + words
words = temp
filtered_text = []
for w in words:
path = w + ".mp4"
f = finders.find(path)
#splitting the word if its animation is not present in database
if not f:
for c in w:
filtered_text.append(c)
#otherwise animation of word
else:
AI BASED DEAF ASSISTANCE SYSTEM 25
 filtered_text.append(w)
words = filtered_text;
return render(request,'animation.html',{'words':words,'text':text})
else:
return render(request,'animation.html')
def signup_view(request):
if request.method == 'POST':
form = UserCreationForm(request.POST)
if form.is_valid():
user = form.save()
login(request,user)
# log the user in
return redirect('animation')
else:
form = UserCreationForm()
return render(request,'signup.html',{'form':form})
def login_view(request):
if request.method == 'POST':
form = AuthenticationForm(data=request.POST)
if form.is_valid():
#log in user
user = form.get_user()
login(request,user)
if 'next' in request.POST:
return redirect(request.POST.get('next'))
else:
return redirect('animation')
else:
form = AuthenticationForm()
return render(request,'login.html',{'form':form})

def logout_view(request):
logout(request)
return redirect("home")

AI BASED DEAF ASSISTANCE SYSTEM 26

 6.3 SAMPLE CODE
Manage.py
#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys
def main():
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'A2SL.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(sys.argv)
if __name__ == '__main__':
main()
Setup.py
import setuptools

setuptools.setup(
name='audio-speech-to-sign-language-converter',
version='0.1.0',
description='Python project',
author='Jigar Gajjar',
author_email='jigargajjar.55jg@gmail.com',
url='https://github.com/jigargajjar55/Audio-Speech-To-Sign-Language-Converter',
packages=setuptools.find_packages(),
setup_requires=['nltk', 'joblib','click','regex','sqlparse','setuptools'],
)

AI BASED DEAF ASSISTANCE SYSTEM 27

 6.4 WEB APPLICATION
#home.html
{% extends 'base.html' %}
{% load static %}
{% block content %}
<video width="500" height="380" class="center" autoplay loop>
<source src= "{% static 'Hello.mp4' %}" type="video/mp4">
Your browser does not support the video tag.
</video>
<div class="form-style" align="middle" >
<a href="{% url 'animation' %}"><button class="button">Click to Start</button></a>
</div>
{% endblock %}
#login.html
{% extends 'base.html' %)
{% block content %}
<div class="form-style">
<h1>Log in</h1>
<form class="site-form" action="." method="post">
{% csrf_token %}
{{ form }}
{% if request.GET.next %}
<input type="hidden" name="next" value="{{ request.GET.next }}">
{% endif %}
<input class="submit" type="submit" value="Log in">
</form>
</div>
{% endblock %}
#signup.html
{% extends 'base.html' %}

AI BASED DEAF ASSISTANCE SYSTEM 28

 {% block content %}
<div class="form-style">
<h1>Sign Up</h1>
<form class="site-form" action="." method="post">
{% csrf_token %}
{{ form }}
<br><br>
<input class="submit" type="submit" value="Sign Up">
</form>
</div>
<script type="text/javascript">
document.getElementsByTagName("span")[0].innerHTML="";
document.getElementsByTagName("span")[1].innerHTML="";
</script>
{% endblock %}
#about.html
{% extends 'base.html' %}

{% block content %}

<h2>VERSION 1.0.0</h2>
<hr>
<h2>We are just a bunch of Enthusiastic people,who wants to help The Society.</h2>
<hr>
<h2>Our Creator Team:</h2>
<hr>
<ul class="td">
<li>Ravali Vootla</li>
<li>Vasu Dubbanaboina</li>
<li>Mani Srinivas Gajula</li>
<li>Likith Yerramshetty</li>
</ul>
<hr>
<footer><p class="td">Thank you, For visiting our website</p></footer>
AI BASED DEAF ASSISTANCE SYSTEM 29
 {% endblock %}

#animation.html
{% extends 'base.html' %}
{% load static %}

{% block content %}
<div class="split left">
<h2 align="center">Enter Text or Use Mic</h2>
<br>
<form action="" method="post" align="left">
{% csrf_token %}
<input type="text" name="sen" class="mytext" id="speechToText" placeholder="">
<button type="button" name="button" class="mic" onclick="record()"><img src="{% static
'mic3.png' %}" height="32px" width="38px" /></button>
&nbsp&nbsp&nbsp&nbsp
<input type="submit" name="submit" class="submit">

</form>
<br>
<table cellspacing="20px">
<tr>
<td class="td">The text that you entered is:</td>
<td class="td">{{ text }}</td>
</tr>
<tr>
<td class="td">Key words in sentence:</td>
<td class="td">
<ul class="td" id="list" align="center">
{% for word in words %}
<li id="{{ i }}" style="margin-right: 8px">{{ word }}</li>
{% endfor %}
</ul>
</td>
</tr>
AI BASED DEAF ASSISTANCE SYSTEM 30
 </table></div>
<div class="split right">
<h2 align="center">Sign Language Animation</h2>

<div style="text-align:center"> &nbsp&nbsp

<button class="submit" onclick="playPause()">Play/Pause</button>
<!-- LETS USE ONLY PLAY PAUSE BUTTON WITH 720 *420 ,IT FITS THE SCREEN ASHISH

<button onclick="makeBig()">Big</button>
<button onclick="makeSmall()">Small</button>
<button onclick="makeNormal()">Normal</button>-->

<video id="videoPlayer" width="600" height="350" preload="auto" autoplay>

<source src="" type="video/mp4">
Your browser does not support HTML5 video.
</video>
</div></div>

<script>
//webkitSpeechRecognition api for speech to text conversion
function record(){
var recognition = new webkitSpeechRecognition();
recognition.lang='en-IN';

recognition.onresult = function(event){
console.log(event)
document.getElementById('speechToText').value = event.results[0][0].transcript;
}
recognition.start();
}
function play()
{
var videoSource = new Array();
var videos = document.getElementById("list").getElementsByTagName("li");
var j;
AI BASED DEAF ASSISTANCE SYSTEM 31
 for(j=0;j<videos.length;j++)
{
videoSource[j] = "/static/" + videos[j].innerHTML +".mp4";
}

var i = 0; // define i
var videoCount = videoSource.length;

function videoPlay(videoNum)
{
document.getElementById("list").getElementsByTagName("li")[videoNum].style.color =
"#09edc7";
document.getElementById("list").getElementsByTagName("li")[videoNum].style.fontSize =
"xx-large";
document.getElementById("videoPlayer").setAttribute("src", videoSource[videoNum]);
document.getElementById("videoPlayer").load();
document.getElementById("videoPlayer").play();

}
document.getElementById('videoPlayer').addEventListener('ended', myHandler, false);
document.getElementById("list").getElementsByTagName("li")[0].style.color = "#09edc7";
document.getElementById("list").getElementsByTagName("li")[0].style.fontSize = "xx-large";

videoPlay(0); // play the video

function myHandler()
{
document.getElementById("list").getElementsByTagName("li")[i].style.color = "#feda6a";
document.getElementById("list").getElementsByTagName("li")[i].style.fontSize = "20px";
i++;
if (i == videoCount)
{
document.getElementById("videoPlayer").pause();
}
else
AI BASED DEAF ASSISTANCE SYSTEM 32
 {
videoPlay(i);
}
}
}
/* LETS USE ONLY PLAY PAUSE BUTTON WITH 720 *420 ,IT FITS THE SCREEN
function makeBig() {
document.getElementById("videoPlayer").width = 560;
document.getElementById("videoPlayer").height = 360;
}

function makeSmall() {
document.getElementById("videoPlayer").width = 320;
document.getElementById("videoPlayer").height = 180;
}

function makeNormal() {
document.getElementById("videoPlayer").width = 420;
document.getElementById("videoPlayer").height =280;
}*/
function playPause(){
if (document.getElementById("videoPlayer").paused){
play();}
else{
document.getElementById("videoPlayer").pause();}
}
</script>
{% endblock %}
#base.html
{% load static %}
<!DOCTYPE html>
<html>
<head>
<title>Homepage</title>
</head>
AI BASED DEAF ASSISTANCE SYSTEM 33
 <div style="background-color:#404040;color: blueviolet;padding:10 10 1 10;border: 1px blueviolet
groove;margin-bottom:0;">

<h1 align=center color="blueviolet">Audio To Sign Language Tool</h1>

</div>
<br>
<body>
<ul id="nav">
<li class="li"><a class="active" href="{% url 'home' %}">Home</a></li>
<li class="li"><a href="{% url 'animation' %}">Convertor</a></li>
{% if not user.is_authenticated %}
<li class="li"><a href="{% url 'signup' %}">Sign Up</a></li>
{% endif %}
{% if user.is_authenticated %}
<li class="li"><a href="{% url 'logout' %}">Log-Out</a></li>
{% else %}
<li class="li"><a href="{% url 'login' %}">Log-in</a></li>
{% endif %}
<li class="li"><a href="{% url 'contact' %}">Contact</a></li>
<li class="li"><a href="{% url 'about' %}">About</a></li>
</ul>
<div class="wrapper" >
{% block content %}
{% endblock %}
</div>
</body>
</html>
#contact.html
{% extends 'base.html' %}
{% block content %}
<h2>VERSION 1.0.0</h2>
<hr>
<h2>CONTACT US</h2>
<p class="td">For any queries regarding this website contact us on following:</p>
<p class="td">Our Email ID:ravalivootla026@gmail.com</p>
AI BASED DEAF ASSISTANCE SYSTEM 34
 <!-- <p class="td">Twitter handle:@audio2sl</p> -->
<p class="td">Contact number:7842770523</p>
<hr>
<p class="td">Thank you, For visiting our website</p>
{% endblock %}

AI BASED DEAF ASSISTANCE SYSTEM 35

 TESTING

AI BASED DEAF ASSISTANCE SYSTEM 36

 7.TESTING
7.1 Unit Testing
Unit testing involves testing individual components of the system to ensure they function correctly.
The Hearing Impairment Assistant system has several components, such as speech recognition, text
processing, ISL representation, and GUI interaction. Each component is tested separately to verify its
accuracy and reliability.
Testing Components:
• Speech Recognition: Verifies that the system accurately captures and converts speech to text.
• Text Processing: Ensures that punctuation is removed correctly and text is properly converted to
lowercase.
• ISL Representation: Validates that the system correctly matches the speech-to-text output with the
ISL GIF or alphabet images.
• GUI Interaction: Tests the functionality of the buttons and verifies smooth navigation.
7.2 Integration Testing
Integration testing ensures that different modules of the system work together as expected. This
involves testing the interaction between speech recognition, text processing, and ISL representation
components.
Integration Testing Steps:
1.Speech Recognition + Text Processing: Ensure that the recognized speech is accurately processed
into lowercase text with no punctuation.
1. Text Processing + ISL Representation: Verify that processed text is correctly matched with
ISL GIFs or alphabet images.
2. GUI + Backend Integration: Ensure the GUI buttons correctly initiate speech recognition and
display ISL images or GIFs.
7.3 Performance Testing
Performance testing evaluates how efficiently the system processes speech and displays ISL
representations. It ensures that the system responds in real-time with minimal delay.
Performance Metrics:
• Speech Recognition Time: The time taken to capture and convert speech into text.
• ISL Representation Time: The time taken to load and display the corresponding ISL GIF or
alphabet images.
• Memory and CPU Usage: Measure the resource consumption during runtime.

AI BASED DEAF ASSISTANCE SYSTEM 37

 RESULTS

AI BASED DEAF ASSISTANCE SYSTEM 38

 8. RESULT

• The Hearing Impairment Assistant system effectively converts live speech into Indian Sign Language
(ISL) representations using images and GIFs.
• It accurately captures and processes speech using the Google Speech Recognition API, converting it
into lowercase text while removing punctuation for better matching with the ISL vocabulary.
• The system displays the corresponding ISL GIF for recognized words and shows alphabet images
letter by letter for unrecognized words.
• The GUI provides two options: "Live Voice" to initiate speech recognition and display the ISL
translation, and "All Done!" to exit the application.

Fig 8.1.1

Fig8.1.1 represents User interface of the Audio to Sign Language Tool showcasing the avatar ready to
perform sign language animations based on user input.

AI BASED DEAF ASSISTANCE SYSTEM 39

 Fig 8.2.1
Fig 8.2.1 represents Login page of the Audio to Sign Language Tool allowing users to securely
access the system to convert audio or text into sign language animations.

AI BASED DEAF ASSISTANCE SYSTEM 40

 Fig 8.3.1
Fig 8.3.1 represents Text or voice input is processed and translated into animated Indian Sign Language
gestures by the virtual avatar, enhancing accessibility for deaf users,i.e “have your dinner”.

AI BASED DEAF ASSISTANCE SYSTEM 41

 Fig 8.4.1

Fig 8.4.1 represents another example for Text or voice input is processed and translated into animated
Indian Sign Language gestures by the virtual avatar, enhancing accessibility for deaf users,i.e “maintain
silence”.

AI BASED DEAF ASSISTANCE SYSTEM 42

 Fig 8.5.1
Fig 8.5.1 represents another example for Text or voice input is processed and translated into animated Indian Sign
Language gestures by the virtual avatar, enhancing accessibility for deaf users,i.e “Honey”.

AI BASED DEAF ASSISTANCE SYSTEM 43

 CONCLUSION

AI BASED DEAF ASSISTANCE SYSTEM 44

 9.CONCLUSION
In conclusion , the propsed system an AI-based Deaf Assistance System is designed to
convert English audio into sign language, enabling real-time communication for deaf individuals.
By leveraging advanced AI technologies like deep learning and speech recognition, the system
processes spoken language and translates it into visual sign language through avatars or animations.
It provides seamless, accessible interactions across various environments, from healthcare to public
services, reducing the reliance on human interpreters. The system promotes inclusivity and
empowers deaf individuals to engage in everyday conversations, enhancing their social and
professional experiences. Additionally, its scalability and cost-effective nature make it a viable
solution for widespread adoption.

AI BASED DEAF ASSISTANCE SYSTEM 45

 BIBLIOGRAPHY

AI BASED DEAF ASSISTANCE SYSTEM 46

 10. BIBLIOGRAPHY
[1] Nicoletta Adamo-Villani and Ronnie B. Wilbur. 2015. ASL-Pro: American Sign Language
Animation with Prosodic Elements. In Universal Access in Human-Computer Interaction. Access
to Interaction, Margherita Antona and Constantine Stephanidis (Eds.). Springer International
Publishing, Cham, 307–318.
[2] M Ebrahim Al-Ahdal and Md Tahir Nooritawati. 2012. Review in sign language recognition
systems. In 2012 IEEE Symposium on Computers & Informatics (ISCI). IEEE, 52–57.
[3] Sedeeq Al-khazraji, Larwan Berke, Sushant Kafe, Peter Yeung, and Matt Huenerfauth. 2018.
Modeling the Speed and Timing of American Sign Language to Generate Realistic Animations. In
Proceedings of the 20th International ACM SIGACCESS Conference on Computers and
Accessibility. ACM, 259–270.
[4] Anwar AlShammari, Asmaa Alsumait, and Maha Faisal. 2018. Building an Interactive E-
Learning Tool for Deaf Children: Interaction Design Process Framework. In 2018 IEEE Conference
on e-Learning, e-Management and e-Services (IC3e). IEEE, 85–90.
[5] UN General Assembly. 2006. Convention on the Rights of Persons with Disabilities. GA Res
61 (2006), 106.

AI BASED DEAF ASSISTANCE SYSTEM 47