© 2023 IJRAR May 2023, Volume 10, Issue 2 www.ijrar.

org (E-ISSN 2348-1269, P- ISSN 2349-5138)

Real-Time Sign Language Recognition using Deep

Learning and Computer Vision: A Flask-based
SLR System
1Sagar More, 2Aditya Chaudhari, 3Pravin Mandlik, 4Shubham Bhoknal
5
Prof. Pratima Chougule
1-4
Student, 5Project Guide
1-4
Student of Computer Engineering Department, Dr. D. Y. Patil College of Engineering and Innovation,
Varale, Pune, Maharashtra, India
5
Assistant Professor, Department of Artificial Intelligence and Data Science, Dr. D. Y. Patil College of
Engineering and Innovation, Varale, Pune, Maharashtra, India
Abstract: Sign language plays a pivotal role in facilitating communication for individuals with speech and hearing impairments.
However, understanding and interpreting sign language poses challenges for those unfamiliar with this visual language. Existing
sign language recognition systems rely on either sensor data or camera-based approaches, focusing primarily on American Sign
Language and limited to static pattern recognition. This paper presents an innovative approach to sign language recognition that
harnesses the power of deep learning. The proposed system aims to bridge the communication gap by accurately interpreting various
sign languages in real-time. By leveraging cutting-edge techniques, this project seeks to create a more inclusive environment,
allowing seamless interaction between sign language users and individuals without sign language proficiency. The project's
outcomes have the potential to revolutionize communication and accessibility for individuals with speech and hearing impairments,
paving the way for more comprehensive and adaptable sign language recognition systems. Our system implements a Flask web
application that utilizes a pre-trained deep learning model for real-time image classification using a webcam feed. The application
loads a pre-trained model using Keras and captures video frames from the webcam using OpenCV. Each frame is preprocessed by
resizing, normalizing, and feeding it into the model for prediction. The predicted class name and confidence score are extracted and
displayed on the frames. The processed frames are encoded as JPEG images and streamed as a video feed through Flask routes. The
application provides a user-friendly interface for real-time image classification using the webcam, demonstrating the integration of
Flask, OpenCV, and Keras libraries.

Keywords – Deep Learning, Flask, Keras, OpenCV, ISL, ASL

I. INTRODUCTION
Sign language is a fully developed language with a distinct structure that sets it apart from spoken languages. The global
population of deaf individuals exceeded 70 million in 2017, as reported by the "World Federation of the Deaf." Sign language serves
as the primary mode of communication for the hearing-impaired, with only a select group of individuals, including relatives, activists,
and educators, possessing the ability to understand and utilize it. Communication among the hearing-impaired community relies
heavily on text and visual expressions. Sign Language Recognition has emerged as an area of active research and development,
aiming to bridge the communication divide between sign language users and those who are unfamiliar with sign language. Sign
languages encompass intricate visual-gestural systems employed by the deaf and hard-of-hearing community to convey complex
meanings and thoughts effectively. However, the understanding and interpretation of sign language pose significant challenges for
individuals who lack exposure to this unique language form. The primary goal of Sign Language Recognition is to develop intelligent
systems capable of accurately recognizing and understanding sign language gestures. By harnessing advanced technologies such as
computer vision, machine learning, and deep learning, these systems can analyze video recordings or real-time camera feeds to
recognize and translate sign language gestures into text or spoken language.

IJRAR23B3375 International Journal of Research and Analytical Reviews (IJRAR) 237

© 2023 IJRAR May 2023, Volume 10, Issue 2 www.ijrar.org (E-ISSN 2348-1269, P- ISSN 2349-5138)
II. LITERATURE REVIEW
Author Name / Year Paper Title Proposed Description Accuracy
Method
Qazi Mohammad Areeb, Helping Hearing- Convolutional for detecting hand classification model
Maryam, Mohammad Impaired in Emergency Neural Networks gestures, CNN is achieved 98%
Nadeem, Roobaea Situations: A Deep (CNN), recurrent applied for image accuracy and
Alroobaea, And Faisal Learning-Based neural networks recognition and detection model
Anwer,2022 Approach (RNN,) and Long processing object achieved 99.6%
short-term detection, feature accuracy.
memory (LSTM) learning, and
sequence prediction
with RNN
Rachana Patil, Vivek Indian Sign Language SGDC (stochastic classification or The Convolutional
Patil, Abhishek recognition using gradient descent) recognition of images Neural Network
and Adam
Bahuguna, and Mr. Conventional Neural that extract features model achieved a
(Combination of
Gaurav Datkhile,2021 Network. Adagrad and from- contour, 95% accuracy
RMSProp) geometrical features
such as position,
angle, distance, etc.
Color feature,
histogram.
Farman Shah, Sign Language Support Vector The technique accuracy of Speeded
Muhammad Saqlain Recognition Using Machine (SVM) employs the idea of Up Robust Features
Shah, Waseem Akram, Multiple Kernel multiple kernel (SURF) is only 15%.
Awais Manzoor, Rasha Learning: A Case Study learning utilization.
Orban Mahmoud, And of Pakistan Sign
Diaa Salama Language
Abdelminaam,2021
Sarthak Sharma, Preet Real-Time Sign Convolutional For Real-time Sign High accuracy rate
Kaur Nagi, Rahul Ahuja, Language Detection and Neural Network Language Gesture
Poorti Rajani, Senior Recognition Recognition Systems
Asst. Prof. Kavita Different tools are
Namdev,2021 used which are Deep
Learning, OpenCV,
TensorFlow, Keras
Sevgi Z. Gurbuz, 2021 American Sign Frequency warped RF sensing to 95% recognition
Language Recognition cepstral provide non-contact accuracy for 5 signs
coefficients
Using RF Sensing ASL recognition and 72% accuracy
(FWCC),
minimum capabilities in for 20 signs
redundancy support of ASL-
maximum sensitive smart
relevance environments
(mRMR)
Saleh Aly And Walaa DeepArSLR: A Novel CSOM Resnet-50 is utilized DeepLabv3+
Aly, 2020 Signer-Independent (Convolutional as a backbone
self-organizing : 89.5%
Deep Learning network to learn
MAP). Resnet-50
Framework for Isolated hand-shaped features
Arabic Sign Language in the DeepLabv3+
Gestures Recognition model.
Purva Chaitanya Badhe, Artificial Neural Otsu’s global RGB videos as a The training
Vaishali Kulkarni, 2020 Network-based Indian thresholding, Fast database. accuracy is 98% and
Fourier Transform
Sign Language the validation
(FFT)
Recognition using accuracy is 63%
handcrafted features
Kusumika Krori Dutta, Machine Learning K-Nearest Images were trained K-NN: 100%
Sunny Arokia Swamy Techniques for Indian Neighbor (K-NN), separately using recognition
Bellary, 2017 Sign Language Back Propagation KNN techniques and Back Propagation:
Recognition Backpropagation was 94-96%
IJRAR23B3375 International Journal of Research and Analytical Reviews (IJRAR) 238
© 2023 IJRAR May 2023, Volume 10, Issue 2 www.ijrar.org (E-ISSN 2348-1269, P- ISSN 2349-5138)
used for training and Accuracy of
feature extraction. recognition.

III. METHODOLOGY

Fig 3.1: Basic Methodology of Model

3.1 Data Collection
The process of gathering information in this algorithm involves capturing screenshots from the video feed and using landmarks
to identify specific frames that meet certain criteria. These frames are then saved as pictures to facilitate various tasks like training
a machine learning model to detect sign language.

Fig 3.1.1: Dataset used for training.

3.2 Preprocessing
After pre-processing, the pre-processed image data is ready to be fed into the loaded model for prediction. These pre-processing
steps ensure that the captured frames are transformed into a suitable format for the model to process and make accurate predictions.

3.3 Feature Extraction

Feature extraction is implicitly performed by the pre-trained model itself when making predictions on the preprocessed image
data. The code provided focuses on preprocessing the frames from the camera and using the pre-trained model to perform
predictions rather than explicitly performing separate feature extraction steps.

3.4 Classification
The classification is performed by passing the preprocessed image through the pre-trained model, obtaining the class probabilities,
and selecting the class with the highest probability as the predicted class. The code then displays the predicted class and confidence
score on the image before streaming it as a response to the web application.

3.5 Recognition
The code uses the captured frames, preprocesses them, feeds them into the pre-trained Keras model for prediction, and annotates.
the frames with the predicted class and confidence score. The annotated frames are then streamed as a video feed in real-time.
recognition using Flask.

IV. System Analysis

The described system provides a real-time sign language recognition solution using Flask, a web framework, and Keras, a deep
learning library. The system employs computer vision techniques to capture video frames from a camera feed and processes them for
classification. The captured frames are preprocessed by resizing, converting to the appropriate color space, and normalizing the pixel
values. This ensures consistency and compatibility with the pre-trained Keras model.
The pre-trained Keras model is loaded, which has been previously trained on a labeled dataset. The model predicts the class
probabilities for the preprocessed image frames. The highest probability class is selected as the predicted class label, and the
corresponding confidence score is computed. These predictions are overlaid on the frames using OpenCV, allowing real-time
visualization of the classification results.
The system utilizes Flask to create a web application that serves the processed frames as a video feed. It leverages Flask's capabilities
to handle HTTP requests and responses, allowing users to access the video feed through a web browser. The Flask application defines
routes for the home page, video feed, and a quit endpoint. The video feed route streams the processed frames as a multipart response,
enabling real-time display of the classification results on the client-side.

IJRAR23B3375 International Journal of Research and Analytical Reviews (IJRAR) 239

© 2023 IJRAR May 2023, Volume 10, Issue 2 www.ijrar.org (E-ISSN 2348-1269, P- ISSN 2349-5138)
The System involves evaluating the efficiency and performance of the real-time image classification system. Key metrics for
analysis include the frame processing time, classification accuracy, and system responsiveness. The frame processing time
measures the time taken to preprocess and classify each frame, influencing the system's real-time capabilities. The classification
accuracy indicates the reliability of the model's predictions, which can be assessed by comparing the predicted labels with ground
truth labels. The system's responsiveness refers to the ability to handle concurrent requests and deliver a smooth video stream with
minimal latency.
II. RESULT
The result of this project is a sign language recognition system that utilizes deep learning models and computer vision techniques
to accurately recognize and interpret sign language gestures in real-time. The system demonstrates high accuracy and performance
in recognizing a variety of sign language gestures, allowing for effective communication between sign language users and non-
sign language users. The project lays the groundwork for future advancements in the field of sign language recognition, with
potential applications in accessibility, communication technology, and inclusive interaction.

Fig 5.1: Prediction of Sign Language

III. CONCLUSION
The sign language recognition project presents a valuable solution to bridge the communication gap between sign language users
and non-sign language users. Through the utilization of deep learning models and computer vision techniques, the project
demonstrates the ability to accurately recognize and interpret sign language gestures in real-time. The system's performance and
accuracy are enhanced by preprocessing techniques, model training, and optimization.

The project's future scope encompasses a wide range of possibilities for further development and expansion. These include
recognizing a broader range of gestures beyond sign language, supporting multiple sign languages, incorporating real-time
translation capabilities, improving the user interface, optimizing performance, developing a mobile application, expanding the
dataset, integrating with accessibility initiatives, conducting real-world testing, and fostering collaboration through open-source
contributions.

By addressing these future directions, the sign language recognition project can advance the field of accessibility and
communication technology, promoting inclusivity and empowering sign language users to engage more effectively with the world
around them. The project serves as a foundation for further research and development in sign language recognition, paving the
way for innovative applications and solutions that have the potential to positively impact the lives of individuals with hearing or
speech impairments.

IV. FUTURE SCOPE

The system can be further developed to recognize and interpret a broader range of sign language gestures beyond the initial set.
This can involve expanding the gesture vocabulary or incorporating specialized gestures for specific domains or applications. The
project can be extended to support multiple sign languages, considering the unique gestures and structures of each language. This
would involve collecting and labeling datasets for different sign languages and training language-specific models.

Integrating the sign language recognition system with smart devices opens new possibilities for seamless and hands-free
interaction. By incorporating the system into devices such as smartwatches, smart glasses, or voice assistants, users can access its
functionality in a convenient and intuitive manner.

Continuously collecting user feedback and data to improve the gesture recognition model can enhance the system's accuracy and
adaptability. This can involve implementing user feedback mechanisms, crowdsourcing data collection, and leveraging active
learning techniques.

IJRAR23B3375 International Journal of Research and Analytical Reviews (IJRAR) 240

© 2023 IJRAR May 2023, Volume 10, Issue 2 www.ijrar.org (E-ISSN 2348-1269, P- ISSN 2349-5138)
V. ACKNOWLEDGMENT
We extend our heartfelt gratitude to Dr. Suresh Mali, the Principal of DYPCOEI (Dr. DY Patil College of Engineering and
Innovation), for his kind assistance and unwavering motivation throughout our project. His guidance and support were
instrumental in our success, and we are truly grateful for his mentorship.
Our profound gratitude goes out to Dr. Alpana Adsul, the head of the Department of Computer Engineering. Her invaluable
advice and continuous support played a crucial role in shaping our project. We are thankful for her expertise and dedication to
fostering our growth as computer engineering students.
We would like to convey our sincere appreciation to Dr. Deepali Sale, the project coordinator, for her helpful insights and
constructive comments during the planning and development stages. Her input significantly enhanced the quality of our work,
and we are grateful for her guidance throughout the project.
Lastly, we would like to express our heartfelt thanks to Prof. Pratima Chougule, our guide, for her exceptional guidance and
ongoing oversight. Her expertise and mentorship were vital in steering us in the right direction and ensuring the successful
completion of our project. Additionally, we are grateful to her for providing us with important information about job
opportunities related to our project and for assisting us in the final stages of its completion.
Overall, we are immensely grateful to Dr. Suresh Mali, Dr. Alpana Adsul, Dr. Deepali Sale, and Prof. Pratima Chougule for their
unwavering support, guidance, and contributions to our project. Their assistance has been invaluable, and we are indebted to them
for their immense help and encouragement.

REFERENCES
[1] Sagar More, Aditya Chaudhari, Pravin Mandlik, Shubham Bhoknal, A Review On Different Technical Appproaches Of Sign
Language Recognition, International Research Journal Of Modernization In Engineering Technology And Science, Volume 4,2022,
E-Issn: 2582-5208
[2] Qazi Mohammad Areeb, Maryam, Mohammad Nadeem, Roobaea Alroobaea, And Faisal Anwer, Helping Hearing-Impaired In
Emergency Situations: A Deep Learning-Based Approach, Institute Of Electrical And Electronics Engineers Access, Volume 10,
2022, Pp 8502-8517.
[3] Rachana Patil, Vivek Patil, Abhishek Bahuguna And Mr. Gaurav Datkhile, Indian Sign Language Recognition Using
Conventional Neural Network, International Conference On Automation, Computing And Communication,2021, India, Pp 1-5.

[4] Farman Shah, Muhammad Saqlain Shah, Waseem Akram, Awais Manzoor, Rasha Orban Mahmoud And Diaa Salama
Abdelminaam, Sign Language Recognition Using Multiple Kernel Learning: A Case Study Of Pakistan Sign Language, Institute
Of Electrical And Electronics Engineers Access, Volume 9, 2021, Pp 67548-67558.

[5] Sarthak Sharma, Preet Kaur Nagi, Rahul Ahuja, Poorti Rajani, Senior Asst. Prof. Kavita Namdev, Realtime Sign Language
Detection, And Recognition, International Journal For Research In Applied Science & Engineering Technology, Volume 9, 2021,
Pp 1944-1948.

[6] Sevgi Z. Gurbuz, American Sign Language Recognition Using Rf Sensing, Ieee Sensors Journal, Volume 21 No. 3, 2021, Pp
3763-3775.

[7] Saleh Aly And Walaa Aly, Deeparslr: A Novel Signer-Independent Deep Learning Framework For Isolated Arabic Sign
Language Gestures Recognition, Institute Of Electrical And Electronics Engineers Access, Volume 4, 2020, Pp 1-14.

[8] Purva Chaitanya Badhe, Vaishali Kulkarni, Artificial Neural Network Based Indian Sign Language Recognition Using
Handcrafted Features, Institute Of Electrical And Electronics Engineers, 2020.

[9] Kusumika Krori Dutta, Sunny Arokia Swamy Bellary, Machine Learning Techniques For Indian Sign Language Recognition,
Institute Of Electrical And Electronics Engineers, 2017, Pp 333-336

IJRAR23B3375 International Journal of Research and Analytical Reviews (IJRAR) 241