www.ijcrt.

org © 2023 IJCRT | Volume 11, Issue 5 May 2023 | ISSN: 2320-2882

GESTURE RECOGNITION BASED VIRTUAL

MOUSE AND KEYBOARD
1
Ajay Sarwade, 2Gautami Sirsul, 3Shruti Salaskar, 4Vrushali Chennur
1
BE Computer, 2 BE Computer, 3 BE Computer, 4 BE Computer
1
Department of Computer Engineering,
1
Department of Computer Engineering, Ahmednagar, India

Abstract:
Now a days computer vision has reached its pinnacle, where a computer can identify its owner using a simple program of image
processing. In this stage of development, people are using this vision in many aspects of day to day life, like Face Recognition,
Color detection, Automatic car, etc. In this project, computer vision is used in creating an Optical mouse and keyboard using hand
gestures. The camera of the computer will read the image of different gestures performed by a person’s hand and according to the
movement of the gestures the Mouse or the cursor of the computer will move, even perform right and left clicks using different
gestures. Similarly, the keyboard functions may be used with some different gestures, like using one finger
gesture for alphabet select and four-figure gesture to swipe left and right. It will act as a virtual mouse and keyboard with no wire
or external devices. The only hardware aspect of the project is a web-cam and the coding is done on python using Anaconda
platform. Here the Convex hull defects are first generated and then using the defect calculations an algorithm is generated and
mapping the mouse and keyboard functions with the defects. Mapping a couple of them with the mouse and keyboard, the
computer will understand the gesture shown by the user and act accordingly.

I. INTRODUCTION

The Computer webcam is capturing the video of the person sitting in front of thecomputer, there will be a small green box which
will be generated in the middle of the screen. In that green box, the objects shown will be processed by the code and matched with
it if it matches then a red colored border will be generated, which means the computer has identified the object and then by moving
the object the mouse cursor can be moved. This will not only help in the security of the computer but also help in generating a
virtual computational experience. Here in the place of different objects, using hand gestures one gesture will be moving the cursor,
the different gesture will be used for right click and different for left click, similarly with a simple gesture can do the keyboard
functions virtually that may have been done on some keyboard as a physical aspect. It the gesture does not match the box will show

Motivation
Gestures- Common in personal communication
GIVES A WIDE SCOPE IN DEVELOPING A UNIQUE WAY OF HUMAN-MACHINE INTERACTION PREPARE YOUR PAPER BEFORE STYLING

Problem Definition
Generally for personal use in computers and laptops we use a physical mouse or touchpads invented a long time ago and in this
project requirement for external hardware is completely eliminated by using human computer interaction technology we
detect hand movements and gestures and eye features for mouse movements and mouse events.

IJCRT2305927 International Journal of Creative Research Thoughts (IJCRT) www.ijcrt.org h576

www.ijcrt.org © 2023 IJCRT | Volume 11, Issue 5 May 2023 | ISSN: 2320-2882
Paper 1. Research on the Hand Gesture Recognition Based on Deep Learning”
Authors:Jing-Hao Sun,Ting-Ting Ji,Shu-Bin Zhang with the rapid development of computer vision, the demand for interaction
between human and machine is becoming more and more extensive. Since hand gestures are able to express enriched information,
the hand gesture recognition is widely used in robot control, intelligent furniture and other aspects. The paper realizes the
segmentation of hand gestures by establishing the skin color model and AdaBoost classifier based on haar according to the
particularity of skin color for hand gestures, as well as the denaturation of hand gestures with one frame of video being cut for
analysis. In this regard, the human hand is segmentd from the complicated background, the realtime hand gesture tracking is also
realized by CamShift algorithm. Then, the area of hand gestures which has been detected in real time is recognized by convolutional
neural network so as to realize the recognition of 10 common digits. Experiments show 98.3% accuracy.

Paper 2. Dynamic and Personalized Keyboard for Eye Tracker Typing”

Authors:Kadir Akdeniz1 , Zehra C， ataltepe1,2. Patients who suffer from Amyotrophic lateral sclerosis (ALS) or stroke cannot
talk and express their everyday basic needs and requests. They can communicate using eye trackers since they can still use their
eyes and sometimes move their heads. This study suggests new methods for improvements in both speed and ease of use for eye
tracker softwares. The first one is letter prediction to improve the speed, and second one is a new design that obviates the need of
blinking with eye trackers, thus providing more comfortable and longer sessions of writing.

”Paper 3.Algorithm for decoding visual gestures for an assistive virtual keyboard” Authors:Rafael Augusto Da Silva, Member,
IEEE, and Antonio Cl ˆ audio Paschoarelli Veiga, ´ Member, IEEE.Text production is one of the most frequent activities on a
computer, a trivial task that can be limiting for individuals affected by severe neuromotor disorders such as Amy-
otrophic Lateral Sclerosis (ALS) that can lead to Locked-in syndrome (LIS). These individuals need augmentative and alternative
communication tools, since they may have only the eye movements as a form of communication and interaction with the
outside world. This work investigates methods of interaction based on eye movement tracking and presents a virtual keyboard that
utilizes gaze detection as a text input. It describes the development of the shape detection algorithm for the assistive keyboard, typed
word voting from a Brazilian Portuguese lexicon and preliminary results on the decoding algorithm

”Paper 4. Virtual Mouse Control Using Colored Finger Tips and Hand Gesture Recognition” Authors:Vantukala VishnuTeja
Reddy1 , Thumma Dhyanchand2 , Galla Vamsi Krishna3. In human-computer interaction, virtual mouse implemented with finger
tip recognition and hand gesture tracking based on image in a live video is one of the studies. In this paper, virtual mouse control
using finger tip identification and hand gesture recognition is proposed. This study consists of two methods for tracking the fingers,
one is by using colored caps and other is by hand gesture detection. This include s three main steps that are finger detection using
color identification, hand gesture tracking and implementation on on-screen cursor. In this study, hand gesture tracking
is generated through the detection of the contour and formation of a convex hull around it. Features of hands are extracted with the
area ratio of contour and hull formed. Detailed tests are performed to check this algorithm in real world sc enarios.

”Paper 5. I-Keyboard: Fully Imaginary Keyboard on Touch Devices Empowered by Deep Neural Decoder” Authors:Ue-Hwan
Kim , Sahng-Min Yoo , and Jong-Hwan Kim , Fellow, IEEE. Text entry aims to provide an effective and efficient pathway for
humans to deliver their messages to computers. With the advent of mobile computing, the recent focus
of text-entry research has moved from physical keyboards to soft keyboards.
KSE-100 index is used as proxy of market risk. KSE-100 index contains top 100 firms which are selected on the bases of
their market capitalization. Beta is the measure of systematic risk and has alinear relationship with return (Horn, 1993). High risk
is associated with high return (Basu, 1977, Reiganum, 1981 and Gibbons, 1982). Fama and MacBeth (1973) suggested the existence
of a significant linear positive relation between realized return and systematic risk as measured by β. But on the other side some
empirical results showed that high risk is not associated with high return (Michailidis et al. 2006, Hanif, 2009). Mollah and Jamil
(2003) suggested thatrisk-return relationship is notlinear perhaps due to high volatility.

Project scope
• The camera of the computer will read the image of different gestures performed by a person’s hand and according to the movement
of the gestures the Mouse or the cursor of the computer will move, even perform right and left clicks using different
gestures

Software Quality Attributes

• Our software has many quality attribute that are given below:-dx • Adaptability: This software is adaptable by all users.
• Availability: This software is freely available to all users. The availability of the software is easy for everyone.
• Maintainability: After the deployment of the project if any error occurs then it can be easily maintained by the software developer.
• Reliability: The performance of the software is better which will increase the reliabilityof the Software.
• User Friendliness: Since, the software is a GUI application; the output generated
is much user friendly in its behavior.
• Integrity: Integrity refers to the extent to which access to software or data by
unauthorized persons can be controlled.

ANALYSIS MODEL (SDLC MODEL)

IJCRT2305927 International Journal of Creative Research Thoughts (IJCRT) www.ijcrt.org h577

www.ijcrt.org © 2023 IJCRT | Volume 11, Issue 5 May 2023 | ISSN: 2320-2882
The waterfall model is a classical model used in system development life cycle to create a system with a linear and sequential
approach. It is termed as waterfall because the model develops systematically from one phase to another in a downward
fashion.
An effective System Development Life Cycle (SDLC) should result in a high quality system that meets customer expectations,
reaches completion within time and cost evaluations, and works effectively and efficiently in the current and planned Information
echnology infrastructure. System Development Life Cycle (SDLC) is a conceptual model which includes policies and procedures
for developing or altering systems throughout their life cycles. SDLC is used by analysts to develop an information
system. SDLC includes the following activities:

Figure :

DATA FLOW DIAGRAM:

In data flow diagram, we show that flow of data in our system in DFD0 we show that base DFD in which rectangle present input
as well as output and circle show our system , in DFD1 we show actual input and actual output of system input of our system is text
or image and output is rumor detected like wise in DFD 2 we present operation of user as well as admin.

IJCRT2305927 International Journal of Creative Research Thoughts (IJCRT) www.ijcrt.org h578

www.ijcrt.org © 2023 IJCRT | Volume 11, Issue 5 May 2023 | ISSN: 2320-2882

Dig : 2

Dig : 3

UML DIAGRAMS :
Unified Modeling Language is a standard language for writing software blue prints. The UML may be used to visualize, specify,
construct and document the artifacts of a software intensive system. The Number of UML Diagram is available.
• Use case Diagram.
• Activity Diagram.
• Sequence Diagram.
• class Diagram.
• ER Diagram.

IJCRT2305927 International Journal of Creative Research Thoughts (IJCRT) www.ijcrt.org h579

www.ijcrt.org © 2023 IJCRT | Volume 11, Issue 5 May 2023 | ISSN: 2320-2882
Use case Diagram.
A use case diagram can identify the different types of users of a system and the different use cases and will often be accompanied
by other types of diagrams as well. The use cases are represented by either circles or ellipses.

Activity Diagram.

Activity diagrams are graphical representations of workflows of step wise activities and actions with support for choice, iteration
and concurrency.

IJCRT2305927 International Journal of Creative Research Thoughts (IJCRT) www.ijcrt.org h580

www.ijcrt.org © 2023 IJCRT | Volume 11, Issue 5 May 2023 | ISSN: 2320-2882
Sequence Diagram
A sequence diagram shows object interactions arranged in time sequence. It depicts the objects involved in the scenario and the
sequence of messages exchanged be- tween the objects needed to carry out the functionality of the scenario.

Conclusion :

This paper is proposing a system to recognize the hand gesture and Eyes Tracker replace the mouse and keyboard function.
That includes the movement of the mouse cursor, the drag and click with the keyboard features like printing alphabets and
other keyboard functions. The process of skin segmentation is utilized to separate the colour/image of hand with its
background. In general, the proposed algorithm can detect and recognize hand gesture so that it can operate mouse and
keyboard features and also create a real world user interface. 3d printing, Architectural drawings and even doing medical
operations from anywhere to everywhere. This project can be easily applied and its application can be very vast in medical
science where computation is required but couldn’t fully be implemented due to lack of human computer interaction.

REFERENCE :
http://www.iosrjournals.org/iosrjce/papers/Vol10%20issue5/C01051016.pdf?id=139
2. S. Sadhana Rao,” Sixth Sense Technology”, Proceedings of the International Conference on Communication
and Computational Intelligence– 2010, pp.336-339.
• 3. Game P. M., Mahajan A.R,”A gestural user interface to Interact with computer system ”, International Journal
on Science and Technology (IJSAT) Volume II, Issue I, (Jan.- Mar.) 2011, pp.018 – 027.
• 4. International Journal of Latest Trends in Engineering and Technology Vol. (7)Issue(4), pp.055-062.
• 5. Imperial Journal of Interdisciplinary Research (IJIR) Vol-3, Issue-4, 2017. 6. Christy, A., Vaithyasubramanian,
S., Mary, V.A., Naveen Renold, J. (2019),” Artificial intelligence based automatic decelerating vehicle control
system to avoid misfortunes “, International Journal of Advanced Trends in Computer Science and Engineering,
Vol. 8, Issue.
• 6, Pp. 3129-3134.
7. G. M. Gandhi and Salvi, "Artificial Intelligence Integrated Blockchain For Training Autonomous Cars," 2019 Fifth
International Conference on Science Technology

IJCRT2305927 International Journal of Creative Research Thoughts (IJCRT) www.ijcrt.org h581