1

CHAPTER 1: INTRODUCTION

1.1 MOTIVATION:

Recent improvements in gesture detection and hand tracking have brought about both
opportunities and challenges. Try out a few of these options while outlining challenges
and exciting future possibilities for virtual reality and user engagement. Given the
popularity of COVID19, the goal of this research is to reduce interactions between people
and the reliance on technology to operate computers. These results will motivate
additional research and, in the long run, support the use of virtual environments.

1.2 PROBLEM STATEMENT:

Gesture detection and hand tracking have brought about both opportunities and
challenges. Try out a few of these options while outlining challenges and exciting future
possibilities for virtual reality and user engagement. Given the popularity ofCOVID19,
the goal of this research is to reduce interactions between people and the reliance on
technology to operate computers. These results will motivate additional research and, in
the long run, support the use of virtual environments. In the proposed era, there are no
such restrictions and gesture recognition may be used as a substitute. It may be possible
to click and drag objects during this adventure using a variety of hand actions. The
suggested project's input technique will only require a camera.

1.3 OBJECTIVES:

• Make human-computer interaction easier: By using hand gestures and voice

commands, the computer requires almost no direct contact.
• Provide an alternative interface: For people who have difficulty using a traditional
mouse or keyboard.
• Be user-friendly and accessible: For people with physical restrictions or discomfort
from using a normal mouse.
• Limit the spread of infections: By eliminating the requirement for physical touch with
shared computer equipment.
• Be cost-effective: A user-friendly alternative to traditional input devices.
 2

1.4 SCOPE:

• This project involves developing an innovative interface for controlling a computer's mouse
using hand gestures detected through a webcam.
• By leveraging computer vision and machine learning, the system will interpret these
gestures and translate them into standard mouse functions such as cursor movement,
clicking, double-clicking, right-clicking, and scrolling.
• This will provide users with an alternative, potentially more intuitive, way to interact with
their computers without the need for a physical mouse.
 3

CHAPTER 2: LITERATURE SURVEY PAPERS

Related studies have been undertaken and worked upon in this domain.
The proposed system is built upon those developments and they act as the backbone of our
mechanism.

Title: "A Comprehensive Review on Hand Gesture Recognition Systems: Sensors,

Technologies, and Challenges"
Author: Araga S, Thimmaiah V.
Year: 2020
Methodology: The survey categorized hand gesture recognition systems based on the types
of sensors used (e.g., RGB cameras, depth sensors, wearable sensors). It evaluated various
computer vision techniques such as convolutional neural networks (CNNs), deep learning
approaches, and traditional feature extraction methods. The study highlighted current
challenges and future research directions.

Title: "Hand Gesture Recognition Using Deep Learning: A Review"

Author: Adithya V, Das S.
Year: 2021
Methodology: This survey focused on the application of deep learning models for hand
gesture recognition. It reviewed CNNs, RNNs, and hybrid models, discussing their
architectures, training techniques, and performance on different datasets. The paper also
examined data augmentation strategies and transfer learning to improve model
generalization.

Title: "Advances in Vision-Based Hand Gesture Recognition Systems"

Author: Li X, Zhang Y.
Year: 2022
Methodology: The paper reviewed advancements in vision-based hand gesture recognition
systems, including the integration of machine learning and artificial intelligence techniques.
It provided an overview of gesture datasets, preprocessing techniques, feature extraction
methods, and classification algorithms. The survey emphasized real-time processing and
system robustness.
 4

Title: "Deep Learning for Hand Gesture Recognition: A Comprehensive Review"

Author: Gupta P, Singh R.
Year: 2023
Methodology: This literature survey provided an in-depth analysis of deep learning
approaches for hand gesture recognition. It compared various deep learning frameworks,
including TensorFlow and PyTorch, and explored the use of generative adversarial
networks (GANs) for synthetic data generation. The review also discussed the impact of
hardware accelerators like GPUs and TPUs on training efficiency.

Title: "Emerging Trends in Hand Gesture Recognition Using Vision-Based Techniques"

Author: Chen H, Liu Z.
Year: 2023
Methodology: The study examined emerging trends in hand gesture recognition,
particularly focusing on vision-based techniques. It covered state-of-the-art methods such
as 3D convolutional networks and attention mechanisms.

Title : "Deep Learning Approaches for Hand Gesture Recognition: A Review"

Author :Chen S, Jin X.
Year :2019
Methodology: Analyzed various deep learning techniques applied to hand gesture
recognition, including convolutional neural networks (CNNs) and recurrent neural
networks (RNNs). Discussed the advantages and limitations of deep learning in gesture
recognition.
 5

CHAPTER 3: SYSTEM REQUIREMENTS AND SPECIFICATIONS:

3.1 FUNCTIONAL REQUIREMENTS:

1. Hand Gesture Recognition:
• Detect hand movements using a webcam or other camera devices.
• Recognize specific gestures for mouse functions such as moving the
cursor,clicking, double-clicking, right-clicking, and scrolling.
2. Cursor Movement:
• Translate hand movements into corresponding cursor movements on the
screen.
• Ensure smooth and responsive cursor motion.
3. Mouse Clicks:
• Implement gestures for left-click, right-click, and double-click actions.
• Ensure accurate and timely recognition of click gestures.
4. Scrolling:
• Recognize gestures for vertical and horizontal scrolling.
• Allow for both smooth and step-based scrolling.
5. Custom Gestures:
• Provide the ability to define and use custom gestures for specific actions.
• Include a user-friendly interface for gesture configuration.

3.2 NON-FUNCTIONAL REQUIREMENTS

1. Performance:
• Ensure real-time processing of gestures with minimal latency.
• Optimize for efficient use of system resources.
2. Accuracy:
• Achieve high accuracy in gesture recognition to prevent false positives and
negatives.
• Use machine learning algorithms and data sets to improve recognition
rates.
3. Usability:
• Design an intuitive user interface for setting up and calibrating the virtual
mouse.
• Provide clear instructions and feedback to the user.
4. Compatibility:
• Ensure compatibility with various operating systems (Windows, macOS,
Linux).
• Support multiple camera types and resolutions.
5. Security:
• Ensure the system does not access or store unnecessary user data.
• Provide secure methods for gesture data handling and processing.
 6

3.3 SOFTWARE AND HARDWARE REQUIREMENTS:

• Hardware requirements:

• A webcam or built-in camera to capture hand gestures.

• Dual Core(2.00)GHz and Above Processor is required.

• 4GB and above RAM is Required.

• 256GB HDD/SDD Secondary storage is required.

• Software Libraries:

• Utilize OpenCV for computer vision tasks and Media Pipe framework for
hand tracking and gesture recognition.

• Programming Languages:

• Python is commonly used for its ease of integration with OpenCV

and MediaPipe.
•

• Algorithms:
• Implementing Machine Learning algorithms like SOTA(State-of-the-
Art) forgesture recognition.

• Other Software Requirements:-

• Operating System:-Windows 7 and above.

• Tool:- Anaconda Navigator and Jupyter notebook.

 7

CHAPTER 4: SYSTEM ANALYSIS:

4.1 Detailed Description of the Project:

The system aims to develop a finger-only virtual mouse that tracks hand movements and
utilizes various finger combinations to perform mouse operations. Unlike existing systems,
which rely on static hand recognition, this system dynamically tracks hand movements and
finger positions for more versatile and intuitive control. It does not require additional
gadgets or sensors, making it affordable and user-friendly. This system leverages computer
vision techniques using the OpenCV library and the MediaPipe framework for hand
tracking and gesture recognition. Machine learning algorithms are employed to accurately
differentiate between hand motions and finger tips. The flexibility of the MediaPipe
framework allows for scalability across different platforms, including desktops and mobile
devices.

4.2 Existing System:

The current virtual mouse control system allows users to execute basic mouse operations
via hand recognition technology. These operations include controlling the mouse pointer,
left-clicking, right-clicking, and dragging using hand gestures. However, the potential of
hand recognition beyond these basic functions remains largely unexplored. Existing
systems typically rely on static hand recognition, where predefined hand shapes correspond
to specific actions. However, this approach has limitations in terms of accuracy, gesture
recognition capabilities, and compatibility with different hand shapes and movements.

4.3 Proposed System:

The proposed system aims to develop a finger-only virtual mouse that tracks hand
movements and utilizes various finger combinations to perform mouse operations. Unlike
existing systems, which rely on static hand recognition, this system dynamically tracks hand
movements and finger positions for more versatile and intuitive control. It does not require
additional gadgets or sensors, making it affordable and user-friendly. This systemleverages
computer vision techniques using the OpenCV library and the MediaPipe framework for
hand tracking and gesture recognition. Machine learning algorithms are employed to
accurately differentiate between hand motions and finger tips. The flexibility of the
MediaPipe framework allows for scalability across different platforms, including desktops
and mobile devices.

Advantages:

• Intuitive and user-friendly interface.

• Enhanced accessibility for users.

• Flexibility in recognizing various hand gestures- Cost-effective solution comparedto

traditional input devices
 8

4.4 Modules:

1. Camera Interface Module: Captures video frame from camera and processes frames for
further analysis

2. Hand/Feature Detection Module: Detects hand or specific features (like fingertips) in

the video feed and uses image processing techniques such as contour detection, color
segmentation, or machine learning models.

3. Gesture Recognition Module: Detects hand or specific features (like fingertips) in the
video feed. It detects hand or specific features (like fingertips) in the video feed. Recognizes
specific gestures or movements of the hand and maps gestures to corresponding mouse
actions (click, drag, scroll).

4. Control Module: Sends the translated gestures as mouse events to the operating system.
Handles actions like left click, right click, drag-and-drop, and scrolling.

5. User Interface Module: display mouse actions on the screen.

 9

CHAPTER 5: SYSTEM DESIGN

5.1 Architectural Structure

Fig 5.1 Architecture structure of gesture controlled virtual mouse

The architecture flow chart of gesture controlled virtual mouse using computer vision algorithms
begins with a video capture module, where the camera capture the real-time video feed and the video
is processed and detected
 10

5.2 Use Case Diagram:

Fig 5.2 Use Case Diagram of Hand Gesture Detection

The computer vision algorithms would depict various user interactions such as control cursor, click,
drag and drop and scroll. These use cases illustrate how a user can employ hand gesture to perform
typical mouse functions, with the system processing these gestures through computer vision
techniques.
 11

5.3 System Architecture

Fig 5.3 System Architecture for Gesture Controlled Virtual Mouse

This includes a camera module to capture real time hand movements connected to a processing unit
that runs computer vision algorithms to detect and interpret gestures. This processing unit interface
with the operating system to translate these gestures into corresponding mouse movements and
actions on the screen.
 12

CHAPTER 6: SYSTEM IMPLEMENTATION

6.1 Methodlogy
Python has been used for the overall development of the proposed system. It makes use of the
python library OpenCV for utilising Computer Vision and its algorithms to recognize hand
gestures. It makes use of CNN implemented by Mediapipe for recognizing hand landmarks and
coordinates. The whole mechanism works in 3 phases: detecting and tracing, recognition, and
function execution.

Fig 6.1 Block Diagram for the Process of Gesture Recognition

Fig 6.2 Mediapipe Hand Tip Recognition Layout

 13

6.2 ALGORITHM:

Step 1: Start the program.

Step 2: Open the file and using that file location go to cmd.

Step 3: Using some libraries installed for the code run the program.

Step 4: Initialize the system and start the video capturing of WEBCAM.

Step 5: Capture frames using WECAM.

Step 6: Detect Hands and Hand Tips using Media pipe and OpenCV.

Step 7: Detect which finger is UP.

Step 8: Recognizing the Gesture.

Step 9: Perform mouse operations according to gesture.Step 10:

Stop the program.
 14

6.3 PSEUDOCODE

IMPORT libraries (Opencv2, mediapipe, pyautogui, math, etc.)

DEFINE ENUM Gest:
FIST = 0
PINKY = 1
RING = 2
MID = 4
LAST3 = 7
INDEX = 8
FIRST2 = 12
LAST4 = 15
THUMB = 16
PALM = 31
V_GEST = 33
TWO_FINGER_CLOSED = 34
PINCH_MAJOR = 35
PINCH_MINOR = 36

DEFINE ENUM HLabel:

MINOR = 0
MAJOR = 1

CLASS HandRecog:
INITIALIZE attributes:
finger = 0
ori_gesture = Gest.PALM
prev_gesture = Gest.PALM
frame_count = 0
hand_result = None
hand_label = None

FUNCTION update_hand_result(hand_result):
SET self.hand_result = hand_result

FUNCTION get_signed_dist(point):
COMPUTE signed distance between two points based on y-coordinates

FUNCTION get_dist(point):
COMPUTE distance between two points

FUNCTION get_dz(point):
COMPUTE absolute difference on z-axis between two points

FUNCTION set_finger_state():
COMPUTE finger state based on distances between landmarks

FUNCTION get_gesture():
DETERMINE gesture based on finger state
HANDLE fluctuations and return stable gesture
 15

CLASS Controller:
INITIALIZE attributes:
tx_old = 0
ty_old = 0
flag = False
grabflag = False
pinchmajorflag = False
pinchminorflag = False
pinchstartxcoord = None
pinchstartycoord = None
pinchdirectionflag = None

prevpinchlv = 0
pinchlv = 0
framecount = 0
prev_hand = None
pinch_threshold = 0.3

FUNCTION getpinchylv(hand_result):
COMPUTE and RETURN vertical pinch distance

FUNCTION getpinchxlv(hand_result):
COMPUTE and RETURN horizontal pinch distance

FUNCTION changesystembrightness():
ADJUST screen brightness based on pinch level

FUNCTION changesystemvolume():
ADJUST system volume based on pinch level

FUNCTION scrollVertical():
PERFORM vertical scrolling based on pinch level

FUNCTION scrollHorizontal():
PERFORM horizontal scrolling based on pinch level

FUNCTION get_position(hand_result):
COMPUTE and RETURN smoothed cursor position based on hand landmarks

FUNCTION pinch_control_init(hand_result):
INITIALIZE pinch gesture attributes

FUNCTION pinch_control(hand_result, controlHorizontal, controlVertical):

HANDLE pinch gesture and invoke appropriate control functions

FUNCTION handle_controls(gesture, hand_result):

EXECUTE actions based on detected gesture
UPDATE mouse actions and system controls
 16

CLASS GestureController:
INITIALIZE attributes:
gc_mode = 0
cap = None
CAM_HEIGHT = None
CAM_WIDTH = None
hr_major = None
hr_minor = None
dom_hand = True

FUNCTION classify_hands(results):
DETERMINE major and minor hand based on hand classification

FUNCTION start():
OPEN video capture
INITIALIZE HandRecog objects (hr_major, hr_minor)
SETUP Mediapipe Hands

WHILE camera is open:

READ video frame

CONVERT frame to RGB and process with Mediapipe
UPDATE HandRecog objects with hand landmarks
GET gesture from HandRecog objects
HANDLE controls based on detected gesture
DRAW hand landmarks on frame
DISPLAY frame
BREAK loop if ENTER key is pressed
RELEASE video capture and CLOSE windows

MAIN:
CREATE instance of GestureController
CALL start() method on instance
 17

6.4 PREDEFINED PARAMETERS:

 18

6.5 Implementation:
In the proposed system, MediaPipe and OpenCV have been used to further the advancement
in interactions between humans and machines. The proposed system performs mouse
functions via hand gestures by using Computer Vision techniques which is a shift from theexisting
wired and wireless models.

The main gestures for using functionalities of this system are:

1. Neutral Gesture: Used for stopping the ongoing gesture.

2. Move Cursor: Using this gesture, the operator can navigate the cursor to the desired
location. Speed of the cursor’s movement is proportional to the speed of moving hand.
3. Left Click: Gesture to perform a single left-click
4. Right Click: Gesture to perform a single right-click.
5. Double Click: Gesture to perform a double-click.
6. Scrolling: It is a dynamic gesture to scroll horizontally and vertically. The speed of
scrolling is proportional to the distance by which the pinch gesture has moved from its
starting point. Vertical and horizontal scrolling is performed by vertical and horizontal
pinch movements respectively.
7. Drag and Drop: Gesture used for dragging and dropping files, icons, etc.
8. Multiple Item Selection: It is used for selecting multiple items.
9. Volume Control: This is a dynamic gesture for volume control. The rate of
increase/decrease of volume changes in accordance with the distance moved by the
pinch gesture from its starting point.
10. Brightness Control: It is a dynamic gesture to control brightness. The rate of
increase/decrease of brightness is proportional to the distance moved by the pinch
gesture from its starting point.
 19

CHAPTER 7: SYSTEM TESTING

7.1 Introduction:
Testing is the way toward running a framework with the expectation of discovering blunders.
Testing upgrades the uprightness of the framework by distinguishing the deviations in plans
and blunders in the framework. Testing targets distinguishing blunders – prom zones. This
aides in the avoidance of mistakes in the framework. Testing additionally adds esteems to theitem
by affirming the client's necessity.
The primary intention is to distinguish blunders and mistake get-prom zones in a framework.
Testing must be intensive and all around arranged. A somewhat tried framework is as terrible
as an untested framework. Furthermore, the cost of an untested and under-tried framework is
high. The execution is the last and significant stage. It includes client preparation, framework
testing so as to guarantee the effective running of the proposed framework. The client tests
the framework and changes are made by their requirements. The testing includes the testing
of the created framework utilizing different sorts of information. While testing, blunders are
noted and rightness is the mode.
Objectives of testing:

Testing in a cycle of executing a program with the expectation of discovering mistakes. A

effective experiment is one that reveals an up 'til now unfamiliar blunder.

Framework testing is a phase of usage, which is pointed toward guaranteeing that the
framework works accurately and productively according to the client's need
before the live activity initiates. As expressed previously, testing is indispensable to the
achievement of a framework. Framework testing makes the coherent presumption that if all the
framework is right, the objective will be effectively accomplished. A progression of testsare
performed before the framework is prepared for the client acknowledgment test.

7.2 Testing methods:

System testing is a stage of implementation. This helps the weather system works accurately
and efficiently before live operation commences. Testing is vital to the success of the system.
The candidate system is subject to a variety of tests: online response, volume, stress, recovery,
security, and usability tests series of tests are performed for the proposed system areready for
user acceptance testing.
Unit Testing:

Unit testing chiefly centers around the littlest unit of programming plan. This is known asmodule
testing. The modules are tried independently. The test is done during the
programming stage itself. In this progression, every module is discovered to be working
acceptably as respects the normal yield from the module.
Integration Testing:

Mix testing is an efficient methodology for developing the program structure, while
simultaneously leading tests to reveal blunders related with the interface. The goal is to take
 20

unit tried modules and manufacture a program structure. All the modules are joined and triedin
general.
Output Testing:

Subsequent to performing approval testing, the following stage is yield trying of the proposed
framework, since no framework could be valuable on the off chance
that it doesn't create the necessary yield in a particular configuration. The yield design on the
screen is discovered to be right. The organization was planned in the framework
configuration time as indicated by the client needs. For the printed copy likewise, the yield
comes according to the predefined prerequisites by the client. Subsequently yield testing
didn't bring about any amendment for the framework.
User Acceptance Testing:

Client acknowledgment of a framework is the vital factor for the achievement of any
framework. The framework viable is tried for client acknowledgment by continually staying
in contact with the imminent framework clients at the hour of creating and making changes at
whatever point required.
 21

7.3 Test Cases

The users test the developed system when changes are made according to the needs. The testing
phase involves the testing of the developed system using various kinds of data. Anelaborate
testing of data is prepared and system is tested using the test data.
 22

Pre - Apply Pre- Image are As Pass

Processing methods processing converted
Expected.
like, noise for image successfully
removal,
gray
conversion

Pre - Apply Pre- Image are Open-cv Fail

Processing methods processing converted library is not
like, noise for image successfully detecting and
removal, preprocessing
gray it
conversion

Detecting Train model Check for Model Model trained Pass

object model loss trained successfully
using and accuracy successfully
neural
network

Identify the Detect the Check for Predicted Result is Pass

eye and given object drowsiness result shown.
track the
classification
object
with accuracy

Identify the Detect the Check for Predicted Object fail to Pass
eye and given object drowsiness result track
track the
classification
object
with accuracy
 23

CHAPTER 8: SCREENSHOTS

Fig 1 Hand recognition

Fig 2 Cursor movement

 24

Fig 3 Left Click

Fig 4 Right Click

 25

Fig 5 Multiple Items Selection

Fig 6 Drag Items

 26

Fig 7 Drop Items

Fig 8 Brightness Control

 27

Fig 9 Volume Control

 28

CHAPTER 9: CONCLUSION AND FUTURE ENHANCEMENT

Conclusion:

The virtual mouse system's main objective is to eliminate the need for a hardware mouse by
allowing users to manage mouse cursor functions with hand gestures instead. The described
method can be used with a webcam or an integrated camera that analyses framesto recognise
hand movements and hand tips and execute specific mouse actions. The model has some
shortcomings, including some difficulties with dragging and clicking to select text and a slight
loss of precision in right-click mouse capabilities. We will now concentrate on improving the
fingertip identification algorithm in order to overcome these limitations.

Future enhancements:
Limitations of the proposed system are that there is a reduction in the precision of the right-
sided clicking operation of the mouse. Also, the ability to drag objects is not smooth. These
are a few limitations in the mechanism which are being worked upon. In addition, we are
developing our system by adding voice assistant functionalities which will make the Human
Computer Interaction (HCI) even more intuitive and further advance its progress.
 29

REFERENCES:
[1] Wu, H. Cai X, Ma, J. and Zhang, X., 2022, September. Towards Natural
Virtual Mouse with mm-Wave Radar. In 2022 19th European Radar Conference
(EuRAD) (pp. 45-48), IEEE.
[2] Liang, Zifei, Choong H. Lee Tanzil M. Arefin, Zijun Dong, Piotr Walczak,
Song-Hai Shi, Florian Knoll, Yulin Ge, Leslie Ying, and Jiangyang Zhang.
"Virtual mouse brain histology from multi-contrast MRI via deep learning." Elife
11 (2022): e72331.
[3] Joy Guha, Shreya Kumari, Prof. Shiv Kumar Verma, “AI Virtual Mouse Using
Hand Gesture Recognition”, Applied Science and Engineering Technology,
Volume IJRASET41981, IJRASET, April 2022.
[4] S.Shriram, “Deep Learning-Based Real-Time AI Virtual Mouse System Using
Computer Vision to Avoid COVID-19 Spread”, Applied Sciences, Volume 2021,
Article ID 8133076, 31 Jul 2021, Hindawi.
[5] J. Katona, “A review of human–computer interaction and virtual reality
research fields in cognitive Info Communications,” Applied Sciences, vol. 11, no.
6,2021.
[6] D.-S. Tran, N.-H. Ho, H.-J. Yang, S.-H. Kim, and G. S. Lee, “Real-time virtual
mouse system using RGB-D images and fingertip detection,” Multimedia Tools
and Applications, vol. 80, no. 7, pp. 10473–10490, 2021.
[7] B. J. Boruah, A. K. Talukdar and K. K. Sarma, "Development of a Learning-
aid tool using Hand Gesture Based Human Computer Interaction System," 2021
Advanced Communication Technologies and Signal Processing (ACTS), 2021,
pp. 1-5.
[8] V. V. Reddy, T. Dhyanchand, G. V. Krishna and S. Maheshwaram, "Virtual
Mouse Control Using Colored Finger Tips and Hand Gesture Recognition," 2020
IEEE-HYDCON, 2020, pp. 1-5.
[9] J. T. Camillo Lugaresi, “MediaPipe: A Framework for Building Perception
Pipelines,” 2019.
[10] Real-time Finger Tracking and Contour Detection for Gesture Recognition
using OpenCV, Dr. Ruchi Manish Gurav; Dr. Premanand K. Kadbe [2015],
International Conference on Industrial Instrumentation and Control (ICIC).