CHAPTER-I

1) INTRODUCTION

1.1 GENERAL

A popular trend in computer-human interaction is the natural ability to recognize different faces. Prior
to recently, computer vision difficulties were highly difficult. However, with the introduction of modern
technology, these issues have significantly improved, including the ability to detect changes in light caused
by age, hair, and other accessories. Face recognition software, on the other hand, makes it easier to
recognize and authenticate persons based just on their facial traits. Therefore, it's crucial to interpret their
actions and facial features. Because of these characteristics and expressions, the emotions on a human
face can be categorized. Artificial intelligence systems are being used as a result of recent technological
advancements because they can recognize and comprehend emotions from facial features. Therefore, this
is an attempt to demonstrate the existence of the most recent technological advancements in deep learning
or convolution neural network models for human-computer interaction. Many techniques are needed to
identify and categorize human faces, but deep learning technique performs better than other techniques
due to its vast array of datasets and quick computation speed. Preprocessing, detection, orientation, feature
extraction, emotion classification are some of the processes that are typically involved in face recognition
and classification. Deep learning model performs these tasks more efficiently than handy computations

1
 1.2 DEEP LEARNING

The area of machine learning that uses artificial neural network architecture as its foundation is called
deep learning. Layers of linked nodes called neurons are used by an artificial neural network, or ANN, to
process and learn from the input data.

A fully connected deep neural network consists of an input layer, one or more hidden layers, and
their connections made sequentially. Each neuron receives input from the neurons in the layer above it or
from the input layer. The output of one neuron is used as the input by neurons in the layer below it, and
so on, until the final layer produces the output of the network. The layers of the neural network modify
the input data in a variety of nonlinear ways, allowing the network to learn complex representations of the
original data. Today Because of its success in a range of applications, including computer vision, natural
language processing, and reinforcement learning, deep learning has grown to become one of the most
well-known and visible subfields within machine learning.
Applications:
• Object detection and recognition
• Image classification
• Image segmentation

2
 1.3: CONVOLUTIONAL NEURAL NETWORKS

Deep Learning has shown to be an extremely potent tool over the last few decades due to its capacity to
handle massive volumes of data. Particularly in pattern recognition, the use of hidden layers has surpassed
interest in more conventional methods. Convolutional Neural Networks, or CNN are among the most
widely used deep neural networks in deep learning, particularly for Computer Vision applications.
Convolutional neural networks (CNN) are a class of deep neural networks used in deep learning that are
most frequently used to analyse visual imagery. Nowadays, matrix multiplications come to mind when we
think of a neural network, but CNN does not operate in this manner. It makes use of a unique method
known as convolution. Convolution, as we know it now, is a mathematical operation on two functions that
yields a third function that describes how one's shape is altered by another.

Kernels in convolutional neural network (CNN) models are used to identify an image's outline or border
functions. The weights in this model are arranged in an array of values to create and achieve the desired
properties. Each CNN model allots space to ascertain the image control that needs to be identified. The
product is computed and determined using the image's location since the values in the image indicate the
degree to which the convolution operation depends.

3
 1.4 OBJECTIVES

• Face detection
• Face recognition
• Emotion classification

4
 CHAPTER-II

2) IMPLEMENTATION

2.1 METHOLOGY

• First image is captured through external USB camera

• Preprocessing image
• Extracting important features using viola-jones algorithm
• Face recognition using haar-cascade algorithm
• Train the learning algorithms
• Using the trained model to detect facial emotion in real time

Flowchart and design specifications:

5
 • Capture the image through camera
The captured image is boxed in, converted to a binary pattern to specify it as a feature vector, and stored
in a database. The images are trained to match the input image and classify the expression of facial features
as happy, disgusted, sad, angry, and neutral. There are seven steps in the training process, which are as
follows: loading the dataset, preprocessing, augmenting the data as a feature vector, building and
compiling the design model, training and storing the feature vector, checking that the test model is valid.
• Face detection using Viola-jones algorithm
Fundamentally, the Viola-Jones algorithm makes use of Haar-like features, which are a collection of
straightforward rectangular filters that are used to capture intensity variations in an image. The basis for
identifying patterns like edges, corners, and texture variations is provided by these features. The
algorithm's genius is in how it uses an integral image, which is a representation of the original image that
makes it possible to quickly calculate the sum of the values of all the pixels in any rectangular area. This

6
integral image not only speeds up the calculation of Haar-like features but also significantly improves the
computational effectiveness of the algorithm. The Viola-Jones algorithm's cascade classifier structure is
yet another significant advancement. Each stage, which consists of several stages, is a powerful classifier.
The algorithm runs faster and more efficiently thanks to the cascade, which enables the quick rejection of
non-object regions. To reduce computational load and enable real-time applications, thresholding is
applied at each stage, with only regions that pass the threshold moving on to the next.
• Features are extracted and image is recognised using convolutional neural network
• Then compare whether the image is matched with image present in database or not
• If the image is matched then display image as known image
• If the image is not matched then display as unknown image and store in database for next time.

The 16-layer VGG-16 convolutional neural network is renowned for its simplicity. For image
classification, it makes use of 2x2 max-pooling layers and 3x3 convolutional filters. The architecture is
computationally demanding but efficient, with fully connected and cascading layers. It is often used for
transfer learning in a variety of computer vision tasks because it has been pre-trained on datasets such as
ImageNet. VGG-16 is still important for comprehending the fundamentals of deep neural networks, even
with more recent architectures

7
 2.2 PROCESS

2.2.1 BUILDING THE DATABASE TO CLASSIFY THE EMOTION USING CNN

Constructing a dataset with a convolution neural network layer for emotion classification
The suggested work's free dataset can be found at the provided link, http://www.kdef.se/.
The KDEF data set, shown in Figure 8, contained 4900 images. Of those, 1999 were eliminated because
they were in skewed positions, leaving 2901 images for training after the dataset was divided into 70%
and 30%, respectively. In a similar vein, my own dataset of 100 photos was also produced.

KDEF Data set

2.2.2 CREATING DATABASE WITH OPEN CV LIBRARY AND OTHER RESOURCES

A folder should be created, named, and two Python files, such as face_recognize.py and create_data.py,
should then be created. To verify for errors, the code is copied into the final source file and run. To support
the face features, copy the xml file into the project directory in a similar manner. The
haarcascade_frontalface_default.xml file needs to be copied.
Dlib : This cross-platform library was created with C++ programming. It supports the face features with
the help of the Histogram of Oriented Gradients and linear SVM for face recognition.
Code:

8
The VGG caffe model Convolution neural networks are trained and supported by this model, which is
imported for model definition, data preparation, and training. When writing Python scripts, this model is
employed to forecast the data that is not utilized. The CNN Caffe model in Python code is provided
below.

Numbskull, pandas, and get-pip This installation package for the Python programming language is an
open-source library tool.
open CV, SVM and pickle: These are library for computer vision, computation operations and feature
vectors. Imported library functions for python

9
 2.2.3 TRAINING PROCEDURE FOR RECOGNITION AND CLASSIFICATION

In order to train the VGG caffe model to classify emotions, the training and testing data are divided into
two models. The VGG caffe model creates feature vectors with support vector machines. In order to train
the classifier, a 224 x 224 image with preprocess rescaling and feature vector extraction is used. The
following is the main code for the feature extracted to train the classifier.

2.2.4 BUILDING ALL THE FILES TO PYTHON AND ANACONDA SOFTWARE PLATFORM

The embeddings.py file is used to generate and embed the images, and the encoded_images_data.csv file
is used to encode the library of facial features.
Below is Python 3.5 for face recognition with emotion

10
 3) RESULT

Three phases—face detection, face recognition, and face classification—are applied to the results.
The explanation of each step is provided below for your convenience. The newest software and
databases based on computer technology are used to recognize human faces in real time. Objects,
backgrounds, and landscapes are all involved in the human face detection process. The CNN model is
used to build facial recognition software, from which facial features are extracted and categorized using
the VGG 16 model. A person's face can be classified according to its expression, which includes disgust,
sadness, happiness, fear, and neutrality. On the LCD screen, matching is shown based on the classified
expression.
The task is divided into three stages. In the first stage, the camera reads the image and takes a picture
using the viola jones algorithm, OpenCV, haar-cascade detection. The RGB image was taken, then
processed using a feature extraction database to enable facial recognition. A matrix containing values
ranging from 0 to 255 is created from the image. In order to perform matrix manipulations in accordance
with the CNN model, the image resolution of 1024*1024*3 is reduced to 227*227*3, and the training
model of database features is tested for recognition of the human face.
Face detection: As seen in figures 9 and 10 below, the face is first tested using a 1024 x 1024 resolution
and the Viola Jones Algorithm.

11
Face recognition is done in the second step with the aid of a built-in dataset. The trained model of the
dataset is compared with the real-time images. If the photos can be identified by comparison with the
created dataset, they are given names. Training.py and images-data.csv are needed to accomplish image
recognition. The training portion is used to label the recognized images, as shown in figure 11.

12
The trained model is further classified to recognize the facial emotion and classify each expression using
VGG 16 and the svc_FER_linear_2.sav model in the third phase of the proposed work, which is face
classification. With clf, a convolution neural network (CNN) generates the face encodings. To identify the
human face's facial expression in Python, all models are categorized using their weights in the
main_FER_V2.py file. In the end, the program is executed in various stages to load the KDEF model and
identify the expressions of emotion on the face, as illustrated in figure 12.
The Anaconda software uses the Python file face_recognition with emotion.py to categorize the emotional
expressions on faces.
Anger, sadness, happiness, fear, disgust, and surprise are among the facial emotion expressions that are
categorized in the proposed work and are depicted in the figure 19.

13
 4) CONCLUSION

4.1 CONCLUSION

The goal of the proposed work is to create a real-time system for the detection, identification, and
classification of human faces. The results above display the seven states in which the classified
expressions are represented. The functionality was tested using Python 3.5 and Anaconda software. The
viola jones and haar-cascade algorithms were used for face detection.
For face recognition and classification, a convolution neural network model was employed in conjunction
with the KDEF dataset and VGG 16. The CV and additional libraries function as assistance for Python
programming.
Numerous industries, including education, business, medicine, and electronics, use this application
extensively. The KDEF dataset and VGG 16 are designed to accomplish face recognition and
classification. The seven facial expressions listed above depict various human conditions.
Several applications that are directly related to autism can be useful in deciphering a person's or child's
expressions. Through the proposed work, it would also be easy to evaluate a student's emotions through
E-learning techniques.

14
 4.2 FUTURE SCOPE

The field of face recognition has a promising future. Experts predict that in the upcoming years, this
technology will grow at an incredible rate and bring in enormous sums of money. The primary sources of
revenue that will be significantly impacted are security and surveillance. Additionally, public buildings,
schools, and private industries are now welcoming it with open arms.
In order to prevent fraud in debit and credit card purchases and payments, particularly those made online,
it is anticipated that banks and retailers will also implement it in the upcoming years. This technology
would close the gaps left by the widely used, insufficient password system. Robots with facial recognition
capabilities might eventually make an appearance. They can be useful in finishing jobs that are too hard
or impractical for people.

15
 5) REFERENCES

[1] Ken Nozaki, HisaoIshibuchi 1996. Adaptive fuzzy rule based Classification systems, IEEE
transactions on Fuzzy systems, Vol.4, No.3.
[2] Farid Ghareh Mohammadi, Mohammed Saniee Abadeh 2014. Image Steganalysis using abee colony
based feature selection Algorithm. Engineering Applications of AI. Science Direct, pp. 35-43. [3] Liu, F
L C Y, 2015. Improving Steganalysis by Fusing SVM Classifiers for JPEG Images. IEEE, pp. 185-190.
[4] Chhikara, M K, 2016. Significance of feature selection for image Steganalysis.IEEE.
[5] Bin Li.2011 A survey on Image steganography and Steganalysis,Journal of Information Hiding and
Multimedia signal Processing, Ubiquitous International, vol2 No.3 pp. 142-172.

16
17