2021 9th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO)

Amity University, Noida, India. Sep 3-4, 2021

Emotion-Based Music Recommendation System

2021 9th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) | 978-1-6654-1703-7/21/$31.00 ©2021 IEEE | DOI: 10.1109/ICRITO51393.2021.9596276

Vijay Prakash Sharma1, Azeem Saleem Gaded2, Deevesh Chaudhary3, Sunil kumar4, Shikha Sharma5
1,3. Assistant Professor, IT department, Manipal University Jaipur
2. Student, IT department, Manipal University Jaipur
4. Professor, Manipal University Jaipur
5. Assistant Professor, Poornima University, Jaipur
1. vijayprakashsharma@gmail.com 2. azeem.179302041@muj.manipal.edu 3.
Choudhary.deevesh@gmail.com 4. skvasistha@gmail.com 5. er.shikhasharma1986@gmail.com

ABSTRACT- Music is the form of art known to have a Depression is one of the prominent states of illness that
greater connection with a person’s emotion. It has got a unique has increased its impact everywhere. But, unfortunately, this
ability to lift up one’s mood. If a user receives a state of mind cannot only be treated with medication but also
recommendation based on his preference, it will also improve requires a mood enhancer. Moreover, a depressed person
his listing experience. Music recommendations have existed for cannot be left alone as it induces suicidal threats. Thus, I
a long time. Still, in most scenarios, the recommendation is wish that I could help such people using technology.
decided after learning the user preferences over time, like
looking at their past song preferences, the amount of time they Human needs different types of music in every other
listen to the music, etc. situation. At times of physical work, he wants different
music. When a person wants to relax, he wants to listen to
This paper suggests a neural network-based approach to different types of music. So, the music recommendation
song recommendation where their facial expressions detect a system can be applied in different areas like physical work,
person's mood. This approach is more efficient than the stress, music therapy, studying, etc. In this work, we
existing ones and eases users' work of first searching and introduce the model of the customized emotion-driven music
creating a specific playlist. Facial expressions play a crucial recommendation system. AI and generalized music
role in detecting a person's mood. A webcam or camera is used
recommendation and therapy are involved in this model.
to picture a face, and input is extracted from that picture. This
This paper clarifies the approach of applying emotion-driven
input is also used for determining an individual's mood.
personalization during the music recommendation process.
Keywords: Face Expression, Emotion, music,
II. LITERATURE SURVEY
Recommendation, CNN model,
Renuka R et al. [1] suggested a model based upon
I. INTRODUCTION changes in the various face curvatures and the intensities of
the pixels associated with those curvatures. Artificial Neural
An automation system for understanding and analysis of
Networks (ANN) were trained to define emotions.
music is a new prospect in music information retrieval. As
we know, music content has a vast variety and vibrancy; Zeng et al.[2] centered on a variety of methods for
many researchers from different fields like computer science, handling audio and/or visual records of emotional state
signal system, mathematics, and statistics are working on displays. Happiness, sorrow, fear, anger, disgust, and
this area. Recommending music also improves the user's surprise are among the emotion categories represented by the
listening experience. Music recommendations have existed effect. The paper gives a thorough analysis of audio/visual
for a long time. Still, in most scenarios, the recommendation computing techniques.
is decided after learning the user preferences over time, like
looking at their past song preferences, The amount of time Parul Tambe et al. [3] suggested an approach that
they listen to the music, etc. In this paper, we suggest a automated user-music player interactions by learning all of
different approach to song recommendation where their the user's preferences, moods, and activities and
facial expressions detect a person's mood. This approach is recommending songs as a result. The device recorded users'
more efficient than the existing ones and eases users' work of distinct facial expressions to evaluate their emotions and
first searching and then creating a specific playlist. For determine the music genre.
detecting a person’s mood, the facial expression is a crucial Binbin Hu et al. [4] proposed a Markov Decision Process
point. A webcam or camera is used to picture a face, and model for music recommendation and considered the music
input is extracted from that picture. This input is also used recommendation a playlist recommendation task. They
for determining an individual's mood. proposed RLWRec, a novel reinforcement learning-based
Based on a person’s mood system, prepare a music model fro exploting the optimal playlist approach.
catalog that eliminates manual efforts to create a playing list Deger Ayata et al. [5] provides a framework for emotion-
and segregate or group songs into different lists. This system based music recommendation that learns a user's emotion
helps in generating an appropriate playlist according to a using physiological data obtained via wearable sensors. A
person's mood. An emotion-based music recommendation wearable computing device embedded with specific types of
system aims to scan and interpret the user's emotions and sensors, namely galvanic skin response (GSR) and photo
create a playlist based on the parameters. Thus, our proposed plethys mography physiological(PPG) sensors, is used to
system concentrates on identifying human emotions for classify a user’s emotion.
creating an emotion-based music player.

978-1-6654-1703-7/21/$31.00 ©2021 IEEE 1

Authorized licensed use limited to: Amrita School of Engineering. Downloaded on September 04,2023 at 04:04:40 UTC from IEEE Xplore. Restrictions apply.
 Renata. L. Rossa et al.[6] suggested enhanced Sentiment i. Existing systems for facial expression recognition
Metric (eSM), a sentiment intensity-based music in real time are time and storage-intensive.
recommendation system, which is a lexicon based sentiment ii. Existing solutions are ineffective in creating a
metric combined with a user-profile-based correction factor. playlist depending on a user's current emotional
Sentences posted on social media are used to extract the
sentiments of the people, and the music recommendation state.
engine is run on mobile devices using a simple framework iii. Existing systems used extra hardware and sensors,
that recommends songs based on the intensity of the current which increases the system's overall cost.
user's emotions.
The proposed methodology in this paper aims to
ShanthaShalini et al. [7]. have proposed a dynamic overcome the above shortcomings by developing the
mechanism for music recommendations based on human automated music recommendation system based upon
emotions. Songs for each emotion are trained based on each human facial expression recognition using CNN.
human listening behavior. Using a combination of feature
extraction and machine learning algorithms, the emotion of a III. PROPOSED SYSTEM
real face is recognized. After the mood is determined from The complete system divides into three parts i. face
the input image, appropriate music for that mood is played to detector, ii. emotion detector from face iii. music
keep the users entertained. recommendation system based on the mood.
Shlok et al.[8] presented EMP, a cross-platform emotion-
based music player that recommends songs based on the
user's current mood. EMP's adaptive music recommendation
system features emotion context reasoning capabilities to
give smart mood-based music recommendations. The
Emotion Module uses a photo of the user's face as input and
uses a deep learning algorithm to detect their mood. It has an
accuracy percentage of 90.23 percent.
Anukriti D. [9] proposed a system consisting of three
modules for an emotion-based music recommendation
system: facial expression recognition module, audio emotion
recognition module, and system integration module. The
system integration module creates and queries a meta-data
file to map the other two modules. It used SVM for
classification based upon audio feature extraction.
Deebika S. et al. [10] have implemented a multilayer
perceptron by CNN[11] for minimal processing. To improve
visualization, it employs the back-propagation training
process to activate the filters. Only the eyes and mouth
features are considered to express recognition to achieve
real-time performance and reduce time complexity. The
retrieved eyes and mouth feature points serve as the input for
efficient support vector machine training.
Jhang [12] developed an improved Deep Residual
Network (ResNet-38) to detect user emotion and recommend
film and pop songs. The input layer receives a 48*48
grayscale image, processed by ResNet-38, with a one-
dimensional vector of size seven as the output corresponding
to 7 different emotions.
Chang S. et al. [13] introduced a personalized music
recommendation system (PMRS) based on a CNN technique
that categorizes music into distinct genres based on the audio Fig. 1. Flow diagram of the proposed system.
signal beats. It uses collaborative filtering (CF)
recommendation system that combines the CNN output with We use HAAR cascades for face detection. First, we take
the log files to provide the user with music the picture from the webcam. Then HAAR cascades scan
recommendations. the complete image and return the position of detected faces
as Rect (x,y,w,h).
While several methods for recognizing facial emotions
have been developed, very few systems have been
designed to manage the recommendation of a music playlist
using human emotions. Existing designs for playlist
generation using human emotions depend on either
additional hardware and sensors or human speech. The
methodology provided in this paper tries to address the
following issues in the existing literature:

2
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on September 04,2023 at 04:04:40 UTC from IEEE Xplore. Restrictions apply.
Fig. 2. Flow graph of face detection module
Fig. 4. Neural Network model for emotion detection
Emotion Detection: The architecture of the emotion
detection system is shown in fig. 3. IV. IMPLEMENTATION AND RESULTS

Fig. 3. Architecture of the Emotion Detection system Fig. 5. Detecting Face

We design six layers CNN model for emotion detection.

The number of filters in the first layer is 32, in the second
layer 64, in third layer 128, in fourth layer 256, and the fifth
and sixth layer contains 512 filters. The filter size in every
layer is the same, and it is 3X3. After every two CNN layers,
we include a max-pooling layer.
This model classifies the image into seven classes:
neutral, sad, disgust, happy, fear, surprise, angry.
Music Recommendation
The emotion detection model displays the detected
emotion/mood of the user. Users can change their facial
expressions continuously, so the system waits for seven to
ten seconds to detect proper expression. Various subfolders
are made in the data folder containing music of different
genres, which are subclassified manually. The music player
automatically plays a random song from the detected
emotion song folder.

Fig. 6. Detecting different Emotions/expressions of the face.

3
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on September 04,2023 at 04:04:40 UTC from IEEE Xplore. Restrictions apply.
 Fig. 10. No of Images of each class in the testing set

Fig. 7. Message for emotion detection "Happy" and play the song for a
happy mood

Fig. 11. Sample images of various emotions

VI. CONCLUSION AND FUTURE SCOPE

In this paper, we proposed a system for music
recommendation based on human emotion(facial
expression). Proposed system consists of two consecutive
CNN models. First CNN model was used for detecting seven
different categories of emotions, and second CNN model
recommending music based upon those emotions. In future,
we can add more human emotion as well as ideal face
reactions. We can further extend this model for movie
recommendation system based on different emotions.
REFERENCES
[1] R. R. Londhe, P. Vrushshen, and Pawar, “Analysis of Facial
Fig. 8. Message for emotion detection "Neutral" and play the song for
Expression and Recognition Based On Statistical Approach,” Jan.
neutral mood.
2012.
[2] Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, “A survey of
V. DATASET affect recognition methods: audio, visual, and spontaneous
The dataset contains a total of 35,887 images; each image expressions.,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1,
pp. 39–58, Jan. 2009.
is of size 48X48. These images are divided into seven
classes. First, Dataset split into train and test set of 80:20 [3] P. Tambe, Y. Bagadia, T. Khalil, and N. U. Shaikh, “Advanced
Music Player with Integrated Face Recognition Mechanism,”
ratio. The number of images in the training and test class is undefined, 2015.
shown in fig9 and fig 10. [4] B. Hu, C. Shi, and J. Liu, “Playlist recommendation based on
reinforcement learning,” in Intelligence Science I, vol. 510, Z. Shi, B.
Goertzel, and J. Feng, Eds. Cham: Springer International Publishing,
2017, pp. 172–182.
[5] D. Ayata, Y. Yaslan, and M. E. Kamasak, “Emotion based music
recommendation system using wearable physiological sensors,” IEEE
Trans. Consumer Electron., vol. 64, no. 2, pp. 196–203, May 2018.
[6] R. L. Rosa, D. Z. Rodriguez, and G. Bressan, “Music
recommendation system based on user’s sentiments extracted from
social networks,” IEEE Trans. Consumer Electron., vol. 61, no. 3, pp.
359–367, Aug. 2015.
[7] S. K. Et. al., “Facial Emotion Based Music Recommendation System
using computer vision and machine learning techiniques,”
TURCOMAT, vol. 12, no. 2, pp. 912–917, Apr. 2021.
[8] S. Gilda, H. Zafar, C. Soni, and K. Waghurdekar, “Smart music
player integrating facial emotion recognition and music mood
recommendation,” in 2017 International Conference on Wireless
Communications, Signal Processing and Networking (WiSPNET),
2017, pp. 154–158.
Fig. 9. No of images of each class in the training set
[9] A. Dureha, “An Accurate Algorithm for Generating a Music Playlist
based on Facial Expressions,” IJCA, vol. 100, no. 9, pp. 33–39, Aug.
2014.

4
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on September 04,2023 at 04:04:40 UTC from IEEE Xplore. Restrictions apply.
[10] S. Deebika, K. A. Indira, and Jesline, “A machine learning based
music player by detecting emotions,” in 2019 Fifth International
Conference on Science Technology Engineering and Mathematics
(ICONSTEM), 2019, pp. 196–200.
[11] D. Chaudhary, S. Kumar, and V. S. Dhaka, “Estimating crowd size
for public place surveillance using deep learning,” in Deep learning
and big data for intelligent transportation: enabling technologies and
future trends, vol. 945, K. R. Ahmed and A. E. Hassanien, Eds.
Cham: Springer International Publishing, 2021, pp. 175–197.
[12] J. Zhang, “Movies and Pop Songs Recommendation System by
Emotion Detection through Facial Recognition,” J. Phys.: Conf. Ser.,
vol. 1650, p. 032076, Oct. 2020.
[13] S.-H. Chang, A. Abdul, J. Chen, and H.-Y. Liao, “A personalized
music recommendation system using convolutional neural networks
approach,” in 2018 IEEE International Conference on Applied System
Invention (ICASI), 2018, pp. 47–49.

5
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on September 04,2023 at 04:04:40 UTC from IEEE Xplore. Restrictions apply.