Multi-label Emotion Classification using Machine Learning and

Deep Learning Methods

a
Drashti Kher and Kalpdrum Passi
School of Engineering and Computer Science, Laurentian University, Sudbury, Ontario, Canada

Keywords: Multi-label Emotion Classification, Twitter, Python, Deep Learning, Machine Learning, Naïve Bayes, SVM,
Random Forest, KNN, GRU based RNN, Ensemble Methods, One-way ANOVA.

Abstract: Emotion detection in online social networks benefits many applications like personalized advertisement
services, suggestion systems etc. Emotion can be identified from various sources like text, facial expressions,
images, speeches, paintings, songs, etc. Emotion detection can be done by various techniques in machine
learning. Traditional emotion detection techniques mainly focus on multi-class classification while ignoring
the co-existence of multiple emotion labels in one instance. This research work is focussed on classifying
multiple emotions from data to handle complex data with the help of different machine learning and deep
learning methods. Before modeling, first data analysis is done and then the data is cleaned. Data pre-
processing is performed in steps such as stop-words removal, tokenization, stemming and lemmatization, etc.,
which are performed using a Natural Language Processing toolkit (NLTK). All the input variables are
converted into vectors by naive text encoding techniques like word2vec, Bag-of-words, and term frequency-
inverse document frequency (TF-IDF). This research is implemented using python programming language.
To solve multi-label emotion classification problem, machine learning, and deep learning methods were used.
The evaluation parameters such as accuracy, precision, recall, and F1-score were used to evaluate the
performance of the classifiers Naïve Bayes, support vector machine (SVM), Random Forest, K-nearest
neighbour (KNN), GRU (Gated Recurrent Unit) based RNN (Recurrent Neural Network) with Adam
optimizer and Rmsprop optimizer. GRU based RNN with Rmsprop optimizer achieves an accuracy of 82.3%,
Naïve Bayes achieves highest precision of 0.80, Random Forest achieves highest recall score of 0.823, SVM
achieves highest F1 score of 0.798 on the challenging SemEval2018 Task 1: E-c multi-label emotion
classification dataset. Also, One-way Analysis of Variance (ANOVA) test was performed on the mean values
of performance metrics (accuracy, precision, recall, and F1-score) on all the methods.

1 INTRODUCTION Emotion can be detected from the data with the help
of data mining techniques, machine learning
With the increasing popularity of online social media, techniques and with the help of neural networks
people like expressing their emotions or sharing (Avetisyan, H and Bruna, Ondej and Holub, Jan, 2016).
meaningful events with other people on the social From the examination it was expressed that emotion
network platforms such as twitter, Facebook, detection approaches can be classified into three
personal notes, blogs, novels, emails, chat messages, following types: keyword based or lexical based,
and news headlines (Xiao Zhang, Wenzhong Li1 and learning based and hybrid. The most commonly used
Sanglu Lu, 2017). classifiers, such as SVM, naive bayes and hybrid
Emotion is a strong feeling that deriving from algorithms (Avetisyan, H and Bruna, Ondej and Holub,
person's mood or interactions with each other. Many Jan, 2016). Emotion mining is very interesting topic in
ways are available for detecting emotions from the many studies such as cognitive science, neuroscience,
textual data, for example social media has made our and psychology (Yadollahi, Ali and Shahraki, Ameneh
life easier and by pressing just one button everyone Gholipour and Zaiane, Osmar R, 2017). Whereas emotion
can share personal opinion with the whole world. mining from text is still in its early stages and still has

a
https://orcid.org/0000-0002-7155-7901

128
Kher, D. and Passi, K.
Multi-label Emotion Classification using Machine Learning and Deep Learning Methods.
DOI: 10.5220/0011532400003318
In Proceedings of the 18th International Conference on Web Information Systems and Technologies (WEBIST 2022), pages 128-135
ISBN: 978-989-758-613-2; ISSN: 2184-3252
Copyright c 2022 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
 Multi-label Emotion Classification using Machine Learning and Deep Learning Methods

a long way to proceed, developing systems that can 1.1.1 Machine Learning based Approach
detect emotions from text has many applications.
The intelligent tutoring system can decide on For the machine learning models, data cleaning,
teaching materials, based on users mental state and text preprocessing, stemming, and lemmatization
feelings in E-learning applications. The computer can on the raw data were performed. The text data was
monitor users emotions to suggest appropriate music transformed to vectors by using the TF-IDF
or movies in human computer interaction (Yadollahi, method, then multiple methods were used-to
Ali and Shahraki, Ameneh Gholipour and Zaiane, Osmar R, predict each emotion. SVM, Naive Bayes, Random
2017). Moreover, output of an emotion-mining system Forest, and KNN classifiers were used extensively
can serve as input to the other systems. For instance, to build the machine learning solution. After all the
Rangel and Rosso (Yadollahi, Ali and Shahraki, Ameneh training, various performance metrics measures
Gholipour and Zaiane, Osmar R, 2017 )( Rangel and were plotted for each model concerning every
Paolo Rosso,2016) use the emotions identified within emotion label as a bar plot.
the text for author identification, particularly
identifying the writers age and gender. Lastly, 1.1.2 Deep Learning based Approach
however not the least, psychologists can understand
patients emotions and predict their state of mind For the deep learning, dataset is loaded, then
consequently. On a longer period of time, they are preprocessed, and encoded to perform deep learning
able to detect if a patient is facing depression, stress techniques on it. From this research shows that RNN
that is extremely helpful since he/she can be referred based model performs well on text data, GRU model
to counselling services (Yadollahi, Ali and Shahraki, was built with an attention mechanism to solve the
Ameneh Gholipour and Zaiane, Osmar R, 2017). There is problem by training for multiple epochs to obtain
analysis on detecting emotions from text, facial the best accuracy.
expressions, images, speeches, paintings, songs, etc.
Among all, voice recorded speeches and facial
expressions contain the most dominant clues and have 2 DATA AND PREPROCESSING
largely been studied (Carlos Busso, Zhigang Deng,
Serdar Yildirim, Murtaza Bulut, Chul Min Lee, Abe In this research, 10,983 English tweets were used for
Kazemzadeh, Sungbok Lee, Ulrich Neumann, and multi-label emotion classification from (“SemEval-
Shrikanth Narayanan, 2004)( Alicja Wieczorkowska, 2018”, 2018), (Mohammed, S., M.; Bravo-Marquez,
Piotr Synak, and Zbigniew W. Ra´s., 2006). Some F.; Salameh, M.; Kiritchenko, S, 2018). The dataset
types of text can convey emotions such as personal of emotions classification includes the eight basic
notes, emails, blogs, novels, news headlines, and chat emotions (joy, sadness, anger, fear, trust, disgust,
messages. Specifically, popular social networking surprise, and anticipation) as per Plutchik (1980)
websites such as Facebook, Twitter, Myspace are (Jabreel M., Moreno A, 2019) emotion model, as well
appropriate places to share one’s feelings easily and as a few other emotions that are common in tweets
largely. which are love, optimism, and pessimism. Moreover,
python 3.7.4 version was used for data preprocessing,
1.1 Multi-label Classification for multi-label emotion classification, and data
Emotion Classification visualization.
Data preprocessing is the most crucial data mining
Emotion mining is a multi-label classification technique that transforms the raw data into a useful
problem that requires predicting several emotion and efficient format. Real-world information is
scores from a given sequence data. Any given frequently inconsistent, incomplete, or missing in
sequence data can possess more than one emotion, so specific behaviours and is likely to contain lots of
the problem can be posed as a multi-label errors. It is a demonstrated technique of resolving
classification problem rather than a multi-class such issues. It prepares raw data for further
classification problem. Both machine learning and processing. Different tools are available for data
deep learning were used in this research to solve the preprocessing. Data preprocessing is divided into a
problem. few stages which is show in Figure 1.

129
WEBIST 2022 - 18th International Conference on Web Information Systems and Technologies

endings of words and returns the dictionary form of

the word, which is also known as “Lemma”.
Lemmatization also uses wordnet, which is a lexical
knowledge base. Lemmatization is performed after
stemming, and it is performed on the tokenized
words.
Figure 1: Structure of Data Preprocessing.

The data preprocessing steps that are performed 3 METHODS

before starting machine learning and deep learning
methods are as follows.
Machine learning and a deep learning-based
Data Cleaning: For data cleaning, sometimes tweets approaches were used to solve the multi-label
possess certain usernames, URLs, hashtags, emotion recognition problem on emotion
whitespace, Punctuations, etc., which is not helpful in classification from twitter data. Both machine
machine learning algorithms to get better accuracy. learning and deep learning algorithms were applied
Then, remove all noisy data from every tweet. All after applying domain knowledge-based data
special characters are replaced with spaces. This step cleaning, NLP based data preprocessing, and feature
is performed as special characters do not help much engineering techniques. Different feature
in machine learning modeling. Every tweet is engineering and preprocessing techniques were
transformed into lower case. Also, duplicate tweets applied for both the solutions.
are identified and removed.
3.1 Machine Learning Methods for
Remove Stop Words: Stop words are words that are Emotion Classification
finalized in the Natural Language Processing (NLP)
step. “Stop words” or “Stop word lists” consists of The most popular machine learning methods such as
those words which are very commonly used in a Naïve bayes, SVM, Random Forest, and KNN have
Language, not just English. Stop word removal is been discussed in this section. For the Machine
important because it helps the machine learning learning models, data cleaning, text preprocessing,
models to focus on more important words which stemming, and lemmatization on the raw data were
result in more accurate prediction. Stop word removal performed. Feature engineering converts the
also helps to avoid problems like the curse of text/string data to a format that machine learning
dimensionality as it reduces the dimensionality of the algorithms would interpret. It is an important step
data. It is important to note that there is a total of 179 before applying any of the mentioned machine
stop words available in the English language using learning algorithms. The overview of applying
NLTK library (Manmohan singh, 2020). machine learning techniques to the emotion
classification labeled data and analysis is shown in
Tokenization: In simple terms, tokenization is a Figure 2.
process of turning sequence data into tokens. It is the
most important natural language processing pipeline.
It turns a meaningful piece of text into a string char
named tokens.

Stemming: Stemming is a process of turning

inflected words into their stemmed form. Stemming
also helps to produce morphological variants of a
base word. Stemming is the part of the word which Figure 2: Overview of applying machine learning
adds inflected word with suffixes or prefixes such as techniques.
(-ed, -ize, -s, -de, mis). So, stemming results in words
that are not actual words. Stemming is created by Feature Engineering: The cleaned and preprocessed
removing the suffixes or prefixes used with a word. tokens of tweets are obtained after all the
preprocessing where each token is a “string”.
Lemmatization: The key to this process is linguistics Machine learning models cannot work with strings,
and it depends on the morphological analysis of each they only work with numbers. The tokens are
word. Lemmatization removes the inflectional

130
 Multi-label Emotion Classification using Machine Learning and Deep Learning Methods

transformed into numbers by using the methods given similar to the available categories. It reserves all the
below. available data and classifies a new data point based
Bag of Words (BOW) on the similarity. This means when new data comes
Term frequency and Inverse document frequency out then it can be easily classified into a well suite
(TF-IDF) category by using K- NN algorithm. It can be used for
It is always a better idea to use TF-IDF rather than Classification as well as for Regression but mostly it
BOW as the TF-IDF feature engineering technique is used for the Classification problems. KNN
also preserves some semantic nature of the sequence. algorithm at the training phase just stores the dataset
For this research, the TF-IDF feature engineering and when it gets new data, then it classifies that data
technique was used to encode tokens as numbers. into a different category that is much similar to the
new data.
Naïve Bayes: Naive Bayes is a machine learning
classifier and it used to solve classification problems. 3.2 Deep Learning based Emotion
It uses Bayes theorem extensively for training. It can Classification
solve diagnostic and predictive problems. Bayesian
Classification provides a useful point of view for Deep learning adjusts a multilayer approach to the
evaluating and understanding many learning hidden layers of the neural network. In machine
algorithms. It calculates explicit probabilities for learning approaches, features are defined and
hypothesis, and it is robust to noise in input extracted either manually or by making use of feature
information (Hemalatha, Dr. G. P Saradhi Varma, Dr. selection methods. In any case, features are learned
A. Govardhan, 2013). In this multilabel classification, and extricated automatically in deep learning,
single Naive Bayes model is trained for predicting achieving better accuracy and performance. Figure 3
each output variable. shows the overview of deep learning technique. deep
learning currently provides the best solutions to many
Support Vector Machine: The support vector problems in the fields of image and speech
machine is a supervised learning distance-based recognition, as well as in NLP.
model. It is extensively used for classification and
regression. The main aim of SVM is to find an
optimal separating hyperplane that correctly
classifies data points and separates the points of two
classes as far as possible, by minimizing the risk of
misclassifying the unseen test samples and training
samples (García-Gonzalo, E., Fernández-Muñiz, Z.,
García Nieto, P.J., Bernardo Sánchez, A., Menéndez
Fernández, M, 2016). It means that two classes have
maximum distance from the separating hyperplane. Figure 3: Overview of applying deep learning techniques.
Random forest: It is an ensemble learning method
for classification and regression. Each tree is grown Feature Exrtraction: Feature extraction is the name
with a random parameter and the final output is for methods that combine and/or select variables
achieved by aggregating over the ensemble (R. Gajjar into features, effectively reducing the amount of data
and T. Zaveri, 2017). As the name suggests, It is a that must be processed, while still accurately and
classifier that contains a number of decision trees on completely describing the original data set.
different subsets of the given dataset and takes the
average to improve the predictive accuracy of that Word Embedding: Word embeddings are the texts
dataset. Rather than depending on one decision tree, changed into numbers and there may be different
the random forest takes the prediction from each tree numerical representations of the same content. As it
and based on the majority votes of predictions, and it turns out, most of the machine learning algorithms
predicts the final output. and deep learning architectures are unable to process
strings or plain text in their raw form (NSS, 2017).
K-Nearesr Neighbor: K-Nearest Neighbor is one of They require numbers as inputs to perform any sort of
the simplest Machine Learning algorithms based on work, which is classification, regression etc.
Supervised Learning technique. It assumes the Moreover, with the huge amount of data that is
similarity between the new data and available data present within the text format, it is basic to extract
and put the new data into the category that is the most knowledge out of it and build applications (NSS,

131
WEBIST 2022 - 18th International Conference on Web Information Systems and Technologies

2017). So, word embeddings are used for converting to Naïve Bayes. For precision, machine learning
all text documents into a numeric format. methods achieved better result compared to deep
learning methods. For deep learning models, GRU
Word2vec: It could be a two-layer neural net that based RNN with RmsProp optimizer (0.59)
processes text (Pathmind Inc., 2022) . The text corpus performed well compare to Adam optimizer (0.52).
takes as an input, and its output may be a set of
vectors. Whereas it is not a deep neural network it
turns text into a numerical form that deep neural
network can process. The main purpose and
usefulness of Word2vec is to group the vectors of
similar words together in vector space (Pathmind Inc.,
2022).

Gated Recurrent Unit based Recurrent Neural

Network: In this research, Simple recurrent neural
networks are not used because they do not have long
term dependencies. The way to solve gated recurrent Figure 4: Precision of various algorithms at emotion
units used. For solving, the vanishing gradient category.
problem of a standard RNN, GRU uses two gates:
update gate and reset gate. GRUs can be trained on Figure 5 shows that the Random Forest classifier
data stored for a long time without removing achieved the best performance with respect to recall
irrelevant data or cleaning the data. (0.819) for average of all emotions. Also, SVM and
Naïve Bayes perform well with a recall of 0.81 and
0.815, respectively. Moreover, K-nearest Neighbor
4 RESULTS (KNN) classifier has low recall value for trust (0.465)
and surprise (0.384) emotion but overall KNN
performed well with an average recall of 0.749. For
The following evaluation parameters were used to
deep learning methods, GRU based RNN with
evaluate the performance of the classifiers.
RmsProp optimizer (0.632) performed well compare
Accuracy: It is a ratio of correctly predicted
to Adam optimizer (0.452). Figure 5 shows the recall
emotion class to the total number of observation
of the classifiers for each emotion category.
emotion class.
Precision: It is a ratio of correctly predicted
emotion class to the total number of positive
predicted class.
Recall: It is a ratio of correctly predicted positive
emotion class to all observation in true actual class.
F1 score: F1 score is the degree of calculating the
weighted average of precision and recall. It ranges
between 0 to 1 and it is considered perfect when it is
1 which means that the model has low false positives
and low false negatives.
Confusion Matrix: A confusion matrix is used for
summarizing the performance of a classification Figure 5: Recall of algorithms at emotion category.
algorithm.
The most commonly used performance evaluation Figure 6 shows that the support vector machine
metrics for classification problems are accuracy, (SVM) classifier achieved the best performance with
Precision, recall and F1 score. Evaluation parameters respect to F1 score (0.798) for average of all
are measured with the help of confusion matrix. emotions. Moreover, K-nearest Neighbor (KNN)
Figure 4 shows that the Naïve Bayes classifier classifier has quite low result (0.671) compared to
achieved the best performance with respect to Random Forest (0.794), Naïve Bayes (0.762), and
precision (0.80) on average of all emotions. SVM. For deep learning models, both the models
Moreover, KNN method has high precision for performed similar in all emotions. But GRU based
Pessimism (0.951) emotion compared to the other RNN with RmsProp optimizer (0.595) performed
methods but did not perform well overall compared well compare to Adam optimizer (0.486). Figure 6

132
 Multi-label Emotion Classification using Machine Learning and Deep Learning Methods

shows F1 score of the classifiers for each emotion prediction. These ensemble methods considered in
category. this research are parallel in nature which means all the
models are independent of each other. Figure 7 shows
that both ensemble techniques achieved the best result
with respect to precision (0.818, 0.813), recall (0.829,
0.83) and F1 score (0.789, 0.799) for average of all
emotions respectively. Moreover, both the ensemble
techniques perform better than any individual
method. Figure 7 compares performance metrics of
ensemble methods against other individual
algorithms.

Figure 6: F1 score of algorithms at emotion category.

Notice that GRU based RNN with RmsProp,

Random Forest and SVM perform relatively better
over other methods. The efficacy of the task is
achieved through the ensemble modelling. In
ensemble modelling, the predictions of different
models are combined to produce improved
performance over any individual model in classifying Figure 7: Comparison of performance metrics of algorithms
the emotions. This approach helps in reducing the against ensemble methods.
variance and improves the generalization. The
following two popular ensemble techniques have Mohammed et al. (Jabreel M., Moreno A,
been used in this study: (i) majority voting, and (ii) 2019) achieved 0.59 accuracy, 0.57 precision,
weighted average. 0.61 recall, and 0.56 F1 score using GRU based
In majority voting approach, predictions of RNN classifier, which was used in this research
different algorithms have been combined and the as a reference. In comparison, different
majority vote is predicted. In weighted average classifiers were used for measuring all evaluation
approach, predictions of algorithms have been parameters for emotion classification labeled
combined with certain weightage. The weightage data set. GRU based RNN with RmsProp
of each algorithm is generally assigned based on the optimizer classifier gave high accuracy for multi-
individual performance of that algorithm on the data. label emotion classification from emotion
In this research, F1 score of the algorithm is classification dataset (SemEval-2018), even
considered to be its weight. though, other methods give better performance.
The ensemble methods combine the predictions of Table 1 shows that the comparison of all methods
all the other methods to produce an improved for emotion classification dataset.

Table 1: Comparison of all methods.

GRU based GRU based
Naïve Random RNN with RNN with
Number Parameters SVM KNN
Bayes Forest Adam RmsProp
Optimizer Optimizer
1 Accuracy 0.809 0.815 0.819 0.757 0.79 0.823
2 Precision 0.80 0.794 0.794 0.762 0.526 0.596
3 Recall 0.812 0.815 0.82 0.75 0.452 0.632
4 F1-score 0.762 0.798 0.794 0.67 0.486 0.595
5 AUC 0.79 0.81 0.79 0.59 0.81 0.84

133
WEBIST 2022 - 18th International Conference on Web Information Systems and Technologies

Table 2: ANOVA test results on performance metrics.

Majority Weighted
Naïve Random voting average
Metric SVM KNN RmsProp Adam P-value
Bayes Forest Method method
mean mean
Precision 0.80 0.798 0.80 0.736 0.607 0.539 0.819 0.814 6.85*10-9
Recall 0.812 0.819 0.824 0.763 0.588 0.463 0.829 0.832 1.72 *10-8
F1 Score 0.766 0.80 0.801 0.70 0.581 0.497 0.789 0.802 1.36 * 10-14
Accuracy 0.812 0.819 0.824 0.763 0.827 0.795 0.817 0.805 1.4 * 10-5

Overall, the better performance is achieved by significant performance differences (S. Rajaraman,
using machine learning methods for all evaluation Sameer K. Antani, 2020).
parameters. But GRU based RNN with Rmsprop Table 2 summarizes the ANOVA test results for
optimizer performed the best in terms of accuracy, performance metrics. It is observed that the P -values
with the highest accuracy (0.823) compared to other are lower than 0.05 for the performance metrics. This
classifiers. The results also show a huge improvement means that the methods are statistically significant
compared to the results of Mohammed et al. (Jabreel (null hypothesis H0 is rejected) when evaluated on the
M., Moreno A, 2019) for the same dataset. Figure 8 basis of these performance metrics. F1 score is the
shows that comparison of all evaluation parameters consonant mean of both precision and recall. It is a
using different classifiers. better measure of incorrectly classified cases and used
when it needs to maintain higher precision and recall
instead of just focussing on one. In this study, the
mean value of F1 score is higher for weighted average
ensemble method (0.802) compared to that of
majority voting ensemble method (0.789). This
shows, that weighted average method has proved to
be the best model in view of achieving higher F1
score and model built using weighted average method
would result in higher F1 score over other methods.

Figure 8: Comparison of all methods.

5 CONCLUSIONS
Statistical Analysis: To conclude, or to choose the
best method from these ensemble methods as well as In this research, twitter data was analysed for emotion
all classifiers, statistical one-way ANOVA test was classification. Since each tweet is associated with
performed. Test for statistical significance helps to multiple emotions not just limited to one, this
measure whether the difference between the problem has been formulated as multi-label emotion
performance metrics observed via all methods is classification. The popular machine learning
significant or not. classifiers and GRU based Recurrent Neural Network
In this research, One-way Analysis of Variance with Adam and RmsProp optimizer were used to
(ANOVA) test is performed on the mean values of solve multi-label emotion classification problem.
performance metrics on all the methods (shown in The popular ensemble techniques such as
Table 2). The null hypothesis (H0) states that all Majority voting and Weighted average methods were
models demonstrate similar performance. H0 is used for reducing the variance and improve the
accepted if no statistically significant difference (P > generalization. These methods have been proved to
0.05) is observed in the mean value of the be more accurate in terms of all the performance
performance metrics for the different models under metrics (accuracy, precision, recall, and F1 score).
study. The alternate hypothesis (H1) is accepted and Also, One-way Analysis of Variance (ANOVA) test
H0 is rejected if a statistically significant performance is performed on the mean values of performance
difference (P < 0.05) is found to exist (S. Rajaraman, metrics on all the methods.
Sameer K. Antani, 2020). One-way ANOVA is an From the results, it is concluded that accuracy
omnibus test and needs a post-hoc study to identify increased from 0.59 to 0.823 using GRU based RNN
all the methods demonstrating this statistically with RmsProp optimizer classifier which is 23.3%
(0.233) higher, precision increased from 0.57 to 0.80
using Naive Bayes classifier which is 23% (0.23)

134
 Multi-label Emotion Classification using Machine Learning and Deep Learning Methods

higher, recall increased from 0.56 to 0.82 using Neumann, and Shrikanth Narayanan. (2004). Analysis of
Random Forest classifier which is 26% (0.26) more emotion recognition using facial expressions, speech,
and F1 score increased from 0.56 to 0.798 using SVM and multimodal information, In Proceedings of the 6th
which is 23.8% (0.238) higher than Mohammed et al. International Conference on Multimodal Interfaces.
ACM, pp. 205–211
(Alicja Wieczorkowska, Piotr Synak, and Zbigniew Alicja Wieczorkowska, Piotr Synak, and Zbigniew W.
W. Ra´s., 2006) research paper results on emotion Ra´s. (2006). Multi-label classification of emotions in
classification dataset (SemEval-2018). Highest value music, In
of AUC (0.84) was achieved for GRU based RNN Intelligent Information Processing and Web Mining.
with RmsProp optimizer. For visualization, Springer, pp. 307–315
Matplotlib library was used in Jupyter Notebook to SemEval-2018 Task 1: Affect in Tweets (Emotion
compare all the results using machine learning and Classification Dataset):
deep learning methods. https://competitions.codalab.org/competitions/17751#learn
_the_details-datasets
Future Work: In the future, the present analysis can Mohammed, S., M.; Bravo-Marquez, F.; Salameh, M.;
Kiritchenko, S. (2018). Semeval-2018 task 1: Affect in
be extended by adding more feature extraction
Tweets, In
parameters and different models can be applied and Proceedings of the 12th InternationalWorkshop on
tested on different datasets. The present research Semantic Evaluation, New Orleans, LA, USA, pp. 1–
focusses on establishing the relations between the 17
tweet and emotion labels. More research can be done Jabreel M., Moreno A. (2019). A Deep Learning-Based
in the direction of exploring relations between the Approach for Multi-Label Emotion Classification in
phrases of tweet and emotion label. Transfer learning Tweets, Appl. Sci. 9:1123. doi: 10.3390/app9061123
with some existing pre-trained models for Manmohan singh. (2020). Stop the stopwords using
classification and data fusion from different data different python libraries,
https://medium.com/towards-artificial-
sources can be a good direction to explore to improve
intelligence/stop-the-stopwords-using-different-
the robustness and accuracy. In this study, dataset python-libraries-ffa6df941653
comes from only twitter source, but other social Hemalatha, Dr. G. P Saradhi Varma, Dr. A. Govardhan.
networks can be used for creating this type of dataset. (2013). Sentiment Analysis Tool using Machine
For this research, emotion classification dataset was Learning Algorithms,
used from the research paper of Mohammed et al., but IJETTCS, Vol 2, Issue 2
new dataset can be created to explore the same García-Gonzalo, E., Fernández-Muñiz, Z., García Nieto,
problem. P.J., Bernardo Sánchez, A., Menéndez Fernández, M.
(2016). Hard- Rock Stability Analysis for Span
Design in Entry-Type Excavations with Learning
Classifiers, 9, 531,
REFERENCES DOI: https://doi.org/10.3390/ma9070531
R. Gajjar and T. Zaveri. (2017). Defocus blur radius
Xiao Zhang, Wenzhong Li1, Sanglu Lu. (2017). Emotion classification using random forest classifier, 2017
detection in online social network based on multi-label International Conference on Innovations in
learning, Electronics, Signal Processing and Communication
Database Systems for Advanced Applications- 22nd (IESC), pp.219-223, DOI: https://doi.org/10.
International Conference, pp. 659-674 1109/IESPC.2017.8071896
Avetisyan, H and Bruna, Ondej and Holub, Jan. (2016). NSS (2017). “An intuitive understanding of Word
Overview of existing algorithms for emotion Embedding: From Count vectors to word2vec”,
classification https://www.analyticsvidhya.com/blog/2017/06/word-
Uncertainties in evaluations of accuracies, Journal of embeddings-count-word2veec
Physics: Conference Series, vol:772 “A Beginner’s guide to word2vec and neural word
Yadollahi, Ali and Shahraki, Ameneh Gholipour and embeddings”, https://wiki.pathmind.com/word2vec
Zaiane, Osmar R. (2017). Current State of Text S. Konstadinov. (2017). Understanding GRU networks,
Sentiment Analysis from Opinion to Emotion https://towardsdatascience.com/understanding-gru-
Mining, ACM. Survey , pp.1-25 networks- 2ef37df6c9be
Rangel and Paolo Rosso. (2016). On the impact of emotions S. Rajaraman, Sameer K. Antani. (2020). Modality-specific
on author profiling, Information Processing & deep learning model ensembles toward improving TB
Management 52, pp.73–92 detection in chest radiographs, IEEE access:
Carlos Busso, Zhigang Deng, Serdar Yildirim, Murtaza practical innovations, open solutions vol.8 :27318-
Bulut, Chul Min Lee, Abe Kazemzadeh, Sungbok Lee, 27326, DOI: 10.1109/access.2020.2971257.
Ulrich

135