Jurnal Teknik Informatika (JUTIF) DOI: https://doi.org/10.52436/1.jutif.2024.5.3.

1366
Vol. 5, No. 3, June 2024, pp. 669-674 p-ISSN: 2723-3863
e-ISSN: 2723-3871

COMPARISON PERFORMANCE OF WORD2VEC, GLOVE, FASTTEXT USING

SUPPORT VECTOR MACHINE METHOD FOR SENTIMENT ANALYSIS
Margaretha Anjani1, Helena Nurramdhani2
1
Informatics, Faculty of Computer Science, Universitas Pembangunan Nasional "Veteran" Jakarta, Indonesia
2
Information System, Faculty of Computer Science, Universitas Pembangunan Nasional "Veteran" Jakarta,
Indonesia
Email: 1margarethanjani@upnvj.ac.id, 2helenairmanda@upnvj.ac.id

(Article received: September 02, 2023; Revision: October 07, 2023; published: May 18, 2024)

Abstract

Spotify is a digital audio service that provides music and podcasts. Reviews received by the application can affect
users who will download the application. The unstructured characteristic of review text is a challenge in text
processing. To produce a valid sentiment analysis, word embedding is required. The data set that is owned is
divided by a ratio of 80:20 for training data and testing data. The method used for feature expansion is Word2Vec,
GloVe, and FastText and the method used in classification is Support Vector Machine (SVM). The three word
embedding methods were chosen because they can capture semantic, syntactic, and contextual meanings around
words when compared to traditional engineering features such as Bag of Word. The best performance evaluation
results show that the GloVe model produces the best performance compared to other word embeddings with an
accuracy value of 85%, a precision value of 90%, a recall value of 79%, and an f1-score of 85%.

Keywords: fasttext, glove, sentiment analysis, support vector machine, word2vec.

1. INTRODUCTION Sentiment analysis is included in the textual data

processing section which carries out analysis based
This current life surrounded by various
on evaluations, sentiments, opinions, and behavior, as
technological successes which are increasingly
well as the emotions of a person who is a source for
rapidly increasing, of course, increasing human needs
obtaining information about sentiments that can have
so as to produce a variety of the latest media. The
positive or negative values contained in opinions [3].
latest media is a means of telecommunication
Analysis, sentiment produces benefits and influences
technology with digitization and can be accessed
on research and applications rapidly [4]. Based on
widely which can be reached anywhere and anytime
research conducted by [5] there are various variations
[1]. With a wide selection of audio media that can be
that can be used when carrying out an analysis, one
used today it is often confusing for audio media users
example of which is by applying the word embedding
to find out which audio media is of high quality.
method in the analysis. The word embedding method
Spotify is one of the online audio media with a
also allows matching words that are similar based on
lot of digital enthusiasts that presents music and
measuring the semantic distance between the vectors
podcasts with a complete and new list that can be
embedded in the word [6]. Word embedding
listened to both offline and online [2]. With
describes every word in the document in vector form.
conditions of intense competition between the Spotify
Where the vector represents the projection of the
application and several other online audio media, it
word in the vector space. The position of the learned
has become a major concern in improving and
word comes from the text or the words around it.
perfecting an application. In connection with the
Word embedding allows you to retrieve the semantic
presence of Google Play reviews, it can launch the
and syntactical meanings of words.
views and opinions of the user's assessment of the
The analysis carried out is a performance
application. Reviews are sentences or text that
comparison of the word-embedding method which
contain comments or ratings on the application.
will be applied to the Support Vector Machine
However, some users often find it difficult to
algorithm for reviews of the use of the Spotify
understand actual information because the
application on the Google Play Store. The word
information provided by users is in text form. Where
embedding methods used are the Word2Vec, GloVe,
the text must be able to tell the views of the reviewer.
and FastText methods. After that, the researcher
Because of this, a lot of research has been done on
draws conclusions to prove which word embedding
machine learning to develop software that can assist
method has the highest levels of accuracy, precision,
with analysis.
recall, and f1-score.

669
670 Jurnal Teknik Informatika (JUTIF), Vol. 5, No. 3, June 2024, pp. 669-674

2. RESEARCH METHODS 2.2.1. Case Folding

The literature review continues throughout the Case folding is a step to equalize all letters into
data destruction process leading up to the lowercase letters so as not to make mistakes during
classification and serves as a guide for conducting the the tokenizing step.
research. Sources of literary studies in the form of e-
books, national magazines, and some international 2.2.2. Data Cleaning
magazines [7]. Research method can be seen in
Data cleaning is a steps to reduce noise by
Figure 1.
eliminating or deleting all numbers, links,
punctuation, and symbols as well as other than letters
of the alphabet.

2.2.3. Tokenizing
Tokenizing is a step to cut or separate sentences
into words for word.

2.2.4. Normalization
Normalization is a step for changing abbreviated
words into standard words, the researcher makes a
dictionary containing abbreviated words and standard
words.

2.2.5. Stopword Removal

Stopword Removal is a step to take important
Figure 1. The Research Steps words from the results of the previous tokenizing
step. This step uses two techniques, including stop
Based on Figure 1, there are 5 (five) steps to be lists (removing unnecessary words) and word lists
carried out in this reasearch, including data collection (saving important words).
and labelling, preprocessing data, feature extraction
with word embedding, split data and classification,
2.2.6. Stemming
and model evaluation. The discussion of each step is
as follows: Stemming is a step to return words that have
basic words by removing affixes to these words.
2.1. Data Collection and Labelling
2.3. Feature Extraction Word Embedding
Data collection uses scraping techniques. Where
web scraping is an activity of collecting data Word embedding is a method of inserting
automatically from the internet by writing a program words, then converting words into continuous vector
that is stored in a file with a certain format and parsing based on a predetermined length, so that they are not
the data to extract the necessary information [8]. limited by a larger vocabulary. The word embedding
Labeling is done on reviews with a rating of 1, 2 and model allows us to associate similar words by
3 will be labeled negative or a value of 0, while measuring the semantic distance between vectors
reviews with a rating of 4 and 5 will be labeled embedded in these words [7].
positive or a value of 1 [9].
2.3.1. Word2Vec
2.2. Preprocessing
Word2Vec is a method that studies each word in
Text preprocessing is the process of converting a corpus to generate individual word vectors that
unstructured data into structured data. The purpose of depend on the local statistical word context. The
text preprocessing is that the data to be used has vector representation has the property of a
smaller dimensions, noise will be reduced, and the relationship with the related word during the training
data will become more structured so that the data can process [14]. The result of the vector grouping of
be further processed [10]. A collection of several similar words is a word dictionary which contains a
documents that have unstructured format can also be collection of word similarity [15].
called corpus [11]. Text mining has several steps
including text extraction using a certain technique, 2.3.2. GloVe
text preprocessing or commonly known as text
GloVe or what can be called Global Vector is an
processing, and text weighting or indexing, as well as
algorithm to get a vector representation of words [16].
text [12]. According to [13] the following are the
GloVe is a method that studies every word in a corpus
steps of text preprocessing:
 Margaretha Anjani, et al., COMPARISON PERFORMANCE OF WORD2VEC … 671

to generate vectors for individual words that depend a. True Positive (TP) is positive data that classified
on the context of global statistical words. Glove has a with positive data, so it is called true or true as
great similarity between vocabulary and other positive data.
vocabulary, so this value can be used to expand the b. True Negative (TN) is negative data that
features of sentiment analysis [17]. classified with negative data, so it is called true
or true as negative data.
2.3.3. FastText c. False Positive (FP) is negative data that
classified with positive data, so it is called false
FastText is a method that learns or applies n- or wrong as positive data.
gram characters to produce a numeric representation. d. False Negative (FN) is positive data that
Where FastText has the advantage of relatively faster classified with negative data, so it is called false
processing time and is able to process words that do or wrong as negative data.
not appear in the dictionary (vocabulary), which in
Word2Vec can cause errors [18]. The confusion matrix is used to calculate
performance metrics, where calculate the
2.4. Classification Support Vector Machine performance of the model that has been generated.
Support Vector Machine (SVM) is a technique The following is an understanding of performance
for making predictions in classification and metrics:
regression [19]. SVM is a classification method using 1. Accuracy
the basic concept which is a harmonious combination Accuracy represents accuracy of the model
derived from computational theory in the previous when it classifies correctly or in other words it
decades. SVM itself requires positive and negative produces the ratio of correct predictions to all existing
training sets to make the best decision when dividing data. Accuracy compares the total True Positive, and
positive data with negative data in an n-dimensional True Negative to the total data.
space or what is commonly called a hyperplane [20].
𝑇𝑃+𝑇𝑁
According by [21], Example of a hyperplane in class 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = (1)
𝑇𝑃+𝑇𝑁+𝐹𝑃+𝐹𝑁
can be seen in Figure 2.
2. Precision
Precision represents accuracy of the data on the
predicted results given by the model. Precision
compares total True Positives to total True Positives
and False Positives.

𝑇𝑃
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = (2)
𝑇𝑃+𝐹𝑃

3. Recall
Recall represents the success of the model in
Figure 2. Hyperplane Separator
recovering information or in other words the ratio that
is predicted to be positive compared to the overall
2.5. Confussion Matrix positive data results. Precision compares total True
Positives to total True Positives and False Negatives.
The confusion matrix will show the total sample
in the right class and the total sample in the wrong 𝑇𝑃
𝑅𝑒𝑐𝑎𝑙𝑙 = (3)
class. The True Positive (TP) and True Negative (TN) 𝑇𝑃+𝐹𝑁
values will refer to the total sample with the class that 4. F1-Score
is estimated to be correct. Whereas the False Positive F1-Score represents the average division of the
(FP) and True Negative (FN) valueswill refer to the weighted precision and recall.
total sample with a class that will be estimated to be
wrong [22]. Combinations of actual values and 2×𝑃×𝑅
𝐹1 − 𝑆𝑐𝑜𝑟𝑒 = (4)
prediction values can be seen in Table 1. 𝑃+𝑅

Table 1. Confussion Matrix 3. RESULTS AND DISCUSSIONS

Prediction Class
Positive Negative 3.1. Data Collection and Labelling
True Positive False Positive
Positive
Actual (TP) (FP) The data collection step is carried out using a
Class True Negative True Negative scraping technique on the Google Play Store. The
Negative
(FN) (TN)
scraping technique is carried out using the Python
programming language which uses the google-play-
Table 1 describes the confusion matrix class scraper library. The scraping technique was carried
which contains 4 elements in it with the provisions for out on October 18 2022 and April 17 2023 with
setting the element values as follows:
672 Jurnal Teknik Informatika (JUTIF), Vol. 5, No. 3, June 2024, pp. 669-674

review data from the Spotify application as many as jaringan udah bagus udah bagus pake
1512 of the latest reviews and in Indonesian which pake data seluler ga data seluler ga bisa
bisa pake wifi juga pake wifi juga ga
were saved using the csv format. The dataset was ga bisa.. :( bisa
taken from 09 October 2022 to 17 October 2022 and Tokenizing kok ga bisa login [kok, ga, bisa,
01 April 2023 to 16 April 2023. And stored in the sih padahal jaringan login, sih, padahal,
form of csv data format which has columns totaling 3 udah bagus pake jaringan, udah,
data seluler ga bisa bagus, pake, data,
variables, namely Comment, Rating, and Date. The pake wifi juga ga seluler, ga, bisa,
result of scraping the Spotify application review data bisa pake, wifi, juga, ga,
can be seen in Figure 3. bisa]
Normalization [kok, ga, bisa, login, [kok, tidak, bisa,
sih, padahal, masuk, sih, padahal,
jaringan, udah, jaringan, sudah,
bagus, pake, data, bagus, pakai, data,
seluler, ga, bisa, seluler, tidak, bisa,
pake, wifi, juga, ga, pakai, internet,
bisa] juga, tidak, bisa]
Stopword [kok, tidak, bisa, [masuk, jaringan,
Removal masuk, sih, padahal, bagus, pakai, data,
jaringan, sudah, seluler, pakai,
Figure 3. Scraping Data Results bagus, pakai, data, internet]
seluler, tidak, bisa,
The collected Spotify application review data pakai, internet, juga,
will proceed to the labeling stage. Data labeling into tidak, bisa]
two positive reviews and negative reviews. The data Stemming [masuk, jaringan, [masuk, jaring,
bagus, pakai, data, bagus, pakai, data,
labeling is adjusted to the rating on the application seluler, pakai, seluler, pakai,
review on the Google Play Store, where ratings 1, 2, internet] internet]
and 3 will be labeled as negative sentiment or with a
value of 0, while ratings 4 and 5 will be labeled as 3.3. Feature Extraction
positive sentiment or with a value of 1. In The
labeling process is not only adjusted to the rating on The Word2Vec, GloVe, and FastText methods
the review, but also verified by Indonesian users who as word embedding at the feature extraction step use
validate the assessment of a review data whether it is the genism library which provides Word2Vec,
appropriate to include positive or negative sentences GloVe, and FastText calculations to form vectors
based on a subjective point of view by Indonesian containing similarity weights. The implementation
users. The final results of the total labeling of the uses a window with a value of 3 as the range in
spotify application reviews can be seen in Table 2 similarity and determines the size of the vector with a
below. length of 1000. Then the average or mean calculation
of all word vectors in the sentence will be applied to
Table 2. Total Labelling represent the embedding vector set can be seen in
Sentiment Negative Positive Table 4.
Total 756 756
Table 4. Vector Result of Word2Vec, Glove, and FastText
Table 2 is the total results of labeling the review Models
data used, there were 756 reviews labeled negative Kata Word2Vec GloVe FastText
sentiment and 756 reviews labeled positive sentiment. keren -0.000088 0.042555 -0.001348
bagus -0.000053 -0.003743 -0.001396
With a total review data of 1512 reviews which were baru -0.000092 0.129566 -0.001374
labeled manually by matching ratings and reviews. putar -0.000105 -0.015709 -0.001581
lagu -0.000089 -0.015919 -0.001844
banget -0.000096 -0.099364 -0.001394
3.2. Preprocessing
The preprocessing step is carried out in several 3.4. Classification
steps to be able to remove noise and words that have
no meaning. The steps to be carried out include case The classification process uses 80% training
folding, data cleaning, tokenizing, normalization, data and 20% testing data which are distributed
stopword removal, and stemming as shown in Table randomly and fairly. The classifier uses the
3. classification_report library from sklearn. After the
feature extraction modeling process using training
Table 3. Preprocessing data, the model will be tested using data testing to
Preprocessing Sebelum Sesudah determine the performance of the model by
Case Folding Kok ga bisa login kok ga bisa login
classifying the test data.
sih? Padahal sih? padahal
jaringan udah bagus jaringan udah bagus
pake data seluler ga pake data seluler ga 3.5. Model Evaluation
bisa pake wifi juga bisa pake wifi juga
ga bisa.. :( ga bisa.. :( Based on results of performance evaluation of
Data Cleaning kok ga bisa login kok ga bisa login three word embedding models, there is a comparison
sih? padahal sih padahal jaringan
 Margaretha Anjani, et al., COMPARISON PERFORMANCE OF WORD2VEC … 673

between the word embedding model trials between the SVM algorithm has the best performance, then in
Word2Vec, GloVe, and FastText. The values that will the second position is the FastText word embedding
be compared from the evaluation results are accuracy, model using the SVM algorithm, and the last position
precision, recall, and f-1 score. The results of the Word2Vec word embedding model using the SVM
model evaluation calculations are written in Table 5. algorithm for processing spotify application review
data. This can be proven by the increase in each
Table 5. Comparision Evaluation Results
evaluation value, where the GloVe model is in the
N Data Word Accu Prec Rec F1-
o Embedding racy ision all Score first rank, the FastText model is in the second rank,
1 Data Word2Vec 65% 74% 44% 56% and the Word2Vec model is in the third rank.
Testi GloVe 85% 90% 79% 85% The Glove model has the highest performance
ng FastText 71% 76% 62% 68% evaluation score due to the efficient use of statistics,
2 Data Word2Vec 64% 72% 46% 57%
Trai GloVe 89% 92% 85% 88% where GloVe combines global statistics to get word
ning FastText 78% 81% 73% 77% vectors, unlike Word2Vec and FastText which rely
on local statistics to get word vectors. The second
Table 5. shows the obtained results of the position FastText models which have the second
comparison of the three feature extraction model highest performance evaluation value can be due to
experiments using the evaluation results including being an extension of the Word2Vec model which
accuracy values, precision values, recall values and treats each word as a composition of n-gram
f1-score values. Of the three feature extraction model characters. Meanwhile, the Word2Vec model is the
experiments, the GloVe model produced the best oldest model that treats every word in the corpus to
evaluation results on data testing and training data, produce a vector for each word.
then the FastText model on data testing and training
data, and the last Word2Vec model on data testing 4. CONCLUSION
and training data. The comparative description of the
three word embedding models using Data Testing can The performance evaluation results are seen
be seen in Figure 4 and also the comparative using from the level of accuracy, precision, recall, and f1-
Data Training can be seen in Figure 5. score applying the confusion matrix calculation, the
Word2Vec method produces a accuracy of 65%, a
precision of 74%, a recall of 44%, and a f1-score of
56%. The GloVe method produces a accuracy of
85%, a precision of 90%, a recall of 79%, and a f1-
score of 85%. While the FastText method has an
accuracy of 71%, a precision of 76%, a recall of 62%,
and a f1-score of 68%. Based on the evaluation results
above, the use of the GloVe method is better than the
FastText and Word2Vec methods for processing the
Spotify application review dataset using the SVM
algorithm in this study. Based on the evaluation
results between the application of the word
embedding method, comparisons can be made. The
GloVe method produces the highest evaluation value
Figure 4. Graph Model Evaluation using Data Testing than the Word2Vec and FastText methods for
processing spotify application review data with the
SVM algorithm in this study, due to the use of
efficient statistics, where GloVe combines global
statistics to obtain word vectors, unlike Word2Vec
and FastText which rely on local statistics to get the
word vector.

REFERENCES
[1] D. McQuail, Teori Komunikasi Massa
McQuail 1, 6E (6th ed.). Salemba Humanika,
2011.
[2] Spotify, About Us, Spotify, 2022
Figure 5. Graph Model Evaluation using Data Training https://www.spotify.com/us/about-
us/contact/ (accessed Sep 16, 2022).
Based on Figure 4 and Figure 5, the comparative [3] F. V. Sari and A. Wibowo, "Analisis
description of the three word embedding models that Sentimen Pelanggan Toko Online Jd.Id
have been implemented, it can be understood that in Menggunakan Metode Naïve Bayes
this study, the GloVe word embedding model using Classifier Berbasis Konversi Ikon Emosi"
674 Jurnal Teknik Informatika (JUTIF), Vol. 5, No. 3, June 2024, pp. 669-674

Jurnal SIMETRIS, vol. 10, no. 2, pp. 2252– [13] E. Sonalitha, S. R. Asriningtias, and A.
4983, 2019. Zubair, Text Mining (Pertama). Graha Ilmu,
[4] G. A. Buntoro, "Analisis Sentimen Calon 2021.
Gubernur DKI Jakarta 2017 Di Twitter," [14] A. S. Girsang, Word Embedding dengan
Integer Journal, vol. 2, no. 1, pp. 32–41, Word2vec, 2020.
2017. https://t.co/jrvaMsgBdH https://mti.binus.ac.id/2020/11/17/word-
[5] A. Nurdin, B. A. S. Aji, A. Bustamin, and Z. embedding-dengan-
Abidin, "Perbandingan Kinerja Word word2vec/#:~:text=Word%20embeddings%
Embedding Word2vec, Glove, Dan Fasttext 20adalah%20proses%20konversi%20kata%
Pada Klasifikasi Teks", Jurnal 20yang%20berupa,merepresentasikan%20se
TEKNOKOMPAK, vol. 14, no. 2, pp. 74–79, buah%20titik%20pada%20space%20dengan
2020. %20dimensi%20tertentu (accessed Sep 30,
2022)
[6] M. D. Rhman, A. Dhunaidy, and f.
Mahananto, "Penerapan Weighted Word [15] H. F. Naufal and E. B. Setiawan, "Ekspansi
Embedding pada Pengklasifikasian Teks Fitur Pada Analisis Sentimen Twitter Dengan
Berbasis Recurrent Neural Network untuk Pendekatan Metode Word2Vec," E-
Layanan Pengaduan Perusahaan Proceeding of Engineering, vol. 8, no. 5, pp.
Transportasi," JURNAL SAINS DAN SENI 10339, 2021.
ITS, vol. 10, no. 1, pp. 2337–3520, 2021. [16] J. Pennington, R. Socher, and C. D. Manning,
[7] S. Fransiska, Rianto, and A. I. Gufroni, GloVe: Global Vectors for Word
"Sentiment Analysis Provider by.U on Representation, 2014.
Google Play Store Reviews with TF-IDF and https://nlp.stanford.edu/projects/glove/
Support Vector Machine (SVM) Method," (accessed Sep 20, 2022).
Scientific Journal of Informatics, vol. 7, no. [17] M. D. D. Sreya, and E. B. Setiawan,
2, pp. 2407–7658, 2020. "Penggunaan Metode GloVe untuk Ekspansi
http://journal.unnes.ac.id/nju/index.php/sji Fitur pada Analisis Sentimen Twitter dengan
[8] R. Mitchell, Web Scraping with Python: Naïve Bayes dan Support Vector Machine,"
Collecting More Data from the Modern Web E-Proceeding of Engineering, vol. 9, no. 3,
(A. MacDonald, Ed.; Second Edition). 2022.
O’Reilly Media, Inc, 2018. [18] A. S. Girsang, Word Embedding dengan
[9] H. Nguyen, A. Veluchamy, M. L. Diop and FastText, 2021.
R. Iqbal, "Comparative Study of Sentiment https://mti.binus.ac.id/2021/12/31/word-
Analysis with Product Reviews Using embedding-dengan-
Machine Learning and Lexicon-Based fasttext/#:~:text=Word%20embedding%20m
Approaches," SMU Data Science Review, enangkap%20informasi%20semantik%20da
vol. 1, no. 4, 2018. n%20kata%20sintaksis%2C,oleh%20Facebo
https://scholar.smu.edu/datasciencereviewA ok%20yang%20dapat%20digunakan%20unt
vailableat:https://scholar.smu.edu/datascienc uk%20word%20embedding (accessed Sep
ereview/vol1/iss4/7http://digitalrepository.s 20, 2022).
mu.edu. [19] B. Santosa, Data Mining : Terbaik
[10] D. A. Fauziah, A. Maududie, and I. Nuritha, Pemanfaatan Data untuk Keperluan Bisnis
"Klasifikasi Berita Politik Menggunakan (Ed. 1, Cet. 1). Graha Ilmu, 2007.
Algoritma K-nearst Neighbor (Classification [20] I. Cholissodin, Sutrisno, A. A. Soebroto, U.
of Political News Content using K-Nearest Hasanah, and Y. I. Febiola, AI, Machine
Neighbor)," BERKALA SAINSTEK, vol. 6, Learning & Deep Learning, 2019.
no. 2, pp. 106–114, 2018. https://www.researchgate.net/publication/34
[11] B. Titania, PENERAPAN METODE TEXT 8003841
MINING DAN SOCIAL NETWORK [21] A. Pratama, Klasifikasi Kondisi Detak
ANALYSIS PADA JEJARING SOSIAL Jantung Berdasarkan Hasil Pemeriksaan
TWITTER, 2020. Elektronikardiografi (EKG) Menggunakan
[12] A. M. Pravina, I. Cholissodin, and P. P. Binary Decision Tree-Support Vector
Adikara, "Analisis Sentimen Tentang Opini Machin (BDT-SVM), 2016.
Maskapai Penerbangan pada Dokumen [22] K. N. Utami and E. B. Setiawan, "Ekspansi
Twitter Menggunakan Algoritme Support Fitur dengan FastText pada Klasisikadi Topik
Vector Machine (SVM)," Jurnal dengan Metode Naïve Bayes-Support Vector
Pengembangan Teknologi Informasi Dan Machine (NBSVM) di Twitter," E-
Ilmu Komputer, vol. 3, no. 3, pp. 2789–2797, Proceeding of Engineering, vol. 9, no. 3, pp.
2019. http://j-ptiik.ub.ac.id 1872, 2022. https://t.co/C1SAKZKniG.