Abstract
The utilization of word embeddings—powerful models computed through Neural Network architectures that encode words as vectors—has witnessed rapid growth across various Natural Language Processing applications, encompassing semantic analysis, information retrieval, dependency parsing, question answering, and machine translation. The efficacy of these tasks is strictly linked to the quality of the embeddings, underscoring the critical importance of evaluating and selecting optimal embedding models. While established procedures and benchmarks exist for intrinsic evaluation, the authors note a conspicuous absence of comprehensive evaluations of intrinsic embedding quality across multiple tasks. This paper introduces vec2best, a unified tool encompassing state-of-the-art intrinsic evaluation tasks across diverse benchmarks. vec2best furnishes the user with an extensive evaluation of word embedding models. It represents a framework for evaluating word embeddings trained using various methods and hyper-parameters on a range of tasks from the literature. The tool yields a holistic evaluation metric for each model called the PCE (Principal Component Evaluation). We conducted evaluations on 135 word embedding models, trained using GloVe, fastText, and word2vec, across four tasks integrated into vec2best (similarity, analogy, categorization, and outlier detection), along with their respective benchmarks. Additionally, we leveraged vec2best to optimize embedding hyper-parameter configurations in a real-world scenario. vec2best is conveniently accessible as a pip-installable Python package.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data Availability
The datasets generated during and analyzed during the current study will be available in a public GitHub repository, and the URL will be provided after the paper’s acceptance.
Notes
The package is available at https://pypi.org/project/vec2best/
only for fastText and word2vec.
H2020-SC6-TRANSFORMATIONS-2018-2019-2020, grant agreement no. 101004703, https://www.h2020-pillars.eu/
The OECD Programme for the International Assessment of Adult Competencies (PIAAC) is an international survey designed to assess skills of adults aged 16 to 65.
ESCO (European Skills, Competences, Qualifications, and Occupations) is the European multilingual classification of Skills, Competences, and Occupations. It acts as a dictionary, describing, identifying, and classifying professional occupations and skills relevant to the EU labor market and education and training.
Belgium, Cyprus, Czech Republic, Denmark, France, Germany, Greece, Ireland, Italy, Lithuania, Netherlands, Poland, Slovak Republic, Slovenia, Spain, Sweden, and the United Kingdom (UK)
References
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: NeurIPS (2013)
Wang, B., Wang, A., Chen, F., Wang, Y., Kuo, C.-C.J.: Evaluating word embedding models: methods and experimental results. APSIPA transactions on signal and information processing 8 (2019)
Schnabel, T., Labutov, I., Mimno, D., Joachims, T.: Evaluation methods for unsupervised word embeddings. In: EMNLP (2015)
Camacho-Collados, J., Pilehvar, M.T.: From word to sense embeddings: a survey on vector representations of meaning. Journal of Artificial Intelligence Research 63, 743–788 (2018)
Baroni, M., Dinu, G., Kruszewski, G.: Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In: ACL (2014)
Bakarov, A.: A survey of word embeddings evaluation methods (2018)
Giabelli, A., Malandri, L., Mercorio, F., Mezzanzanica, M., Nobani, N.: Embeddings evaluation using a novel measure of semantic similarity. Cognitive Computation, 1–15 (2022)
Giabelli, A., Malandri, L., Mercorio, F., Mezzanzanica, M., Seveso, A.: Neo: A tool for taxonomy enrichment with new emerging occupations. In: The Semantic Web–ISWC 2020: 19th International Semantic Web Conference, Athens, Greece, November 2–6, 2020, Proceedings, Part II 19, pp. 568–584 (2020). Springer
Gladkova, A., Drozd, A.: Intrinsic evaluations of word embeddings: what can we do better? In: Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pp. 36–42 (2016)
Levy, O., Goldberg, Y., Dagan, I.: Improving distributional similarity with lessons learned from word embeddings. TACL 3 (2015)
Caselles-Dupré, H., Lesaint, F., Royo-Letelier, J.: Word2vec applied to recommendation: Hyperparameters matter. In: RECSYS (2018)
Torregrossa, F., Claveau, V., Kooli, N., Gravier, G., Allesiardo, R.: On the correlation of word embedding evaluation metrics. In: Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pp. 4789–4797 (2020)
Torregrossa, F., Allesiardo, R., Claveau, V., Kooli, N., Gravier, G.: A survey on training and evaluation of word embeddings. International Journal of Data Science and Analytics 11, 85–103 (2021)
Lai, S., Liu, K., He, S., Zhao, J.: How to generate a good word embedding. IEEE Intelligent Systems 31(6), 5–14 (2016) 10.1109/MIS.2016.45
Faruqui, M., Dyer, C.: Community evaluation and exchange of word vectors at wordvectors. org. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 19–24 (2014)
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5, 135–146 (2017)
Pennington, J., Socher, R., Manning, C.: Glove: global vectors for word representation. In: EMNLP (2014)
Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In NAACL. Association for Computational Linguistics New Orleans, Louisiana, USA (2018)
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Roy, A., Pan, S.: Incorporating extra knowledge to enhance word embedding. In: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 4929–4935 (2021)
Asudani, D.S., Nagwani, N.K., Singh, P.: Impact of word embedding models on text analytics in deep learning environment: a review. Artificial Intelligence Review, 1–81 (2023)
Formica, A., Taglino, F.: Semantic relatedness in DBpedia: a comparative and experimental assessment. Information Sciences 621, 474–505 (2023)
Zhang, M., Palade, V., Wang, Y., Ji, Z.: Word representation using refined contexts. Applied Intelligence 52(11), 12347–12368 (2022)
Jameel, S., Schockaert, S.: Word and document embedding with vMF-mixture priors on context word vectors. (2019). ACL
Yang, D., Li, N., Zou, L., Ma, H.: Lexical semantics enhanced neural word embeddings. Knowledge-Based Systems 252, 109298 (2022)
An, H., Liu, X., Zhang, D.: Learning bias-reduced word embeddings using dictionary definitions. In: Findings of the Association for Computational Linguistics: ACL 2022, pp. 1139–1152 (2022)
Zheng, J., Wang, Y., Wang, G., Xia, J., Huang, Y., Zhao, G., Zhang, Y., Li, S.: Using context-to-vector with graph retrofitting to improve word embeddings. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8154–8163. Association for Computational Linguistics, Dublin, Ireland (2022). 10.18653/v1/2022.acl-long.561 . https://aclanthology.org/2022.acl-long.561
Camacho-Collados, J., Navigli, R.: Find the word that does not belong: a framework for an intrinsic evaluation of word vector representations. In: Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pp. 43–50 (2016)
Corcoran, P., Palmer, G., Arman, L., Knight, D., Spasić, I.: Creating welsh language word embeddings. Applied Sciences 11(15), 6896 (2021)
Jolliffe, I.T., Cadima, J.: Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 374(2065), 20150202 (2016)
Hadj Taieb, M.A., Zesch, T., Ben Aouicha, M.: A survey of semantic relatedness evaluation datasets and procedures. Artificial Intelligence Review 53(6), 4407–4448 (2020)
Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., Ruppin, E.: Placing search in context: the concept revisited. In: Proceedings of the 10th International Conference on World Wide Web, pp. 406–414 (2001)
Rubenstein, H., Goodenough, J.B.: Contextual correlates of synonymy. Communications of the ACM 8(10), 627–633 (1965)
Luong, M.-T., Socher, R., Manning, C.D.: Better word representations with recursive neural networks for morphology. In: Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pp. 104–113 (2013)
Bruni, E., Tran, N.-K., Baroni, M.: Multimodal distributional semantics. Journal of artificial intelligence research 49, 1–47 (2014)
Radinsky, K., Agichtein, E., Gabrilovich, E., Markovitch, S.: A word at a time: computing word relatedness using temporal semantic analysis. In: Proceedings of the 20th International Conference on World Wide Web, pp. 337–346 (2011)
Hill, F., Reichart, R., Korhonen, A.: Simlex-999: evaluating semantic models with (genuine) similarity estimation. Computational Linguistics 41(4) (2015)
Miller, G.A., Charles, W.G.: Contextual correlates of semantic similarity. Language and cognitive processes 6(1), 1–28 (1991)
Halawi, G., Dror, G., Gabrilovich, E., Koren, Y.: Large-scale learning of word relatedness with constraints. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1406–1414 (2012)
Yang, D., Powers, D.: Verb similarity on the taxonomy of wordnet. In: The Third International WordNet Conference: GWC 2006 (2006). Masaryk University
Baker, S., Reichart, R., Korhonen, A.: An unsupervised model for instance level subcategorization acquisition. In: EMNLP, pp. 278–289 (2014)
Gerz, D., Vulić, I., Hill, F., Reichart, R., Korhonen, A.: Simverb-3500: a large-scale evaluation set of verb similarity. arXiv preprint arXiv:1608.00869 (2016)
Camacho-Collados, J., Pilehvar, M.T., Collier, N., Navigli, R.: Semeval-2017 task 2: multilingual and cross-lingual semantic word similarity. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 15–26 (2017)
Agirre, E., Alfonseca, E., Hall, K., Kravalova, J., Pasca, M., Soroa, A.: A study on similarity and relatedness using distributional and wordnet-based approaches (2009)
Allen, C., Hospedales, T.: Analogies explained: towards understanding word embeddings. In: International Conference on Machine Learning, pp. 223–231 (2019). PMLR
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
Mikolov, T., Yih, W.-t., Zweig, G.: Linguistic regularities in continuous space word representations. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 746–751 (2013)
Jurgens, D., Mohammad, S., Turney, P., Holyoak, K.: Semeval-2012 task 2: measuring degrees of relational similarity. In: * SEM 2012: The First Joint Conference on Lexical and Computational Semantics–Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 356–364 (2012)
Hanson, S.J., Bauer, M.: Conceptual clustering, categorization, and polymorphy. Machine Learning 3, 343–372 (1989)
Almuhareb, A.: Attributes in lexical acquisition. PhD thesis, University of Essex (2006)
Baroni, M., Lenci, A.: How we blessed distributional semantic evaluation. In: Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, pp. 1–10 (2011)
Baroni, M., Murphy, B., Barbu, E., Poesio, M.: Strudel: a corpus-based semantic model based on properties and types. Cognitive science 34(2), 222–254 (2010)
Baroni, M., Evert, S., Lenci, A.: Bridging the gap between semantic theory and computational simulations: Proceedings of the esslli workshop on distributional lexical semantics. Hamburg, Germany: FOLLI (2008)
Blair, P., Merhav, Y., Barry, J.: Automated generation of multilingual clusters for the evaluation of distributed representations. arXiv preprint arXiv:1611.01547 (2016)
Ninio, F.: A simple proof of the Perron-Frobenius theorem for positive symmetric matrices. Journal of Physics A: General Physics 9(8), 1281–1282 (1976) 10.1088/0305-4470/9/8/017
Jastrzebski, S., Leśniak, D., Czarnecki, W.M.: How to evaluate word embeddings? on importance of data efficiency and simple supervised tasks. arXiv preprint arXiv:1702.02170 (2017)
Guo, Y., Langer, C., Mercorio, F., Trentini, F.: Skills mismatch, automation, and training: evidence from 17 European countries using survey data and online job ads. In: CESifo Forum, vol. 23, pp. 11–15 (2022)
Boselli R, Cesarini M, Mercorio F, Mezzanzanica M. Classifying online Job advertisements through machine learning. Future Gener Comput Syst. 2018;86:319–28.
Colombo E, Mercorio F, Mezzanzanica M. AI meets labor market: Exploring the link between automation and skills. Inf Econ Policy. 2019;47:27–37.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare no competing interests.
Research Involving Human Participants or Animals
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ascari, R., Giabelli, A., Malandri, L. et al. A Fistful of Vectors: A Tool for Intrinsic Evaluation of Word Embeddings. Cogn Comput 16, 949–963 (2024). https://doi.org/10.1007/s12559-023-10235-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12559-023-10235-3