Skip to main content
Springer Nature Link
Log in

MAPA BiLSTM-BERT: multi-aspects position aware attention for aspect level sentiment analysis

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Sentiment categorization at the aspect level tries to provide fine-grained sentiment polarities for distinct aspects inside a sentence. Some issues remain unresolved in the previous work. First, the specific position context is not fully addressed. Second, the distinct aspect of an opinionated sentence is evaluated independently. Also, the present, attentive approaches neglect the disruption caused by all the aspects in the same sentence while measuring the current aspect attention vector. We proposed multi-aspect-specific position attention bidirectional long short-term memory (MAPA BiLSTM)-bidirectional encoder representations from transformers (BERT) model to address these issues. The MAPA BiLSTM BERT introduces the explicit multiple aspect position-aware attention between the aspect word and the closest context words, also BERT aspect-specific attention investigates how to model multiple aspects using the aspect position attention mechanism. The parallel fused MAPA BiLSTM-BERT gains the multiple aspect contextual knowledge in a sentence for aspect classification. We conduct an empirical evaluation of the proposed method on the laptop review, restaurant review (SemEval2014 datasets), Twitter review, and multi aspect multi-sentiment (MAMS) dataset; results indicate a significant performance improvement over state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

Data used in the research article are available from a reasonable research point of view request.

References

  1. Wankhade M, Rao ACS, Kulkarni C (2022) A survey on sentiment analysis methods, applications, and challenges. Artif Intell Rev 55:5731–5780

    Article  Google Scholar 

  2. Majumder N, Bhardwaj R, Poria S, Gelbukh A, Hussain A (2020) Improving aspect-level sentiment analysis with aspect extraction. Neural Comput Appl 1–11

  3. Wu T, Tang S, Zhang R, Cao J, Zhang Y (2020) Cgnet: a light-weight context guided network for semantic segmentation. IEEE Trans Image Process 30:1169–1179

    Article  Google Scholar 

  4. Nazir A, Rao Y, Wu L, Sun L (2020) Issues and challenges of aspect-based sentiment analysis: a comprehensive survey. IEEE Trans Affect Comput

  5. Xue W, Li T (2018) Aspect based sentiment analysis with gated convolutional networks. arXiv preprint arXiv:1805.07043

  6. Wang Y, Huang M, Zhu X, Zhao L (2016) Attention-based lstm for aspect-level sentiment classification. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp 606–615

  7. Tang D, Qin B, Feng X, Liu T (2015) Effective LSTMs for target-dependent sentiment classification. arXiv preprint arXiv:1512.01100

  8. Chen P, Sun Z, Bing L, Yang W (2017) Recurrent attention network on memory for aspect sentiment analysis. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp 452–461

  9. Chen P, Sun Z, Bing L, Yang W (2017) Recurrent attention network on memory for aspect sentiment analysis. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp 452–461

  10. Li X, Bing L, Lam W, Shi B (2018) Transformation networks for target-oriented sentiment classification. arXiv preprint arXiv:1805.01086

  11. Rockt aschel T, Grefenstette E, Hermann KM, Kocisky T, Blunsom P (2015) Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664

  12. Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

  13. Tang D, Qin B, Liu T (2016) Aspect level sentiment classification with deep memory network. arXiv preprint arXiv:1605.08900

  14. Wankhade M, Annavarapu CSR, Verma MK (2021) CBVoSD: context based vectors over sentiment domain ensemble model for review classification. J Supercomput 1–37

  15. Wang Y, Huang M, Zhu X, Zhao L (2016) Attention-based LSTM for aspect-level sentiment classification. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp 606–615

  16. Wang S, Mazumder S, Liu B, Zhou M, Chang Y (2018) Target-sensitive memory networks for aspect sentiment classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 957–967

  17. Kiritchenko S, Zhu X, Cherry C, Mohammad S (2014) NRC-Canada-2014: detecting aspects and sentiment in customer reviews. In: Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp 437–442

  18. Tang D, Qin B, Liu T (2016) Aspect level sentiment classification with deep memory network. arXiv preprint arXiv:1605.08900

  19. Guo X, Zhang G, Wang S, Chen Q (2020) Multi-way matching based fine-grained sentiment analysis for user reviews. Neural Comput Appl 32(10):5409–5423

    Article  Google Scholar 

  20. Liu B (2012) Sentiment analysis and opinion mining. Synth Lect Hum lang Technol 5(1):1–167

    Article  Google Scholar 

  21. Yang M, Qu Q, Shen Y, Lei K, Zhu J (2020) Cross-domain aspect/sentiment-aware abstractive review summarization by combining topic modeling and deep reinforcement learning. Neural Comput Appl 32(11):6421–6433

    Article  Google Scholar 

  22. Ahmed M, Chen Q, Li Z (2020) Constructing domain-dependent sentiment dictionary for sentiment analysis. Neural Comput Appl 32(18):14719–14732

    Article  Google Scholar 

  23. Chen P, Sun Z, Bing L, Yang W (2017) Recurrent attention network on memory for aspect sentiment analysis. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp 452–461

  24. Jin Z, Yang Y, Liu Y (2020) Stock closing price prediction based on sentiment analysis and LSTM. Neural Comput Appl 32(13):9713–9729

    Article  Google Scholar 

  25. Tang D, Qin B, Feng X, Liu T (2015) Effective LSTMs for target-dependent sentiment classification. arXiv preprint arXiv:1512.01100

  26. Wu H, Zhang Z, Shi S, Wu Q, Song H (2022) Phrase dependency relational graph attention network for aspect-based sentiment analysis. Knowl Based Syst 236:107736

    Article  Google Scholar 

  27. Zhang B, Hu Y, Xu D, Li M, Li M (2022) SKG-learning: a deep learning model for sentiment knowledge graph construction in social networks. Neural Comput Appl 1–20

  28. Zhang C, Li Q, Song D (2019) Aspect-based sentiment classification with aspect-specific graph convolutional networks. arXiv preprint arXiv:1909.03477

  29. Li X, Lam W (2017) Deep multi-task learning for aspect term extraction with memory interaction. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp 2886–2892

  30. Gu S, Zhang L, Hou Y, Song Y (2018) A position-aware bidirectional attention network for aspect-level sentiment analysis. In: Proceedings of the 27th International Conference on Computational Linguistics, pp 774–784

  31. Tang D, Qin B, Liu T (2016) Aspect level sentiment classification with deep memory network. arXiv preprint arXiv:1605.08900

  32. Gu S, Zhang L, Hou Y, Song Y (2018) A position-aware bidirectional attention network for aspect-level sentiment analysis. In: Proceedings of the 27th International Conference on Computational linguistics, pp 774–784

  33. Xu L, Li H, Lu W, Bing L (2020) Position-aware tagging for aspect sentiment triplet extraction. arXiv preprint arXiv:2010.02609

  34. Gu S, Zhang L, Hou Y, Song Y (2018) A position-aware bidirectional attention network for aspect-level sentiment analysis. In: Proceedings of the 27th International Conference on Computational Linguistics, pp 774–784

  35. Tang D, Qin B, Liu T (2016) Aspect level sentiment classification with deep memory network. arXiv preprint arXiv:1605.08900

  36. Jiang L, Yu M, Zhou M, Liu X, Zhao T (2011) Target-dependent twitter sentiment classification. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp 151–160

  37. Kiritchenko S, Zhu X, Cherry C, Mohammad S (2014) NRC-Canada-2014: detecting aspects and sentiment in customer reviews. In: Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp 437–442

  38. Dong L, Wei F, Tan C, Tang D, Zhou M, Xu K (2014) Adaptive recursive neural network for target-dependent twitter sentiment classification. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (volume 2: Short Papers), pp 49–54

  39. Hu M, Liu B (2004) Mining and summarizing customer reviews. In: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 168–177

  40. Ren F, Feng L, Xiao D, Cai M, Cheng S (2020) Dnet: a lightweight and efficient model for aspect based sentiment analysis. Expert Syst Appl 151:113393

    Article  Google Scholar 

  41. Wankhade M, Rao ACS (2022) Bi-directional lstm attention mechanism for sentiment classification. In: 2022 2nd Asian Conference on Innovation in Technology (ASIANCON), pp 1–6. IEEE

  42. Ma D, Li S, Zhang X, Wang H (2017) Interactive attention networks for aspect-level sentiment classification. arXiv preprint arXiv:1709.00893

  43. Huang B, Ou Y, Carley KM (2018) Aspect level sentiment classification with attention-over-attention neural networks. In: International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation, pp 197–206. Springer

  44. Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder–decoder for statistical machine translation. arXiv preprint arXiv:1406.1078

  45. Wankhade M, Rao ACS (2022) Opinion analysis and aspect understanding during COVID-19 pandemic using BERT-Bi-LSTM ensemble method. Sci Rep 12(1):17095

    Article  Google Scholar 

  46. Cai Y, Huang Q, Lin Z, Xu J, Chen Z, Li Q (2020) Recurrent neural network with pooling operation and attention mechanism for sentiment analysis: a multi-task learning approach. Knowl Based Syst 203:105856

    Article  Google Scholar 

  47. Wei J, Liao J, Yang Z, Wang S, Zhao Q (2020) Bilstm with multi-polarity orthogonal attention for implicit sentiment analysis. Neurocomputing 383:165–173

    Article  Google Scholar 

  48. Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

  49. Mnih V, Heess N, Graves A et al. (2014) Recurrent models of visual attention. In: Advances in Neural Information Processing Systems, pp 2204–2212

  50. Chen P, Sun Z, Bing L, Yang W (2017) Recurrent attention network on memory for aspect sentiment analysis. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp 452–461

  51. Wang X, Chen X, Tang M, Yang T, Wang Z (2020) Aspect-level sentiment analysis based on position features using multilevel interactive bidirectional GRU and attention mechanism. Discrete Dynamics in Nature and Society 2020

  52. Lou Y, Zhang Y, Li F, Qian T, Ji D (2020) Emoji-based sentiment analysis using attention networks. ACM Trans Asian Low Resour Lang Inf Process (TALLIP) 19(5):1–13

    Article  Google Scholar 

  53. Xi C, Lu G, Yan J (2020) Multimodal sentiment analysis based on multi-head attention mechanism. In: Proceedings of the 4th International Conference on Machine Learning and Soft Computing, pp 34–39

  54. Basiri ME, Nemati S, Abdar M, Cambria E, Acharya UR (2021) ABCDM: an attention-based bidirectional CNN–RNN deep model for sentiment analysis. Future Gener Comput Syst 115:279–294

    Article  Google Scholar 

  55. Cambria E. Li Y, Xing FZ, Poria S, Kwok K (2020) Senticnet 6: ensemble application of symbolic and subsymbolic ai for sentiment analysis. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp 105–114

  56. Ribani R, Marengoni M (2019) A survey of transfer learning for convolutional neural networks. In: 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images Tutorials (SIBGRAPI-T), pp 47–57. IEEE

  57. Xu H, Li S, Hu R, Li S, Gao S (2018) From random to supervised: a novel dropout mechanism integrated with global information. arXiv preprint arXiv:1808.08149

  58. Li X, Bing L, Lam W, Shi B (2018) Transformation networks for target-oriented sentiment classification. arXiv preprint arXiv:1805.01086

  59. Zhao A, Yu Y (2021) Knowledge-enabled BERT for aspect-based sentiment analysis. Knowl Based Syst 227:107220

    Article  Google Scholar 

  60. Tay Y, Tuan LA, Hui SC (2018) Learning to attend via word-aspect associative fusion for aspect-based sentiment analysis. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32

  61. Mehta S, Islam MR, Rangwala H, Ramakrishnan N (2019) Event detection using hierarchical multi-aspect attention. In: The World Wide Web Conference, pp 3079–3085

  62. Jiang Q, Chen L, Xu R, Ao X, Yang M (2019) A challenge dataset and effective models for aspect-based sentiment analysis. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp 6280–6285

  63. Pang B, Lee L, Vaithyanathan S (2002) Thumbs up? sentiment classification using machine learning techniques. arXiv preprint cs/0205070

  64. Kiritchenko S, Zhu X, Cherry C, Mohammad S (2014) Nrc-canada-2014: Detecting aspects and sentiment in customer reviews. In: Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp 437–442

  65. Tang D, Qin B, Liu T (2016) Aspect level sentiment classification with deep memory network. arXiv preprint arXiv:1605.08900

  66. Ma B, Yuan H, Wu Y (2017) Exploring performance of clustering methods on document sentiment analysis. J Inf Sci 43(1):54–74

    Article  Google Scholar 

  67. Fan F, Feng Y, Zhao D (2018) Multi-grained attention network for aspect-level sentiment classification. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 3433–3442

  68. Li X, Bing L, Lam W, Shi B (2018) Transformation networks for target-oriented sentiment classification. arXiv preprint arXiv:1805.01086

  69. Liu Q, Zhang H, Zeng Y, Huang Z, Wu Z (2018) Content attention model for aspect based sentiment analysis. In: Proceedings of the 2018 World Wide Web Conference, pp 1023–1032

  70. Ma X, Zeng J, Peng L, Fortino G, Zhang Y (2019) Modeling multi-aspects within one opinionated sentence simultaneously for aspect-level sentiment analysis. Future Gener Comput Syst 93:304–311

    Article  Google Scholar 

  71. Xu Q, Zhu L, Dai T, Yan C (2020) Aspect-based sentiment classification with multi-attention network. Neurocomputing 388:135–143

    Article  Google Scholar 

  72. Zhou J, Huang JX, Hu QV, He L (2020) SK-GCN: modeling syntax and knowledge via graph convolutional network for aspect-level sentiment classification. Knowl Based Syst 205:106292

    Article  Google Scholar 

  73. Wang K, Shen W, Yang Y, Quan X, Wang R (2020) Relational graph attention network for aspect-based sentiment analysis. arXiv preprint arXiv:2004.12362

  74. Zhang M, Qian T (2020) Convolution over hierarchical syntactic and lexical graphs for aspect level sentiment analysis. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 3540–3549

  75. Devlin J, Chang M-W, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  76. Rietzler A, Stabinger S. Opitz P, Engl S (2019) Adapt or get left behind: domain adaptation through BERT language model finetuning for aspect-target sentiment classification. arXiv preprint arXiv:1908.11860

  77. Song Y, Wang J, Jiang T, Liu Z, Rao Y (2019) Attentional encoder network for targeted sentiment classification. arXiv preprint arXiv:1902.09314

  78. Chen F, Yuan Z, Huang Y (2020) Multi-source data fusion for aspect-level sentiment classification. Knowl Based Syst 187:104831

    Article  Google Scholar 

  79. Yang H, Li K (2022) PyABSA: open framework for aspect-based sentiment analysis. arXiv preprint arXiv:2208.01368

  80. Veyseh APB, Nour N, Dernoncourt F, Tran QH, Dou D, Nguyen TH (2020) Improving aspect-based sentiment analysis with gated graph convolutional networks and syntax-based regulation. arXiv preprint arXiv:2010.13389

  81. Liang B, Su H, Gui L, Cambria E, Xu R (2022) Aspect-based sentiment analysis via affective knowledge enhanced graph convolutional networks. Knowl Based Syst 235:107643

    Article  Google Scholar 

Download references

Funding

Not applicable.

Author information

Authors and Affiliations

  1. Department of Computer Science, Indian Institute of Technology (Indian School of Mines), Dhanbad, 826004, Jharkhand, India

    Mayur Wankhade & Chandra Sekhara Rao Annavarapu

  2. Machine Intelligence Research Labs, Auburn, WA, 98071, USA

    Ajith Abraham

Authors
  1. Mayur Wankhade
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chandra Sekhara Rao Annavarapu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ajith Abraham
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

MW: Data curtion, conceptualization, validation, implementations, methodology, writing—original draft. AA: Methodology, conceptualization, editing, review, validation, supervision. ACSR: Review, validation, editing, methodology, supervision, conceptualization.

Corresponding author

Correspondence to Mayur Wankhade.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

Not applicable

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wankhade, M., Annavarapu, C.S.R. & Abraham, A. MAPA BiLSTM-BERT: multi-aspects position aware attention for aspect level sentiment analysis. J Supercomput 79, 11452–11477 (2023). https://doi.org/10.1007/s11227-023-05112-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-023-05112-7

Keywords

Search

Navigation

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy