Abstract
Domain adaptation for named entity recognition (NER) has achieved remarkable results in recent years. However, most previous domain adaptation methods mainly focus on a single source domain without considering all available annotation data. When domain shift between the target and the single source domain is too large, performance would drastically decline. In addition, samples from different domains may have diverse degrees of relevance to the target domain, which requires further consideration. To tackle these problems, we propose a multi-domain adaptation model with multi-aspect relevance learning for NER. Specifically, a multi-aspect relevance learning method is proposed to promote the effect of multi-domain adaptation in NER. Meanwhile, a novel strategy called BERT-based cross-domain pre-training is applied to reduce the dependence on large-scale domain related data. We conduct experiments on five target domains, and the experimental results have achieved a new state-of-the-art performance in terms of all metrics, which proves the effectiveness of our model.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aharoni, R., & Goldberg, Y. (2020). Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 7747–7763).
Alsentzer, E., Murphy, J., Boag, W., Weng, W.H., Jindi, D., Naumann, T., & McDermott, M. (2019). Publicly available clinical BERT embeddings. In Proceedings of the 2nd clinical natural language processing workshop (pp. 72–78).
Baevski, A., Edunov, S., Liu, Y., Zettlemoyer, L., & Auli, M. (2019). Cloze-driven pretraining of self-attention networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) (pp. 5363–5372).
Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. Preprint retrieved from https://arxiv.org/abs/1409.0473
Chen, X., & Cardie, C. (2018). Multinomial adversarial networks for multi-domain text classification. In Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: Human language technologies, volume 1 (long papers) (pp. 1226–1240).
Daumé III, H. (2007). Frustratingly easy domain adaptation. In Proceedings of the 45th annual meeting of the association of computational linguistics (pp. 256–263).
Devlin, J., Chang, M.W., Lee, K., & Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) (pp. 4171–4186).
Du, C., Sun, H., Wang, J., Qi, Q., & Liao, J. (2020). Adversarial and domain-aware BERT for cross-domain sentiment analysis. In Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 4019–4028).
Ganin, Y., & Lempitsky, V. (2015). Unsupervised domain adaptation by backpropagation. In Proceedings of the 32nd international conference on international conference on machine learning (Vol. 37, pp. 1180–1189).
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020). Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 8342–8360).
Han, X., & Eisenstein, J. (2019). Unsupervised domain adaptation of contextualized embeddings for sequence labeling. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) (pp. 4229–4239).
Hirschman, L., & Gaizauskas, R. (2001). Natural language question answering: The view from here. Natural Language Engineering, 7(4), 275.
Howard, J., & Ruder, S. (2018). Universal language model fine-tuning for text classification. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers) (pp. 328–339).
Jia, C., Liang, X., & Zhang, Y. (2019). Cross-domain NER using cross-domain language modeling. In Proceedings of the 57th annual meeting of the association for computational linguistics (pp. 2464–2474).
Jia, C., & Zhang, Y. (2020). Multi-cell compositional LSTM for NER domain adaptation. In Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 5906–5917).
Lee, J., Yoon, W., Kim, S., Kim, D., So, C., & Kang, J. (2020). Biobert: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics (Oxford, England), 36(4), 1234–1240.
Lee, J. Y., Dernoncourt, F., & Szolovits, P. (2018). Transfer learning for named-entity recognition with neural networks. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018).
Lewis, D. D., Yang, Y., Rose, T. G., & Li, F. (2004). Rcv1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5, 361–397.
Li, S., & Zong, C. (2008). Multi-domain sentiment classification. In Proceedings of ACL-08: HLT, short papers (pp. 257–260).
Liu, Z., Xu, Y., Yu, T., Dai, W., Ji, Z., Cahyawijaya, S., Madotto, A., & Fung, P. (2020). Crossner: Evaluating cross-domain named entity recognition. Preprint retrieved from https://arxiv.org/abs/2012.04373
Logeswaran, L., Chang, M. W., Lee, K., Toutanova, K., Devlin, J., & Lee, H. (2019). Zero-shot entity linking by reading entity descriptions. In Proceedings of the 57th annual meeting of the association for computational linguistics (pp. 3449–3460).
Luo, Y., Xiao, F., & Zhao, H. (2020). Hierarchical contextualized representation for named entity recognition. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, pp. 8441–8448).
Mou, L., Meng, Z., Yan, R., Li, G., Xu, Y., Zhang, L., & Jin, Z. (2016). How transferable are neural networks in NLP applications? In Proceedings of the 2016 conference on empirical methods in natural language processing (pp. 479–489).
Phang, J., Févry, T., & Bowman, S.R. (2018). Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. Preprint retrieved from https://arxiv.org/abs/1811.01088
Sang, E. T. K. (2002). Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In COLING-02: The 6th conference on natural language learning 2002 (CoNLL-2002).
Sarawagi, S. (2008). Information extraction. Now Publishers Inc.
Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019). How to fine-tune BERT for text classification? In China national conference on Chinese computational linguistics (pp. 194–206). Springer.
Swayamdipta, S., Peters, M., Roof, B., Dyer, C., & Smith, N. A. (2019). Shallow syntax in deep water. Preprint retrieved from https://arxiv.org/abs/1908.11047
Wang, J., Kulkarni, M., & Preoţiuc-Pietro, D. (2020). Multi-domain named entity recognition with genre-aware and agnostic inference. In Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 8476–8488).
Wu, F., & Huang, Y. (2015) . Collaborative multi-domain sentiment classification. In 2015 IEEE international conference on data mining (pp. 459–468). IEEE.
Yadav, V., & Bethard, S. (2018). A survey on recent advances in named entity recognition from deep learning models. In Proceedings of the 27th international conference on computational linguistics (pp. 2145–2158).
Yang, Z., Salakhutdinov, R., & Cohen, W. W. (2017). Transfer learning for sequence tagging with hierarchical recurrent networks. Preprint retrieved from https://arxiv.org/abs/1703.06345
Yu, J., Bohnet, B., & Poesio, M. (2020). Named entity recognition as dependency parsing. Preprint retrieved from https://arxiv.org/abs/2005.07150
Zhang, Y., & Yang, J. (2018). Chinese NER using lattice LSTM. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers) (pp. 1554–1564).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Li, J., Liu, J., Chen, Y. et al. Multi-domain adaptation for named entity recognition with multi-aspect relevance learning. Lang Resources & Evaluation 57, 803–818 (2023). https://doi.org/10.1007/s10579-022-09590-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10579-022-09590-8