Abstract
Text style transfer (TST) is an NLP task with a long history and a broad range of applications. Recently, it has seen success with the use of large pretrained language models (LMs). However, the size of contemporary LMs often makes fine-tuning for downstream tasks infeasible. For this reason, methods of controllable text generation (CTG) which do not aim at fine-tuning the original LM have received attention for solving TST tasks. In this work, we contribute to this line of research and adapt an existing CTG method, CAIF, for TST. The original CAIF is based on reweighting the logits of the generative LM according to a free-form style attribute classifier. To allow its use for TST, we replace the standard LM with a model capable of paraphrasing, making corresponding changes. We refer to the resulting unsupervised method as ParaCAIF. We illustrate its applicability by experimenting with detoxification, a relatively new yet practical TST subtask. We work with detoxification in two languages: Russian and English. For both languages, ParaCAIF significantly reduces the toxicity of the generated paraphrases as compared to plain paraphrasers. To the best of our knowledge, it is the first work that adapts a CTG method for Russian detoxification. For English, ParaCAIF outperforms an analogous adapted CTG method, ParaGeDi, in terms of style transfer accuracy. Although the overall performance of ParaCAIF remains lower than that of supervised approaches, it has a broader range of application, as it does not require parallel data which are not readily available for many TST tasks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
References
Belchikov, A.: Russian language toxic comments (2019). https://www.kaggle.com/datasets/blackmoon/russian-language-toxic-comments
Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the dangers of stochastic parrots: can language models be too big? In: FAccT 2021 (2021). https://doi.org/10.1145/3442188.3445922
Dale, D.:
(2021). https://habr.com/ru/post/564916/Dale, D.:
(2021). https://habr.com/ru/post/562064/Dale, D., et al.: Text Detoxification Using Large Pre-trained Neural Models (2021). https://aclanthology.org/2021.emnlp-main.629
Dathathri, S., et al.: Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2020). https://openreview.net/forum?id=H1edEyBKDS
Dementieva, D., et al.: Methods for Detoxification of Texts for the Russian Language (2021). https://arxiv.org/abs/2105.09052
Dementieva, D., et al.: RUSSE-2022: findings of the first Russian detoxification task based on parallel corpora (2022)
Deng, Y., et al.: Residual Energy-Based Models for Text Generation (2020). https://openreview.net/forum?id=B1l4SgHKDH
Feng, F., Yang, Y., Cer, D., Arivazhagan, N., Wang, W.: Language-agnostic BERT sentence embedding (2020)
Fenogenova, A.: Russian Paraphrasers: Paraphrase with Transformers (2021). https://aclanthology.org/2021.bsnlp-1.2
Hallinan, S., Liu, A., Choi, Y., Sap, M.: Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts (2022). https://arxiv.org/abs/2212.10543
Jigsaw: Toxic Comment Classification Challenge (2018). https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge
Jigsaw: Jigsaw Unintended Bias in Toxicity Classification (2019). https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification
Jigsaw: Jigsaw Multilingual Toxic Comment Classification (2020). https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification
Jin, D., Jin, Z., Hu, Z., Vechtomova, O., Mihalcea, R.: Deep Learning for Text Style Transfer: A Survey (1) (2022). https://aclanthology.org/2022.cl-1.6
John, V., Mou, L., Bahuleyan, H., Vechtomova, O.: Disentangled Representation Learning for Non-Parallel Text Style Transfer (2019). https://www.aclweb.org/anthology/P19-1041.pdf
Konodyuk, N., Tikhonova, M.: Continuous prompt tuning for Russian: how to learn prompts efficiently with RuGPT3? In: Burnaev, E., et al. (eds.) AIST 2021. CCIS, vol. 1573, pp. 30–40. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-15168-2_3
Krause, B., et al.: GeDi: Generative Discriminator Guided Sequence Generation (2021). https://aclanthology.org/2021.findings-emnlp.424
Krishna, K., Wieting, J., Iyyer, M.: Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020). https://aclanthology.org/2020.emnlp-main.55
Kuratov, Y., Arkhipov, M.: Adaptation of deep bidirectional multilingual transformers for Russian language (2019)
Laugier, L., Pavlopoulos, J., Sorensen, J., Dixon, L.: Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021). https://doi.org/10.18653/v1/2021.eacl-main.124
Liu, A., et al.: DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts (2021). https://arxiv.org/abs/2105.03023
Liu, Y., et al.: RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019). https://arxiv.org/abs/1907.11692
Mueller, J., Gifford, D.K., Jaakkola, T.S.: Sequence to better sequence: continuous revision of combinatorial structures. In: Proceedings of Machine Learning Research (2017). https://proceedings.mlr.press/v70/mueller17a.html
Prabhumoye, S., Black, A.W., Salakhutdinov, R.: Exploring Controllable Text Generation Techniques (2020). https://aclanthology.org/2020.coling-main.1
Raffel, C., et al.: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019). https://arxiv.org/abs/1910.10683
Reimers, N., Gurevych, I.: Making monolingual sentence embeddings multilingual using knowledge distillation (2020)
Rubtsova, Y.: Avtomaticheskoye postroyeniye i analiz korpusa korotkikh tekstov (postov mikroblogov) dlya zadachi razrabotki i trenirovki tonovogo klassifikatora. inzheneriya znaniy i tekhnologii semanticheskogo veba (2012)
Nogueira dos Santos, C., Melnyk, I., Padhi, I.: Fighting Offensive Language on Social Media with Unsupervised Text Style Transfer (2018). https://aclanthology.org/P18-2031
Semiletov, A.: Toxic Russian comments (2020). https://www.kaggle.com/datasets/alexandersemiletov/toxic-russian-comments
Shen, T., Lei, T., Barzilay, R., Jaakkola, T.: Style transfer from non-parallel text by cross-alignment (2017)
Sitdikov, A., Balagansky, N., Gavrilov, D., Markov, A.: Classifiers are Better Experts for Controllable Text Generation (2022). https://arxiv.org/abs/2205.07276
Thakur, N., Reimers, N., Daxenberger, J., Gurevych, I.: Augmented SBERT: data augmentation method for improving bi-encoders for pairwise sentence scoring tasks (2020)
Warstadt, A., Singh, A., Bowman, S.R.: Neural Network Acceptability Judgments (2019). https://aclanthology.org/Q19-1040
Weng, L.: Controllable Neural Text Generation (2021). https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/
Wieting, J., Berg-Kirkpatrick, T., Gimpel, K., Neubig, G.: Beyond BLEU: Training Neural Machine Translation with Semantic Similarity (2019). https://aclanthology.org/P19-1427
Wieting, J., Gimpel, K.: ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2018). https://aclanthology.org/P18-1042
Wu, X., Zhang, T., Zang, L., Han, J., Hu, S.: “Mask and Infill”: Applying Masked Language Model to Sentiment Transfer (2019). https://arxiv.org/pdf/1908.08039
Xue, L., et al.: mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2021). https://aclanthology.org/2021.naacl-main.41
Zhang, H., Song, H., Li, S., Zhou, M., Song, D.: A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models (2022). https://arxiv.org/abs/2201.05337
Zhao, J.J., Kim, Y., Zhang, K., Rush, A.M., LeCun, Y.: Adversarially regularized autoencoders. In: Proceedings of Machine Learning Research (2018). https://proceedings.mlr.press/v80/zhao18b.html
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
A ParaCAIF vs. Reranking
Table 4 displays the comparison results of ParaCAIF model and plain paraphraser with added reranking procedure..
B ParaCAIF Russian Detoxification Examples
Tables 5 and 6 demonstrate examples of sentences from RUSSE Detox detoxified by ParaCAIF ruT5 with different \(\alpha \) values. We observe that \(\alpha = -5\) tends to delete all strong obscene words, while a less strict \(\alpha \) of \(-1\) allows some direct toxicity (e.g., блин ‘damn it’).
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Orlov, E., Apishev, M. (2024). Paraphrasers and Classifiers: Controllable Text Generation for Text Style Transfer. In: Ignatov, D.I., et al. Analysis of Images, Social Networks and Texts. AIST 2023. Lecture Notes in Computer Science, vol 14486. Springer, Cham. https://doi.org/10.1007/978-3-031-54534-4_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-54534-4_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-54533-7
Online ISBN: 978-3-031-54534-4
eBook Packages: Computer ScienceComputer Science (R0)