Abstract
Multilingual neural machine translation (MNMT) with a single encoder-decoder model has attracted much interest due to its simple deployment and low training cost. However, the all-shared translation model often yields degraded performance due to the modeling capacity limitations and language diversity. Moreover, it has been revealed in recent studies that the shared parameters lead to negative language interference although they may also facilitate knowledge transfer across languages. In this work, we propose an adaptive architecture for multilingual modeling, which divides the parameters in MNMT sub-layers into shared and language-specific ones. We train the model to learn and balance the shared and unique features with different degrees of parameter sharing. We evaluate our model on one-to-many and many-to-one translation tasks. Experiments on IWSLT dataset show that our proposed model remarkably outperforms the multilingual baseline model and achieves comparable or even better performance compared with the bilingual model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
Signature: BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.4.14.
References
Aharoni, R., Johnson, M., Firat, O.: Massively multilingual neural machine translation. In: ACL 2019 (2019)
Arivazhagan, N., et al.: Massively multilingual neural machine translation in the wild: findings and challenges. arXiv preprint arXiv:1907.05019 (2019)
Bapna, A., Firat, O.: Simple, scalable adaptation for neural machine translation. In: EMNLP-IJCNLP 2019 (2019)
Blackwood, G., Ballesteros, M., Ward, T.: Multilingual neural machine translation with task-specific attention. In: COLING 2018 (2018)
Conneau, A., et al.: Unsupervised cross-lingual representation learning at scale. In: ACL 2020 (2020)
Dabre, R., Chu, C., Kunchukuttan, A.: A survey of multilingual neural machine translation. ACM Comput. Surv. (CSUR) 53(5), 1–38 (2020)
Dong, D., Wu, H., He, W., Yu, D., Wang, H.: Multi-task learning for multiple language translation. In: ACL-IJCNLP 2015 (2015)
Firat, O., Cho, K., Bengio, Y.: Multi-way, multilingual neural machine translation with a shared attention mechanism. In: NAACL 2016 (2016)
Firat, O., Sankaran, B., Al-onaizan, Y., Yarman Vural, F.T., Cho, K.: Zero-resource translation with multi-lingual neural machine translation. In: EMNLP 2016 (2016)
Ha, T.L., Niehues, J., Waibel, A.: Toward multilingual neural machine translation with universal encoder and decoder. In: Proceedings of IWSLT 2016 (2016)
Johnson, M., et al.: Google’s multilingual neural machine translation system: enabling zero-shot translation. Trans. Assoc. Comput. Linguist. 5, 339–351 (2017)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Lu, Y., Keung, P., Ladhak, F., Bhardwaj, V., Zhang, S., Sun, J.: A neural interlingua for multilingual machine translation. In: Proceedings of the Third Conference on Machine Translation: Research Papers (2018). https://doi.org/10.18653/v1/W18-6309
Michel, P., Levy, O., Neubig, G.: Are sixteen heads really better than one? arXiv preprint arXiv:1905.10650 (2019)
Post, M.: A call for clarity in reporting BLEU scores. In: Proceedings of the Third Conference on Machine Translation: Research Papers (2018). https://doi.org/10.18653/v1/W18-6319
Sachan, D., Neubig, G.: Parameter sharing methods for multilingual self-attentional translation models. In: Proceedings of the Third Conference on Machine Translation: Research Papers (2018). https://doi.org/10.18653/v1/W18-6327
Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: ACL 2016 (2016)
Tan, X., Chen, J., He, D., Xia, Y., Qin, T., Liu, T.Y.: Multilingual neural machine translation with language clustering. In: EMNLP-IJCNLP 2019 (2019)
Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6000–6010 (2017)
Vázquez, R., Raganato, A., Tiedemann, J., Creutz, M.: Multilingual NMT with a language-independent attention bridge. In: Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019) (2019)
Wang, Y., Zhang, J., Zhai, F., Xu, J., Zong, C.: Three strategies to improve one-to-many multilingual translation. In: EMNLP 2018 (2018)
Wang, Y., Zhou, L., Zhang, J., Zhai, F., Xu, J., Zong, C.: A compact and language-sensitive multilingual translation method. In: ACL 2019 (2019)
Wang, Z., Lipton, Z.C., Tsvetkov, Y.: On negative interference in multilingual models: findings and a meta-learning treatment. In: EMNLP 2020 (2020)
Zhang, B., Bapna, A., Sennrich, R., Firat, O.: Share or not? Learning to schedule language-specific capacity for multilingual translation. In: ICLR 2021 (2021)
Zhang, B., Williams, P., Titov, I., Sennrich, R.: Improving massively multilingual neural machine translation and zero-shot translation. In: ACL 2020 (2020)
Zoph, B., Knight, K.: Multi-source neural translation. In: NAACL 2016 (2016)
Zoph, B., Yuret, D., May, J., Knight, K.: Transfer learning for low-resource neural machine translation. In: EMNLP 2016 (2016)
Acknowledgments
The research work has been supported by the National Key Research and Development Program of China No. 2020AAA0108004 and the Natural Science Foundation of China under Grant No. U1936109.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, J., Huang, K., Li, J., Liu, H., Huang, D. (2021). Adaptive Transformer for Multilingual Neural Machine Translation. In: Wang, L., Feng, Y., Hong, Y., He, R. (eds) Natural Language Processing and Chinese Computing. NLPCC 2021. Lecture Notes in Computer Science(), vol 13028. Springer, Cham. https://doi.org/10.1007/978-3-030-88480-2_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-88480-2_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88479-6
Online ISBN: 978-3-030-88480-2
eBook Packages: Computer ScienceComputer Science (R0)
