Title: Cross-lingual Language Model Pretraining
Authors: Guillaume Lample, Alexis Conneau
Published: 22nd January 2019 (Tuesday) @ 13:22:34
Link: http://arxiv.org/abs/1901.07291v1
Abstract
Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we obtain 34.3 BLEU on WMTâ16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMTâ16 Romanian-English, outperforming the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.
Release: Cross-lingual pretraining sets new state of the art
Follows work:
- UMT: Unsupervised machine translation A novel approach to provide fast, accurate translations for more languages
- LASER: Zero-shot transfer across 93 languages Open-sourcing enhanced LASER library
- XNLI: Facebook, NYU expand available languages for natural language understanding systems