Papers of the day   All papers

Unsupervised Cross-lingual Representation Learning at Scale


Alexis Conneau: Our new paper: Unsupervised Cross-lingual Representation Learning at Scale We release XLM-R, a Transformer MLM trained in 100 langs on 2.5 TB of text data. Double digit gains on XLU benchmarks + strong per-language performance (~XLNet on GLUE). [1/6]

5 replies, 428 likes

Yann LeCun: XLM-R: Amazing results on XLU and GLUE benchmarks from Facebook AI: large transformer network trained on 2.5TB of text from 100 languages.

1 replies, 204 likes

Alexis Conneau: Two papers accepted this year at #ACL2020 :) The first one on Unsupervised Cross-lingual Representation Learning at Scale (XLM-R) is a new SOTA on XLU benchmarks; and shows that multilinguality doesn't imply losing monolingual performance. (1/3)

1 replies, 155 likes

Guillaume Lample: XLM-R, the large scale version of XLM. Super impressive results. A single model trained on 2.5TB of data handles 100 languages, and outperforms mBERT by more than 10% on several classification benchmarks, with up to 21% accuracy on low-resource languages like Swahili and Urdu.

1 replies, 134 likes

Kartikay Khandelwal: Excited to share that my first first-author paper - “Unsupervised Cross-lingual Representation Learning at Scale” got accepted at @aclmeeting! #acl2020nlp Link: In this work, we present XLM-R - a SOTA multilingual model in 100 languages. #benderrule

2 replies, 83 likes

Thomas Wolf: Nice work by @alex_conneau @kakemeister and co. on pretraining multilingual language models to overcome the curse of multilinguality. Pretty impressive to see the resulting 100-languages model challenge strong English-only models like XLNet & RoBERTa 👇

1 replies, 64 likes

Kartikay Khandelwal: Really excited to share new work! XLM-R: A multilingual model in 100 languages, trained on 2TB of data! SOTA on cross-lingual benchmarks AND competitive with monolingual models on GLUE! We also explore how to effectively train these models! My first first author NLP paper! :)

1 replies, 64 likes

Ves Stoyanov: We released XLM-R (XLM-Roberta) it achieves new state of the art results on cross-lingual NLI, QA and NER. I am particularly excited about the huge improvement on low-resource languages.

0 replies, 63 likes

Roee Aharoni: Very happy to see more massively-multilingual work coming out. The world needs more non-English NLP!

0 replies, 17 likes

Kartikay Khandelwal: Really excited for #acl2020nlp! We’ll be talking all things XLM-R in our QA session, so stop by and say hi! July 8th: 10-11AM Pacific Time Session 14A 1-2PM Pacific Time Session 15A Talk: Paper: 1/3

1 replies, 14 likes

Myle Ott: Now available in fairseq:

0 replies, 5 likes

CeShine 😷: They found that applying a Sentence Piece model on raw text data for all languages is enough. No need for extra tokenization steps.

0 replies, 5 likes

roadrunner01: Unsupervised Cross-lingual Representation Learning at Scale pdf: abs:

0 replies, 4 likes

Stefan: XLM-RoBERTa is out 😍 Thanks to the fairseq-team 🤗 #nlp

0 replies, 3 likes

Dennis Aumiller: @anwagnerdreas @DanHLawReporter @seb_ruder @christof77 @stanfordnlp @TDataScience @LDKconference @huggingface According to the paper (Table 6,, it was trained on 350M latin tokens. Also the disclaimer that multi-lingual models tend to trade optimal performance in a single language for the multi-lingual capabilities, so re-training might be slightly better.

0 replies, 3 likes

Stefan: "Unsupervised Cross-lingual Representation Learning at Scale" is out now:

0 replies, 2 likes

AUEB NLP Group: Next AUEB NLP Group meeting, Tue 16 June, 17:00-18:30, "Cross-lingual Language Model Pretraining", Conneau & Lample (NeurIPS 2019, and "Unsupervised Cross-lingual Representation Learning at Scale", Conneau et al. (ACL 2020,

1 replies, 2 likes

Kartikay Khandelwal: @sigtyp_acl Sharing our work on Unsupervised Cross—lingual Representation learning at scale!

0 replies, 1 likes

Kartikay Khandelwal: You can find the paper here:

0 replies, 1 likes


Found on Nov 07 2019 at

PDF content of a computer science paper: Unsupervised Cross-lingual Representation Learning at Scale