Papers of the day   All papers

Unsupervised Cross-lingual Representation Learning at Scale

Comments

Alexis Conneau: Our new paper: Unsupervised Cross-lingual Representation Learning at Scale https://arxiv.org/pdf/1911.02116.pdf We release XLM-R, a Transformer MLM trained in 100 langs on 2.5 TB of text data. Double digit gains on XLU benchmarks + strong per-language performance (~XLNet on GLUE). [1/6] https://t.co/0RX1ljGuri

5 replies, 428 likes


Yann LeCun: XLM-R: Amazing results on XLU and GLUE benchmarks from Facebook AI: large transformer network trained on 2.5TB of text from 100 languages.

1 replies, 204 likes


Alexis Conneau: Two papers accepted this year at #ACL2020 :) The first one on Unsupervised Cross-lingual Representation Learning at Scale http://arxiv.org/abs/1911.02116 (XLM-R) is a new SOTA on XLU benchmarks; and shows that multilinguality doesn't imply losing monolingual performance. (1/3) https://t.co/8DotzbbkDm

1 replies, 155 likes


Guillaume Lample: XLM-R, the large scale version of XLM. Super impressive results. A single model trained on 2.5TB of data handles 100 languages, and outperforms mBERT by more than 10% on several classification benchmarks, with up to 21% accuracy on low-resource languages like Swahili and Urdu.

1 replies, 134 likes


Kartikay Khandelwal: Excited to share that my first first-author paper - “Unsupervised Cross-lingual Representation Learning at Scale” got accepted at @aclmeeting! #acl2020nlp Link: https://arxiv.org/pdf/1911.02116.pdf In this work, we present XLM-R - a SOTA multilingual model in 100 languages. #benderrule

2 replies, 83 likes


Thomas Wolf: Nice work by @alex_conneau @kakemeister and co. on pretraining multilingual language models to overcome the curse of multilinguality. Pretty impressive to see the resulting 100-languages model challenge strong English-only models like XLNet & RoBERTa 👇 https://twitter.com/alex_conneau/status/1192490719031656448 https://t.co/VPJ5QIbPUK

1 replies, 64 likes


Kartikay Khandelwal: Really excited to share new work! XLM-R: A multilingual model in 100 languages, trained on 2TB of data! SOTA on cross-lingual benchmarks AND competitive with monolingual models on GLUE! We also explore how to effectively train these models! My first first author NLP paper! :)

1 replies, 64 likes


Ves Stoyanov: We released XLM-R (XLM-Roberta) it achieves new state of the art results on cross-lingual NLI, QA and NER. I am particularly excited about the huge improvement on low-resource languages.

0 replies, 63 likes


Roee Aharoni: Very happy to see more massively-multilingual work coming out. The world needs more non-English NLP!

0 replies, 17 likes


Kartikay Khandelwal: Really excited for #acl2020nlp! We’ll be talking all things XLM-R in our QA session, so stop by and say hi! July 8th: 10-11AM Pacific Time Session 14A 1-2PM Pacific Time Session 15A Talk: https://virtual.acl2020.org/paper_main.747.html Paper: https://arxiv.org/pdf/1911.02116.pdf 1/3

1 replies, 14 likes


Myle Ott: Now available in fairseq: https://github.com/pytorch/fairseq/tree/master/examples/xlmr

0 replies, 5 likes


CeShine 😷: They found that applying a Sentence Piece model on raw text data for all languages is enough. No need for extra tokenization steps.

0 replies, 5 likes


roadrunner01: Unsupervised Cross-lingual Representation Learning at Scale pdf: https://arxiv.org/pdf/1911.02116.pdf abs: https://arxiv.org/abs/1911.02116 https://t.co/l5XiXJrBsZ

0 replies, 4 likes


Stefan: XLM-RoBERTa is out 😍 Thanks to the fairseq-team 🤗 #nlp https://github.com/pytorch/fairseq/tree/master/examples/xlmr

0 replies, 3 likes


Dennis Aumiller: @anwagnerdreas @DanHLawReporter @seb_ruder @christof77 @stanfordnlp @TDataScience @LDKconference @huggingface According to the paper (Table 6, https://arxiv.org/pdf/1911.02116.pdf), it was trained on 350M latin tokens. Also the disclaimer that multi-lingual models tend to trade optimal performance in a single language for the multi-lingual capabilities, so re-training might be slightly better.

0 replies, 3 likes


Stefan: "Unsupervised Cross-lingual Representation Learning at Scale" is out now: https://arxiv.org/abs/1911.02116

0 replies, 2 likes


AUEB NLP Group: Next AUEB NLP Group meeting, Tue 16 June, 17:00-18:30, "Cross-lingual Language Model Pretraining", Conneau & Lample (NeurIPS 2019, https://papers.nips.cc/paper/8928-cross-lingual-language-model-pretraining.pdf) and "Unsupervised Cross-lingual Representation Learning at Scale", Conneau et al. (ACL 2020, https://arxiv.org/abs/1911.02116).

1 replies, 2 likes


Kartikay Khandelwal: @sigtyp_acl Sharing our work on Unsupervised Cross—lingual Representation learning at scale! https://twitter.com/kakemeister/status/1246525877149667335?s=21

0 replies, 1 likes


Kartikay Khandelwal: You can find the paper here: https://arxiv.org/pdf/1911.02116.pdf

0 replies, 1 likes


Content

Found on Nov 07 2019 at https://arxiv.org/pdf/1911.02116.pdf

PDF content of a computer science paper: Unsupervised Cross-lingual Representation Learning at Scale