Papers of the day   All papers

ALBERT: A LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS

Comments

Hugging Face: So, we read ALBERT. @remilouf took some notes for you 👇 Paper: https://arxiv.org/abs/1909.11942 Also in 🤗 transformers: https://github.com/huggingface/transformers https://t.co/LLaNema3mc

6 replies, 429 likes


Kirk Borne: Google Open-Sources ALBERT Natural Language Model: https://www.infoq.com/news/2020/01/google-albert-ai-nlp/ —————— #NLProc #NLU #NLG #AI #MachineLearning #DeepLearning #TensorFlow #Algorithms #BigData #DataScience —————— Research paper: https://arxiv.org/abs/1909.11942 https://t.co/9F7SzKEOOY

1 replies, 33 likes


Miles Brundage: P.S. See also the ALBERT paper, which shows stronger results on these metrics for a similarly sized model, using a different approach (using parameters better to begin with vs. compressing a big model later): https://arxiv.org/abs/1909.11942

0 replies, 32 likes


arXiv CS-CL: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations http://arxiv.org/abs/1909.11942

0 replies, 11 likes


arXiv CS-CL: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations http://arxiv.org/abs/1909.11942

0 replies, 10 likes


Leonid Boytsov: [thread] Another very interesting efficiency paper is the one that introduces ALBERT.https://arxiv.org/abs/1909.11942 1. Factorized embedding matrix. Very clever idea that permits untying embedding and hidden layer sizes. 2. Cross-layer parameter sharing (e.g., attention parameters)

1 replies, 9 likes


BioDecoded: ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations | Google AI Blog https://ai.googleblog.com/2019/12/albert-lite-bert-for-self-supervised.html https://arxiv.org/abs/1909.11942 #NLP #DeepLearning https://t.co/hWC0o3E4Ki

0 replies, 8 likes


Santosh ML: This is one of the best ML summaries I have ever seen

0 replies, 7 likes


Husein Zolkepli: Hi everyone! Malaya released ALBERT-Base for Bahasa Malaysia / Manglish / Rojak / Bahasa Indonesia. Original paper for ALBERT, https://arxiv.org/abs/1909.11942 Can read more about ALBERT-Bahasa from here and how to start use ALBERT-Bahasa, https://github.com/huseinzol05/Malaya/tree/3.0/pretrained-model/albert

0 replies, 7 likes


Carlo Lepelaars: Finally got around to reading up on some recent NLP papers. Currently reading: ALBERT: https://arxiv.org/pdf/1909.11942.pdf RoBERTa: https://arxiv.org/pdf/1907.11692.pdf XLNet: https://arxiv.org/pdf/1906.08237.pdf BERTje (Dutch BERT model): https://arxiv.org/pdf/1912.09582.pdf Do you have any other NLP paper recommendations?

1 replies, 7 likes


arXiv CS-CL: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations http://arxiv.org/abs/1909.11942

0 replies, 5 likes


AUEB NLP Group: Next AUEB NLP Group meeting, Tue Oct 8, 17:15-19:00, *IPLab* (http://nlp.cs.aueb.gr/contact.html): Discussion of RoBERTa (https://arxiv.org/abs/1907.11692) and ALBERT (https://arxiv.org/abs/1909.11942). Coordinator: Ilias Chalkidis @KiddoThe2B. Study the papers before the meeting. All welcome.

0 replies, 4 likes


Colin Raffel: @iskander @kchonyc @sleepinyourhat @yoavgo @SergeyFeldman @jeremyphoward Re: parameter sharing: https://arxiv.org/abs/1807.03819 and https://arxiv.org/abs/1909.11942 Re: cheaper attention, use domain-specific sparsity if you can, e.g. https://arxiv.org/abs/2004.05150 Re: tasks, use teacher-forced max likelihood if you can; T2T has some nice toy problems https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/data_generators/algorithmic.py

1 replies, 4 likes


arXiv CS-CL: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations http://arxiv.org/abs/1909.11942

0 replies, 3 likes


arXiv CS-CL: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations http://arxiv.org/abs/1909.11942

0 replies, 2 likes


AUEB NLP Group: Next AUEB NLP Group meeting, Tue Oct 15, 17:15-19:00, *IPLab* (http://nlp.cs.aueb.gr/contact.html): Part II of discussion of RoBERTa (https://arxiv.org/abs/1907.11692) and ALBERT (https://arxiv.org/abs/1909.11942). Coordinator: Ilias Chalkidis. Study the papers before the meeting. All welcome.

0 replies, 2 likes


Christopher Ackerman: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations https://arxiv.org/pdf/1909.11942.pdf Google open-sourced A Lite Bert (ALBERT), a deep-learning natural language processing (NLP) model https://www.infoq.com/news/2020/01/google-albert-ai-nlp/ https://devopedia.org/bert-language-model https://t.co/vOsRGbSB9t

0 replies, 1 likes


arXiv CS-CL: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations http://arxiv.org/abs/1909.11942

0 replies, 1 likes


Guenter Bartsch: Looks like both #PyTorch as well as #TensorFlow implementations of ALBERT https://arxiv.org/pdf/1909.11942.pdf have been open sourced: https://github.com/lonePatient/albert_pytorch https://github.com/brightmart/albert_zh

0 replies, 1 likes


arXiv CS-CL: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations http://arxiv.org/abs/1909.11942

0 replies, 1 likes


Content

Found on Dec 11 2019 at https://arxiv.org/pdf/1909.11942.pdf

PDF content of a computer science paper: ALBERT: A LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS