Quoc Le: Pretty amazing progress on speech recognition thanks to pre-training and self-training with unlabeled data.
Key ingredients: Large conformer architecture + wave2vec2.0 pretraining + Noisy Student Training
Link: https://arxiv.org/abs/2010.10504 https://t.co/vBIlQWx2Us
4 replies, 539 likes
Loren Lugosch: New SOTA for LibriSpeech using Transducers (+ contrastive pre-training, self-training, and other tricks).
Think I might write that Transducer explainer after all.
2 replies, 48 likes
Leo Boytsov: It's a WWW (wild-wild-wild) world! Google and Facebook report nearly simultaneously on success in using self-supervision and self-training in speech recognition. Ouch.
0 replies, 16 likes
Leo Boytsov: Great improvements in ASR thanks to self-supervised training.
1 replies, 13 likes
Ruoming Pang: 1.4%/2.6% on LibriSpeech with Conformer + Noisy Student + Wav2Vec: https://arxiv.org/abs/2010.10504
0 replies, 9 likes
HoxoMaxwell! 🎃: Conformer 👇
Gulati, Convolution-augmented Transformer for Speech Recognition, 2020
0 replies, 6 likes
Jim Dowling: Amazing to @quocleix do ablation studies on a 1.1bn param network. In a talk with @sinash93 next month at the @Data_AI_Summit , we will explain how we parallelize Ablation studies using Maggy - https://databricks.com/session_eu20/parallel-ablation-studies-for-machine-learning-with-maggy-on-apache-spark https://t.co/O1QUyTGTmz
1 replies, 4 likes
Daisuke Okanohara: Noisy student training (generate labeled dataset using a trained model, and train a student w/ noise) is also effective in speech recognition tasks, using a giant Conformer (Self-attention+Conv.) model. Achieve new SOTA on the LibriSpeech data set. https://arxiv.org/abs/2010.10504
0 replies, 4 likes
Takuya Yoshioka: Stunning.
0 replies, 1 likes
0 replies, 1 likes
Found on Oct 21 2020 at https://arxiv.org/pdf/2010.10504.pdf