Papers of the day   All papers

Distilling the Knowledge in a Neural Network

Comments

May 17 2019 Oriol Vinyals

The distillation paper was actually rejected from #NeurIPS2014 : ) As I'm sure many people are working towards #NeurIPS2019, optimize for long term impact, not probability of acceptance.
8 replies, 623 likes


Sep 25 2019 Jeff Dean

@hardmaru No opinion on favorite or not, but this paper @geoffreyhinton, @OriolVinyalsML, & I submitted to NeurIPS'14 was rejected (~2K citations): Distilling the Knowledge in a Neural Network https://arxiv.org/abs/1503.02531 2/3 said "1: This work is incremental and unlikely to have much impact"
8 replies, 516 likes


May 17 2019 ML Review

Distilling the Knowledge in a Neural Network #NIPS2014 By Geoffrey Hinton @OriolVinyalsML @jeffdean On transferring knowledge from an ensemble or from a large highly regularized model into a smaller, distilled model https://arxiv.org/abs/1503.02531
0 replies, 194 likes


Sep 26 2019 hardmaru

You can reject a paper but you can’t reject an idea 💡
3 replies, 191 likes


Oct 17 2019 Dariusz Kajtoch

Distillation can produce smaller, faster and cheaper models that have comparable generalization performance than their complex, cumbersome counterparts. #NLProc #DeepLearning #DataScience https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1910.01108 @huggingface https://t.co/ktlhIvgNJu
1 replies, 108 likes


Sep 20 2019 Oriol Vinyals

@colinraffel "Distilling the Knowledge in a Neural Network" by Hinton, Dean and myself : ( https://arxiv.org/abs/1503.02531
0 replies, 69 likes


Sep 25 2019 brianmcfee

There's some irony in a field largely based on gradient descent with small step sizes frowning upon incremental work
1 replies, 62 likes


Sep 25 2019 Pablo Samuel Castro

reminder that even the best of us have their papers rejected. keep at it if you enjoy it, despite what grumpy reviewers say.
0 replies, 24 likes


Content