Papers of the day   All papers

Distilling the Knowledge in a Neural Network


May 17 2019 Oriol Vinyals

The distillation paper was actually rejected from #NeurIPS2014 : ) As I'm sure many people are working towards #NeurIPS2019, optimize for long term impact, not probability of acceptance.
8 replies, 625 likes

Sep 25 2019 Jeff Dean

@hardmaru No opinion on favorite or not, but this paper @geoffreyhinton, @OriolVinyalsML, & I submitted to NeurIPS'14 was rejected (~2K citations): Distilling the Knowledge in a Neural Network 2/3 said "1: This work is incremental and unlikely to have much impact"
8 replies, 520 likes

May 17 2019 ML Review

Distilling the Knowledge in a Neural Network #NIPS2014 By Geoffrey Hinton @OriolVinyalsML @jeffdean On transferring knowledge from an ensemble or from a large highly regularized model into a smaller, distilled model
0 replies, 195 likes

Sep 26 2019 hardmaru

You can reject a paper but you can’t reject an idea 💡
3 replies, 191 likes

Oct 17 2019 Dariusz Kajtoch

Distillation can produce smaller, faster and cheaper models that have comparable generalization performance than their complex, cumbersome counterparts. #NLProc #DeepLearning #DataScience @huggingface
1 replies, 108 likes

Sep 20 2019 Oriol Vinyals

@colinraffel "Distilling the Knowledge in a Neural Network" by Hinton, Dean and myself : (
0 replies, 69 likes

Sep 25 2019 brianmcfee

There's some irony in a field largely based on gradient descent with small step sizes frowning upon incremental work
1 replies, 62 likes

Sep 25 2019 Pablo Samuel Castro

reminder that even the best of us have their papers rejected. keep at it if you enjoy it, despite what grumpy reviewers say.
0 replies, 24 likes