Papers of the day   All papers

ON EMPIRICAL COMPARISONS OF OPTIMIZERS FOR DEEP LEARNING

Comments

Oct 15 2019 Roger Grosse

In deep learning research, the sky turns out to be blue, but only if you measure it very carefully. Interesting meta-scientific paper on evaluating neural net optimizers, by Choi et al. https://arxiv.org/pdf/1910.05446.pdf
2 replies, 203 likes


Oct 16 2019 Sebastian Raschka

"On Empirical Comparisons of Optimizers for Deep Learning" => "As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate" https://arxiv.org/abs/1910.05446 https://t.co/hUIGMyshkC
2 replies, 183 likes


Oct 15 2019 Dmytro Mishkin

Tl;dr: Adam >> SGD, if you tune eps, momentum and lr schedule for it
0 replies, 14 likes


Oct 16 2019 arxiv

On Empirical Comparisons of Optimizers for Deep Learning. http://arxiv.org/abs/1910.05446 https://t.co/QVxeQCpAOL
0 replies, 10 likes


Oct 15 2019 Daisuke Okanohara

In NN optimization, ignored metaparameters are actually important. Especially, "eps" is often used with default 1e-8, but the optimal value can be 1~10^4. With full metaparameter search, ADAM and NADAM outperform SGD and momentum. https://arxiv.org/abs/1910.05446
0 replies, 9 likes


Content