Papers of the day   All papers



Oct 15 2019 Roger Grosse

In deep learning research, the sky turns out to be blue, but only if you measure it very carefully. Interesting meta-scientific paper on evaluating neural net optimizers, by Choi et al.
2 replies, 203 likes

Oct 16 2019 Sebastian Raschka

"On Empirical Comparisons of Optimizers for Deep Learning" => "As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate"
2 replies, 183 likes

Oct 15 2019 Dmytro Mishkin

Tl;dr: Adam >> SGD, if you tune eps, momentum and lr schedule for it
0 replies, 14 likes

Oct 15 2019 Daisuke Okanohara

In NN optimization, ignored metaparameters are actually important. Especially, "eps" is often used with default 1e-8, but the optimal value can be 1~10^4. With full metaparameter search, ADAM and NADAM outperform SGD and momentum.
0 replies, 13 likes

Oct 16 2019 arxiv

On Empirical Comparisons of Optimizers for Deep Learning.
0 replies, 10 likes

Mar 05 2020 Delip Rao

2. "Did you optimize your hyperparameters?" With compute costs coming down, it is becoming more affordable to run hyperparameter optimization, as long as you stay away from the Sesame Street. It would be interesting to condition this based on where the authors are coming from.
1 replies, 7 likes