Papers of the day   All papers

Do We Need Zero Training Loss A‰er Achieving Zero Training Error?

Comments

hardmaru: Do We Need Zero Training Loss After Achieving Zero Training Error? By not letting the training loss to go to zero, model will “random walk” with the same non-zero loss and drift into an area with a flat loss landscape that leads to better generalization. https://arxiv.org/abs/2002.08709 https://t.co/sMJPZLo4bO

14 replies, 790 likes


Shawn Presser: Most underrated ML hack of this century: Loss getting too low? loss = abs(loss - x) + x where x is a value like 0.2. Presto, your loss is no longer <0.2. Set it to whatever you want. It completely stabilized our biggan runs. This is "flood loss" https://arxiv.org/abs/2002.08709

13 replies, 494 likes


Takashi Ishida: Really happy to announce that our flooding paper 🌊 has been accepted to ICML2020!! #ICML2020 @icmlconf Joint work with Ikko Yamane, Tomoya Sakai, Gang Niu, and Masashi Sugiyama. Looking forward to presenting it in the conference next month!

0 replies, 38 likes


tsauri: @ericjang11 @karpathy well only recently we know we can "freeze" validation loss with something like loss = (loss + 0.2).abs() - 0.2 https://twitter.com/hardmaru/status/1252159376502091776

0 replies, 26 likes


YipingLu_2prime: And you can see epoch wise double descent here

0 replies, 4 likes


akira: https://arxiv.org/pdf/2002.08709.pdf A study to avoid zero learning loss by making the learning as Loss = ABS(Loss-b)+b so as not to fall below the setting value http://b.It can be used in conjunction with other regularization methods, and is effective in many task. https://t.co/6CmHHLmfa0

0 replies, 3 likes


Content

Found on Apr 20 2020 at https://arxiv.org/pdf/2002.08709.pdf

PDF content of a computer science paper: Do We Need Zero Training Loss A‰er Achieving Zero Training Error?