Papers of the day   All papers

Do We Need Zero Training Loss A‰er Achieving Zero Training Error?


hardmaru: Do We Need Zero Training Loss After Achieving Zero Training Error? By not letting the training loss to go to zero, model will “random walk” with the same non-zero loss and drift into an area with a flat loss landscape that leads to better generalization.

14 replies, 790 likes

Shawn Presser: Most underrated ML hack of this century: Loss getting too low? loss = abs(loss - x) + x where x is a value like 0.2. Presto, your loss is no longer <0.2. Set it to whatever you want. It completely stabilized our biggan runs. This is "flood loss"

13 replies, 494 likes

Takashi Ishida: Really happy to announce that our flooding paper 🌊 has been accepted to ICML2020!! #ICML2020 @icmlconf Joint work with Ikko Yamane, Tomoya Sakai, Gang Niu, and Masashi Sugiyama. Looking forward to presenting it in the conference next month!

0 replies, 38 likes

tsauri: @ericjang11 @karpathy well only recently we know we can "freeze" validation loss with something like loss = (loss + 0.2).abs() - 0.2

0 replies, 26 likes

YipingLu_2prime: And you can see epoch wise double descent here

0 replies, 4 likes

akira: A study to avoid zero learning loss by making the learning as Loss = ABS(Loss-b)+b so as not to fall below the setting value http://b.It can be used in conjunction with other regularization methods, and is effective in many task.

0 replies, 3 likes


Found on Apr 20 2020 at

PDF content of a computer science paper: Do We Need Zero Training Loss A‰er Achieving Zero Training Error?