hardmaru: Do We Need Zero Training Loss After Achieving Zero Training Error?
By not letting the training loss to go to zero, model will “random walk” with the same non-zero loss and drift into an area with a flat loss landscape that leads to better generalization.
14 replies, 790 likes
Shawn Presser: Most underrated ML hack of this century:
Loss getting too low?
loss = abs(loss - x) + x
where x is a value like 0.2.
Presto, your loss is no longer <0.2.
Set it to whatever you want. It completely stabilized our biggan runs.
This is "flood loss" https://arxiv.org/abs/2002.08709
13 replies, 494 likes
Takashi Ishida: Really happy to announce that our flooding paper 🌊 has been accepted to ICML2020!! #ICML2020 @icmlconf
Joint work with Ikko Yamane, Tomoya Sakai, Gang Niu, and Masashi Sugiyama. Looking forward to presenting it in the conference next month!
0 replies, 38 likes
tsauri: @ericjang11 @karpathy well only recently we know we can "freeze" validation loss with something like
loss = (loss + 0.2).abs() - 0.2
0 replies, 26 likes
YipingLu_2prime: And you can see epoch wise double descent here
0 replies, 4 likes
A study to avoid zero learning loss by making the learning as Loss = ABS(Loss-b)+b so as not to fall below the setting value http://b.It can be used in conjunction with other regularization methods, and is effective in many task. https://t.co/6CmHHLmfa0
0 replies, 3 likes
Found on Apr 20 2020 at https://arxiv.org/pdf/2002.08709.pdf