Papers of the day   All papers

Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited

Comments

Andrew Gordon Wilson: Overparameterization isn't mysterious, if we stop parameter counting as a proxy for complexity. Our new paper "Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited": https://arxiv.org/abs/2003.02139. With Wesley Maddox, @g_benton_. 1/7 https://t.co/Cf2msbacWx

3 replies, 586 likes


Andrew Gordon Wilson: Effective dimension compares favourably to popular path-norm and PAC-Bayes flatness measures, including double descent and width-depth trade-offs! We have just posted this new result in section 7 of our paper on posterior contraction in BDL: https://arxiv.org/abs/2003.02139. 1/16 https://t.co/T8i2BKQqbO

1 replies, 80 likes


Andrew Gordon Wilson: Interesting thread tracing the origins of double descent.

0 replies, 17 likes


Andrew Gordon Wilson: @davidwhogg You may be interested in https://arxiv.org/abs/2002.08791, where we show that Bayesian model averaging mitigates double descent (as predicted by Sec 1 & 3). We also provide an explanation for DD in https://arxiv.org/abs/2003.02139.

0 replies, 11 likes


Dmitry Kobak: PPPS. @andrewgwils linked below (https://twitter.com/andrewgwils/status/1243988946931060737) to his new preprint (https://arxiv.org/abs/2003.02139) citing even earlier work by Opper for "non-monotonic generalization capability"! Here is Opper et al. 1990 https://iopscience.iop.org/article/10.1088/0305-4470/23/11/012: https://t.co/fLAE2KlQ3o

0 replies, 7 likes


Andrew Gordon Wilson: @hippopedoid We have some early references on double descent in our recent paper (https://arxiv.org/abs/2003.02139), where we also show that effective dimension closely tracks double descent in the regime with low training loss. Increasing the number of parameters (in this way) decreases complexity. https://t.co/x67MYV3qt2

2 replies, 5 likes


arXiv in review: #ICML2020 Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited. (arXiv:2003.02139v2 [cs\.LG] UPDATED) http://arxiv.org/abs/2003.02139

0 replies, 3 likes


Statistics Papers: Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited. http://arxiv.org/abs/2003.02139

0 replies, 2 likes


HotComputerScience: Most popular computer science paper of the day: "Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited" https://hotcomputerscience.com/paper/rethinking-parameter-counting-in-deep-models-effective-dimensionality-revisited https://twitter.com/andrewgwils/status/1235377114976849923

0 replies, 2 likes


Brundage Bot: Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited. Wesley J. Maddox, Gregory Benton, and Andrew Gordon Wilson http://arxiv.org/abs/2003.02139

1 replies, 1 likes


Content

Found on Mar 05 2020 at https://arxiv.org/pdf/2003.02139.pdf

PDF content of a computer science paper: Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited