Papers of the day   All papers

PRUNING NEURAL NETWORKS AT INITIALIZATION: WHY ARE WE MISSING THE MARK?

Comments

Jonathan Frankle: Several methods have recently been proposed for pruning neural networks at initialization. In our new paper (@KDziugaite, @roydanroy, @mcarbin), we rigorously study these methods to determine why they "miss the mark" and underperform pruning after training https://arxiv.org/abs/2009.08576 https://t.co/e3loMBlOoj

5 replies, 354 likes


Daniel Roy: New work with colleagues at @element_ai and @MIT_CSAIL on pruning at initialization. Key experiments shed light on what’s missing from current methods. This is apparently a hot topic as we’ve learned several other teams were hot on the trail.

3 replies, 117 likes


Michael Carbin: Our new work demonstrating there's still a ways to go on pruning at initialization: the community seems to only know which layers -- but not which individual weights -- to prune. With a flurry of activity around these ideas, I look forward to other teams' findings as well!

0 replies, 94 likes


Sara Hooker: This remains an open exciting research question -- how do we start sparse on large scale realistic datasets and still converge to reasonable performance.

0 replies, 51 likes


Daisuke Okanohara: Current methods for pruning NNs at initialization (SNIP, GraSP, SynFlow) perform similarly even when shuffling pruning masks within each layer or reinitializing weights, which indicates they only determine the fraction of weights to prune per-layer. https://arxiv.org/abs/2009.08576

0 replies, 9 likes


ALife Papers: PRUNING NEURAL NETWORKS AT INITIALIZATION: WHY ARE WE MISSING THE MARK? "Recent work has explored the possibility of pruning neural networks at initialization [...] Although these methods surpass random pruning, they remain below magnitude pruning" https://arxiv.org/pdf/2009.08576.pdf https://t.co/lGq2kpSSYu

0 replies, 3 likes


akira: https://arxiv.org/abs/2009.08576 A study that evaluated methods that prune from initialization. They are consistently perform worse than methods that prune after training. In addition, shuffling weights within each layer or reinitializing weights results in equal or better accuracy. https://t.co/8qnU1t1mlP

0 replies, 1 likes


arxiv: Pruning Neural Networks at Initialization: Why are We Missing the Mark?. http://arxiv.org/abs/2009.08576 https://t.co/7JtorDq8ke

0 replies, 1 likes


arXiv CS-CV: Pruning Neural Networks at Initialization: Why are We Missing the Mark? http://arxiv.org/abs/2009.08576

0 replies, 1 likes


Content

Found on Sep 21 2020 at https://arxiv.org/pdf/2009.08576.pdf

PDF content of a computer science paper: PRUNING NEURAL NETWORKS AT INITIALIZATION: WHY ARE WE MISSING THE MARK?