Papers of the day   All papers

Movement Pruning: Adaptive Sparsity by Fine-Tuning

Comments

Victor Sanh: Excited to share our latest work on extreme pruning in the context of transfer learning 🧀 95% of the original perf with only ~5% of remaining weights in the encoder💪 Paper: https://arxiv.org/abs/2005.07683 With amazing collaborators @Thom_Wolf & @srush_nlp at @huggingface [1/7] https://t.co/X2VnG3JvuI

2 replies, 625 likes


Thomas Wolf: Victor is releasing his new research work on extreme pruning of pretrained models! I really loved this project! A very deep dive to understand why & how standard pruning methods fail in the context of Transfer Learning and how we can do a lot better! Check his detailed thread👇

0 replies, 125 likes


Sasha Rush: New 🤗 preprint on pruning for transfer learning ("fine-pruning"). Exploits a simple idea: magnitude pruning stops making sense if weights don't really move. https://t.co/c7Uq0ZCPJX

0 replies, 85 likes


Leo Boytsov: "95% of the original perf with only ~5% remaining weights in the encoder!" is a great result by a team of @huggingface researchers https://arxiv.org/pdf/2005.07683.pdf

0 replies, 50 likes


Mario Kostelac: I took 30min to find another proof of this - https://arxiv.org/abs/2005.07683. 3% of weights, 95%+ accuracy. (paper by @SanhEstPasMoi , @Thom_Wolf, @srush_nlp from @huggingface)

1 replies, 36 likes


Mario Kostelac: Our ML models are getting bigger every day, but we're just scratching the surface! 👇 is a great example of that! Facebook managed to remove 96% of the model weights, without degrading perceived audio quality. https://t.co/iOaVHx00XA

3 replies, 10 likes


Aran Komatsuzaki: Movement Pruning: Adaptive Sparsity by Fine-Tuning Achieves minimal accuracy loss with down to only 3% of the model parameters. https://arxiv.org/abs/2005.07683 https://t.co/Lsa5C4zJuz

1 replies, 9 likes


Victor Sanh: Paper: https://arxiv.org/abs/2005.07683 Code & Weights will be released very soon! Stay tuned! In the meantime, here’s a sneak peek at the memory size compressions: [7/7] https://t.co/7FGokZAsBR

2 replies, 4 likes


LumenAI: #Pruning #DeepLearning

0 replies, 3 likes


arXiv CS-CL: Movement Pruning: Adaptive Sparsity by Fine-Tuning http://arxiv.org/abs/2005.07683

0 replies, 2 likes


arXiv CS-CL: Movement Pruning: Adaptive Sparsity by Fine-Tuning http://arxiv.org/abs/2005.07683

0 replies, 1 likes


Content

Found on May 18 2020 at https://arxiv.org/pdf/2005.07683.pdf

PDF content of a computer science paper: Movement Pruning: Adaptive Sparsity by Fine-Tuning