Papers of the day   All papers

Training with Quantization Noise for Extreme Model Compression

Comments

Edouard Grave: Training with QuantNoise allows to strongly compress neural networks: 80.0% accuracy on ImageNet in 3.3MB, 82.5MB accuracy on MNLI in 14MB. Blog: https://ai.facebook.com/blog/training-with-quantization-noise-for-extreme-model-compression/ Paper: https://arxiv.org/abs/2004.07320

2 replies, 384 likes


Delip Rao: Have your cake and eat it too. Training with quantization noise seems work wonders for large language model compression while minimizing sacrifice on downstream #NLProc task accuracy. Nice work from FAIR folks https://arxiv.org/abs/2004.07320 https://t.co/vBBYafvZlC

1 replies, 47 likes


Smerity: Incredibly impressive model compression: QuantNoise is mash-up between DropConnect and Straight-Through Estimation. Take your weights, apply quantization to a subset (~DropConnect), and then pretend you never did that on the backward pass (STE). https://t.co/UuqmCp8ben

0 replies, 41 likes


arxiv: Training with Quantization Noise for Extreme Fixed-Point Compression. http://arxiv.org/abs/2004.07320 https://t.co/Wb0EP4ha4j

0 replies, 4 likes


ML and Data Projects To Know: 📙 Training with Quantization Noise for Extreme Model Compression Authors: Angela Fan, @PierreStock, Benjamin Graham, @EXGRV, Remi Gribonval, @hjegou, Armand Joulin Paper Link: https://arxiv.org/abs/2004.07320 Blog Post: https://ai.facebook.com/blog/training-with-quantization-noise-for-extreme-model-compression/

0 replies, 2 likes


Daisuke Okanohara: For NN model compression with quantization, they propose to quantize "the part" of NN in the forward pass and use the original weight in the backward pass (=straight through estimator) to achieve better compression/accuracy tradeoff. https://arxiv.org/abs/2004.07320

0 replies, 2 likes


Content

Found on Apr 23 2020 at https://arxiv.org/pdf/2004.07320.pdf

PDF content of a computer science paper: Training with Quantization Noise for Extreme Model Compression