Papers of the day   All papers

Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift


Sep 11 2019 David Page

The paper that introduced Batch Norm combines clear intuition with compelling experiments (14x speedup on ImageNet!!) So why has 'internal covariate shift' remained controversial to this day? Thread πŸ‘‡
14 replies, 1179 likes

Sep 11 2019 Jeremy Howard

This is the best distillation of recent (and old!) research on batchnorm I've seen. There is so much to learn about training mechanics by studying this thread and the links it contains.
3 replies, 695 likes

Sep 12 2019 Roger Grosse

Excellent overview of the (widely misunderstood) mechanism hypothesized in the original batch norm paper, as well as recent empirical evidence that strongly supports it.
2 replies, 137 likes

Sep 12 2019 Yann LeCun

Nice tweethread on the effect of variable centering in deep nets (like batch norm).
0 replies, 100 likes

Sep 12 2019 Yann LeCun

Nice thread on the effects of variable normalization in neural nets (e.g. with batch norm).
0 replies, 46 likes

Sep 12 2019 Frank Dellaert

Wondering about batch norm? Read this amazing thread.
0 replies, 18 likes

Sep 11 2019 Andrei Bursuc

Nice summary of the knowledge and understanding we have over BatchNorm from both recent and classic perspectives at the time of its publication
1 replies, 15 likes

Sep 13 2019 Brandon Rohrer

A fun deep dive on β€œWhy Batch Norm?”
0 replies, 10 likes

Sep 12 2019 Lavanya

Cool thread on the intuition behind why BatchNorm works so well in practice! πŸ€·πŸΌβ€β™€οΈ
0 replies, 10 likes

Sep 11 2019 Thread Reader App

@soldni Hola there is your unroll: Thread by @dcpage3: "The paper that introduced Batch Norm combines clear intuition with compelling experiments (14x […]" Talk to you soon. πŸ€–
0 replies, 7 likes

Sep 12 2019 (((Ω„()(Ω„() 'yoav))))

(another!) great thread re batch norm.
0 replies, 4 likes

Sep 12 2019 Aakash Kumar Nain πŸ”Ž

Nice thread
0 replies, 4 likes

May 04 2019 Rob Flickenger ⚑

TIL that some ML researchers had an idea (batch normalization) that worked, then tacked on a post-hoc mathematical analysis to claim that it reduced "internal covariate shift". The technique works, but not for that reason.
1 replies, 2 likes

Oct 19 2019 Brandon Rohrer

@dcpage3 offers keen insights into how batch norm works. I particularly resonated with his meta-conclusions: 1) When studying neural networks, use simple examples. You can iterate faster and see what’s going on more easily. 2) VISUALIZE. h/t @jeremyphoward
0 replies, 2 likes

Aug 02 2019 The Phenomenon

To my ML/AI/DS folks here: What are your best three articles in this field? It can be blog posts or published papers or a combination of both.
1 replies, 1 likes