Papers of the day   All papers

Evolving Normalization-Activation Layers


Hanxiao Liu: New paper: Evolving Normalization-Activation Layers. We use evolution to design new layers called EvoNorms, which outperform BatchNorm-ReLU on many tasks. A promising use of AutoML to discover fundamental ML building blocks. Joint work with @DeepMind

7 replies, 650 likes

hardmaru: List of papers about automating improvements for deep learning: • better architectures from known building blocks • better activation functions • better learning rules than sgd/adam • better data augmentation strategies • better loss functions • better normalization layers

1 replies, 623 likes

Quoc Le: Cool results from our collaboration with colleagues at @DeepMind on searching for new layers as alternatives for BatchNorm-ReLU. Excited with the potential use of AutoML for discovering novel ML concepts from low level primitives.

3 replies, 337 likes

Raphael Meudec: Spent the weekend on implementing EvoNorm S0 and B0 with @TensorFlow 2.0 and running some ResNet18 trainings over CIFAR 10 & 100. 💻 Code available here : 📈 TensorBoard (w/ and w/o data aug) 📝 Paper:

2 replies, 112 likes

Jeff Dean (@🏡): Some nice work from @Hanxiao_6 Andrew Brock Karen Simonyan and @quocleix (joint work between @GoogleAI and @DeepMind) on evolving new normalization techniques that outperform batchnorm on a variety of tasks. Evolution is the new norm!

0 replies, 108 likes

Diganta Misra ツ: Recently @GoogleAI and @DeepMind released a paper shortly called EvoNorm (Paper Link - I tried implementing it on @PyTorch. GitHub Link -

1 replies, 65 likes

Thang Luong: Nice ideas of using (a) multiple architectures in the search objective for generalization & (b) a light weight proxy task on CIFAR-10 but rerank final candidates with ImageNet. EvoNorm seems to work pretty well across batch sizes! by @Hanxiao_6, @quocleix, & @DeepMind colleagues.

0 replies, 33 likes

roadrunner01: Evolving Normalization-Activation Layers pdf: abs:

1 replies, 31 likes

Xander Steenbrugge: @_brohrer_ There's a new drop-in TF layer from Google Brain / DeepMind that broadly outperforms BN and has an online variant. Paper: Great explainer video by @labs_henry:

1 replies, 24 likes

Daisuke Okanohara: Optimal normalization-activation layers are searched with multi-objective evolution. Found EvoNorm-B0 uses the normalization by the max of batch/instance variances and no activation. EvoNorm-S0 (no batch dependencies) is similar to GN+Swish.

0 replies, 8 likes

Lavanya 🦋: 📜 The Evolving Normalization-Activation Layers paper by @Hanxiao_6 et all – 👩‍🔬 Interactive @weights_biases report with results – 👩‍💻 Github repo to reproduce results –

0 replies, 5 likes

Sayak Paul: Thanks to @CShorten30 for his awesome video on the paper and I definitely recommend checking it out: Link to the original paper: @GoogleAI @GoogleDevsIN @GoogleDevExpert 5/5

1 replies, 4 likes

LDV Capital: Great thought, @charlesxjyang21! Another #autoML paper evaluated on the same set of benchmarks – #ImageNet & #CIFAR is acceptable as long they still pose a difficult/ relevant problem

0 replies, 3 likes

Shanqing Cai: Nice use of the new Graphs support of to show the computation graph underlying EvoNorm!

0 replies, 3 likes

arXiv CS-CV: Evolving Normalization-Activation Layers

0 replies, 3 likes

Weights & Biases: Experimental summary of my implementation of EvoNorm layers proposed in 📕Read: ✍️Code: #MachineLearning #DeepLearning

0 replies, 3 likes

👨‍🔬👨‍💻 Fabien Tarrade 💥🚀: Excellent video from @ykilcher on"Evolving Normalization-Activation Layers" This is about this paper by @Hanxiao_6, Andrew Brock, Karen Simonyan and Quoc V. Le

0 replies, 2 likes

OGAWA, Tadashi: => NAS (AutoML), Google Accelerator-aware NAS, Mar 5, 2020 BigNAS, Mar 24 EvoNorms: Evolving Normalization-Activation Layers, Apr 28 MobileDets, Apr 30 NAS 2020

1 replies, 2 likes

OGAWA, Tadashi: => "AutoML at Google and Future Directions", Quoc V. Le, Google, Invited, ICLR WS on Neural Architecture Search, Apr 26, 2020 AutoML-Zero, Mar 6 2020 Song Han, IEEE Micro, Jan/Feb 2020

1 replies, 0 likes


Found on Apr 08 2020 at

PDF content of a computer science paper: Evolving Normalization-Activation Layers