Papers of the day   All papers

Evolving Normalization-Activation Layers

Comments

Hanxiao Liu: New paper: Evolving Normalization-Activation Layers. We use evolution to design new layers called EvoNorms, which outperform BatchNorm-ReLU on many tasks. A promising use of AutoML to discover fundamental ML building blocks. http://arxiv.org/abs/2004.02967 Joint work with @DeepMind https://t.co/A98YVNzrtg

7 replies, 650 likes


hardmaru: List of papers about automating improvements for deep learning: • better architectures from known building blocks • better activation functions • better learning rules than sgd/adam • better data augmentation strategies • better loss functions • better normalization layers

1 replies, 623 likes


Quoc Le: Cool results from our collaboration with colleagues at @DeepMind on searching for new layers as alternatives for BatchNorm-ReLU. Excited with the potential use of AutoML for discovering novel ML concepts from low level primitives.

3 replies, 337 likes


Raphael Meudec: Spent the weekend on implementing EvoNorm S0 and B0 with @TensorFlow 2.0 and running some ResNet18 trainings over CIFAR 10 & 100. 💻 Code available here : https://www.github.com/sicara/tf2-evonorm 📈 TensorBoard (w/ and w/o data aug) https://tensorboard.dev/experiment/QmwLVEBvSd2k9pN1AjTZsg/#scalars 📝 Paper: https://arxiv.org/abs/2004.02967

2 replies, 112 likes


Jeff Dean (@🏡): Some nice work from @Hanxiao_6 Andrew Brock Karen Simonyan and @quocleix (joint work between @GoogleAI and @DeepMind) on evolving new normalization techniques that outperform batchnorm on a variety of tasks. Evolution is the new norm!

0 replies, 108 likes


Diganta Misra ツ: Recently @GoogleAI and @DeepMind released a paper shortly called EvoNorm (Paper Link - https://arxiv.org/pdf/2004.02967.pdf). I tried implementing it on @PyTorch. GitHub Link - https://github.com/digantamisra98/EvoNorm

1 replies, 65 likes


Thang Luong: Nice ideas of using (a) multiple architectures in the search objective for generalization & (b) a light weight proxy task on CIFAR-10 but rerank final candidates with ImageNet. EvoNorm seems to work pretty well across batch sizes! by @Hanxiao_6, @quocleix, & @DeepMind colleagues.

0 replies, 33 likes


roadrunner01: Evolving Normalization-Activation Layers pdf: https://arxiv.org/pdf/2004.02967.pdf abs: https://arxiv.org/abs/2004.02967 https://t.co/pGcuisODw1

1 replies, 31 likes


Xander Steenbrugge: @_brohrer_ There's a new drop-in TF layer from Google Brain / DeepMind that broadly outperforms BN and has an online variant. Paper: https://arxiv.org/abs/2004.02967 Great explainer video by @labs_henry: https://www.youtube.com/watch?v=RFn5eH5ZCVo https://t.co/jYEyxuQrCv

1 replies, 24 likes


Daisuke Okanohara: Optimal normalization-activation layers are searched with multi-objective evolution. Found EvoNorm-B0 uses the normalization by the max of batch/instance variances and no activation. EvoNorm-S0 (no batch dependencies) is similar to GN+Swish. https://arxiv.org/abs/2004.02967

0 replies, 8 likes


Lavanya 🦋: 📜 The Evolving Normalization-Activation Layers paper by @Hanxiao_6 et all – https://arxiv.org/abs/2004.02967 👩‍🔬 Interactive @weights_biases report with results – https://app.wandb.ai/sayakpaul/EvoNorm-TensorFlow2/reports/EvoNorm-layers-in-TensorFlow-2--Vmlldzo4Mzk3MQ?utm_source=social_twitter&utm_medium=report&utm_campaign=report_author 👩‍💻 Github repo to reproduce results – https://github.com/sayakpaul/EvoNorms-in-TensorFlow-2

0 replies, 5 likes


Sayak Paul: Thanks to @CShorten30 for his awesome video on the paper and I definitely recommend checking it out: https://www.youtube.com/watch?v=RFn5eH5ZCVo. Link to the original paper: https://arxiv.org/abs/2004.02967. @GoogleAI @GoogleDevsIN @GoogleDevExpert 5/5

1 replies, 4 likes


LDV Capital: Great thought, @charlesxjyang21! Another #autoML paper evaluated on the same set of benchmarks – #ImageNet & #CIFAR is acceptable as long they still pose a difficult/ relevant problem https://arxiv.org/abs/2004.02967

0 replies, 3 likes


Shanqing Cai: Nice use of the new Graphs support of http://tensorboard.dev to show the computation graph underlying EvoNorm!

0 replies, 3 likes


arXiv CS-CV: Evolving Normalization-Activation Layers http://arxiv.org/abs/2004.02967

0 replies, 3 likes


Weights & Biases: Experimental summary of my implementation of EvoNorm layers proposed in https://arxiv.org/pdf/2004.02967.pdf. 📕Read: https://app.wandb.ai/sayakpaul/EvoNorm-TensorFlow2/reports/EvoNorm-layers-in-TensorFlow-2--Vmlldzo4Mzk3MQ ✍️Code: https://github.com/sayakpaul/EvoNorms-in-TensorFlow-2 #MachineLearning #DeepLearning

0 replies, 3 likes


👨‍🔬👨‍💻 Fabien Tarrade 💥🚀: Excellent video from @ykilcher on"Evolving Normalization-Activation Layers" https://youtu.be/klPuEHCKG9M. This is about this paper https://arxiv.org/abs/2004.02967 by @Hanxiao_6, Andrew Brock, Karen Simonyan and Quoc V. Le

0 replies, 2 likes


OGAWA, Tadashi: => NAS (AutoML), Google Accelerator-aware NAS, Mar 5, 2020 https://arxiv.org/abs/2003.02838 BigNAS, Mar 24 https://arxiv.org/abs/2003.11142 EvoNorms: Evolving Normalization-Activation Layers, Apr 28 https://arxiv.org/abs/2004.02967 MobileDets, Apr 30 https://arxiv.org/abs/2004.14525 NAS 2020 https://twitter.com/ogawa_tter/status/1257367888668786691 https://t.co/aUnXSVLOsk

1 replies, 2 likes


OGAWA, Tadashi: => "AutoML at Google and Future Directions", Quoc V. Le, Google, Invited, ICLR WS on Neural Architecture Search, Apr 26, 2020 https://slideslive.com/38926392/automl-at-google-and-future-directions AutoML-Zero, Mar 6 2020 https://arxiv.org/abs/2003.03384 Song Han, IEEE Micro, Jan/Feb 2020 https://ieeexplore.ieee.org/abstract/document/8897011 https://twitter.com/ogawa_tter/status/1144518821220212736 https://t.co/Qpm2WrMisK

1 replies, 0 likes


Content

Found on Apr 08 2020 at https://arxiv.org/pdf/2004.02967.pdf

PDF content of a computer science paper: Evolving Normalization-Activation Layers