Hanxiao Liu: New paper: Evolving Normalization-Activation Layers.
We use evolution to design new layers called EvoNorms, which outperform BatchNorm-ReLU on many tasks. A promising use of AutoML to discover fundamental ML building blocks.
Joint work with @DeepMind https://t.co/A98YVNzrtg
7 replies, 651 likes
hardmaru: List of papers about automating improvements for deep learning:
• better architectures from known building blocks
• better activation functions
• better learning rules than sgd/adam
• better data augmentation strategies
• better loss functions
• better normalization layers
1 replies, 624 likes
Quoc Le: Cool results from our collaboration with colleagues at @DeepMind on searching for new layers as alternatives for BatchNorm-ReLU. Excited with the potential use of AutoML for discovering novel ML concepts from low level primitives.
3 replies, 340 likes
Raphael Meudec: Spent the weekend on implementing EvoNorm S0 and B0 with @TensorFlow 2.0 and running some ResNet18 trainings over CIFAR 10 & 100.
💻 Code available here : https://www.github.com/sicara/tf2-evonorm
📈 TensorBoard (w/ and w/o data aug) https://tensorboard.dev/experiment/QmwLVEBvSd2k9pN1AjTZsg/#scalars
📝 Paper: https://arxiv.org/abs/2004.02967
2 replies, 113 likes
Jeff Dean (@🏡): Some nice work from @Hanxiao_6 Andrew Brock Karen Simonyan and @quocleix (joint work between @GoogleAI and @DeepMind) on evolving new normalization techniques that outperform batchnorm on a variety of tasks.
Evolution is the new norm!
0 replies, 108 likes
Diganta Misra ツ: Recently @GoogleAI and @DeepMind released a paper shortly called EvoNorm (Paper Link - https://arxiv.org/pdf/2004.02967.pdf). I tried implementing it on @PyTorch. GitHub Link - https://github.com/digantamisra98/EvoNorm
1 replies, 65 likes
Thang Luong: Nice ideas of using (a) multiple architectures in the search objective for generalization & (b) a light weight proxy task on CIFAR-10 but rerank final candidates with ImageNet. EvoNorm seems to work pretty well across batch sizes! by @Hanxiao_6, @quocleix, & @DeepMind colleagues.
0 replies, 33 likes
roadrunner01: Evolving Normalization-Activation Layers
abs: https://arxiv.org/abs/2004.02967 https://t.co/pGcuisODw1
1 replies, 31 likes
Xander Steenbrugge: @_brohrer_ There's a new drop-in TF layer from Google Brain / DeepMind that broadly outperforms BN and has an online variant.
Great explainer video by @labs_henry: https://www.youtube.com/watch?v=RFn5eH5ZCVo https://t.co/jYEyxuQrCv
1 replies, 24 likes
Daisuke Okanohara: Optimal normalization-activation layers are searched with multi-objective evolution. Found EvoNorm-B0 uses the normalization by the max of batch/instance variances and no activation. EvoNorm-S0 (no batch dependencies) is similar to GN+Swish. https://arxiv.org/abs/2004.02967
0 replies, 8 likes
Lavanya 🦋: 📜 The Evolving Normalization-Activation Layers paper by
@Hanxiao_6 et all – https://arxiv.org/abs/2004.02967
👩🔬 Interactive @weights_biases report with results – https://app.wandb.ai/sayakpaul/EvoNorm-TensorFlow2/reports/EvoNorm-layers-in-TensorFlow-2--Vmlldzo4Mzk3MQ?utm_source=social_twitter&utm_medium=report&utm_campaign=report_author
👩💻 Github repo to reproduce results – https://github.com/sayakpaul/EvoNorms-in-TensorFlow-2
0 replies, 5 likes
Sayak Paul: Thanks to @CShorten30 for his awesome video on the paper and I definitely recommend checking it out: https://www.youtube.com/watch?v=RFn5eH5ZCVo.
Link to the original paper: https://arxiv.org/abs/2004.02967.
@GoogleAI @GoogleDevsIN @GoogleDevExpert
1 replies, 4 likes
Shanqing Cai: Nice use of the new Graphs support of http://tensorboard.dev to show the computation graph underlying EvoNorm!
0 replies, 3 likes
LDV Capital: Great thought, @charlesxjyang21! Another #autoML paper evaluated on the same set of benchmarks – #ImageNet & #CIFAR is acceptable as long they still pose a difficult/ relevant problem https://arxiv.org/abs/2004.02967
0 replies, 3 likes
OGAWA, Tadashi: =>
NAS (AutoML), Google
Accelerator-aware NAS, Mar 5, 2020 https://arxiv.org/abs/2003.02838
BigNAS, Mar 24 https://arxiv.org/abs/2003.11142
EvoNorms: Evolving Normalization-Activation Layers, Apr 28 https://arxiv.org/abs/2004.02967
MobileDets, Apr 30 https://arxiv.org/abs/2004.14525
NAS 2020 https://twitter.com/ogawa_tter/status/1257367888668786691 https://t.co/aUnXSVLOsk
1 replies, 2 likes
👨🔬👨💻 Fabien Tarrade 💥🚀: Excellent video from @ykilcher on"Evolving Normalization-Activation Layers" https://youtu.be/klPuEHCKG9M. This is about this paper https://arxiv.org/abs/2004.02967 by @Hanxiao_6, Andrew Brock, Karen Simonyan and Quoc V. Le
0 replies, 2 likes
OGAWA, Tadashi: =>
"AutoML at Google and Future Directions", Quoc V. Le, Google, Invited, ICLR WS on Neural Architecture Search, Apr 26, 2020 https://slideslive.com/38926392/automl-at-google-and-future-directions
AutoML-Zero, Mar 6 2020 https://arxiv.org/abs/2003.03384
Song Han, IEEE Micro, Jan/Feb 2020 https://ieeexplore.ieee.org/abstract/document/8897011
1 replies, 0 likes
Found on Apr 08 2020 at https://arxiv.org/pdf/2004.02967.pdf