Papers of the day   All papers

ON THE RELATIONSHIP BETWEEN SELF-ATTENTION AND CONVOLUTIONAL LAYERS

Comments

Jean-Baptiste Cordonnier: Very happy to share our latest work accepted at #ICRL2020: we prove that a Self-Attention layer can express any CNN layer. 1/5 📄Paper: https://openreview.net/pdf?id=HJlnC1rKPB 🍿Interactive website : https://epfml.github.io/attention-cnn/ 🖥Code: https://github.com/epfml/attention-cnn 📝Blog: http://jbcordonnier.com/posts/attention-cnn/ https://t.co/X1rNS1JvPt

5 replies, 1156 likes


hardmaru: Self-attention is proving to be a really good unified prior for both image and sequence processing. It can prob also learn useful representations for images that are difficult for conv layers to learn. See author’s thread for blog post and web demo: https://twitter.com/jb_cordonnier/status/1215581826187743232?s=21

1 replies, 139 likes


Torsten Scholak: After reading @chrmanning et al’s paper on where bert looks at, https://arxiv.org/abs/1906.04341, this makes intuitive sense to me

1 replies, 75 likes


(not Hugging for the moment) Face: On the Relationship between Self-Attention and Convolutional Layers @jb_cordonnier, @loukasa_tweet, & Jaggi show that multi-head attention layers can and often do perform convolution, linking two of the most popular operations in modern neural networks. https://openreview.net/pdf?id=HJlnC1rKPB https://t.co/z53zMOJ0NI

2 replies, 50 likes


Maks Sorokin 🦾: @hardmaru @iclr_conf @jb_cordonnier Author's tweet: https://twitter.com/jb_cordonnier/status/1215581826187743232?s=21

0 replies, 6 likes


Chaitanya Joshi: @ChrSzegedy Similar results for computer vision: https://twitter.com/jb_cordonnier/status/1215581826187743232

1 replies, 0 likes


Content

Found on Jan 10 2020 at https://openreview.net/pdf?id=HJlnC1rKPB

PDF content of a computer science paper: ON THE RELATIONSHIP BETWEEN SELF-ATTENTION AND CONVOLUTIONAL LAYERS