Papers of the day   All papers

Stand-Alone Self-Attention in Vision Models


Jun 17 2019 niki parmar

New Paper: Stand-Alone Self-Attention in Vision Models Can attention work as a stand-alone primitive for vision models? We develop a pure self-attention model by replacing the spatial convolutions in a ResNet by a simple, local self-attention layer.
15 replies, 458 likes

Sep 07 2019 niki parmar

Our paper got accepted to #Neurips!! Code release coming soon, keep an eye out :)
1 replies, 206 likes

Jun 17 2019 William Fedus

Additional evidence of the transformer/self-attention as a useful computational primitive in vision tasks such as ImageNet classification and COCO detection. Future work is exciting: "...we hope to unify convolution and self-attention to best combine their unique advantages"
0 replies, 23 likes

Jan 10 2020 Jean-Baptiste Cordonnier

Our work explains the recent success of Transformer architecture applied to vision: Attention Augmented Convolutional Networks. @IrwanBello et al., 2019. Stand-Alone Self-Attention in Vision Models. Ramachandran et al., 2019. 3/5
1 replies, 17 likes

Jun 17 2019 Ashish Vaswani

Pure content based interactions are competitive for vision models. Lot's of exciting work to be done in this research area.
2 replies, 14 likes

Sep 08 2019 Daisuke Okanohara

For image recognition tasks, they showed that local self-attention is competitive or superior to convolution in higher layers, and full attention model can achieve similar performance as ConvNet. Better absolute/relative position encoding is required.
0 replies, 8 likes

Sep 10 2019 HotComputerScience

Most popular computer science paper of the day: "Stand-Alone Self-Attention in Vision Models"
0 replies, 4 likes


PDF content of a computer science paper: Stand-Alone Self-Attention in Vision Models"