Papers of the day   All papers

Stand-Alone Self-Attention in Vision Models

Comments

Jun 17 2019 niki parmar

New Paper: Stand-Alone Self-Attention in Vision Models https://arxiv.org/abs/1906.05909 Can attention work as a stand-alone primitive for vision models? We develop a pure self-attention model by replacing the spatial convolutions in a ResNet by a simple, local self-attention layer. https://t.co/A2NDeivGmt
13 replies, 450 likes


Sep 07 2019 niki parmar

Our paper got accepted to #Neurips!! Code release coming soon, keep an eye out :)
1 replies, 206 likes


Jun 17 2019 William Fedus

Additional evidence of the transformer/self-attention as a useful computational primitive in vision tasks such as ImageNet classification and COCO detection. Future work is exciting: "...we hope to unify convolution and self-attention to best combine their unique advantages"
0 replies, 23 likes


Jun 17 2019 Ashish Vaswani

Pure content based interactions are competitive for vision models. Lot's of exciting work to be done in this research area.
2 replies, 14 likes


Sep 08 2019 Daisuke Okanohara

For image recognition tasks, they showed that local self-attention is competitive or superior to convolution in higher layers, and full attention model can achieve similar performance as ConvNet. Better absolute/relative position encoding is required. https://arxiv.org/abs/1906.05909
0 replies, 8 likes


Sep 10 2019 HotComputerScience

Most popular computer science paper of the day: "Stand-Alone Self-Attention in Vision Models" https://hotcomputerscience.com/paper/stand-alone-self-attention-in-vision-models https://twitter.com/nikiparmar09/status/1140649942505013249
0 replies, 4 likes


Content