Papers of the day   All papers

Big Self-Supervised Models are Strong Semi-Supervised Learners

Comments

Ting Chen: SimCLRv2: an improved self-supervised approach for semi-supervised learning. On ImageNet with 1% of the labels, it achieves 76.6% top-1, a 22% relative improvement over previous SOTA. https://arxiv.org/abs/2006.10029 Joint work with @skornblith, @kswersk, @mo_norouzi, @geoffreyhinton https://t.co/Fv6IyFCpNu

2 replies, 426 likes


Denny Britz: “Bigger Models Are More Label-Efficient” - They beat supervised models on ImageNet with just 10% of the labels: https://arxiv.org/abs/2006.10029 OpenAI’s Image GPT made a similar point. Interesting b/c intuitively this is not obvious. Maybe bigger = representations are more disentangled? https://t.co/f1QJawYO0O

9 replies, 323 likes


Sayak Paul: Here's a list of my favorite recent papers on transfer learning for vision: - BigTransfer: https://arxiv.org/abs/1912.11370 - VirTex: https://arxiv.org/abs/2006.06666 - SimCLRv2: https://arxiv.org/abs/2006.10029 - Self-training: https://arxiv.org/abs/2006.06882 Would love to see a T5-like paper for vision.

2 replies, 217 likes


Anand Agarwal: An improved approach for semi-supervised learning! Thrilled to see these new developments in #MachineLearning and self-paced #education https://arxiv.org/abs/2006.10029

0 replies, 169 likes


Simon Kornblith: Unsupervised pretraining works much better for semi-supervised learning when the model is big. A model 10x bigger than ResNet-50 needs 10x fewer labels to get the same accuracy. But once the big model is trained, we can distill it to back to ResNet-50 with little loss in accuracy

0 replies, 77 likes


Mohammad Norouzi: Big Self-Supervised Models are Strong Semi-Supervised Learners.

0 replies, 64 likes


Simone Scardapane: *Big Self-Supervised Models are Strong Semi-Supervised Learners* by @tingchenai @skornblith @kswersk @mo_norouzi @geoffreyhinton Title says it all: scaling up SimCLR + distillation yields impressive semi-supervised results on ImageNet. https://arxiv.org/abs/2006.10029 https://t.co/pORJk5JDs3

0 replies, 29 likes


Theodore Galanos: @Thom_Wolf I'm not sure this pertains only to CV, it doesn't really, but the self-supervised 'revolution' has been fascinating to watch. Too many papers to have here but some highlights are: https://arxiv.org/abs/1911.05722, https://arxiv.org/abs/2002.05709, https://arxiv.org/abs/2004.11362, https://arxiv.org/abs/2006.10029

0 replies, 27 likes


Blake Richards: This is really interesting, and leads me to emphasise something that we often forget when discussing human intelligence: the human brain has *way* more synapses than the biggest current ANNs. If we had the compute to do human brain size ANNs, I wonder what we could achieve?

7 replies, 26 likes


Evgenii Zheltonozhskii: Self-supervised learning sees a strong boost in performance: first BYOD and now SimCLRv2 by @tingchenai @skornblith @kswersk @mo_norouzi @geoffreyhinton. Starts to make sense to talk about top-1 performace on 1% of ImageNet -- 76.6% for top model.

1 replies, 20 likes


Emtiyaz Khan: Perhaps bigger models give rise to smoother functions? This is an interesting study nevertheless.

0 replies, 10 likes


Aran Komatsuzaki: Big Self-Supervised Models are Strong Semi-Supervised Learners SimCLRv2 on Resnet-50 achieves 73.9% top-1 accuracy in Imagenet with 1% labels, a 10× improvement in label efficiency over the previous sota https://arxiv.org/pdf/2006.10029.pdf https://t.co/oKuV9QDk6F

2 replies, 10 likes


Janne Spijkervet: During the 3-month lockdown, research on self-supervised representation learning has surely not halted! After reading the fascinating BYOL this morning from @DeepMind , I find SimCLRv2 in my inbox this evening by @tingchenai et al.! :) https://arxiv.org/pdf/2006.10029.pdf

0 replies, 5 likes


Brundage Bot: Big Self-Supervised Models are Strong Semi-Supervised Learners. Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton http://arxiv.org/abs/2006.10029

1 replies, 4 likes


akira: https://arxiv.org/abs/2006.10029 They proposed SimCLRv2, which uses only a small number of labels and performs as well or better than supervised learning. It consists of three stages: unsupervised learning, FineTune, and self-training distillation using unlabeled data. https://t.co/KjLY9ufPUF

0 replies, 3 likes


Vicente Reyes-Puerta: This updated framework takes the “unsupervised pretrain, supervised fine-tune” paradigm 👍 for me it is like a massive transfer learning approach 🤔 great paper, waiting for the pretrained model to be available on @github 😁 https://arxiv.org/pdf/2006.10029.pdf #DeepLearning #MachineLearning

0 replies, 1 likes


Peter Organisciak: Somewhat expected, but interesting find from image ML - the bigger a pre-trained model gets, the fewer labels fine-tuning needs. https://arxiv.org/abs/2006.10029 So you can compensate for few labels with a bigger model.

1 replies, 1 likes


Content

Found on Jun 19 2020 at https://arxiv.org/pdf/2006.10029.pdf

PDF content of a computer science paper: Big Self-Supervised Models are Strong Semi-Supervised Learners