Ting Chen: SimCLRv2: an improved self-supervised approach for semi-supervised learning. On ImageNet with 1% of the labels, it achieves 76.6% top-1, a 22% relative improvement over previous SOTA.
Joint work with @skornblith, @kswersk, @mo_norouzi, @geoffreyhinton https://t.co/Fv6IyFCpNu
2 replies, 426 likes
Denny Britz: “Bigger Models Are More Label-Efficient” - They beat supervised models on ImageNet with just 10% of the labels: https://arxiv.org/abs/2006.10029
OpenAI’s Image GPT made a similar point. Interesting b/c intuitively this is not obvious. Maybe bigger = representations are more disentangled? https://t.co/f1QJawYO0O
9 replies, 323 likes
Sayak Paul: Here's a list of my favorite recent papers on transfer learning for vision:
- BigTransfer: https://arxiv.org/abs/1912.11370
- VirTex: https://arxiv.org/abs/2006.06666
- SimCLRv2: https://arxiv.org/abs/2006.10029
- Self-training: https://arxiv.org/abs/2006.06882
Would love to see a T5-like paper for vision.
2 replies, 217 likes
Anand Agarwal: An improved approach for semi-supervised learning! Thrilled to see these new developments in #MachineLearning and self-paced #education
0 replies, 169 likes
Simon Kornblith: Unsupervised pretraining works much better for semi-supervised learning when the model is big. A model 10x bigger than ResNet-50 needs 10x fewer labels to get the same accuracy. But once the big model is trained, we can distill it to back to ResNet-50 with little loss in accuracy
0 replies, 77 likes
Mohammad Norouzi: Big Self-Supervised Models are Strong Semi-Supervised Learners.
0 replies, 64 likes
Simone Scardapane: *Big Self-Supervised Models are Strong Semi-Supervised Learners*
by @tingchenai @skornblith @kswersk @mo_norouzi @geoffreyhinton
Title says it all: scaling up SimCLR + distillation yields impressive semi-supervised results on ImageNet.
0 replies, 29 likes
Theodore Galanos: @Thom_Wolf I'm not sure this pertains only to CV, it doesn't really, but the self-supervised 'revolution' has been fascinating to watch. Too many papers to have here but some highlights are: https://arxiv.org/abs/1911.05722, https://arxiv.org/abs/2002.05709, https://arxiv.org/abs/2004.11362, https://arxiv.org/abs/2006.10029
0 replies, 27 likes
Blake Richards: This is really interesting, and leads me to emphasise something that we often forget when discussing human intelligence: the human brain has *way* more synapses than the biggest current ANNs. If we had the compute to do human brain size ANNs, I wonder what we could achieve?
7 replies, 26 likes
Evgenii Zheltonozhskii: Self-supervised learning sees a strong boost in performance: first BYOD and now SimCLRv2 by @tingchenai @skornblith @kswersk @mo_norouzi @geoffreyhinton. Starts to make sense to talk about top-1 performace on 1% of ImageNet -- 76.6% for top model.
1 replies, 20 likes
Emtiyaz Khan: Perhaps bigger models give rise to smoother functions? This is an interesting study nevertheless.
0 replies, 10 likes
Aran Komatsuzaki: Big Self-Supervised Models are Strong Semi-Supervised Learners
SimCLRv2 on Resnet-50 achieves 73.9% top-1 accuracy in Imagenet with 1% labels, a 10× improvement in label efficiency over the previous sota
2 replies, 10 likes
Janne Spijkervet: During the 3-month lockdown, research on self-supervised representation learning has surely not halted! After reading the fascinating BYOL this morning from @DeepMind , I find SimCLRv2 in my inbox this evening by @tingchenai et al.! :) https://arxiv.org/pdf/2006.10029.pdf
0 replies, 5 likes
Brundage Bot: Big Self-Supervised Models are Strong Semi-Supervised Learners. Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton http://arxiv.org/abs/2006.10029
1 replies, 4 likes
They proposed SimCLRv2, which uses only a small number of labels and performs as well or better than supervised learning. It consists of three stages: unsupervised learning, FineTune, and self-training distillation using unlabeled data. https://t.co/KjLY9ufPUF
0 replies, 3 likes
Vicente Reyes-Puerta: This updated framework takes the “unsupervised pretrain, supervised fine-tune” paradigm 👍 for me it is like a massive transfer learning approach 🤔 great paper, waiting for the pretrained model to be available on @github 😁
0 replies, 1 likes
Peter Organisciak: Somewhat expected, but interesting find from image ML - the bigger a pre-trained model gets, the fewer labels fine-tuning needs. https://arxiv.org/abs/2006.10029 So you can compensate for few labels with a bigger model.
1 replies, 1 likes
Found on Jun 19 2020 at https://arxiv.org/pdf/2006.10029.pdf