Papers of the day   All papers

Big Self-Supervised Models are Strong Semi-Supervised Learners


Ting Chen: SimCLRv2: an improved self-supervised approach for semi-supervised learning. On ImageNet with 1% of the labels, it achieves 76.6% top-1, a 22% relative improvement over previous SOTA. Joint work with @skornblith, @kswersk, @mo_norouzi, @geoffreyhinton

2 replies, 426 likes

Denny Britz: “Bigger Models Are More Label-Efficient” - They beat supervised models on ImageNet with just 10% of the labels: OpenAI’s Image GPT made a similar point. Interesting b/c intuitively this is not obvious. Maybe bigger = representations are more disentangled?

9 replies, 323 likes

Sayak Paul: Here's a list of my favorite recent papers on transfer learning for vision: - BigTransfer: - VirTex: - SimCLRv2: - Self-training: Would love to see a T5-like paper for vision.

2 replies, 217 likes

Anand Agarwal: An improved approach for semi-supervised learning! Thrilled to see these new developments in #MachineLearning and self-paced #education

0 replies, 169 likes

Simon Kornblith: Unsupervised pretraining works much better for semi-supervised learning when the model is big. A model 10x bigger than ResNet-50 needs 10x fewer labels to get the same accuracy. But once the big model is trained, we can distill it to back to ResNet-50 with little loss in accuracy

0 replies, 77 likes

Mohammad Norouzi: Big Self-Supervised Models are Strong Semi-Supervised Learners.

0 replies, 64 likes

Simone Scardapane: *Big Self-Supervised Models are Strong Semi-Supervised Learners* by @tingchenai @skornblith @kswersk @mo_norouzi @geoffreyhinton Title says it all: scaling up SimCLR + distillation yields impressive semi-supervised results on ImageNet.

0 replies, 29 likes

Theodore Galanos: @Thom_Wolf I'm not sure this pertains only to CV, it doesn't really, but the self-supervised 'revolution' has been fascinating to watch. Too many papers to have here but some highlights are:,,,

0 replies, 27 likes

Blake Richards: This is really interesting, and leads me to emphasise something that we often forget when discussing human intelligence: the human brain has *way* more synapses than the biggest current ANNs. If we had the compute to do human brain size ANNs, I wonder what we could achieve?

7 replies, 26 likes

Evgenii Zheltonozhskii: Self-supervised learning sees a strong boost in performance: first BYOD and now SimCLRv2 by @tingchenai @skornblith @kswersk @mo_norouzi @geoffreyhinton. Starts to make sense to talk about top-1 performace on 1% of ImageNet -- 76.6% for top model.

1 replies, 20 likes

Emtiyaz Khan: Perhaps bigger models give rise to smoother functions? This is an interesting study nevertheless.

0 replies, 10 likes

Aran Komatsuzaki: Big Self-Supervised Models are Strong Semi-Supervised Learners SimCLRv2 on Resnet-50 achieves 73.9% top-1 accuracy in Imagenet with 1% labels, a 10× improvement in label efficiency over the previous sota

2 replies, 10 likes

Janne Spijkervet: During the 3-month lockdown, research on self-supervised representation learning has surely not halted! After reading the fascinating BYOL this morning from @DeepMind , I find SimCLRv2 in my inbox this evening by @tingchenai et al.! :)

0 replies, 5 likes

Brundage Bot: Big Self-Supervised Models are Strong Semi-Supervised Learners. Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton

1 replies, 4 likes

akira: They proposed SimCLRv2, which uses only a small number of labels and performs as well or better than supervised learning. It consists of three stages: unsupervised learning, FineTune, and self-training distillation using unlabeled data.

0 replies, 3 likes

Vicente Reyes-Puerta: This updated framework takes the “unsupervised pretrain, supervised fine-tune” paradigm 👍 for me it is like a massive transfer learning approach 🤔 great paper, waiting for the pretrained model to be available on @github 😁 #DeepLearning #MachineLearning

0 replies, 1 likes

Peter Organisciak: Somewhat expected, but interesting find from image ML - the bigger a pre-trained model gets, the fewer labels fine-tuning needs. So you can compensate for few labels with a bigger model.

1 replies, 1 likes


Found on Jun 19 2020 at

PDF content of a computer science paper: Big Self-Supervised Models are Strong Semi-Supervised Learners