Jason Lee: Predicting What You Already Know Helps: Provable Self-Supervised Learning
We analyze how predicting parts of the input from other parts (missing patch, missing word, etc.) helps to learn a representation that linearly separates the downstream task.
https://arxiv.org/abs/2008.01064 1/2 https://t.co/ahjaONZlfI
1 replies, 540 likes
Stanford NLP Group: Wonderful to see some theory behind the great success of self-supervised learning. Still trying to get our slow brains around how strong the results are. Cameo for the Stanford Sentiment Treebank—can it become the MNIST of #NLProc? By @jasondeanlee & al. https://arxiv.org/abs/2008.01064 https://t.co/TZd6vL3E7K
1 replies, 191 likes
Phillip Isola: Nice to see more theory on this.
Paraphrasing: the only way to correctly colorize pikachu yellow is to first implicitly recognize that you are looking at a picture of pikachu!
2 replies, 104 likes
Sham Kakade: Great to see some theory on self-supervised learning. Looking forward to reading this one!
1 replies, 55 likes
Qi Lei: When the label (pikachu) captures some of the joint information between the input image and the image patch, predicting the missing part (self-supervised learning) implicitly learns the label.
0 replies, 9 likes
MONTREAL.AI: Predicting What You Already Know Helps: Provable Self-Supervised Learning
Lee et al.: https://arxiv.org/abs/2008.01064
#DeepLearning #MachineLearning #SelfSupervisedLearning https://t.co/glWcB6sz2r
0 replies, 8 likes
Kevin Yang 楊凱筌: Pretraining helps because of approximate independence between inputs and the pretext task given the labels for the downstream task.
@jasondeanlee @QiLei45724485 Nikunj Saunshi, and Jiacheng Zhuo
0 replies, 1 likes
Found on Aug 19 2020 at https://arxiv.org/pdf/2008.01064.pdf