Papers of the day   All papers

A Metric Learning Reality Check

Comments

Eric Jang 🇺🇸🇹🇼: Every once in awhile a paper comes out that makes you breathe a sigh of relief that you don't publish in that field... https://arxiv.org/pdf/2003.08505.pdf "Our results show that when hyperparameters are properly tuned via cross-validation, most methods perform similarly to one another" https://t.co/bnG1bm265p

32 replies, 1846 likes


Zachary Lipton: I suspect most of us doing deep learning are in "that field". Similar qualitatively to language modeling gains attributable to hyperparameter tuning (Melis et al. https://arxiv.org/abs/1707.05589) & other examples we discuss in "Troubling Trends in ML Scholarship"—https://arxiv.org/abs/1807.03341

3 replies, 292 likes


(((ل()(ل() 'yoav)))): (a) important work; (b) is anyone really surprised? i'd imagine this general trend will hold for any task and/or metric people are hill climbing on with DL.

9 replies, 135 likes


Denny Britz: A Metric Learning Reality Check: “[…] state of the art loss functions perform marginally better than, and sometimes on par with, classic methods” 🔥 https://arxiv.org/abs/2003.08505 https://t.co/DACbVP9Eoi

2 replies, 68 likes


Kevin Musgrave: Paper update thread👀 Back in November I wrote an article to point out the problems in deep metric learning https://medium.com/@tkm45/benchmarking-metric-learning-algorithms-the-right-way-90c073a83968 The follow up to this was A Metric Learning Reality Check, and the latest version of the paper was uploaded today. https://arxiv.org/pdf/2003.08505.pdf (1/N) https://t.co/yNNRXRcvyc

2 replies, 49 likes


Edward Grefenstette: Love this. Similar result (different domain) to Melis et al 2017: https://arxiv.org/abs/1707.05589. That paper was pearls before swine for the EMNLP reviewers that read it. Also echos stuff that @Smerity has observed, I believe.

0 replies, 37 likes


Thomas Wolf: @dennybritz So many. Here are a few: CV: https://arxiv.org/abs/1910.04867 Standard splits: https://www.aclweb.org/anthology/P19-1267 Metric Learning: https://arxiv.org/abs/2003.08505 IR: https://sigir.org/wp-content/uploads/2019/01/p040.pdf Pruning: https://arxiv.org/abs/2003.03033 Summarization: http://arxiv.org/abs/1908.08960 Lipton et al: http://arxiv.org/abs/1807.03341 & RL...

0 replies, 34 likes


Serge Belongie: ECCV 2020 version of “Metric Learning Reality Check” now available on arXiv @CS_Cornell collaboration w/@FacebookAI

0 replies, 33 likes


Daniel Lemire: “Deep metric learning papers from the past four years have consistently claimed great advances in accuracy (…), we present experimental results that show that the improvements over time have been marginal at best.“ https://arxiv.org/pdf/2003.08505.pdf via @vhranger

0 replies, 16 likes


X.A. staying 🏡 + 😷 saves lives: "Deep metric learning papers from the past four years have consistently claimed great advances in accuracy(....)We find flaws (...), and (...) show that the improvements over time have been marginal at best."😳https://arxiv.org/abs/2003.08505 - [PAPER] by folks at Cornell and @facebookai

0 replies, 14 likes


Daniël Lakens: There is an important lesson here about reward structures and knowledge generation in science.

0 replies, 13 likes


Andrey Kurenkov 🤖: "We find flaws in the experimental setup of these papers, and propose a new way to evaluate metric learning algorithms. ... improvements over time have been marginal at be" As a largely empirical field, AI needs to get better an empirical research. And to do that, slow down!

0 replies, 12 likes


arxiv: A Metric Learning Reality Check. http://arxiv.org/abs/2003.08505 https://t.co/ynFlk1v35W

0 replies, 12 likes


Tiago Ramalho: A Metric Learning Reality Check "Our results show that when hyperparameters are properly tuned via cross-validation, most methods perform similarly to one another. " https://arxiv.org/abs/2003.08505 https://t.co/eVGtvMuGHx

1 replies, 10 likes


Karl Higley: If you’re learning embeddings for recommendation candidate selection via approximate nearest neighbor search, turns out cutting edge loss functions from the past few years may not be much better than triplet loss (from 2006.)

1 replies, 10 likes


Dagmar Monett: "We find that [20 #ML #AI] papers have drastically overstated improvements ... If a paper attempts to explain the performance gains of its proposed method, & it turns out that [those gains] are non-existent, then their explanation must be invalid as well." https://arxiv.org/pdf/2003.08505

1 replies, 9 likes


Arthur Douillard: A reality check on #DeepLearning Metric Learning: https://arxiv.org/abs/2003.08505 The authors show that the progress of the last decade is mainly due to: 1. Hyperparameters tuning on the test set. 2. Better architectures (GoogleNet -> BN-Inception) I'm astonished! https://t.co/AzOX93QloB

1 replies, 8 likes


Jason Baldridge: Calls @DaniYogatama, @ikekong and @nlpnoah (2015) to mind. https://www.aclweb.org/anthology/D15-1251/

1 replies, 7 likes


Serge Belongie: Metric Learning Reality Check paper @LightningSource @CornellCIS @sernamlim @facebookai in the news https://arxiv.org/abs/2003.08505

0 replies, 7 likes


Mark Nelson: @dennybritz A few: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewPaper/16669 http://gradientscience.org/data_rep_bias/ https://arxiv.org/abs/2003.08505

0 replies, 5 likes


Hamid EBZD: "A Metric Learning Reality Check" https://arxiv.org/abs/2003.08505

0 replies, 4 likes


Sotirios (Sotos) Tsaftaris: When experiments are done on equal footing and each method has proper hyperparameters tuned there is no progress in performance in metric learning. A valuable lesson for everyone working in ML.

0 replies, 4 likes


Kwang Moo Yi: Amazing effort here. Benchmarking IS inportant. Sort of what we also experienced in our recent effort :)

0 replies, 4 likes


arXiv CS-CV: A Metric Learning Reality Check http://arxiv.org/abs/2003.08505

0 replies, 4 likes


Daniel Lowd: “If a paper attempts to explain the performance gains of its proposed method, and it turns out that those performance gains are non-existent, then their explanation must be invalid as well.”

0 replies, 3 likes


ML and Data Projects To Know: 📙 A Metric Learning Reality Check Authors: @LightningSource, @SergeBelongie, @sernamlim Paper: https://arxiv.org/pdf/2003.08505.pdf https://t.co/jVDRDd9PgC

0 replies, 3 likes


Dmytro Mishkin: @ogrisel https://arxiv.org/abs/2003.08505

0 replies, 3 likes


Robert (Munro) Monarch: Wow! When you properly account for tuning and data treatment, there’s been no gain for common computer vision tasks for 15 years. From @LightningSource @SergeBelongie & Ser-Nam Lim, who are presumably under police protection from the ML community https://arxiv.org/pdf/2003.08505.pdf https://t.co/RwaawZPWAl

0 replies, 2 likes


Magda Paschali: A much needed reality check! Intriguing findings and lots of valuable tips for experiment standarization and fairness from @LightningSource et al. https://arxiv.org/abs/2003.08505 https://github.com/KevinMusgrave/powerful-benchmarker

0 replies, 2 likes


MONTREAL.AI: A Metric Learning Reality Check Musgrave et al.: https://arxiv.org/abs/2003.08505 https://arxiv.org/pdf/2003.08505.pdf… "Our results show that when hyperparameters are properly tuned via cross-validation, most methods perform similarly to one another" #ArtificialIntelligence #DeepLearning https://t.co/AmAoHKY83r

0 replies, 1 likes


Akshaj Verma: Press F to pay respects #DeepLearning #MachineLearning

0 replies, 1 likes


QUARANTANAMO: machine learning OR HYPER-PARAMETER OPTIMIZATION you decide

0 replies, 1 likes


Andrew Beam: @tammy_jiang I would say most machine learning papers use these in some form *except* preregistration, which can cause... problems: https://arxiv.org/pdf/2003.08505.pdf

1 replies, 1 likes


jhofmanninger@gmail.com: Sad but not surprising. I suppose similar results can be found for many other tasks

0 replies, 1 likes


عمر فرؤق: Leaderboard Driven Development.

1 replies, 0 likes


Content

Found on May 09 2020 at https://arxiv.org/pdf/2003.08505.pdf

PDF content of a computer science paper: A Metric Learning Reality Check