Papers of the day   All papers

SYSTEMATIC OVERESTIMATION OF MACHINE LEARNING PERFORMANCE IN NEUROIMAGING STUDIES OF DEPRESSION

Comments

Simon Eickhoff: Diagnosis of major depression from structural MRI by machine-learning. Sobering result: 61% accuracy with N=1868 Mimicked smaller studies by subsampling reveals strong effect of training AND test size. Even with N~100, estimated accuracies vary by >20% https://arxiv.org/abs/1912.06686 https://t.co/g2oK2jKu0G

15 replies, 233 likes


Tim Dalgleish: Large scale (N=1868) machine-learning study reveals very poor performance – 61% – in a binary discrimination of depression/no depression, from MRI data. Suggests that the many smaller N studies with much higher predictive power are likely inflated. Not surprised ...

4 replies, 99 likes


Eiko Fried: After 2 "decades of the brain": 1. Accuracy far too low for clinical implementation. 2. Studies test healthy vs depressed groups. Since results likely generalize to other disorders, unclear what we learn about depression in absence of comparing to e.g. anxiety disorders.

3 replies, 89 likes


David Mehler: Latest preprint, showing that test sets require large N to avoid overstimating classification accuraries. Fantastic work by Claas Flint @cl445 & team @UK_Muenster in collabo @INM7_ISN + @TheFlorey #MachineLearning cc @KordingLab @talyarkoni @danilobzdok https://arxiv.org/abs/1912.06686?fbclid=IwAR1-6yqfA_H4SThVcEOAok71q6DQn6CK4PSsh3HLPmyROzQjSEYIIerONNQ https://t.co/lOYkCMqDFt

6 replies, 59 likes


Allen Frances: Lots of "coolest" psychiatry research is useless because it amounts to mindless playing with technical toys (eg MRI & Machine Learning) By combining them & torturing the data, you can publish whole bunch of papers & get grants & promotions. But findings never seem to replicate.

3 replies, 50 likes


Rick Adams: Great study. Time to give up doing case v control classification studies of psychiatric diagnoses using sMRI. All elements of this are flawed: the diagnoses themselves, case v control isn’t clinically useful*, and structural MRI seems so unlikely to hold answers...

1 replies, 42 likes


Allen Frances: Machines will soon beat humans at psych diagnosis: 1)more systematic 2)more data 3) much bigger samples 3)less bias 4)can detect new patterns But current studies: 1) have tiny n's 2)small data base 3)ignore complexity of comorbidity

14 replies, 25 likes


Michael P. Hengartner, PhD: Excellent study with very important implications: 1) MRI has no diagnostic validity in depression, 2) Small studies generate unreliable and severely overestimated effects (as explained and predicted by Button et al https://www.nature.com/articles/nrn3475)

1 replies, 21 likes


KordingLab π›βˆˆπŸ§ : Why do so many small sample fMRI classification studies show such high % correct? Great new analysis. Relates to yesterday's decision on biomarkers.

0 replies, 19 likes


Rebecca Todd: Important reality check.

0 replies, 8 likes


James C.Coyne: Why is this kind of thing still undertaken? Results still inferior to 2 question depression screening, which sucks.

0 replies, 3 likes


Chadi Abdallah: Low but realistic accuracy. It is unlikely that crude estimate of brain structure would explain high variance of complex behavioral syndrome like depression. As a field, it's about time we let go of "diagnosis" and start focusing on what clinically matter -- treatment biomarkers

0 replies, 2 likes


Content

Found on Mar 06 2020 at https://arxiv.org/pdf/1912.06686.pdf

PDF content of a computer science paper: SYSTEMATIC OVERESTIMATION OF MACHINE LEARNING PERFORMANCE IN NEUROIMAGING STUDIES OF DEPRESSION