Papers of the day   All papers

The importance of transparency and reproducibility in artificial intelligence research

Comments

Michael Hoffman: 1/Now on @arxiv: our letter about the recent @GoogleHealth cancer screening paper. "The importance of transparency and reproducibility in artificial intelligence research" by @bhaibeka @robtibshirani @HastieTrevor @johnquackenbush @hugo_aerts and others. https://arxiv.org/abs/2003.00898 https://t.co/Iea8t5YCGG

13 replies, 458 likes


J. Nathan Matias: If companies are going to claim that their products make advances that could save lives, those claims need to be independently verified. This group of scientists offers an important critique of an unverifiable study by Google in Nature in January

4 replies, 48 likes


Florian Markowetz 🇪🇺🇩🇪🇬🇧: This is a wide-spread problem. Here is another example with the same wording from https://www.nature.com/articles/s41591-019-0583-3 https://t.co/rGlc2FJN5i

5 replies, 30 likes


Gary Collins 🇪🇺: Reporting guidelines are being developed (https://tinyurl.com/y2fjohwo) to improve reporting/reproducibility of such AI studies extending https://tinyurl.com/y4kxcm8n, (https://tinyurl.com/y2mkd3tx) to improve AI research for patient benefit https://arxiv.org/abs/1812.10404 @bhaibeka @EQUATORNetwork

0 replies, 26 likes


Gunnar Rätsch: I fully endorse the statements in a commentary (https://arxiv.org/pdf/2003.00898.pdf) discussing the difficulties of reproducing a study by McKinney et al. published in Nature https://www.nature.com/articles/s41586-019-1799-6.

1 replies, 22 likes


Philipp Berens: This paper makes an extremely important point for applying #ArtificialIntelligence in #medicine: code and data have to be released as well (of course protecting patient rights in a suitable manner)

0 replies, 17 likes


Stephanie Hicks: @michaelhoffman @jtleek @arxiv @GoogleHealth @bhaibeka @robtibshirani @HastieTrevor @johnquackenbush @hugo_aerts @ahmedhosny @maqcsociety @LeviWaldron1 @BoWang87 @CmcintoshAi @anshulkundaje @GreeneScientist @wolfgangkhuber @ABrazma Fantastic work everyone! In case it's of interest, I'll just add @StrictlyStat and I discussed similar ideas about the @GoogleHealth cancer screening paper on @CorrespondAuth in our latest podcast episode https://twitter.com/CorrespondAuth/status/1237416755921731584

0 replies, 15 likes


Indranil Mallick: Thanks @michaelhoffman. With ever larger numbers of medical researchers trying to get into #ai #DeepLearning based on these publications, it's time to realise that without external validity, your results are simply science fiction. #transparency

0 replies, 12 likes


Bo Wang: Reproducibility is essential for AI research!

0 replies, 12 likes


The Corresponding Author: Ep 12 now out where @StrictlyStat @stephaniehicks discuss the importance of #reproducibleResearch in #AI re the @GoogleHealth #cancer screening paper 🎧 https://soundcloud.com/the-corresponding-author/episode-12-deep-learning-and-ai-thoughts Also, incredibly cool to see this timely preprint from @bhaibekan et al! 📝 https://arxiv.org/abs/2003.00898

0 replies, 11 likes


Rasmus Kleis Nielsen: Science need to be more transparent and reproducable than this, especially when making great claims (that also happen to be aligned with self-interest).

0 replies, 10 likes


Dr Joseph Delaney: This is a must read and pretty concerning. I am not quite sure why the journal isn't concerned by this behavior by the authors. I understand some of this might corporate secrets, but that isn't how science works (i.e., you need direct replication)

1 replies, 9 likes


Andrew Pruszynski: Critically important work and thread.

0 replies, 8 likes


Prasath Lab CCHMC: Read the full thread! We, a small informatics lab (forget about acute facilities available for a moment), got frustrated when we tried to reproduce this study (among many other recent AI 'breakthrough' papers)! Pure frustration! don't want the data used, just better info. sigh...

1 replies, 7 likes


Anthony Mathelier: This thread!!! Again and again and again!!!

0 replies, 7 likes


Topic: David Ayala 🏳️‍🌈: THIS THREAD IS SO POWERFUL!!!!

0 replies, 6 likes


Michael Hoffman: @ReplicabilityG @topher_batty @nbonneel @JulieDigne @dcoeurjo @nmellado0 Wow, I'm impressed that even 41% of papers had code available. Attitudes probably better than in machine learning. https://arxiv.org/abs/2003.00898

0 replies, 5 likes


Casey Greene: I'm battling @michaelhoffman for the position of Middlest Author in this one. ⚔️⚔️

1 replies, 5 likes


Wiebke Toussaint: Just read this letter on #reproducibility of #AI research. Excellent and important read. The application of technology (in this case AI) in life-critical applications of any kind must be tested and validated. Healthcare is not advertising...

0 replies, 4 likes


Imogen Stafford: A brilliant thread here about transparency and reproducibility in #ArtificialIntelligence for healthcare. How can we ensure correct patient care if we don't uphold these key scientific principles?

0 replies, 4 likes


Andres Marrugo: Nice thread and paper on computational reproducibility in AI research.

0 replies, 4 likes


Manideep Kolla: Great work @bhaibeka @michaelhoffman and team. This could well be one of the most important papers released this year (or in past few years). Reproducibility is so important, especially in AI research and it has many greater implications. I'm currently working on my thesis and...

1 replies, 3 likes


David McGaughey: IBM #Watson sequel? IBM got flamed for making #Watson AI claims that were unbound with reality. Without `code` and `data` impossible to evaluate performance.

1 replies, 3 likes


Julian García: "The importance of transparency and reproducibility in artificial intelligence research": https://arxiv.org/abs/2003.00898 Good stuff.

0 replies, 3 likes


Jessica Chong: a true service to science

0 replies, 3 likes


ReproHack ♻️: Have a listen to this episode of @CorrespondAuth on #ReproducibleResearch and deep learning for clinical applications! Interesting thoughts on sharing data, model weights and code (and which barriers some researchers face when they want to share those).

0 replies, 2 likes


Lockman Fam: Is AI really better at detecting cancer than a radiologist? Hard to tell, as underlying published research doesn't pass the reproducibility test. This revisits the question, when is published research less science than advertising. What peer-review processes support the former?

0 replies, 2 likes


Meaux: Science has to be reproducible, tech architecture in life sciences requires planning & architectures around this problem. There’s systems like @quiltdata @pachyderminc & others that make this problem easier to implement & maintain. This thread discusses failure to do so..

0 replies, 2 likes


Kilian Vos: Great thread on the importance of shared #code and open data for reproductibility, key element to scientific progress #OpenSource #github #AcademicTwitter

0 replies, 1 likes


Brian Winey, PhD DABR: Transparency is vital for the translation of AI into the clinic, especially when it enters the triage and decision process.

0 replies, 1 likes


Professor Booty PhD: Kind of a must read

0 replies, 1 likes


HotComputerScience: Most popular computer science paper of the day: "The importance of transparency and reproducibility in artificial intelligence research" https://hotcomputerscience.com/paper/the-importance-of-transparency-and-reproducibility-in-artificial-intelligence-research https://twitter.com/michaelhoffman/status/1237349469118586891

0 replies, 1 likes


Mike: This thread. This is not a new problem. I've seen many papers where a certain method is listed as "manuscript in preparation". Of course, that MS never comes. It would be unacceptable to do this in the wet lab, and the dry lab should be no different.

0 replies, 1 likes


Content

Found on Mar 10 2020 at https://arxiv.org/pdf/2003.00898.pdf

PDF content of a computer science paper: The importance of transparency and reproducibility in artificial intelligence research