Papers of the day   All papers

The importance of transparency and reproducibility in artificial intelligence research


Michael Hoffman: 1/Now on @arxiv: our letter about the recent @GoogleHealth cancer screening paper. "The importance of transparency and reproducibility in artificial intelligence research" by @bhaibeka @robtibshirani @HastieTrevor @johnquackenbush @hugo_aerts and others.

13 replies, 468 likes

J. Nathan Matias: If companies are going to claim that their products make advances that could save lives, those claims need to be independently verified. This group of scientists offers an important critique of an unverifiable study by Google in Nature in January

4 replies, 48 likes

Florian Markowetz 🇪🇺🇩🇪🇬🇧: This is a wide-spread problem. Here is another example with the same wording from

5 replies, 30 likes

Gary Collins 🇪🇺: Reporting guidelines are being developed ( to improve reporting/reproducibility of such AI studies extending, ( to improve AI research for patient benefit @bhaibeka @EQUATORNetwork

0 replies, 26 likes

Michael Hoffman: 2/We submitted the letter on 1 February. We posted a preprint @arxiv a month later, and I summarize that version of the letter and the problems with Google article and the authors' post-publication behavior this Twitter thread. Check it out.

1 replies, 25 likes

Gunnar Rätsch: I fully endorse the statements in a commentary ( discussing the difficulties of reproducing a study by McKinney et al. published in Nature

1 replies, 22 likes

Philipp Berens: This paper makes an extremely important point for applying #ArtificialIntelligence in #medicine: code and data have to be released as well (of course protecting patient rights in a suitable manner)

0 replies, 17 likes

Stephanie Hicks: @michaelhoffman @jtleek @arxiv @GoogleHealth @bhaibeka @robtibshirani @HastieTrevor @johnquackenbush @hugo_aerts @ahmedhosny @maqcsociety @LeviWaldron1 @BoWang87 @CmcintoshAi @anshulkundaje @GreeneScientist @wolfgangkhuber @ABrazma Fantastic work everyone! In case it's of interest, I'll just add @StrictlyStat and I discussed similar ideas about the @GoogleHealth cancer screening paper on @CorrespondAuth in our latest podcast episode

0 replies, 15 likes

Bo Wang: Reproducibility is essential for AI research!

0 replies, 12 likes

Indranil Mallick: Thanks @michaelhoffman. With ever larger numbers of medical researchers trying to get into #ai #DeepLearning based on these publications, it's time to realise that without external validity, your results are simply science fiction. #transparency

0 replies, 12 likes

The Corresponding Author: Ep 12 now out where @StrictlyStat @stephaniehicks discuss the importance of #reproducibleResearch in #AI re the @GoogleHealth #cancer screening paper 🎧 Also, incredibly cool to see this timely preprint from @bhaibekan et al! 📝

0 replies, 11 likes

Rasmus Kleis Nielsen: Science need to be more transparent and reproducable than this, especially when making great claims (that also happen to be aligned with self-interest).

0 replies, 10 likes

Dr Joseph Delaney: This is a must read and pretty concerning. I am not quite sure why the journal isn't concerned by this behavior by the authors. I understand some of this might corporate secrets, but that isn't how science works (i.e., you need direct replication)

1 replies, 9 likes

Andrew Pruszynski: Critically important work and thread.

0 replies, 8 likes

Prasath Lab CCHMC: Read the full thread! We, a small informatics lab (forget about acute facilities available for a moment), got frustrated when we tried to reproduce this study (among many other recent AI 'breakthrough' papers)! Pure frustration! don't want the data used, just better info. sigh...

1 replies, 7 likes

Anthony Mathelier: This thread!!! Again and again and again!!!

0 replies, 7 likes

Topic: David Ayala 🏳️‍🌈: THIS THREAD IS SO POWERFUL!!!!

0 replies, 6 likes

Casey Greene: I'm battling @michaelhoffman for the position of Middlest Author in this one. ⚔️⚔️

1 replies, 5 likes

Michael Hoffman: @ReplicabilityG @topher_batty @nbonneel @JulieDigne @dcoeurjo @nmellado0 Wow, I'm impressed that even 41% of papers had code available. Attitudes probably better than in machine learning.

0 replies, 5 likes

Imogen Stafford: A brilliant thread here about transparency and reproducibility in #ArtificialIntelligence for healthcare. How can we ensure correct patient care if we don't uphold these key scientific principles?

0 replies, 4 likes

Wiebke Toussaint: Just read this letter on #reproducibility of #AI research. Excellent and important read. The application of technology (in this case AI) in life-critical applications of any kind must be tested and validated. Healthcare is not advertising...

0 replies, 4 likes

Andres Marrugo: Nice thread and paper on computational reproducibility in AI research.

0 replies, 4 likes

Manideep Kolla: Great work @bhaibeka @michaelhoffman and team. This could well be one of the most important papers released this year (or in past few years). Reproducibility is so important, especially in AI research and it has many greater implications. I'm currently working on my thesis and...

1 replies, 3 likes

David McGaughey: IBM #Watson sequel? IBM got flamed for making #Watson AI claims that were unbound with reality. Without `code` and `data` impossible to evaluate performance.

1 replies, 3 likes

Julian García: "The importance of transparency and reproducibility in artificial intelligence research": Good stuff.

0 replies, 3 likes

Jessica Chong: a true service to science

0 replies, 3 likes

Meaux: Science has to be reproducible, tech architecture in life sciences requires planning & architectures around this problem. There’s systems like @quiltdata @pachyderminc & others that make this problem easier to implement & maintain. This thread discusses failure to do so..

0 replies, 2 likes

Neil Saunders: make sure to read entire thread!

0 replies, 2 likes

Lockman Fam: Is AI really better at detecting cancer than a radiologist? Hard to tell, as underlying published research doesn't pass the reproducibility test. This revisits the question, when is published research less science than advertising. What peer-review processes support the former?

0 replies, 2 likes

ReproHack ♻️: Have a listen to this episode of @CorrespondAuth on #ReproducibleResearch and deep learning for clinical applications! Interesting thoughts on sharing data, model weights and code (and which barriers some researchers face when they want to share those).

0 replies, 2 likes

Michael Hoffman: @DingoNates @drevney @deevybee @skdh @MedCrisis Preprint has also been on @arxiv since March:

0 replies, 2 likes

Kilian Vos: Great thread on the importance of shared #code and open data for reproductibility, key element to scientific progress #OpenSource #github #AcademicTwitter

0 replies, 1 likes

Brian Winey, PhD DABR: Transparency is vital for the translation of AI into the clinic, especially when it enters the triage and decision process.

0 replies, 1 likes

HotComputerScience: Most popular computer science paper of the day: "The importance of transparency and reproducibility in artificial intelligence research"

0 replies, 1 likes

Professor Booty PhD: Kind of a must read

0 replies, 1 likes

Mike: This thread. This is not a new problem. I've seen many papers where a certain method is listed as "manuscript in preparation". Of course, that MS never comes. It would be unacceptable to do this in the wet lab, and the dry lab should be no different.

0 replies, 1 likes


Found on Mar 10 2020 at

PDF content of a computer science paper: The importance of transparency and reproducibility in artificial intelligence research