Papers of the day   All papers

Reliance on Metrics is a Fundamental Challenge for AI


Rachel Thomas: The Problem with Metrics is a Fundamental Problem for AI-- paper by me and David Uminsky @DataInstituteSF accepted to EDSC 2020 cc: @craignewmark

9 replies, 402 likes

Rachel Thomas: For those interested in a more academic reference on problems with overemphasizing metrics, please check out my paper (with David Uminsky) 10/

1 replies, 29 likes

Rachel Thomas: which I then expanded in this paper, with more tie-ins to existing literature, more examples, and positive steps:

1 replies, 26 likes

Rachel Thomas: @m_sendhil Thank you for sharing the post Sendhil! I also expanded these ideas into a paper (together with David Uminsky):

0 replies, 25 likes

DataScienceNigeria: Don’t be gamed by AI metrics! Good analytics must be more than optimized metrics. We must humanize our algorithms with context & qualitative insights. WHY? -once a metric becomes a goal, it ceases to be a metric. Read this great work by @math_rachel

0 replies, 22 likes

David Manheim: Good to see people more engaging with this important topic!

1 replies, 8 likes

Rachel Thomas: In this thread, study author @Amanda_Lenhart lists all the researchers linked in the report (my metrics paper is linked in the metrics myth

0 replies, 8 likes

Nick Weir: This has (rightfully) been getting a ton of attention in my TL for its excellent perspective. But IMO, a lot of readers' takes badly miss the point. 1/14

1 replies, 7 likes

Brandon Rohrer: “The unreasonable effectiveness of metric optimization in current AI approaches is a fundamental challenge to the field, and yields an inherent contradiction: solely optimizing metrics leads to far from optimal outcomes.” -@math_rachel and David Uminsky

0 replies, 6 likes

Carlos E. Perez: Interesting paper by @math_rachel that explores the problem with over emphasis of metrics in machine learning (covers Goodhart's Law)

0 replies, 6 likes

Jeff Emmett 🌱: Metrics are just a proxy for what we really care about & can easily be gamed. How do we get healthier metrics? 1. Use a slate of criteria for a fuller picture 2. Combine with qualitative accounts 3. Involve a range of stakeholders, including those who will be most impacted

1 replies, 6 likes

Ben Olsen: Over-indexing on a single metric has always been a problem of #datascience and #analytics - when we move to #AI / #ML, the problem persists. Great work here by @math_rachel and David Uminsky @DataInstituteSF on mitigations and concrete case studies

1 replies, 5 likes

Amanda Lenhart: But metrics aren’t objective & can’t always measure things we value (e.g., wellbeing, human connection). Metrics can miss important groups of users, & lead to bias, discrimination and “a myopic focus on short-term goals.” [12/21]

1 replies, 4 likes

Rachel Thomas: @micbucci Please check out my paper on problems with overemphasizing metrics (& the cited references):

0 replies, 3 likes

i miss the framebuffer: what about 4: rethinking the uses of metrication and “AI” at all?

1 replies, 1 likes

午後のarXiv: "The Problem with Metrics is a Fundamental Problem for AI", Rachel Thomas, David Uminsky

0 replies, 1 likes

Internet Ethics: #ethics #AI #data #tech

0 replies, 1 likes


Found on Feb 21 2020 at

PDF content of a computer science paper: Reliance on Metrics is a Fundamental Challenge for AI