Papers of the day   All papers

Reliance on Metrics is a Fundamental Challenge for AI

Comments

Rachel Thomas: The Problem with Metrics is a Fundamental Problem for AI-- paper by me and David Uminsky @DataInstituteSF accepted to EDSC 2020 https://arxiv.org/abs/2002.08512 cc: @craignewmark https://t.co/UzFP24ySni

9 replies, 402 likes


Rachel Thomas: For those interested in a more academic reference on problems with overemphasizing metrics, please check out my paper (with David Uminsky) 10/ https://twitter.com/math_rachel/status/1230666703656251392?s=19

1 replies, 29 likes


Rachel Thomas: which I then expanded in this paper, with more tie-ins to existing literature, more examples, and positive steps: https://twitter.com/math_rachel/status/1230666703656251392?s=20

1 replies, 26 likes


Rachel Thomas: @m_sendhil Thank you for sharing the post Sendhil! I also expanded these ideas into a paper (together with David Uminsky): https://twitter.com/math_rachel/status/1230666703656251392?s=20

0 replies, 25 likes


DataScienceNigeria: Don’t be gamed by AI metrics! Good analytics must be more than optimized metrics. We must humanize our algorithms with context & qualitative insights. WHY? -once a metric becomes a goal, it ceases to be a metric. Read this great work by @math_rachel https://arxiv.org/pdf/2002.08512.pdf https://t.co/opE7sktsri

0 replies, 22 likes


David Manheim: Good to see people more engaging with this important topic!

1 replies, 8 likes


Rachel Thomas: In this thread, study author @Amanda_Lenhart lists all the researchers linked in the report (my metrics paper is linked in the metrics myth https://arxiv.org/abs/2002.08512): https://twitter.com/Amanda_Lenhart/status/1314597183518375936?s=20

0 replies, 8 likes


Nick Weir: This has (rightfully) been getting a ton of attention in my TL for its excellent perspective. But IMO, a lot of readers' takes badly miss the point. 1/14

1 replies, 7 likes


Brandon Rohrer: “The unreasonable effectiveness of metric optimization in current AI approaches is a fundamental challenge to the field, and yields an inherent contradiction: solely optimizing metrics leads to far from optimal outcomes.” -@math_rachel and David Uminsky

0 replies, 6 likes


Carlos E. Perez: Interesting paper by @math_rachel that explores the problem with over emphasis of metrics in machine learning (covers Goodhart's Law) https://arxiv.org/abs/2002.08512

0 replies, 6 likes


Jeff Emmett 🌱: Metrics are just a proxy for what we really care about & can easily be gamed. How do we get healthier metrics? 1. Use a slate of criteria for a fuller picture 2. Combine with qualitative accounts 3. Involve a range of stakeholders, including those who will be most impacted

1 replies, 6 likes


Ben Olsen: Over-indexing on a single metric has always been a problem of #datascience and #analytics - when we move to #AI / #ML, the problem persists. Great work here by @math_rachel and David Uminsky @DataInstituteSF on mitigations and concrete case studies

1 replies, 5 likes


Amanda Lenhart: But metrics aren’t objective & can’t always measure things we value (e.g., wellbeing, human connection). Metrics can miss important groups of users, & lead to bias, discrimination and “a myopic focus on short-term goals.” [12/21] https://arxiv.org/pdf/2002.08512.pdf

1 replies, 4 likes


Rachel Thomas: @micbucci Please check out my paper on problems with overemphasizing metrics (& the cited references): https://twitter.com/math_rachel/status/1230666703656251392?s=19

0 replies, 3 likes


i miss the framebuffer: what about 4: rethinking the uses of metrication and “AI” at all?

1 replies, 1 likes


午後のarXiv: "The Problem with Metrics is a Fundamental Problem for AI", Rachel Thomas, David Uminsky https://arxiv.org/abs/2002.08512

0 replies, 1 likes


Internet Ethics: #ethics #AI #data #tech

0 replies, 1 likes


Content

Found on Feb 21 2020 at https://arxiv.org/pdf/2002.08512.pdf

PDF content of a computer science paper: Reliance on Metrics is a Fundamental Challenge for AI