Papers of the day   All papers

The Measure of Intelligence


Brian Roemmele: Consider: millions of years ago our antecedents gave a massive sacrifice of their left hemisphere. We lost a tremendous amount of short term memory and replaced it with Broca’s, Wernicke & the phonological loop. But why? So we can—talk. Thus chimpanzees can do this—we can’t:

846 replies, 94703 likes

François Chollet: I've just released a fairly lengthy paper on defining & measuring intelligence, as well as a new AI evaluation dataset, the "Abstraction and Reasoning Corpus". I've been working on this for the past 2 years, on & off. Paper: ARC:

94 replies, 4253 likes

Quoc Le: Want to improve accuracy and robustness of your model? Use unlabeled data! Our new work uses self-training on unlabeled data to achieve 87.4% top-1 on ImageNet, 1% better than SOTA. Huge gains are seen on harder benchmarks (ImageNet-A, C and P). Link:

22 replies, 1416 likes

Emil Wallner: François Chollet’s core point: We can't measure an AI system's adaptability and flexibility by measuring a specific skill. With unlimited data, models memorize decisions. To advance AGI we need to quantify and measure ***skill-acquisition efficiency***. Let’s dig in👇

14 replies, 1089 likes

François Chollet: We're launching a Kaggle competition on the Abstraction & Reasoning Corpus (ARC)! 3 months duration, $20,000 in prizes. Good luck :)

13 replies, 944 likes

hardmaru: Excited to see a @Kaggle competition based on the Abstraction and Reasoning Corpus by @fchollet in “On the Measure of Intelligence” (

2 replies, 558 likes

Gary Marcus: Mini thread: If you haven't already @FChollet's beautiful & insightful paper on intelligence & AI, you should. An elegant distillation of where we are now, & an intriguing proposal for how to make progress.

8 replies, 457 likes

Aurélien Geron: I greatly enjoyed this paper! 👍 It provides conceptual and technical tools to progress towards AGI. I wrote a TFDS dataset so you can load it as a TF dataset simply with tfds.load("arc"). You can play with it in this Colab notebook:

1 replies, 253 likes

Yann LeCun: There can be no intelligence without learning.

29 replies, 244 likes

Noah Guzmán: I really dislike Kaggle under most circumstances, but this is quite an interesting problem. I will likely submit an attempt involving cellular automata.

6 replies, 238 likes

Brandon Rohrer: If you like to think deeply about building and measuring intelligence, I highly recommend this read. It's a cogent review of intelligence measurement, it's challenges, past approaches, and their limitations. @fchollet also takes the brave step of proposing a path forward.

2 replies, 228 likes

Max Kleiman-Weiner: New work from Kevin Ellis is a breakthrough for learning to synthesize programs. DreamCoder learns and extends a DSL and uses it to solve new problems faster. Rediscovers human-like languages: physical laws, vector algebra & functional programming:

0 replies, 223 likes

Martin Görner: Now reading the ARC paper by @fchollet. “On the measure of intelligence” where he proposes a new benchmark for “intelligence” called the “Abstraction and Reasoning corpus”. Highlights below ->

5 replies, 184 likes

François Chollet: One of the benefits of having an explicit, formal definition of intelligence, is to identify what general principles underpin it. A precise definition and measure serve a North Star for research. Using the definition from, the statement below is accurate.

5 replies, 157 likes

Emil Wallner: 'The Measure of Intelligence' contradicts the compute-is-all-we-need narrative, adds historical context, and is accessible to a broad audience. Hats off to @fchollet for creating clear and actionable suggestions to advance AGI. It's a must-read!

4 replies, 128 likes

Florian Huber: Two very knowledgeable articles about current shortcomings in #AI/#ML and promising ways towards more general, robust, and flexible #AI: 1."On the measure of intelligence" @fchollet: 2."The Next Decade in AI" @GaryMarcus:

1 replies, 119 likes

Riva: just read @fchollet's new paper.. so cool... 1) a critique of past and current AI benchmarking 2) reexamines the definitions of intelligence 3) suggests how to quantify generalization difficulty 4) *and* provides a dataset to test for it (..& more!)

4 replies, 116 likes

Ewen Harrison: Please read this important article from @fchollet on the nature of intelligence, how we measure intelligence, and a framework for how we might benchmark it in machines. #DataScience /2/2

3 replies, 99 likes

Leo Brueggeman: Kaggle ML competition launching, $20K in prizes for reasoning models Few things unique: 1) it's inspired by intelligence theory on what path to real #AI would look like 2) few training examples, limiting use of current data hungry #deeplearning models

1 replies, 96 likes

Chris Butler: Really enjoyed this exploration of machine intelligence benchmarking by @fchollet. It made me consider the different mental models (and meta mental models) I think about using and creating in my human practice of decision making.

1 replies, 89 likes

Tim Scarfe: We are discussing @fchollet "On the Measure of Intelligence" ( paper on this weeks @MLStreetTalk. Do you want us to cover your viewpoint? Join our discord and let us know! @ykilcher @CShorten30

1 replies, 72 likes

Suzana Ilić: Wow. Can't wait to read this.

0 replies, 66 likes

David Holz: @SamoBurja @rivatez @jimkxa @fchollet just released a VERY important paper arguing against simple scaling in AI. It's the most convincing piece yet and huge step forward in the discussion of general intelligence

0 replies, 64 likes

Gary Marcus: Yes! Reading these two papers together, comparing and contrasting and seeking your own synthesis, would be a great way to think about the road to a more robust AI.

1 replies, 56 likes

Abel.TM: Great seeing AI focusing on fundamental questions: New approaches for generalization and reasoning (@GaryMarcus) The nature of understanding (@tdietterich) Actionable definitions of intelligence (@fchollet)

2 replies, 47 likes

Braden Schrock: @jwangARK I’m in the middle of reading it, but have found @fchollet most recent paper extremely helpful in thinking about AI benchmarks and how those translate to AI *ability*

0 replies, 45 likes

Deb Raji: Lots of great work on evaluation lately, but it's always framed as if AGI is the goal - what if it wasn’t? What if human-like cognition was not the goal? What if what we use ML for requires a completely different computational process from the way we think?

1 replies, 34 likes

François Chollet: Also, if you're curious why ARC looks the way it does, where it comes from, and what it means to work on solving it, you should check out this paper:

0 replies, 30 likes

Davor Jordacevic: I highly recommend this paper about building and measuring intelligence. Thanks @fchollet for such a nice work. 📄

1 replies, 30 likes

summerfieldlab: there are many excellent points made in this detailed article on natural/machine intelligence from @fchollet. My take home: meaningful definitions of intelligence are inextricably tied up with our [human] experience of the [natural] world.

0 replies, 23 likes

François Chollet: You can use propaganda to fight against ideas, but you can't use propaganda to fight against material reality.

2 replies, 20 likes

Pierre Ferragu: Excellent paper, @fchollet ( What I took from it: 1) The very idea of general intelligence is misleading. intelligence is specific to a context (e.g. human mind is orders of magnitude better at 2D and 3D than 4D)

1 replies, 20 likes

Jessica Joy Kerr: The one button I need is labeled “Select / Off” The nice round button in the center does nothing useful (tells you how long ago it was started, but only for an hour, then it beeps a song of defeat and does nothing) #UX #fixedIt

8 replies, 16 likes

Stanisław Jastrzębski: So do deep networks 'interpolate' or do they 'extrapolate'? :) For context see or @GaryMarcus critique of deep learning; I think most people would classify ImageNet-A as 'extrapolation', but also unclear what is the unlabeled dataset overlap with ImageNetA

0 replies, 15 likes

Kyle McDonald: a new metric for measuring general intelligence. very curious to see the first baseline results on this, and also the deep criticisms of this framing of "intelligence".

2 replies, 15 likes

SchoolOfAIOfficial: If one can measure the intelligence of students perhaps one can help a student to learn more effectively based on their precise needs and areas of difficulty. Read this new paper to learn about another approach to intelligence:

0 replies, 13 likes

LittleBimble: @zacharylipton "On the Measure of Intelligence" by @fchollet

0 replies, 11 likes

Manu Joseph: Been waiting for this to come out since I heard about it in @lexfridman 's podcast. A concrete step towards defining and benchmarking intelligence in our quest for AGI...

1 replies, 10 likes

David Broadhurst: This article by @garymarcus is well worth a read for anyone interested in the future practicalities of effective #AI. But, I can't help but feel that the most productive (rapid) way to improve the general perception is to stop using the term #AI for all #ML systems - misleading.

0 replies, 9 likes

Aakash Kumar Nain: Next paper to read

0 replies, 8 likes

Wolfgang Schröder: Interesting paper by F. Chollet (Google) on the measure of #intelligence intelligence. Chollet describes intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience.

1 replies, 6 likes

Ben Duffy: Hmmm, a counter opinion to the view typically held by the AI/AGI research community. Rather than "a collection of task-specific skills" we should measure broad (and even extreme) generalisation and therefore learning ability. The psychometrics perspective. Printing this.

0 replies, 6 likes

Carl Anderson: I finally got around to reading this. Awesome work. I think it will turn out to be a landmark paper. This is the right way to approach AGI. ARC may not be sufficiently broad but if we can make progress on coding CORE and an approach to solving ARC, we'll be on our way.

1 replies, 6 likes

YorkU Philosophy: The Philosophy Department at York University is hiring an Assistant Professor in Ethics and AI.

0 replies, 5 likes

MichelleRobbins: Excellent summary thread 👇 Grabbed the full @fchollet paper here -> for further exploration #AI #AGI

0 replies, 5 likes

Karel Břinda: A very thoughtful paper about how to define and measure intelligence, with a great overview of the history and challenges as well as of the limitations of common benchmarks.

0 replies, 5 likes

Kristin Andrews: We're hiring! Ethics of AI (and other nonhuman systems) because types of intelligences abound. and

1 replies, 5 likes

Neil Selwyn: PS. there are similarities here to Francois Chollet’s point that “the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs & humans at specific tasks such as board games & video games”: (7/7)

1 replies, 5 likes

Alexandra Matthiesen: François Chollet of Google explores a new means of benchmarking AIs in the Abstraction and Reasoning Corpus, “built upon an explicit set of priors designed to be as close as possible to innate human priors.”

0 replies, 4 likes

Diarmuid O Keeffe: Must be a big deal if Gary Marcus is endorsing...

0 replies, 4 likes

Gema Parreño: Measuring Intelligence by skill-acquisition efficiency ; scope, generalization difficulty, priors, and experience as pieces to be accounted for in characterizing intelligent systems #AGI #Benchmarking #ML #MeasureAndProgress

0 replies, 3 likes

Dagmar Monett: THIS THREAD 👇 #defining #intelligence #measuring #intelligence #AI

0 replies, 3 likes

Volatility Quant: On the Measure of Intelligence (Chollet, 2019) #VolQ #QuantResearch

1 replies, 3 likes • Intelligence is the efficiency with which a learning system turns experience and priors into skill at previously unknown tasks • measure of intelligence must account for priors, experience, & generalization difficulty

2 replies, 2 likes

Pablo Casas: 👌👌👌

0 replies, 2 likes

Ronald Richman: It’s a good day when there is something new to read from @fchollet. Now just need to wait for v2 of the DL with Python book!

0 replies, 2 likes

Stanislav Fort: I really enjoyed reading /On the Measure of Intelligence/ by @fchollet. a) the distinction between a skill and the meta-skill of acquiring new skills, and b) the emphasis on data-efficiency of learning felt refreshing. I highly recommend having a look!

0 replies, 2 likes

Austen Bernstein: thank you @fchollet for your recent paper Society's important problems require true intelligence not skill based AI. "ARC takes the position that... every test task should be novel (measuring the ability to understand a new task, rather than skill)"

0 replies, 2 likes

Margaret Siegien: How to measure the “I” in #AI #ArtificialIntelligence #deepmind @PawlowskiMario 👉

0 replies, 2 likes

PlumeraiHQ: About once a month, researchers at our Amsterdam and London labs run a journal club: one person presents a paper, and then we discuss it as a group. 📄 Last Friday, we read @fchollet’s The Measure of Intelligence:

1 replies, 2 likes

Josh Chen: A piercingly insightful interview by @fchollet about the strengths, weaknesses and value of deep learning, as well as a better set of measures for what should constitute machine "intelligence".

0 replies, 2 likes

Paolo Perazzo: Great thread around @fchollet ARC paper “On the measure of intelligence”:

0 replies, 2 likes

steven t. piantadosi: A way to summarize all this is that "ability" should mean what you do under the “optimal” circumstances for an individual, which will vary person-to-person. “Ability” is not what each person happens to bring to some boring test I just put in front of you

1 replies, 2 likes

PlumeraiHQ: About once a month, researchers at our Amsterdam and London labs run a journal club: one person presents a paper, and then we discuss it as a group. 📄 Last Friday, we read @fchollet’s The Measure of Intelligence:

1 replies, 2 likes

Markus Wulfmeier: Exciting to see additional perspectives on intelligence as measured through adaptation and efficiency of transfer.

0 replies, 1 likes

Philipp Bayer: This is more a book than a paper, dense with thoughts and information on 64 pages: @fchollet on how people focused too much on measuring *specific skills* at the cost of generalization, culminating in a set of tasks similar to IQ-tests for machines

1 replies, 1 likes

JohnMeuser: intelligence is a commodity

0 replies, 1 likes

Edward Craig: The Measure of Intelligence (in AI) - Cornell / long paper #AI

0 replies, 1 likes

anjan kumar: ଏବେଏବେ ମୁଁ ଲମ୍ବା ଲେଖାଟିଏ ଲେଖିଛି ମେଧାକୁ ସଂଜ୍ଞା ଦେବାପାଇଁ ଓ ମାପ କରିବା ପାଇଁ I've just released a fairly lengthy paper on defining & measuring intelligence #ଶୁଆନୁବାଦ @mte2o ପାଇଁ , from @fchollet ‘s tweet .

0 replies, 1 likes

Dr Bernd Porr: A lot of food for thought in it and a brutal honesty where we are with AI at the moment and how to get out of that hole. Paging @StuartJRitchie as there's a lot about IQ tests in there.

0 replies, 1 likes

JohnMeuser: I fear it is too late to change the world enough so as to spare the many from the misguided few what is needed is not a new definition of intelligence all that is needed is a clear and exact science of behavior @ID_AA_Carmack

0 replies, 1 likes

Stephen Pimentel: On the Measure of Intelligence by @fchollet

0 replies, 1 likes

Dmitry Kislyuk: The ARC dataset from @fchollet is a terrific and creative step in the right direction of measuring ML progress. A strong solution to it might bear no resemblance to any current ML system, and that’s a good thing! (mini-thread...)

1 replies, 1 likes

eagerWorks: If you want to go serious about what intelligence is, we highly recommend reading this amazing work by @fchollet. The dataset he provides it’s a really nice bonus to start testing some ideas 💡

0 replies, 1 likes

Valentino Zocca 🇮🇹 🇪🇺: The measure of intelligence by @fchollet #AI #deeplearning #MachineLearning

0 replies, 1 likes

jovo: @tdverstynen

1 replies, 1 likes

pardhasaradhi: @JFPuget I become aware of types of AI after reading @EdKoltz and then started reading the following 'On Measure of Intelligence'

0 replies, 1 likes

JohnMeuser: don’t tell the people of the world though they already know but what are we to do?!?

0 replies, 1 likes


Found on Jan 05 2020 at

PDF content of a computer science paper: The Measure of Intelligence