Papers of the day   All papers

The Measure of Intelligence


François Chollet: I've just released a fairly lengthy paper on defining & measuring intelligence, as well as a new AI evaluation dataset, the "Abstraction and Reasoning Corpus". I've been working on this for the past 2 years, on & off. Paper: ARC:

100 replies, 4491 likes

hardmaru: Excited to see a @Kaggle competition based on the Abstraction and Reasoning Corpus by @fchollet in “On the Measure of Intelligence” (

2 replies, 558 likes

Gary Marcus: Mini thread: If you haven't already @FChollet's beautiful & insightful paper on intelligence & AI, you should. An elegant distillation of where we are now, & an intriguing proposal for how to make progress.

8 replies, 457 likes

Aurélien Geron: I greatly enjoyed this paper! 👍 It provides conceptual and technical tools to progress towards AGI. I wrote a TFDS dataset so you can load it as a TF dataset simply with tfds.load("arc"). You can play with it in this Colab notebook:

1 replies, 253 likes

Noah Guzmán: I really dislike Kaggle under most circumstances, but this is quite an interesting problem. I will likely submit an attempt involving cellular automata.

6 replies, 238 likes

Brandon Rohrer: If you like to think deeply about building and measuring intelligence, I highly recommend this read. It's a cogent review of intelligence measurement, it's challenges, past approaches, and their limitations. @fchollet also takes the brave step of proposing a path forward.

2 replies, 228 likes

Martin Görner: Now reading the ARC paper by @fchollet. “On the measure of intelligence” where he proposes a new benchmark for “intelligence” called the “Abstraction and Reasoning corpus”. Highlights below ->

5 replies, 184 likes

François Chollet: One of the benefits of having an explicit, formal definition of intelligence, is to identify what general principles underpin it. A precise definition and measure serve a North Star for research. Using the definition from, the statement below is accurate.

5 replies, 157 likes

Emil Wallner: 'The Measure of Intelligence' contradicts the compute-is-all-we-need narrative, adds historical context, and is accessible to a broad audience. Hats off to @fchollet for creating clear and actionable suggestions to advance AGI. It's a must-read!

4 replies, 128 likes

Florian Huber: Two very knowledgeable articles about current shortcomings in #AI/#ML and promising ways towards more general, robust, and flexible #AI: 1."On the measure of intelligence" @fchollet: 2."The Next Decade in AI" @GaryMarcus:

1 replies, 119 likes

Riva: just read @fchollet's new paper.. so cool... 1) a critique of past and current AI benchmarking 2) reexamines the definitions of intelligence 3) suggests how to quantify generalization difficulty 4) *and* provides a dataset to test for it (..& more!)

4 replies, 116 likes

Ewen Harrison: Please read this important article from @fchollet on the nature of intelligence, how we measure intelligence, and a framework for how we might benchmark it in machines. #DataScience /2/2

3 replies, 99 likes

Leo Brueggeman: Kaggle ML competition launching, $20K in prizes for reasoning models Few things unique: 1) it's inspired by intelligence theory on what path to real #AI would look like 2) few training examples, limiting use of current data hungry #deeplearning models

1 replies, 96 likes

Chris Butler: Really enjoyed this exploration of machine intelligence benchmarking by @fchollet. It made me consider the different mental models (and meta mental models) I think about using and creating in my human practice of decision making.

1 replies, 89 likes

Tim Scarfe: We are discussing @fchollet "On the Measure of Intelligence" ( paper on this weeks @MLStreetTalk. Do you want us to cover your viewpoint? Join our discord and let us know! @ykilcher @CShorten30

1 replies, 72 likes

Suzana Ilić: Wow. Can't wait to read this.

0 replies, 66 likes

David Holz: @SamoBurja @rivatez @jimkxa @fchollet just released a VERY important paper arguing against simple scaling in AI. It's the most convincing piece yet and huge step forward in the discussion of general intelligence

0 replies, 64 likes

Gary Marcus: Yes! Reading these two papers together, comparing and contrasting and seeking your own synthesis, would be a great way to think about the road to a more robust AI.

1 replies, 56 likes

Abel.TM: Great seeing AI focusing on fundamental questions: New approaches for generalization and reasoning (@GaryMarcus) The nature of understanding (@tdietterich) Actionable definitions of intelligence (@fchollet)

2 replies, 47 likes

Braden Schrock: @jwangARK I’m in the middle of reading it, but have found @fchollet most recent paper extremely helpful in thinking about AI benchmarks and how those translate to AI *ability*

0 replies, 45 likes

Deb Raji: Lots of great work on evaluation lately, but it's always framed as if AGI is the goal - what if it wasn’t? What if human-like cognition was not the goal? What if what we use ML for requires a completely different computational process from the way we think?

1 replies, 34 likes

Davor Jordacevic: I highly recommend this paper about building and measuring intelligence. Thanks @fchollet for such a nice work. 📄

1 replies, 30 likes

François Chollet: Also, if you're curious why ARC looks the way it does, where it comes from, and what it means to work on solving it, you should check out this paper:

0 replies, 30 likes

summerfieldlab: there are many excellent points made in this detailed article on natural/machine intelligence from @fchollet. My take home: meaningful definitions of intelligence are inextricably tied up with our [human] experience of the [natural] world.

0 replies, 23 likes

Pierre Ferragu: Excellent paper, @fchollet ( What I took from it: 1) The very idea of general intelligence is misleading. intelligence is specific to a context (e.g. human mind is orders of magnitude better at 2D and 3D than 4D)

1 replies, 20 likes

Stanisław Jastrzębski: So do deep networks 'interpolate' or do they 'extrapolate'? :) For context see or @GaryMarcus critique of deep learning; I think most people would classify ImageNet-A as 'extrapolation', but also unclear what is the unlabeled dataset overlap with ImageNetA

0 replies, 15 likes

Kyle McDonald: a new metric for measuring general intelligence. very curious to see the first baseline results on this, and also the deep criticisms of this framing of "intelligence".

2 replies, 15 likes

SchoolOfAIOfficial: If one can measure the intelligence of students perhaps one can help a student to learn more effectively based on their precise needs and areas of difficulty. Read this new paper to learn about another approach to intelligence:

0 replies, 13 likes

Melanie Mitchell: @VoxBec @fchollet There is a huge literature in this area, with many different approaches. Here are just a few examples: Hope this helps!

2 replies, 11 likes

LittleBimble: @zacharylipton "On the Measure of Intelligence" by @fchollet

0 replies, 11 likes

Manu Joseph: Been waiting for this to come out since I heard about it in @lexfridman 's podcast. A concrete step towards defining and benchmarking intelligence in our quest for AGI...

1 replies, 10 likes

David Broadhurst: This article by @garymarcus is well worth a read for anyone interested in the future practicalities of effective #AI. But, I can't help but feel that the most productive (rapid) way to improve the general perception is to stop using the term #AI for all #ML systems - misleading.

0 replies, 9 likes

Aakash Kumar Nain: Next paper to read

0 replies, 8 likes

Ben Duffy: Hmmm, a counter opinion to the view typically held by the AI/AGI research community. Rather than "a collection of task-specific skills" we should measure broad (and even extreme) generalisation and therefore learning ability. The psychometrics perspective. Printing this.

0 replies, 6 likes

Carl Anderson: I finally got around to reading this. Awesome work. I think it will turn out to be a landmark paper. This is the right way to approach AGI. ARC may not be sufficiently broad but if we can make progress on coding CORE and an approach to solving ARC, we'll be on our way.

1 replies, 6 likes

Wolfgang Schröder: Interesting paper by F. Chollet (Google) on the measure of #intelligence intelligence. Chollet describes intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience.

1 replies, 6 likes

Karel Břinda: A very thoughtful paper about how to define and measure intelligence, with a great overview of the history and challenges as well as of the limitations of common benchmarks.

0 replies, 5 likes

MichelleRobbins: Excellent summary thread 👇 Grabbed the full @fchollet paper here -> for further exploration #AI #AGI

0 replies, 5 likes

Neil Selwyn: PS. there are similarities here to Francois Chollet’s point that “the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs & humans at specific tasks such as board games & video games”: (7/7)

1 replies, 5 likes

Kristin Andrews: We're hiring! Ethics of AI (and other nonhuman systems) because types of intelligences abound. and

1 replies, 5 likes

Diarmuid O Keeffe: Must be a big deal if Gary Marcus is endorsing...

0 replies, 4 likes

Alexandra Matthiesen: François Chollet of Google explores a new means of benchmarking AIs in the Abstraction and Reasoning Corpus, “built upon an explicit set of priors designed to be as close as possible to innate human priors.”

0 replies, 4 likes

Dagmar Monett: THIS THREAD 👇 #defining #intelligence #measuring #intelligence #AI

0 replies, 3 likes

Volatility Quant: On the Measure of Intelligence (Chollet, 2019) #VolQ #QuantResearch

1 replies, 3 likes

Gema Parreño: Measuring Intelligence by skill-acquisition efficiency ; scope, generalization difficulty, priors, and experience as pieces to be accounted for in characterizing intelligent systems #AGI #Benchmarking #ML #MeasureAndProgress

0 replies, 3 likes

Stanislav Fort: I really enjoyed reading /On the Measure of Intelligence/ by @fchollet. a) the distinction between a skill and the meta-skill of acquiring new skills, and b) the emphasis on data-efficiency of learning felt refreshing. I highly recommend having a look!

0 replies, 2 likes

CMEAI: How do we measure intelligence and what is necessary for AI? Is prediction the primary condition for the human mind, and if so, are neural networks able to replicate this function? @fchollet #artificialintelligence #machinelearning #AI

0 replies, 2 likes • Intelligence is the efficiency with which a learning system turns experience and priors into skill at previously unknown tasks • measure of intelligence must account for priors, experience, & generalization difficulty

2 replies, 2 likes

Pablo Casas: 👌👌👌

0 replies, 2 likes

Suleman Kazi: @FurqanAfsal Current AI application is more 'engineering' and less 'science'. And the metrics we use right now aren't really geared towards abstract intelligence. Here's a really nice (and long) paper defining the measure of intelligence:

1 replies, 2 likes

Austen Bernstein: thank you @fchollet for your recent paper Society's important problems require true intelligence not skill based AI. "ARC takes the position that... every test task should be novel (measuring the ability to understand a new task, rather than skill)"

0 replies, 2 likes

Margaret Siegien: How to measure the “I” in #AI #ArtificialIntelligence #deepmind @PawlowskiMario 👉

0 replies, 2 likes

Ronald Richman: It’s a good day when there is something new to read from @fchollet. Now just need to wait for v2 of the DL with Python book!

0 replies, 2 likes

PlumeraiHQ: About once a month, researchers at our Amsterdam and London labs run a journal club: one person presents a paper, and then we discuss it as a group. 📄 Last Friday, we read @fchollet’s The Measure of Intelligence:

1 replies, 2 likes

PlumeraiHQ: About once a month, researchers at our Amsterdam and London labs run a journal club: one person presents a paper, and then we discuss it as a group. 📄 Last Friday, we read @fchollet’s The Measure of Intelligence:

1 replies, 2 likes

Paolo Perazzo: Great thread around @fchollet ARC paper “On the measure of intelligence”:

0 replies, 2 likes

Markus Wulfmeier: Exciting to see additional perspectives on intelligence as measured through adaptation and efficiency of transfer.

0 replies, 1 likes

Dr Bernd Porr: A lot of food for thought in it and a brutal honesty where we are with AI at the moment and how to get out of that hole. Paging @StuartJRitchie as there's a lot about IQ tests in there.

0 replies, 1 likes

Dmitry Kislyuk: The ARC dataset from @fchollet is a terrific and creative step in the right direction of measuring ML progress. A strong solution to it might bear no resemblance to any current ML system, and that’s a good thing! (mini-thread...)

1 replies, 1 likes

JohnMeuser: I fear it is too late to change the world enough so as to spare the many from the misguided few what is needed is not a new definition of intelligence all that is needed is a clear and exact science of behavior @ID_AA_Carmack

0 replies, 1 likes

pardhasaradhi: @JFPuget I become aware of types of AI after reading @EdKoltz and then started reading the following 'On Measure of Intelligence'

0 replies, 1 likes

Stephen Pimentel: On the Measure of Intelligence by @fchollet

0 replies, 1 likes

anjan kumar: ଏବେଏବେ ମୁଁ ଲମ୍ବା ଲେଖାଟିଏ ଲେଖିଛି ମେଧାକୁ ସଂଜ୍ଞା ଦେବାପାଇଁ ଓ ମାପ କରିବା ପାଇଁ I've just released a fairly lengthy paper on defining & measuring intelligence #ଶୁଆନୁବାଦ @mte2o ପାଇଁ , from @fchollet ‘s tweet .

0 replies, 1 likes

Philipp Bayer: This is more a book than a paper, dense with thoughts and information on 64 pages: @fchollet on how people focused too much on measuring *specific skills* at the cost of generalization, culminating in a set of tasks similar to IQ-tests for machines

1 replies, 1 likes

JohnMeuser: don’t tell the people of the world though they already know but what are we to do?!?

0 replies, 1 likes

jovo: @tdverstynen

1 replies, 1 likes

Edward Craig: The Measure of Intelligence (in AI) - Cornell / long paper #AI

0 replies, 1 likes

eagerWorks: If you want to go serious about what intelligence is, we highly recommend reading this amazing work by @fchollet. The dataset he provides it’s a really nice bonus to start testing some ideas 💡

0 replies, 1 likes

Valentino Zocca 🇮🇹 🇪🇺: The measure of intelligence by @fchollet #AI #deeplearning #MachineLearning

0 replies, 1 likes

JohnMeuser: intelligence is a commodity

0 replies, 1 likes


Found on Nov 06 2019 at

PDF content of a computer science paper: The Measure of Intelligence