Papers of the day   All papers

The Measure of Intelligence


Jan 05 2020 Brian Roemmele

Consider: millions of years ago our antecedents gave a massive sacrifice of their left hemisphere. We lost a tremendous amount of short term memory and replaced it with Broca’s, Wernicke & the phonological loop. But why? So we can—talk. Thus chimpanzees can do this—we can’t:
846 replies, 94703 likes

Nov 06 2019 François Chollet

I've just released a fairly lengthy paper on defining & measuring intelligence, as well as a new AI evaluation dataset, the "Abstraction and Reasoning Corpus". I've been working on this for the past 2 years, on & off. Paper: ARC:
73 replies, 3602 likes

Nov 12 2019 Quoc Le

Want to improve accuracy and robustness of your model? Use unlabeled data! Our new work uses self-training on unlabeled data to achieve 87.4% top-1 on ImageNet, 1% better than SOTA. Huge gains are seen on harder benchmarks (ImageNet-A, C and P). Link:
22 replies, 1416 likes

Nov 11 2019 Emil Wallner

François Chollet’s core point: We can't measure an AI system's adaptability and flexibility by measuring a specific skill. With unlimited data, models memorize decisions. To advance AGI we need to quantify and measure ***skill-acquisition efficiency***. Let’s dig in👇
14 replies, 1089 likes

Dec 10 2019 Gary Marcus

Mini thread: If you haven't already @FChollet's beautiful & insightful paper on intelligence & AI, you should. An elegant distillation of where we are now, & an intriguing proposal for how to make progress.
10 replies, 455 likes

Nov 16 2019 Aurélien Geron

I greatly enjoyed this paper! 👍 It provides conceptual and technical tools to progress towards AGI. I wrote a TFDS dataset so you can load it as a TF dataset simply with tfds.load("arc"). You can play with it in this Colab notebook:
1 replies, 253 likes

Jan 01 2020 Yann LeCun

There can be no intelligence without learning.
29 replies, 244 likes

Nov 06 2019 Brandon Rohrer

If you like to think deeply about building and measuring intelligence, I highly recommend this read. It's a cogent review of intelligence measurement, it's challenges, past approaches, and their limitations. @fchollet also takes the brave step of proposing a path forward.
2 replies, 228 likes

Nov 28 2019 Martin Görner

Now reading the ARC paper by @fchollet. “On the measure of intelligence” where he proposes a new benchmark for “intelligence” called the “Abstraction and Reasoning corpus”. Highlights below ->
5 replies, 176 likes

Jan 01 2020 François Chollet

One of the benefits of having an explicit, formal definition of intelligence, is to identify what general principles underpin it. A precise definition and measure serve a North Star for research. Using the definition from, the statement below is accurate.
5 replies, 157 likes

Nov 11 2019 Emil Wallner

'The Measure of Intelligence' contradicts the compute-is-all-we-need narrative, adds historical context, and is accessible to a broad audience. Hats off to @fchollet for creating clear and actionable suggestions to advance AGI. It's a must-read!
4 replies, 128 likes

Nov 12 2019 Riva

just read @fchollet's new paper.. so cool... 1) a critique of past and current AI benchmarking 2) reexamines the definitions of intelligence 3) suggests how to quantify generalization difficulty 4) *and* provides a dataset to test for it (..& more!)
4 replies, 116 likes

Nov 29 2019 Ewen Harrison

Please read this important article from @fchollet on the nature of intelligence, how we measure intelligence, and a framework for how we might benchmark it in machines. #DataScience /2/2
3 replies, 99 likes

Nov 19 2019 Chris Butler

Really enjoyed this exploration of machine intelligence benchmarking by @fchollet. It made me consider the different mental models (and meta mental models) I think about using and creating in my human practice of decision making.
1 replies, 89 likes

Nov 07 2019 Suzana Ilić

Wow. Can't wait to read this.
0 replies, 66 likes

Nov 08 2019 David Holz

@SamoBurja @rivatez @jimkxa @fchollet just released a VERY important paper arguing against simple scaling in AI. It's the most convincing piece yet and huge step forward in the discussion of general intelligence
0 replies, 64 likes

Nov 08 2019 Abel.TM

Great seeing AI focusing on fundamental questions: New approaches for generalization and reasoning (@GaryMarcus) The nature of understanding (@tdietterich) Actionable definitions of intelligence (@fchollet)
2 replies, 47 likes

Nov 26 2019 Braden Schrock

@jwangARK I’m in the middle of reading it, but have found @fchollet most recent paper extremely helpful in thinking about AI benchmarks and how those translate to AI *ability*
0 replies, 45 likes

Nov 12 2019 Deb Raji

Lots of great work on evaluation lately, but it's always framed as if AGI is the goal - what if it wasn’t? What if human-like cognition was not the goal? What if what we use ML for requires a completely different computational process from the way we think?
1 replies, 34 likes

Nov 08 2019 Davor Jordacevic

I highly recommend this paper about building and measuring intelligence. Thanks @fchollet for such a nice work. 📄
1 replies, 30 likes

Nov 10 2019 Pierre Ferragu

Excellent paper, @fchollet ( What I took from it: 1) The very idea of general intelligence is misleading. intelligence is specific to a context (e.g. human mind is orders of magnitude better at 2D and 3D than 4D)
1 replies, 20 likes

Jan 08 2020 summerfieldlab

there are many excellent points made in this detailed article on natural/machine intelligence from @fchollet. My take home: meaningful definitions of intelligence are inextricably tied up with our [human] experience of the [natural] world.
0 replies, 19 likes

Nov 11 2019 Kyle McDonald

a new metric for measuring general intelligence. very curious to see the first baseline results on this, and also the deep criticisms of this framing of "intelligence".
2 replies, 15 likes

Nov 13 2019 Stanisław Jastrzębski

So do deep networks 'interpolate' or do they 'extrapolate'? :) For context see or @GaryMarcus critique of deep learning; I think most people would classify ImageNet-A as 'extrapolation', but also unclear what is the unlabeled dataset overlap with ImageNetA
0 replies, 15 likes

Nov 07 2019 SchoolOfAIOfficial

If one can measure the intelligence of students perhaps one can help a student to learn more effectively based on their precise needs and areas of difficulty. Read this new paper to learn about another approach to intelligence:
0 replies, 13 likes

Jan 01 2020 LittleBimble

@zacharylipton "On the Measure of Intelligence" by @fchollet
0 replies, 11 likes

Nov 17 2019 Manu Joseph

Been waiting for this to come out since I heard about it in @lexfridman 's podcast. A concrete step towards defining and benchmarking intelligence in our quest for AGI...
1 replies, 10 likes

Nov 06 2019 Aakash Kumar Nain

Next paper to read
0 replies, 8 likes

Jan 13 2020 Wolfgang Schröder

Interesting paper by F. Chollet (Google) on the measure of #intelligence intelligence. Chollet describes intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience.
1 replies, 6 likes

Nov 07 2019 Ben Duffy

Hmmm, a counter opinion to the view typically held by the AI/AGI research community. Rather than "a collection of task-specific skills" we should measure broad (and even extreme) generalisation and therefore learning ability. The psychometrics perspective. Printing this.
0 replies, 6 likes

Nov 16 2019 Carl Anderson

I finally got around to reading this. Awesome work. I think it will turn out to be a landmark paper. This is the right way to approach AGI. ARC may not be sufficiently broad but if we can make progress on coding CORE and an approach to solving ARC, we'll be on our way.
1 replies, 6 likes

Nov 13 2019 Karel Břinda

A very thoughtful paper about how to define and measure intelligence, with a great overview of the history and challenges as well as of the limitations of common benchmarks.
0 replies, 5 likes

Nov 11 2019 YorkU Philosophy

The Philosophy Department at York University is hiring an Assistant Professor in Ethics and AI.
0 replies, 5 likes

Nov 11 2019 MichelleRobbins

Excellent summary thread 👇 Grabbed the full @fchollet paper here -> for further exploration #AI #AGI
0 replies, 5 likes

Nov 11 2019 Kristin Andrews

We're hiring! Ethics of AI (and other nonhuman systems) because types of intelligences abound. and
1 replies, 5 likes

Dec 11 2019 Diarmuid O Keeffe

Must be a big deal if Gary Marcus is endorsing...
0 replies, 4 likes

Dec 25 2019 Alexandra Matthiesen

François Chollet of Google explores a new means of benchmarking AIs in the Abstraction and Reasoning Corpus, “built upon an explicit set of priors designed to be as close as possible to innate human priors.”
0 replies, 4 likes

Nov 12 2019 Gema Parreño

Measuring Intelligence by skill-acquisition efficiency ; scope, generalization difficulty, priors, and experience as pieces to be accounted for in characterizing intelligent systems #AGI #Benchmarking #ML #MeasureAndProgress
0 replies, 3 likes

Nov 06 2019 Dagmar Monett

THIS THREAD 👇 #defining #intelligence #measuring #intelligence #AI
0 replies, 3 likes

Nov 18 2019 PlumeraiHQ

About once a month, researchers at our Amsterdam and London labs run a journal club: one person presents a paper, and then we discuss it as a group. 📄 Last Friday, we read @fchollet’s The Measure of Intelligence:
1 replies, 2 likes

Nov 18 2019 PlumeraiHQ

About once a month, researchers at our Amsterdam and London labs run a journal club: one person presents a paper, and then we discuss it as a group. 📄 Last Friday, we read @fchollet’s The Measure of Intelligence:
1 replies, 2 likes

Nov 07 2019

• Intelligence is the efficiency with which a learning system turns experience and priors into skill at previously unknown tasks • measure of intelligence must account for priors, experience, & generalization difficulty
2 replies, 2 likes

Dec 01 2019 Austen Bernstein

thank you @fchollet for your recent paper Society's important problems require true intelligence not skill based AI. "ARC takes the position that... every test task should be novel (measuring the ability to understand a new task, rather than skill)"
0 replies, 2 likes

Nov 07 2019 Pablo Casas

0 replies, 2 likes

Dec 03 2019 Josh Chen

A piercingly insightful interview by @fchollet about the strengths, weaknesses and value of deep learning, as well as a better set of measures for what should constitute machine "intelligence".
0 replies, 2 likes

Nov 30 2019 Paolo Perazzo

Great thread around @fchollet ARC paper “On the measure of intelligence”:
0 replies, 2 likes

Nov 06 2019 Ronald Richman

It’s a good day when there is something new to read from @fchollet. Now just need to wait for v2 of the DL with Python book!
0 replies, 2 likes

Dec 29 2019 Stanislav Fort

I really enjoyed reading /On the Measure of Intelligence/ by @fchollet. a) the distinction between a skill and the meta-skill of acquiring new skills, and b) the emphasis on data-efficiency of learning felt refreshing. I highly recommend having a look!
0 replies, 2 likes

Dec 05 2019 Margaret Siegien

How to measure the “I” in #AI #ArtificialIntelligence #deepmind @PawlowskiMario 👉
0 replies, 2 likes

Nov 14 2019 Edward Craig

The Measure of Intelligence (in AI) - Cornell / long paper #AI
0 replies, 1 likes

Nov 15 2019 JohnMeuser

intelligence is a commodity
0 replies, 1 likes

Nov 20 2019 Dmitry Kislyuk

The ARC dataset from @fchollet is a terrific and creative step in the right direction of measuring ML progress. A strong solution to it might bear no resemblance to any current ML system, and that’s a good thing! (mini-thread...)
1 replies, 1 likes

Nov 13 2019 Philipp Bayer

This is more a book than a paper, dense with thoughts and information on 64 pages: @fchollet on how people focused too much on measuring *specific skills* at the cost of generalization, culminating in a set of tasks similar to IQ-tests for machines
1 replies, 1 likes

Nov 15 2019 JohnMeuser

I fear it is too late to change the world enough so as to spare the many from the misguided few what is needed is not a new definition of intelligence all that is needed is a clear and exact science of behavior @ID_AA_Carmack
0 replies, 1 likes

Dec 12 2019 jovo

1 replies, 1 likes

Nov 11 2019 Markus Wulfmeier

Exciting to see additional perspectives on intelligence as measured through adaptation and efficiency of transfer.
0 replies, 1 likes

Nov 15 2019 JohnMeuser

don’t tell the people of the world though they already know but what are we to do?!?
0 replies, 1 likes

Dec 19 2019 anjan kumar

ଏବେଏବେ ମୁଁ ଲମ୍ବା ଲେଖାଟିଏ ଲେଖିଛି ମେଧାକୁ ସଂଜ୍ଞା ଦେବାପାଇଁ ଓ ମାପ କରିବା ପାଇଁ I've just released a fairly lengthy paper on defining & measuring intelligence #ଶୁଆନୁବାଦ @mte2o ପାଇଁ , from @fchollet ‘s tweet .
0 replies, 1 likes

Nov 06 2019 Valentino Zocca 🇮🇹 🇪🇺

The measure of intelligence by @fchollet #AI #deeplearning #MachineLearning
0 replies, 1 likes

Dec 11 2019 Dr Bernd Porr

A lot of food for thought in it and a brutal honesty where we are with AI at the moment and how to get out of that hole. Paging @StuartJRitchie as there's a lot about IQ tests in there.
0 replies, 1 likes

Nov 07 2019 eagerWorks

If you want to go serious about what intelligence is, we highly recommend reading this amazing work by @fchollet. The dataset he provides it’s a really nice bonus to start testing some ideas 💡
0 replies, 1 likes