Papers of the day   All papers

The Measure of Intelligence

Comments

Jan 05 2020 Brian Roemmele

Consider: millions of years ago our antecedents gave a massive sacrifice of their left hemisphere. We lost a tremendous amount of short term memory and replaced it with Broca’s, Wernicke & the phonological loop. But why? So we can—talk. Thus chimpanzees can do this—we can’t: https://t.co/CDznxg37p1
846 replies, 94703 likes


Nov 06 2019 François Chollet

I've just released a fairly lengthy paper on defining & measuring intelligence, as well as a new AI evaluation dataset, the "Abstraction and Reasoning Corpus". I've been working on this for the past 2 years, on & off. Paper: https://arxiv.org/abs/1911.01547 ARC: https://github.com/fchollet/ARC https://t.co/bVrmgLAYEv
73 replies, 3602 likes


Nov 12 2019 Quoc Le

Want to improve accuracy and robustness of your model? Use unlabeled data! Our new work uses self-training on unlabeled data to achieve 87.4% top-1 on ImageNet, 1% better than SOTA. Huge gains are seen on harder benchmarks (ImageNet-A, C and P). Link: https://arxiv.org/abs/1911.04252 https://t.co/0umSnX7wui
22 replies, 1416 likes


Nov 11 2019 Emil Wallner

François Chollet’s core point: We can't measure an AI system's adaptability and flexibility by measuring a specific skill. With unlimited data, models memorize decisions. To advance AGI we need to quantify and measure ***skill-acquisition efficiency***. Let’s dig in👇 https://t.co/Y3GjoG9n4e
14 replies, 1089 likes


Dec 10 2019 Gary Marcus

Mini thread: If you haven't already @FChollet's beautiful & insightful paper on intelligence & AI, you should. An elegant distillation of where we are now, & an intriguing proposal for how to make progress. https://arxiv.org/abs/1911.01547
10 replies, 455 likes


Nov 16 2019 Aurélien Geron

I greatly enjoyed this paper! 👍 It provides conceptual and technical tools to progress towards AGI. I wrote a TFDS dataset so you can load it as a TF dataset simply with tfds.load("arc"). You can play with it in this Colab notebook: https://colab.research.google.com/drive/141ceaZhdps3JQj7QvK5Piibasdt7LZNy
1 replies, 253 likes


Jan 01 2020 Yann LeCun

There can be no intelligence without learning. https://twitter.com/tobias_rees/status/1212138636352745472
29 replies, 244 likes


Nov 06 2019 Brandon Rohrer

If you like to think deeply about building and measuring intelligence, I highly recommend this read. It's a cogent review of intelligence measurement, it's challenges, past approaches, and their limitations. @fchollet also takes the brave step of proposing a path forward.
2 replies, 228 likes


Nov 28 2019 Martin Görner

Now reading the ARC paper by @fchollet. https://arxiv.org/abs/1911.01547 “On the measure of intelligence” where he proposes a new benchmark for “intelligence” called the “Abstraction and Reasoning corpus”. Highlights below -> https://t.co/BcUp3WU6YH
5 replies, 176 likes


Jan 01 2020 François Chollet

One of the benefits of having an explicit, formal definition of intelligence, is to identify what general principles underpin it. A precise definition and measure serve a North Star for research. Using the definition from https://arxiv.org/abs/1911.01547, the statement below is accurate.
5 replies, 157 likes


Nov 11 2019 Emil Wallner

'The Measure of Intelligence' contradicts the compute-is-all-we-need narrative, adds historical context, and is accessible to a broad audience. Hats off to @fchollet for creating clear and actionable suggestions to advance AGI. It's a must-read! https://arxiv.org/pdf/1911.01547.pdf
4 replies, 128 likes


Nov 12 2019 Riva

just read @fchollet's new paper.. so cool... 1) a critique of past and current AI benchmarking 2) reexamines the definitions of intelligence 3) suggests how to quantify generalization difficulty 4) *and* provides a dataset to test for it (..& more!) https://arxiv.org/pdf/1911.01547.pdf
4 replies, 116 likes


Nov 29 2019 Ewen Harrison

Please read this important article from @fchollet on the nature of intelligence, how we measure intelligence, and a framework for how we might benchmark it in machines. #DataScience /2/2 https://arxiv.org/abs/1911.01547
3 replies, 99 likes


Nov 19 2019 Chris Butler

Really enjoyed this exploration of machine intelligence benchmarking by @fchollet. It made me consider the different mental models (and meta mental models) I think about using and creating in my human practice of decision making. https://arxiv.org/abs/1911.01547
1 replies, 89 likes


Nov 07 2019 Suzana Ilić

Wow. Can't wait to read this.
0 replies, 66 likes


Nov 08 2019 David Holz

@SamoBurja @rivatez @jimkxa @fchollet just released a VERY important paper arguing against simple scaling in AI. It's the most convincing piece yet and huge step forward in the discussion of general intelligence https://twitter.com/fchollet/status/1192121587467784192
0 replies, 64 likes


Nov 08 2019 Abel.TM

Great seeing AI focusing on fundamental questions: New approaches for generalization and reasoning (@GaryMarcus) http://rebooting.ai/ The nature of understanding (@tdietterich) https://twimlai.com/twiml-talk-315-what-does-it-mean-for-a-machine-to-understand-with-thomas-dietterich/ Actionable definitions of intelligence (@fchollet) https://twitter.com/fchollet/status/1192121587467784192
2 replies, 47 likes


Nov 26 2019 Braden Schrock

@jwangARK I’m in the middle of reading it, but have found @fchollet most recent paper extremely helpful in thinking about AI benchmarks and how those translate to AI *ability* https://twitter.com/fchollet/status/1192121587467784192?s=21
0 replies, 45 likes


Nov 12 2019 Deb Raji

Lots of great work on evaluation lately, but it's always framed as if AGI is the goal - what if it wasn’t? What if human-like cognition was not the goal? What if what we use ML for requires a completely different computational process from the way we think?https://twitter.com/fchollet/status/1192121587467784192
1 replies, 34 likes


Nov 08 2019 Davor Jordacevic

I highly recommend this paper about building and measuring intelligence. Thanks @fchollet for such a nice work. 📄
1 replies, 30 likes


Nov 10 2019 Pierre Ferragu

Excellent paper, @fchollet (http://arxiv.org/abs/1911.01547) What I took from it: 1) The very idea of general intelligence is misleading. intelligence is specific to a context (e.g. human mind is orders of magnitude better at 2D and 3D than 4D)
1 replies, 20 likes


Jan 08 2020 summerfieldlab

there are many excellent points made in this detailed article on natural/machine intelligence from @fchollet. My take home: meaningful definitions of intelligence are inextricably tied up with our [human] experience of the [natural] world.
0 replies, 19 likes


Nov 11 2019 Kyle McDonald

a new metric for measuring general intelligence. very curious to see the first baseline results on this, and also the deep criticisms of this framing of "intelligence".
2 replies, 15 likes


Nov 13 2019 Stanisław Jastrzębski

So do deep networks 'interpolate' or do they 'extrapolate'? :) For context see https://arxiv.org/abs/1911.01547 or @GaryMarcus critique of deep learning; I think most people would classify ImageNet-A as 'extrapolation', but also unclear what is the unlabeled dataset overlap with ImageNetA
0 replies, 15 likes


Nov 07 2019 SchoolOfAIOfficial

If one can measure the intelligence of students perhaps one can help a student to learn more effectively based on their precise needs and areas of difficulty. Read this new paper to learn about another approach to intelligence:
0 replies, 13 likes


Jan 01 2020 LittleBimble

@zacharylipton "On the Measure of Intelligence" by @fchollet https://arxiv.org/abs/1911.01547
0 replies, 11 likes


Nov 17 2019 Manu Joseph

Been waiting for this to come out since I heard about it in @lexfridman 's podcast. A concrete step towards defining and benchmarking intelligence in our quest for AGI...
1 replies, 10 likes


Nov 06 2019 Aakash Kumar Nain

Next paper to read
0 replies, 8 likes


Jan 13 2020 Wolfgang Schröder

Interesting paper by F. Chollet (Google) on the measure of #intelligence intelligence. https://arxiv.org/abs/1911.01547 Chollet describes intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience.
1 replies, 6 likes


Nov 07 2019 Ben Duffy

Hmmm, a counter opinion to the view typically held by the AI/AGI research community. Rather than "a collection of task-specific skills" we should measure broad (and even extreme) generalisation and therefore learning ability. The psychometrics perspective. Printing this.
0 replies, 6 likes


Nov 16 2019 Carl Anderson

I finally got around to reading this. Awesome work. I think it will turn out to be a landmark paper. This is the right way to approach AGI. ARC may not be sufficiently broad but if we can make progress on coding CORE and an approach to solving ARC, we'll be on our way.
1 replies, 6 likes


Nov 13 2019 Karel Břinda

A very thoughtful paper about how to define and measure intelligence, with a great overview of the history and challenges as well as of the limitations of common benchmarks.
0 replies, 5 likes


Nov 11 2019 YorkU Philosophy

The Philosophy Department at York University is hiring an Assistant Professor in Ethics and AI. http://webapps.yorku.ca/academichiringviewer/viewposition.jsp?positionnumber=2028
0 replies, 5 likes


Nov 11 2019 MichelleRobbins

Excellent summary thread 👇 Grabbed the full @fchollet paper here -> https://arxiv.org/pdf/1911.01547.pdf for further exploration #AI #AGI
0 replies, 5 likes


Nov 11 2019 Kristin Andrews

We're hiring! Ethics of AI (and other nonhuman systems) because types of intelligences abound. https://arxiv.org/abs/1911.01547?fbclid=IwAR1kJaH7Duihhv4Afo2HLYuDEJwfYHDYcgPAJvuaPyRA5SizTVrj4E65pkE and https://link.springer.com/article/10.1007/s11023-019-09506-6?fbclid=IwAR3AzRSO1hAWyUCHNgiIQ3T4p80zvfLta4B2AgDekGG6v4cyjydfMx4yQuw
1 replies, 5 likes


Dec 11 2019 Diarmuid O Keeffe

Must be a big deal if Gary Marcus is endorsing...
0 replies, 4 likes


Dec 25 2019 Alexandra Matthiesen

François Chollet of Google explores a new means of benchmarking AIs in the Abstraction and Reasoning Corpus, “built upon an explicit set of priors designed to be as close as possible to innate human priors.” https://arxiv.org/pdf/1911.01547.pdf
0 replies, 4 likes


Nov 12 2019 Gema Parreño

Measuring Intelligence by skill-acquisition efficiency ; scope, generalization difficulty, priors, and experience as pieces to be accounted for in characterizing intelligent systems https://arxiv.org/pdf/1911.01547.pdf #AGI #Benchmarking #ML #MeasureAndProgress
0 replies, 3 likes


Nov 06 2019 Dagmar Monett

THIS THREAD 👇 #defining #intelligence #measuring #intelligence #AI
0 replies, 3 likes


Nov 18 2019 PlumeraiHQ

About once a month, researchers at our Amsterdam and London labs run a journal club: one person presents a paper, and then we discuss it as a group. 📄 Last Friday, we read @fchollet’s The Measure of Intelligence: https://twitter.com/fchollet/status/1192121587467784192?s=21 https://t.co/MDFLv5tWJB
1 replies, 2 likes


Nov 18 2019 PlumeraiHQ

About once a month, researchers at our Amsterdam and London labs run a journal club: one person presents a paper, and then we discuss it as a group. 📄 Last Friday, we read @fchollet’s The Measure of Intelligence: https://twitter.com/fchollet/status/1192121587467784192?s=21 https://t.co/Er0zRNo8KG
1 replies, 2 likes


Nov 07 2019 msb.ai

• Intelligence is the efficiency with which a learning system turns experience and priors into skill at previously unknown tasks • measure of intelligence must account for priors, experience, & generalization difficulty
2 replies, 2 likes


Dec 01 2019 Austen Bernstein

thank you @fchollet for your recent paper https://arxiv.org/pdf/1911.01547.pdf Society's important problems require true intelligence not skill based AI. "ARC takes the position that... every test task should be novel (measuring the ability to understand a new task, rather than skill)"
0 replies, 2 likes


Nov 07 2019 Pablo Casas

👌👌👌
0 replies, 2 likes


Dec 03 2019 Josh Chen

A piercingly insightful interview by @fchollet about the strengths, weaknesses and value of deep learning, as well as a better set of measures for what should constitute machine "intelligence". https://www.zdnet.com/article/keras-creator-chollets-new-direction-for-ai-a-q-a/
0 replies, 2 likes


Nov 30 2019 Paolo Perazzo

Great thread around @fchollet ARC paper “On the measure of intelligence”: https://arxiv.org/abs/1911.01547
0 replies, 2 likes


Nov 06 2019 Ronald Richman

It’s a good day when there is something new to read from @fchollet. Now just need to wait for v2 of the DL with Python book!
0 replies, 2 likes


Dec 29 2019 Stanislav Fort

I really enjoyed reading /On the Measure of Intelligence/ https://arxiv.org/abs/1911.01547 by @fchollet. a) the distinction between a skill and the meta-skill of acquiring new skills, and b) the emphasis on data-efficiency of learning felt refreshing. I highly recommend having a look!
0 replies, 2 likes


Dec 05 2019 Margaret Siegien

How to measure the “I” in #AI #ArtificialIntelligence #deepmind @PawlowskiMario 👉https://arxiv.org/abs/1911.01547 https://bdtechtalks.com/2019/12/03/francois-chollet-arc-ai-measurement/
0 replies, 2 likes


Nov 14 2019 Edward Craig

The Measure of Intelligence (in AI) - Cornell / long paper #AI https://arxiv.org/abs/1911.01547
0 replies, 1 likes


Nov 15 2019 JohnMeuser

intelligence is a commodity https://arxiv.org/abs/1911.01547
0 replies, 1 likes


Nov 20 2019 Dmitry Kislyuk

The ARC dataset from @fchollet is a terrific and creative step in the right direction of measuring ML progress. A strong solution to it might bear no resemblance to any current ML system, and that’s a good thing! https://arxiv.org/pdf/1911.01547.pdf (mini-thread...) https://t.co/Xq10NL4gGE
1 replies, 1 likes


Nov 13 2019 Philipp Bayer

This is more a book than a paper, dense with thoughts and information on 64 pages: https://arxiv.org/abs/1911.01547 @fchollet on how people focused too much on measuring *specific skills* at the cost of generalization, culminating in a set of tasks similar to IQ-tests for machines
1 replies, 1 likes


Nov 15 2019 JohnMeuser

I fear it is too late to change the world enough so as to spare the many from the misguided few what is needed is not a new definition of intelligence all that is needed is a clear and exact science of behavior @ID_AA_Carmack
0 replies, 1 likes


Dec 12 2019 jovo

@tdverstynen https://arxiv.org/abs/1911.01547
1 replies, 1 likes


Nov 11 2019 Markus Wulfmeier

Exciting to see additional perspectives on intelligence as measured through adaptation and efficiency of transfer.
0 replies, 1 likes


Nov 15 2019 JohnMeuser

don’t tell the people of the world though they already know but what are we to do?!?
0 replies, 1 likes


Dec 19 2019 anjan kumar

ଏବେଏବେ ମୁଁ ଲମ୍ବା ଲେଖାଟିଏ ଲେଖିଛି ମେଧାକୁ ସଂଜ୍ଞା ଦେବାପାଇଁ ଓ ମାପ କରିବା ପାଇଁ I've just released a fairly lengthy paper on defining & measuring intelligence #ଶୁଆନୁବାଦ @mte2o ପାଇଁ , from @fchollet ‘s tweet .
0 replies, 1 likes


Nov 06 2019 Valentino Zocca 🇮🇹 🇪🇺

The measure of intelligence by @fchollet #AI #deeplearning #MachineLearning
0 replies, 1 likes


Dec 11 2019 Dr Bernd Porr

A lot of food for thought in it and a brutal honesty where we are with AI at the moment and how to get out of that hole. Paging @StuartJRitchie as there's a lot about IQ tests in there.
0 replies, 1 likes


Nov 07 2019 eagerWorks

If you want to go serious about what intelligence is, we highly recommend reading this amazing work by @fchollet. The dataset he provides it’s a really nice bonus to start testing some ideas 💡
0 replies, 1 likes


Content