Papers of the day   All papers

The Measure of Intelligence

Comments

François Chollet: I've just released a fairly lengthy paper on defining & measuring intelligence, as well as a new AI evaluation dataset, the "Abstraction and Reasoning Corpus". I've been working on this for the past 2 years, on & off. Paper: https://arxiv.org/abs/1911.01547 ARC: https://github.com/fchollet/ARC https://t.co/bVrmgLAYEv

100 replies, 4491 likes


hardmaru: Excited to see a @Kaggle competition based on the Abstraction and Reasoning Corpus by @fchollet in “On the Measure of Intelligence” (https://arxiv.org/abs/1911.01547) https://www.kaggle.com/c/abstraction-and-reasoning-challenge/ https://t.co/cmAJBPOGh2

2 replies, 558 likes


Gary Marcus: Mini thread: If you haven't already @FChollet's beautiful & insightful paper on intelligence & AI, you should. An elegant distillation of where we are now, & an intriguing proposal for how to make progress. https://arxiv.org/abs/1911.01547

8 replies, 457 likes


Aurélien Geron: I greatly enjoyed this paper! 👍 It provides conceptual and technical tools to progress towards AGI. I wrote a TFDS dataset so you can load it as a TF dataset simply with tfds.load("arc"). You can play with it in this Colab notebook: https://colab.research.google.com/drive/141ceaZhdps3JQj7QvK5Piibasdt7LZNy

1 replies, 253 likes


Noah Guzmán: I really dislike Kaggle under most circumstances, but this is quite an interesting problem. I will likely submit an attempt involving cellular automata.

6 replies, 238 likes


Brandon Rohrer: If you like to think deeply about building and measuring intelligence, I highly recommend this read. It's a cogent review of intelligence measurement, it's challenges, past approaches, and their limitations. @fchollet also takes the brave step of proposing a path forward.

2 replies, 228 likes


Martin Görner: Now reading the ARC paper by @fchollet. https://arxiv.org/abs/1911.01547 “On the measure of intelligence” where he proposes a new benchmark for “intelligence” called the “Abstraction and Reasoning corpus”. Highlights below -> https://t.co/BcUp3WU6YH

5 replies, 184 likes


François Chollet: One of the benefits of having an explicit, formal definition of intelligence, is to identify what general principles underpin it. A precise definition and measure serve a North Star for research. Using the definition from https://arxiv.org/abs/1911.01547, the statement below is accurate.

5 replies, 157 likes


Emil Wallner: 'The Measure of Intelligence' contradicts the compute-is-all-we-need narrative, adds historical context, and is accessible to a broad audience. Hats off to @fchollet for creating clear and actionable suggestions to advance AGI. It's a must-read! https://arxiv.org/pdf/1911.01547.pdf

4 replies, 128 likes


Florian Huber: Two very knowledgeable articles about current shortcomings in #AI/#ML and promising ways towards more general, robust, and flexible #AI: 1."On the measure of intelligence" @fchollet: https://arxiv.org/abs/1911.01547 2."The Next Decade in AI" @GaryMarcus: https://arxiv.org/abs/2002.06177

1 replies, 119 likes


Riva: just read @fchollet's new paper.. so cool... 1) a critique of past and current AI benchmarking 2) reexamines the definitions of intelligence 3) suggests how to quantify generalization difficulty 4) *and* provides a dataset to test for it (..& more!) https://arxiv.org/pdf/1911.01547.pdf

4 replies, 116 likes


Ewen Harrison: Please read this important article from @fchollet on the nature of intelligence, how we measure intelligence, and a framework for how we might benchmark it in machines. #DataScience /2/2 https://arxiv.org/abs/1911.01547

3 replies, 99 likes


Leo Brueggeman: Kaggle ML competition launching, $20K in prizes for reasoning models Few things unique: 1) it's inspired by intelligence theory on what path to real #AI would look like https://arxiv.org/pdf/1911.01547.pdf 2) few training examples, limiting use of current data hungry #deeplearning models https://t.co/fc6J5h6TY4

1 replies, 96 likes


Chris Butler: Really enjoyed this exploration of machine intelligence benchmarking by @fchollet. It made me consider the different mental models (and meta mental models) I think about using and creating in my human practice of decision making. https://arxiv.org/abs/1911.01547

1 replies, 89 likes


Tim Scarfe: We are discussing @fchollet "On the Measure of Intelligence" (https://arxiv.org/abs/1911.01547) paper on this weeks @MLStreetTalk. Do you want us to cover your viewpoint? Join our discord and let us know! https://discord.gg/X59rwSB @ykilcher @CShorten30

1 replies, 72 likes


Suzana Ilić: Wow. Can't wait to read this.

0 replies, 66 likes


David Holz: @SamoBurja @rivatez @jimkxa @fchollet just released a VERY important paper arguing against simple scaling in AI. It's the most convincing piece yet and huge step forward in the discussion of general intelligence https://twitter.com/fchollet/status/1192121587467784192

0 replies, 64 likes


Gary Marcus: Yes! Reading these two papers together, comparing and contrasting and seeking your own synthesis, would be a great way to think about the road to a more robust AI.

1 replies, 56 likes


Abel.TM: Great seeing AI focusing on fundamental questions: New approaches for generalization and reasoning (@GaryMarcus) http://rebooting.ai/ The nature of understanding (@tdietterich) https://twimlai.com/twiml-talk-315-what-does-it-mean-for-a-machine-to-understand-with-thomas-dietterich/ Actionable definitions of intelligence (@fchollet) https://twitter.com/fchollet/status/1192121587467784192

2 replies, 47 likes


Braden Schrock: @jwangARK I’m in the middle of reading it, but have found @fchollet most recent paper extremely helpful in thinking about AI benchmarks and how those translate to AI *ability* https://twitter.com/fchollet/status/1192121587467784192?s=21

0 replies, 45 likes


Deb Raji: Lots of great work on evaluation lately, but it's always framed as if AGI is the goal - what if it wasn’t? What if human-like cognition was not the goal? What if what we use ML for requires a completely different computational process from the way we think?https://twitter.com/fchollet/status/1192121587467784192

1 replies, 34 likes


Davor Jordacevic: I highly recommend this paper about building and measuring intelligence. Thanks @fchollet for such a nice work. 📄

1 replies, 30 likes


François Chollet: Also, if you're curious why ARC looks the way it does, where it comes from, and what it means to work on solving it, you should check out this paper: https://arxiv.org/abs/1911.01547

0 replies, 30 likes


summerfieldlab: there are many excellent points made in this detailed article on natural/machine intelligence from @fchollet. My take home: meaningful definitions of intelligence are inextricably tied up with our [human] experience of the [natural] world.

0 replies, 23 likes


Pierre Ferragu: Excellent paper, @fchollet (http://arxiv.org/abs/1911.01547) What I took from it: 1) The very idea of general intelligence is misleading. intelligence is specific to a context (e.g. human mind is orders of magnitude better at 2D and 3D than 4D)

1 replies, 20 likes


Stanisław Jastrzębski: So do deep networks 'interpolate' or do they 'extrapolate'? :) For context see https://arxiv.org/abs/1911.01547 or @GaryMarcus critique of deep learning; I think most people would classify ImageNet-A as 'extrapolation', but also unclear what is the unlabeled dataset overlap with ImageNetA

0 replies, 15 likes


Kyle McDonald: a new metric for measuring general intelligence. very curious to see the first baseline results on this, and also the deep criticisms of this framing of "intelligence".

2 replies, 15 likes


SchoolOfAIOfficial: If one can measure the intelligence of students perhaps one can help a student to learn more effectively based on their precise needs and areas of difficulty. Read this new paper to learn about another approach to intelligence:

0 replies, 13 likes


Melanie Mitchell: @VoxBec @fchollet There is a huge literature in this area, with many different approaches. Here are just a few examples: https://arxiv.org/pdf/1911.01547.pdf https://arxiv.org/pdf/1604.00289.pdf https://melaniemitchell.me/PapersContent/amcas.pdf http://qrg.northwestern.edu/papers/Files/QRG_Dist_Files/QRG_2010/Lovett_CogSci10_Ravens_Final.pdf https://arxiv.org/pdf/1902.00120.pdf Hope this helps!

2 replies, 11 likes


LittleBimble: @zacharylipton "On the Measure of Intelligence" by @fchollet https://arxiv.org/abs/1911.01547

0 replies, 11 likes


Manu Joseph: Been waiting for this to come out since I heard about it in @lexfridman 's podcast. A concrete step towards defining and benchmarking intelligence in our quest for AGI...

1 replies, 10 likes


David Broadhurst: This article by @garymarcus is well worth a read for anyone interested in the future practicalities of effective #AI. But, I can't help but feel that the most productive (rapid) way to improve the general perception is to stop using the term #AI for all #ML systems - misleading.

0 replies, 9 likes


Aakash Kumar Nain: Next paper to read

0 replies, 8 likes


Ben Duffy: Hmmm, a counter opinion to the view typically held by the AI/AGI research community. Rather than "a collection of task-specific skills" we should measure broad (and even extreme) generalisation and therefore learning ability. The psychometrics perspective. Printing this.

0 replies, 6 likes


Carl Anderson: I finally got around to reading this. Awesome work. I think it will turn out to be a landmark paper. This is the right way to approach AGI. ARC may not be sufficiently broad but if we can make progress on coding CORE and an approach to solving ARC, we'll be on our way.

1 replies, 6 likes


Wolfgang Schröder: Interesting paper by F. Chollet (Google) on the measure of #intelligence intelligence. https://arxiv.org/abs/1911.01547 Chollet describes intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience.

1 replies, 6 likes


Karel Břinda: A very thoughtful paper about how to define and measure intelligence, with a great overview of the history and challenges as well as of the limitations of common benchmarks.

0 replies, 5 likes


MichelleRobbins: Excellent summary thread 👇 Grabbed the full @fchollet paper here -> https://arxiv.org/pdf/1911.01547.pdf for further exploration #AI #AGI

0 replies, 5 likes


Neil Selwyn: PS. there are similarities here to Francois Chollet’s point that “the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs & humans at specific tasks such as board games & video games”: https://arxiv.org/abs/1911.01547 (7/7)

1 replies, 5 likes


Kristin Andrews: We're hiring! Ethics of AI (and other nonhuman systems) because types of intelligences abound. https://arxiv.org/abs/1911.01547?fbclid=IwAR1kJaH7Duihhv4Afo2HLYuDEJwfYHDYcgPAJvuaPyRA5SizTVrj4E65pkE and https://link.springer.com/article/10.1007/s11023-019-09506-6?fbclid=IwAR3AzRSO1hAWyUCHNgiIQ3T4p80zvfLta4B2AgDekGG6v4cyjydfMx4yQuw

1 replies, 5 likes


Diarmuid O Keeffe: Must be a big deal if Gary Marcus is endorsing...

0 replies, 4 likes


Alexandra Matthiesen: François Chollet of Google explores a new means of benchmarking AIs in the Abstraction and Reasoning Corpus, “built upon an explicit set of priors designed to be as close as possible to innate human priors.” https://arxiv.org/pdf/1911.01547.pdf

0 replies, 4 likes


Dagmar Monett: THIS THREAD 👇 #defining #intelligence #measuring #intelligence #AI

0 replies, 3 likes


Volatility Quant: On the Measure of Intelligence (Chollet, 2019) https://arxiv.org/abs/1911.01547 #VolQ #QuantResearch https://t.co/mLwawA5A1z

1 replies, 3 likes


Gema Parreño: Measuring Intelligence by skill-acquisition efficiency ; scope, generalization difficulty, priors, and experience as pieces to be accounted for in characterizing intelligent systems https://arxiv.org/pdf/1911.01547.pdf #AGI #Benchmarking #ML #MeasureAndProgress

0 replies, 3 likes


Stanislav Fort: I really enjoyed reading /On the Measure of Intelligence/ https://arxiv.org/abs/1911.01547 by @fchollet. a) the distinction between a skill and the meta-skill of acquiring new skills, and b) the emphasis on data-efficiency of learning felt refreshing. I highly recommend having a look!

0 replies, 2 likes


CMEAI: How do we measure intelligence and what is necessary for AI? Is prediction the primary condition for the human mind, and if so, are neural networks able to replicate this function? @fchollet #artificialintelligence #machinelearning #AI https://arxiv.org/pdf/1911.01547.pdf

0 replies, 2 likes


msb.ai: • Intelligence is the efficiency with which a learning system turns experience and priors into skill at previously unknown tasks • measure of intelligence must account for priors, experience, & generalization difficulty

2 replies, 2 likes


Pablo Casas: 👌👌👌

0 replies, 2 likes


Suleman Kazi: @FurqanAfsal Current AI application is more 'engineering' and less 'science'. And the metrics we use right now aren't really geared towards abstract intelligence. Here's a really nice (and long) paper defining the measure of intelligence: https://arxiv.org/abs/1911.01547

1 replies, 2 likes


Austen Bernstein: thank you @fchollet for your recent paper https://arxiv.org/pdf/1911.01547.pdf Society's important problems require true intelligence not skill based AI. "ARC takes the position that... every test task should be novel (measuring the ability to understand a new task, rather than skill)"

0 replies, 2 likes


Margaret Siegien: How to measure the “I” in #AI #ArtificialIntelligence #deepmind @PawlowskiMario 👉https://arxiv.org/abs/1911.01547 https://bdtechtalks.com/2019/12/03/francois-chollet-arc-ai-measurement/

0 replies, 2 likes


Ronald Richman: It’s a good day when there is something new to read from @fchollet. Now just need to wait for v2 of the DL with Python book!

0 replies, 2 likes


PlumeraiHQ: About once a month, researchers at our Amsterdam and London labs run a journal club: one person presents a paper, and then we discuss it as a group. 📄 Last Friday, we read @fchollet’s The Measure of Intelligence: https://twitter.com/fchollet/status/1192121587467784192?s=21 https://t.co/Er0zRNo8KG

1 replies, 2 likes


PlumeraiHQ: About once a month, researchers at our Amsterdam and London labs run a journal club: one person presents a paper, and then we discuss it as a group. 📄 Last Friday, we read @fchollet’s The Measure of Intelligence: https://twitter.com/fchollet/status/1192121587467784192?s=21 https://t.co/MDFLv5tWJB

1 replies, 2 likes


Paolo Perazzo: Great thread around @fchollet ARC paper “On the measure of intelligence”: https://arxiv.org/abs/1911.01547

0 replies, 2 likes


Markus Wulfmeier: Exciting to see additional perspectives on intelligence as measured through adaptation and efficiency of transfer.

0 replies, 1 likes


Dr Bernd Porr: A lot of food for thought in it and a brutal honesty where we are with AI at the moment and how to get out of that hole. Paging @StuartJRitchie as there's a lot about IQ tests in there.

0 replies, 1 likes


Dmitry Kislyuk: The ARC dataset from @fchollet is a terrific and creative step in the right direction of measuring ML progress. A strong solution to it might bear no resemblance to any current ML system, and that’s a good thing! https://arxiv.org/pdf/1911.01547.pdf (mini-thread...) https://t.co/Xq10NL4gGE

1 replies, 1 likes


JohnMeuser: I fear it is too late to change the world enough so as to spare the many from the misguided few what is needed is not a new definition of intelligence all that is needed is a clear and exact science of behavior @ID_AA_Carmack

0 replies, 1 likes


pardhasaradhi: @JFPuget I become aware of types of AI after reading @EdKoltz and then started reading the following 'On Measure of Intelligence' https://arxiv.org/abs/1911.01547

0 replies, 1 likes


Stephen Pimentel: On the Measure of Intelligence by @fchollet https://arxiv.org/abs/1911.01547

0 replies, 1 likes


anjan kumar: ଏବେଏବେ ମୁଁ ଲମ୍ବା ଲେଖାଟିଏ ଲେଖିଛି ମେଧାକୁ ସଂଜ୍ଞା ଦେବାପାଇଁ ଓ ମାପ କରିବା ପାଇଁ I've just released a fairly lengthy paper on defining & measuring intelligence #ଶୁଆନୁବାଦ @mte2o ପାଇଁ , from @fchollet ‘s tweet .

0 replies, 1 likes


Philipp Bayer: This is more a book than a paper, dense with thoughts and information on 64 pages: https://arxiv.org/abs/1911.01547 @fchollet on how people focused too much on measuring *specific skills* at the cost of generalization, culminating in a set of tasks similar to IQ-tests for machines

1 replies, 1 likes


JohnMeuser: don’t tell the people of the world though they already know but what are we to do?!?

0 replies, 1 likes


jovo: @tdverstynen https://arxiv.org/abs/1911.01547

1 replies, 1 likes


Edward Craig: The Measure of Intelligence (in AI) - Cornell / long paper #AI https://arxiv.org/abs/1911.01547

0 replies, 1 likes


eagerWorks: If you want to go serious about what intelligence is, we highly recommend reading this amazing work by @fchollet. The dataset he provides it’s a really nice bonus to start testing some ideas 💡

0 replies, 1 likes


Valentino Zocca 🇮🇹 🇪🇺: The measure of intelligence by @fchollet #AI #deeplearning #MachineLearning

0 replies, 1 likes


JohnMeuser: intelligence is a commodity https://arxiv.org/abs/1911.01547

0 replies, 1 likes


Content

Found on Nov 06 2019 at https://arxiv.org/pdf/1911.01547.pdf

PDF content of a computer science paper: The Measure of Intelligence