Papers of the day   All papers

Language Models as Knowledge Bases?


Graham Neubig: "Language models as knowledge bases?" they asked: "A cat has four kidneys", replied GPT-2.

23 replies, 590 likes

Fabio Petroni: Can language models replace knowledge bases? In our recent @EMNLP2019 paper we present the LAMA probe for analyzing the knowledge contained in pretrained language models. w/ @_rockt @PSH_Lewis @anton_bakhtin @mindjimmy @riedelcastro from @FacebookAI

4 replies, 467 likes

(((ل()(ل() 'yoav)))): no, they really CANNOT replace knowledge bases yet. they recover correctly only a fraction of the facts in the KB, are restricted to single tokens, and when they don't know the answer they just make something up.

10 replies, 305 likes

Andrew Beam: Question: How do we unify language models and knowledge bases? Answer: Oops, I think we already did... by accident. Language models implicitly contain information that can be queried like a knowledge base. Cool new work from FAIR and UCL: Paper:

2 replies, 215 likes

hardmaru 😷: Language Models as Knowledge Bases? They present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models. h/t @Synced_Global

6 replies, 166 likes

Tim Rocktäschel: Excited to share our work on assessing factual and commonsense knowledge in pretrained language models. Turns out they are surprisingly competitive to knowledge bases that are constructed from recent relation extraction pipelines. Great work by @Fabio_Petroni and the team!

0 replies, 122 likes

WikiResearch: "Language Models as Knowledge Bases?" - an analysis of the relational knowledge present in pretrained language models shows a strong ability of these models to recall factual knowledge - tested on #Wikidata! (Petroni et al., 2019)

1 replies, 70 likes

Patrick Lewis: How much knowledge do language models have? How does that knowledge compare traditional KB? @Fabio_Petroni ‘s LAMA probe suggests that some language models are actually pretty good KBs! Read our @emnlp2019 paper, with @riedelcastro @_rockt @anton_bakhtin @mindjimmy

0 replies, 43 likes

Alex Polozov: I love everything about this. Not only GPT-2 doesn't understand anatomy (of course), but it also starts the claim with four kidneys and proceeds to list a grand total of five!

1 replies, 41 likes

Daisuke Okanohara: Language models (e.g. BERT) are competitive with structured knowledge bases for several NLP tasks. LM requires no schema engineering, easy to exploit more data. How to make an effective question to extract knowledge may be an interesting direction.

0 replies, 21 likes

Jonathan Fly 👾: BERT versus ConceptNET: Time is _______. RIGHT ANSWER: Finite. BERT's GUESSES: short, passing, precious, irrelevant, gone 😔 Bert is a poet. "Language Models as Knowledge Bases?"

1 replies, 19 likes

Ben Lorica 罗瑞卡: "analysis of the factual & commonsense knowledge in publicly available pretrained language models & found that BERT-large is able to recall such knowledge ... at a level remarkably competitive with non-neural and supervised alternatives" 🆕 #Nlproc

1 replies, 15 likes

Andy Way: Anyone who thinks #AI can simply replace humans should have a look at this ... 😂 Jump on anyone who says #NLProc is a ‘solved problem’ just because deep learning is involved!

0 replies, 14 likes

Montreal.AI: Language Models as Knowledge Bases? Petroni et al.: #Transformers #NaturalLanguageProcessing #MachineLearning

0 replies, 9 likes

Andromeda Yelton: So not ONLY is this hilarious but it is an excellent demonstration of why we maybe shouldn't trust AI *too* much because here's what GPT-2 has to say on library policy.

0 replies, 6 likes

Pratik Bhavsar: @omarsar0 I think this is the first one

1 replies, 5 likes

Jeffrey Snover: AI for natural language processing NLP is advancing rapidly but obviously - miles to go before we sleep. Given this description of cats, I won’t be sleeping with both eyes closed for a while. :-)

1 replies, 4 likes

amandalynne (she): meow

0 replies, 4 likes

The Institute for Ethical AI & Machine Learning: Language models as knowledgebases. Interesting perspective on how the potential of languge models can be generalised. In this paper, the authors argue that language models may also be storing relational knowledge present in the training data.

0 replies, 4 likes

Matt Krause: @jgordonfisher @GaryMarcus @huggingface But people *are* making the argument that LMs "know" things:

1 replies, 2 likes

nutanc: GPT2 like many other deep learning models is just good for fun(very costly fun though). Cannot be used in a business environment.

0 replies, 2 likes

Hicham EL BOUKKOURI: Very interesting work by @facebookai & UCL for analysing the factual and common-sense knowledge that is available in large language models such as BERT, ELMo or GPT-2. Paper: GitHub:

0 replies, 1 likes

4InData: #4InData #Transformers #NaturalLanguageProcessing #MachineLearning Language Models as Knowledge Bases? Petroni et al.:

0 replies, 1 likes

arXiv CS-CL: Language Models as Knowledge Bases?

0 replies, 1 likes

Katie Anderson: I'm glad to finally know the scientific reasoning behind my black and white cat's coloring.

0 replies, 1 likes

Andrey Kurenkov 🤖: Super cool! As with the recent paper analyzing memorization in language models, another good question addressed here (how much knowledge about the world they end up containing). Good to see more of this type of research.

1 replies, 1 likes


Found on Sep 26 2019 at

PDF content of a computer science paper: Language Models as Knowledge Bases?