Papers of the day   All papers

Visualizing and Measuring the Geometry of BERT


Jun 09 2019 Christopher Manning

This paper gives some really nice insights and mathematical depth to what had previously (for us) been “the mystery of squared distance” in revealing the representation of parse trees in deep contextual representations (BERT, ELMo, etc.). Great to read!
4 replies, 1146 likes

Jun 07 2019 Martin Wattenberg

How does a neural net represent language? See the visualizations and geometry in this PAIR team paper and blog post
9 replies, 1061 likes

Jun 09 2019 Fernando Pereira

Very cool exploration of the geometry of language embeddings, with some fun math I did not know.
3 replies, 510 likes

Jun 07 2019 Chris Olah

Very cool visualizations of different word senses being represented in later layers of BERT, by @_coenen, Emily Reif, Ann Yuan and collaborators.
2 replies, 332 likes

Jun 10 2019 Jeff Dean

Very nice analysis and visualization from Andy Coenen, Emily Reif, Ann Yuan, @_beenkim, Adam Pearce, @viegasf, and @wattenberg at @GoogleAI's PAIR team about how neural networks models like BERT represent complex grammatical structures in high dimensional spaces.
0 replies, 236 likes

Feb 18 2020 👩‍💻 DynamicWebPaige

✨🗣️ Have you heard a lot about BERT, but are still trying to grok its architecture and how it represents linguistic information internally? 👉 Check out this explanatory paper and blog post from @viegasf and team: 📄 👩‍💻
0 replies, 154 likes

Jun 07 2019 Fernanda Viégas

Analyzing and visualizing syntax trees in the high-dimensional spaces of neural nets. Check out the new PAIR paper on BERT geometry And the blog post on “Language, trees, and geometry in neural networks”
1 replies, 59 likes

Jun 10 2019 John Platt

Very nice NLP and math analysis of BERT word embeddings
0 replies, 20 likes

Jun 08 2019 Andy Coenen

Really enjoyed spending the last month or two poking around in the internals of BERT. It's actually pretty amazing how much syntax-related information is contained in the embeddings and the model itself.
0 replies, 13 likes

Aug 23 2019 Kenji Kondo

Visualization of BERT's representation space by http://Google.AI guys( ). Really impressed to see how it capture contexts clearly, even though it is one of the best cases chosen for demonstration.
0 replies, 9 likes

Jul 23 2019 Montreal.AI

"Visualizing and Measuring the Geometry of BERT" Coenen et al.: Blog post : #ArtificialIntelligence #DeepLearning #MachineLearning
0 replies, 7 likes

Jun 08 2019 arXiv CS-CL

Visualizing and Measuring the Geometry of BERT
0 replies, 5 likes

Jun 09 2019 Álvaro Barbero

Some evidence showing that large language models (BERT) implicitly learn how to do syntactic analysis, producing a kind of syntactic tree in a pythagorean embedding. All of this in an unsupervised way!
0 replies, 3 likes

Jun 07 2019 arXiv CS-CL

Visualizing and Measuring the Geometry of BERT
0 replies, 2 likes

Jun 14 2019 Jelle Zuidema

Jawahar, Sagot, Seddah What does BERT learn about the structure of language? ACL2019 Coenen, Reif, Yuan, Kim, Pearce, Viégas, Wattenberg Visualizing and Measuring the Geometry of BERT Blog:
1 replies, 2 likes

Aug 28 2019 Dylan Bourgeois

@GaryMarcus @Zergylord @wzuidema @rgalhama @LakeBrenden @tallinzen @OpenAI There have been several very interesting works probing the properties of representations generated by BERT-like models recently: , or (among others)
1 replies, 2 likes

Jun 11 2019 brendan chambers

Nice geometrical explanation for the squared distance relation between parsed trees & euclidean contextual embeddings:, building on
0 replies, 1 likes

Jun 07 2019 Dylan Bourgeois

More cool work investigating the linguistic features of BERT! --- "Visualizing and Measuring the Geometry of BERT" by @_coenen, Reif, Yuan et al.
0 replies, 1 likes