Papers of the day   All papers

Visualizing and Measuring the Geometry of BERT

Comments

Jun 09 2019 Christopher Manning

This paper gives some really nice insights and mathematical depth to what had previously (for us) been “the mystery of squared distance” in revealing the representation of parse trees in deep contextual representations (BERT, ELMo, etc.). Great to read!
4 replies, 1146 likes


Jun 07 2019 Martin Wattenberg

How does a neural net represent language? See the visualizations and geometry in this PAIR team paper https://arxiv.org/abs/1906.02715 and blog post https://pair-code.github.io/interpretability/bert-tree/ https://t.co/ZkG81UBHcE
9 replies, 1061 likes


Jun 09 2019 Fernando Pereira

Very cool exploration of the geometry of language embeddings, with some fun math I did not know. https://pair-code.github.io/interpretability/bert-tree/
3 replies, 510 likes


Jun 07 2019 Chris Olah

Very cool visualizations of different word senses being represented in later layers of BERT, by @_coenen, Emily Reif, Ann Yuan and collaborators. https://arxiv.org/pdf/1906.02715.pdf https://t.co/o6sKdPprAJ
2 replies, 332 likes


Jun 10 2019 Jeff Dean

Very nice analysis and visualization from Andy Coenen, Emily Reif, Ann Yuan, @_beenkim, Adam Pearce, @viegasf, and @wattenberg at @GoogleAI's PAIR team about how neural networks models like BERT represent complex grammatical structures in high dimensional spaces.
0 replies, 236 likes


Jun 07 2019 Fernanda Viégas

Analyzing and visualizing syntax trees in the high-dimensional spaces of neural nets. Check out the new PAIR paper on BERT geometry https://arxiv.org/abs/1906.02715 And the blog post on “Language, trees, and geometry in neural networks” https://pair-code.github.io/interpretability/bert-tree/ https://t.co/NhTYvhb8aV
1 replies, 54 likes


Jun 10 2019 John Platt

Very nice NLP and math analysis of BERT word embeddings
0 replies, 20 likes


Jun 08 2019 Andy Coenen

Really enjoyed spending the last month or two poking around in the internals of BERT. It's actually pretty amazing how much syntax-related information is contained in the embeddings and the model itself. https://arxiv.org/pdf/1906.02715.pdf
0 replies, 13 likes


Aug 23 2019 Kenji Kondo

Visualization of BERT's representation space by http://Google.AI guys( https://arxiv.org/abs/1906.02715 ). Really impressed to see how it capture contexts clearly, even though it is one of the best cases chosen for demonstration. https://t.co/VmxqM436EO
0 replies, 9 likes


Jun 08 2019 arXiv CS-CL

Visualizing and Measuring the Geometry of BERT http://arxiv.org/abs/1906.02715
0 replies, 5 likes


Jul 23 2019 Montreal.AI

"Visualizing and Measuring the Geometry of BERT" Coenen et al.: https://arxiv.org/abs/1906.02715 Blog post : https://pair-code.github.io/interpretability/bert-tree/ #ArtificialIntelligence #DeepLearning #MachineLearning https://t.co/YI0I5c1gD7
0 replies, 4 likes


Jun 09 2019 Álvaro Barbero

Some evidence showing that large language models (BERT) implicitly learn how to do syntactic analysis, producing a kind of syntactic tree in a pythagorean embedding. All of this in an unsupervised way!
0 replies, 3 likes


Jun 14 2019 Jelle Zuidema

Jawahar, Sagot, Seddah What does BERT learn about the structure of language? ACL2019 https://hal.inria.fr/hal-02131630/document Coenen, Reif, Yuan, Kim, Pearce, Viégas, Wattenberg Visualizing and Measuring the Geometry of BERT https://arxiv.org/abs/1906.02715 Blog: https://pair-code.github.io/interpretability/bert-tree/
1 replies, 2 likes


Jun 07 2019 arXiv CS-CL

Visualizing and Measuring the Geometry of BERT http://arxiv.org/abs/1906.02715
0 replies, 2 likes


Aug 28 2019 Dylan Bourgeois

@GaryMarcus @Zergylord @wzuidema @rgalhama @LakeBrenden @tallinzen @OpenAI There have been several very interesting works probing the properties of representations generated by BERT-like models recently: https://nlp.stanford.edu/~johnhew/structural-probe.html , https://openreview.net/forum?id=SJzSgnRcKX or https://arxiv.org/abs/1906.02715 (among others)
1 replies, 2 likes


Jun 07 2019 Dylan Bourgeois

More cool work investigating the linguistic features of BERT! --- "Visualizing and Measuring the Geometry of BERT" by @_coenen, Reif, Yuan et al. https://arxiv.org/abs/1906.02715 https://t.co/e4FJ0ao4zw
0 replies, 1 likes


Jun 11 2019 brendan chambers

Nice geometrical explanation for the squared distance relation between parsed trees & euclidean contextual embeddings: https://arxiv.org/abs/1906.02715, building on https://nlp.stanford.edu/~johnhew/structural-probe.html
0 replies, 1 likes


Content