A Structural Probe for Finding Syntax in Word Representations

John Hewitt, Christopher D. Manning


Abstract
Recent work has improved our ability to detect linguistic knowledge in word representations. However, current methods for detecting syntactic knowledge do not test whether syntax trees are represented in their entirety. In this work, we propose a structural probe, which evaluates whether syntax trees are embedded in a linear transformation of a neural network’s word representation space. The probe identifies a linear transformation under which squared L2 distance encodes the distance between words in the parse tree, and one in which squared L2 norm encodes depth in the parse tree. Using our probe, we show that such transformations exist for both ELMo and BERT but not in baselines, providing evidence that entire syntax trees are embedded implicitly in deep models’ vector geometry.
Anthology ID:
N19-1419
Volume:
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Editors:
Jill Burstein, Christy Doran, Thamar Solorio
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4129–4138
Language:
URL:
https://aclanthology.org/N19-1419
DOI:
10.18653/v1/N19-1419
Bibkey:
Cite (ACL):
John Hewitt and Christopher D. Manning. 2019. A Structural Probe for Finding Syntax in Word Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
A Structural Probe for Finding Syntax in Word Representations (Hewitt & Manning, NAACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/N19-1419.pdf
Video:
 https://vimeo.com/361827125
Code
 john-hewitt/structural-probes
Data
Penn Treebank