Unsupervised Learning of Syntactic Structure with Invertible Neural Projections

Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick


Abstract
Unsupervised learning of syntactic structure is typically performed using generative models with discrete latent variables and multinomial parameters. In most cases, these models have not leveraged continuous word representations. In this work, we propose a novel generative model that jointly learns discrete syntactic structure and continuous word representations in an unsupervised fashion by cascading an invertible neural network with a structured generative prior. We show that the invertibility condition allows for efficient exact inference and marginal likelihood computation in our model so long as the prior is well-behaved. In experiments we instantiate our approach with both Markov and tree-structured priors, evaluating on two tasks: part-of-speech (POS) induction, and unsupervised dependency parsing without gold POS annotation. On the Penn Treebank, our Markov-structured model surpasses state-of-the-art results on POS induction. Similarly, we find that our tree-structured model achieves state-of-the-art performance on unsupervised dependency parsing for the difficult training condition where neither gold POS annotation nor punctuation-based constraints are available.
Anthology ID:
D18-1160
Volume:
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Month:
October-November
Year:
2018
Address:
Brussels, Belgium
Editors:
Ellen Riloff, David Chiang, Julia Hockenmaier, Jun’ichi Tsujii
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
1292–1302
Language:
URL:
https://aclanthology.org/D18-1160
DOI:
10.18653/v1/D18-1160
Bibkey:
Cite (ACL):
Junxian He, Graham Neubig, and Taylor Berg-Kirkpatrick. 2018. Unsupervised Learning of Syntactic Structure with Invertible Neural Projections. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1292–1302, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
Unsupervised Learning of Syntactic Structure with Invertible Neural Projections (He et al., EMNLP 2018)
Copy Citation:
PDF:
https://aclanthology.org/D18-1160.pdf
Video:
 https://aclanthology.org/D18-1160.mp4
Code
 jxhe/struct-learning-with-flow
Data
PTB Diagnostic ECG DatabasePenn Treebank