Variance of Average Surprisal: A Better Predictor for Quality of Grammar from Unsupervised PCFG Induction

Lifeng Jin, William Schuler


Abstract
In unsupervised grammar induction, data likelihood is known to be only weakly correlated with parsing accuracy, especially at convergence after multiple runs. In order to find a better indicator for quality of induced grammars, this paper correlates several linguistically- and psycholinguistically-motivated predictors to parsing accuracy on a large multilingual grammar induction evaluation data set. Results show that variance of average surprisal (VAS) better correlates with parsing accuracy than data likelihood and that using VAS instead of data likelihood for model selection provides a significant accuracy boost. Further evidence shows VAS to be a better candidate than data likelihood for predicting word order typology classification. Analyses show that VAS seems to separate content words from function words in natural language grammars, and to better arrange words with different frequencies into separate classes that are more consistent with linguistic theory.
Anthology ID:
P19-1235
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2019
Address:
Florence, Italy
Editors:
Anna Korhonen, David Traum, Lluís Màrquez
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2453–2463
Language:
URL:
https://aclanthology.org/P19-1235
DOI:
10.18653/v1/P19-1235
Bibkey:
Cite (ACL):
Lifeng Jin and William Schuler. 2019. Variance of Average Surprisal: A Better Predictor for Quality of Grammar from Unsupervised PCFG Induction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2453–2463, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Variance of Average Surprisal: A Better Predictor for Quality of Grammar from Unsupervised PCFG Induction (Jin & Schuler, ACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/P19-1235.pdf
Data
Penn Treebank