Rational Recurrences

Hao Peng, Roy Schwartz, Sam Thomson, Noah A. Smith


Abstract
Despite the tremendous empirical success of neural models in natural language processing, many of them lack the strong intuitions that accompany classical machine learning approaches. Recently, connections have been shown between convolutional neural networks (CNNs) and weighted finite state automata (WFSAs), leading to new interpretations and insights. In this work, we show that some recurrent neural networks also share this connection to WFSAs. We characterize this connection formally, defining rational recurrences to be recurrent hidden state update functions that can be written as the Forward calculation of a finite set of WFSAs. We show that several recent neural models use rational recurrences. Our analysis provides a fresh view of these models and facilitates devising new neural architectures that draw inspiration from WFSAs. We present one such model, which performs better than two recent baselines on language modeling and text classification. Our results demonstrate that transferring intuitions from classical models like WFSAs can be an effective approach to designing and understanding neural models.
Anthology ID:
D18-1152
Volume:
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Month:
October-November
Year:
2018
Address:
Brussels, Belgium
Editors:
Ellen Riloff, David Chiang, Julia Hockenmaier, Jun’ichi Tsujii
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
1203–1214
Language:
URL:
https://aclanthology.org/D18-1152
DOI:
10.18653/v1/D18-1152
Bibkey:
Cite (ACL):
Hao Peng, Roy Schwartz, Sam Thomson, and Noah A. Smith. 2018. Rational Recurrences. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1203–1214, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
Rational Recurrences (Peng et al., EMNLP 2018)
Copy Citation:
PDF:
https://aclanthology.org/D18-1152.pdf
Attachment:
 D18-1152.Attachment.pdf
Video:
 https://aclanthology.org/D18-1152.mp4
Code
 Noahs-ARK/rational-recurrences
Data
SST