Dual Fixed-Size Ordinally Forgetting Encoding (FOFE) for Competitive Neural Language Models

Sedtawut Watcharawittayakul, Mingbin Xu, Hui Jiang


Abstract
In this paper, we propose a new approach to employ the fixed-size ordinally-forgetting encoding (FOFE) (Zhang et al., 2015b) in neural languages modelling, called dual-FOFE. The main idea of dual-FOFE is that it allows to use two different forgetting factors so that it can avoid the trade-off in choosing either a small or large values for the single forgetting factor. In our experiments, we have compared the dual-FOFE based neural network language models (NNLM) against the original FOFE counterparts and various traditional NNLMs. Our results on the challenging Google Billion word corpus show that both FOFE and dual FOFE yield very strong performance while significantly reducing the computational complexity over other NNLMs. Furthermore, the proposed dual-FOFE method further gives over 10% improvement in perplexity over the original FOFE model.
Anthology ID:
D18-1502
Volume:
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Month:
October-November
Year:
2018
Address:
Brussels, Belgium
Editors:
Ellen Riloff, David Chiang, Julia Hockenmaier, Jun’ichi Tsujii
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
4725–4730
Language:
URL:
https://aclanthology.org/D18-1502
DOI:
10.18653/v1/D18-1502
Bibkey:
Cite (ACL):
Sedtawut Watcharawittayakul, Mingbin Xu, and Hui Jiang. 2018. Dual Fixed-Size Ordinally Forgetting Encoding (FOFE) for Competitive Neural Language Models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4725–4730, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
Dual Fixed-Size Ordinally Forgetting Encoding (FOFE) for Competitive Neural Language Models (Watcharawittayakul et al., EMNLP 2018)
Copy Citation:
PDF:
https://aclanthology.org/D18-1502.pdf
Video:
 https://aclanthology.org/D18-1502.mp4