Binarized LSTM Language Model

Xuan Liu, Di Cao, Kai Yu


Abstract
Long short-term memory (LSTM) language model (LM) has been widely investigated for automatic speech recognition (ASR) and natural language processing (NLP). Although excellent performance is obtained for large vocabulary tasks, tremendous memory consumption prohibits the use of LSTM LM in low-resource devices. The memory consumption mainly comes from the word embedding layer. In this paper, a novel binarized LSTM LM is proposed to address the problem. Words are encoded into binary vectors and other LSTM parameters are further binarized to achieve high memory compression. This is the first effort to investigate binary LSTM for large vocabulary LM. Experiments on both English and Chinese LM and ASR tasks showed that can achieve a compression ratio of 11.3 without any loss of LM and ASR performances and a compression ratio of 31.6 with acceptable minor performance degradation.
Anthology ID:
N18-1192
Volume:
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Editors:
Marilyn Walker, Heng Ji, Amanda Stent
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2113–2121
Language:
URL:
https://aclanthology.org/N18-1192
DOI:
10.18653/v1/N18-1192
Bibkey:
Cite (ACL):
Xuan Liu, Di Cao, and Kai Yu. 2018. Binarized LSTM Language Model. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2113–2121, New Orleans, Louisiana. Association for Computational Linguistics.
Cite (Informal):
Binarized LSTM Language Model (Liu et al., NAACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/N18-1192.pdf
Data
Penn Treebank