Binarized LSTM Language Model

Xuan Liu, Di Cao, Kai Yu


Abstract
Long short-term memory (LSTM) language model (LM) has been widely investigated for automatic speech recognition (ASR) and natural language processing (NLP). Although excellent performance is obtained for large vocabulary tasks, tremendous memory consumption prohibits the use of LSTM LM in low-resource devices. The memory consumption mainly comes from the word embedding layer. In this paper, a novel binarized LSTM LM is proposed to address the problem. Words are encoded into binary vectors and other LSTM parameters are further binarized to achieve high memory compression. This is the first effort to investigate binary LSTM for large vocabulary LM. Experiments on both English and Chinese LM and ASR tasks showed that can achieve a compression ratio of 11.3 without any loss of LM and ASR performances and a compression ratio of 31.6 with acceptable minor performance degradation.
Anthology ID:
N18-1192
Volume:
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Venues:
HLT | NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2113–2121
URL:
https://www.aclweb.org/anthology/N18-1192
DOI:
10.18653/v1/N18-1192
Bib Export formats:
BibTeX MODS XML EndNote