A Stable Variational Autoencoder for Text Modelling

Ruizhe Li, Xiao Li, Chenghua Lin, Matthew Collinson, Rui Mao


Abstract
Variational Autoencoder (VAE) is a powerful method for learning representations of high-dimensional data. However, VAEs can suffer from an issue known as latent variable collapse (or KL term vanishing), where the posterior collapses to the prior and the model will ignore the latent codes in generative tasks. Such an issue is particularly prevalent when employing VAE-RNN architectures for text modelling (Bowman et al., 2016; Yang et al., 2017). In this paper, we present a new architecture called Full-Sampling-VAE-RNN, which can effectively avoid latent variable collapse. Compared to the general VAE-RNN architectures, we show that our model can achieve much more stable training process and can generate text with significantly better quality.
Anthology ID:
W19-8673
Volume:
Proceedings of the 12th International Conference on Natural Language Generation
Month:
October–November
Year:
2019
Address:
Tokyo, Japan
Editors:
Kees van Deemter, Chenghua Lin, Hiroya Takamura
Venue:
INLG
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
594–599
Language:
URL:
https://aclanthology.org/W19-8673
DOI:
10.18653/v1/W19-8673
Bibkey:
Cite (ACL):
Ruizhe Li, Xiao Li, Chenghua Lin, Matthew Collinson, and Rui Mao. 2019. A Stable Variational Autoencoder for Text Modelling. In Proceedings of the 12th International Conference on Natural Language Generation, pages 594–599, Tokyo, Japan. Association for Computational Linguistics.
Cite (Informal):
A Stable Variational Autoencoder for Text Modelling (Li et al., INLG 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-8673.pdf
Code
 ruizheliUOA/HR-VAE
Data
Penn Treebank