Deep Pyramid Convolutional Neural Networks for Text Categorization

Rie Johnson, Tong Zhang


Abstract
This paper proposes a low-complexity word-level deep convolutional neural network (CNN) architecture for text categorization that can efficiently represent long-range associations in text. In the literature, several deep and complex neural networks have been proposed for this task, assuming availability of relatively large amounts of training data. However, the associated computational complexity increases as the networks go deeper, which poses serious challenges in practical applications. Moreover, it was shown recently that shallow word-level CNNs are more accurate and much faster than the state-of-the-art very deep nets such as character-level CNNs even in the setting of large training data. Motivated by these findings, we carefully studied deepening of word-level CNNs to capture global representations of text, and found a simple network architecture with which the best accuracy can be obtained by increasing the network depth without increasing computational cost by much. We call it deep pyramid CNN. The proposed model with 15 weight layers outperforms the previous best models on six benchmark datasets for sentiment classification and topic categorization.
Anthology ID:
P17-1052
Volume:
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
July
Year:
2017
Address:
Vancouver, Canada
Editors:
Regina Barzilay, Min-Yen Kan
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
562–570
Language:
URL:
https://aclanthology.org/P17-1052
DOI:
10.18653/v1/P17-1052
Bibkey:
Cite (ACL):
Rie Johnson and Tong Zhang. 2017. Deep Pyramid Convolutional Neural Networks for Text Categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 562–570, Vancouver, Canada. Association for Computational Linguistics.
Cite (Informal):
Deep Pyramid Convolutional Neural Networks for Text Categorization (Johnson & Zhang, ACL 2017)
Copy Citation:
PDF:
https://aclanthology.org/P17-1052.pdf
Video:
 https://aclanthology.org/P17-1052.mp4
Data
AG NewsDBpediaYelp