Self-Attention: A Better Building Block for Sentiment Analysis Neural Network Classifiers

Artaches Ambartsoumian, Fred Popowich


Abstract
Sentiment Analysis has seen much progress in the past two decades. For the past few years, neural network approaches, primarily RNNs and CNNs, have been the most successful for this task. Recently, a new category of neural networks, self-attention networks (SANs), have been created which utilizes the attention mechanism as the basic building block. Self-attention networks have been shown to be effective for sequence modeling tasks, while having no recurrence or convolutions. In this work we explore the effectiveness of the SANs for sentiment analysis. We demonstrate that SANs are superior in performance to their RNN and CNN counterparts by comparing their classification accuracy on six datasets as well as their model characteristics such as training speed and memory consumption. Finally, we explore the effects of various SAN modifications such as multi-head attention as well as two methods of incorporating sequence position information into SANs.
Anthology ID:
W18-6219
Volume:
Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis
Month:
October
Year:
2018
Address:
Brussels, Belgium
Editors:
Alexandra Balahur, Saif M. Mohammad, Veronique Hoste, Roman Klinger
Venue:
WASSA
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
130–139
Language:
URL:
https://aclanthology.org/W18-6219
DOI:
10.18653/v1/W18-6219
Bibkey:
Cite (ACL):
Artaches Ambartsoumian and Fred Popowich. 2018. Self-Attention: A Better Building Block for Sentiment Analysis Neural Network Classifiers. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 130–139, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
Self-Attention: A Better Building Block for Sentiment Analysis Neural Network Classifiers (Ambartsoumian & Popowich, WASSA 2018)
Copy Citation:
PDF:
https://aclanthology.org/W18-6219.pdf
Code
 Artaches/SSAN-self-attention-sentiment-analysis-classification