Doubly-Attentive Decoder for Multi-modal Neural Machine Translation

Iacer Calixto, Qun Liu, Nick Campbell


Abstract
We introduce a Multi-modal Neural Machine Translation model in which a doubly-attentive decoder naturally incorporates spatial visual features obtained using pre-trained convolutional neural networks, bridging the gap between image description and translation. Our decoder learns to attend to source-language words and parts of an image independently by means of two separate attention mechanisms as it generates words in the target language. We find that our model can efficiently exploit not just back-translated in-domain multi-modal data but also large general-domain text-only MT corpora. We also report state-of-the-art results on the Multi30k data set.
Anthology ID:
P17-1175
Volume:
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
July
Year:
2017
Address:
Vancouver, Canada
Editors:
Regina Barzilay, Min-Yen Kan
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1913–1924
Language:
URL:
https://aclanthology.org/P17-1175
DOI:
10.18653/v1/P17-1175
Bibkey:
Cite (ACL):
Iacer Calixto, Qun Liu, and Nick Campbell. 2017. Doubly-Attentive Decoder for Multi-modal Neural Machine Translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1913–1924, Vancouver, Canada. Association for Computational Linguistics.
Cite (Informal):
Doubly-Attentive Decoder for Multi-modal Neural Machine Translation (Calixto et al., ACL 2017)
Copy Citation:
PDF:
https://aclanthology.org/P17-1175.pdf
Data
EuroparlFlickr30kMulti30KWMT 2015