Using Target-side Monolingual Data for Neural Machine Translation through Multi-task Learning

Tobias Domhan, Felix Hieber


Abstract
The performance of Neural Machine Translation (NMT) models relies heavily on the availability of sufficient amounts of parallel data, and an efficient and effective way of leveraging the vastly available amounts of monolingual data has yet to be found. We propose to modify the decoder in a neural sequence-to-sequence model to enable multi-task learning for two strongly related tasks: target-side language modeling and translation. The decoder predicts the next target word through two channels, a target-side language model on the lowest layer, and an attentional recurrent model which is conditioned on the source representation. This architecture allows joint training on both large amounts of monolingual and moderate amounts of bilingual data to improve NMT performance. Initial results in the news domain for three language pairs show moderate but consistent improvements over a baseline trained on bilingual data only.
Anthology ID:
D17-1158
Volume:
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Month:
September
Year:
2017
Address:
Copenhagen, Denmark
Editors:
Martha Palmer, Rebecca Hwa, Sebastian Riedel
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
1500–1505
Language:
URL:
https://aclanthology.org/D17-1158
DOI:
10.18653/v1/D17-1158
Bibkey:
Cite (ACL):
Tobias Domhan and Felix Hieber. 2017. Using Target-side Monolingual Data for Neural Machine Translation through Multi-task Learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1500–1505, Copenhagen, Denmark. Association for Computational Linguistics.
Cite (Informal):
Using Target-side Monolingual Data for Neural Machine Translation through Multi-task Learning (Domhan & Hieber, EMNLP 2017)
Copy Citation:
PDF:
https://aclanthology.org/D17-1158.pdf
Code
 awslabs/sockeye