Measuring Immediate Adaptation Performance for Neural Machine Translation

Patrick Simianer, Joern Wuebker, John DeNero


Abstract
Incremental domain adaptation, in which a system learns from the correct output for each input immediately after making its prediction for that input, can dramatically improve system performance for interactive machine translation. Users of interactive systems are sensitive to the speed of adaptation and how often a system repeats mistakes, despite being corrected. Adaptation is most commonly assessed using corpus-level BLEU- or TER-derived metrics that do not explicitly take adaptation speed into account. We find that these metrics often do not capture immediate adaptation effects, such as zero-shot and one-shot learning of domain-specific lexical items. To this end, we propose new metrics that directly evaluate immediate adaptation performance for machine translation. We use these metrics to choose the most suitable adaptation method from a range of different adaptation techniques for neural machine translation systems.
Anthology ID:
N19-1206
Volume:
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Editors:
Jill Burstein, Christy Doran, Thamar Solorio
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2038–2046
Language:
URL:
https://aclanthology.org/N19-1206
DOI:
10.18653/v1/N19-1206
Bibkey:
Cite (ACL):
Patrick Simianer, Joern Wuebker, and John DeNero. 2019. Measuring Immediate Adaptation Performance for Neural Machine Translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2038–2046, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
Measuring Immediate Adaptation Performance for Neural Machine Translation (Simianer et al., NAACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/N19-1206.pdf
Presentation:
 N19-1206.Presentation.pdf
Video:
 https://aclanthology.org/N19-1206.mp4