Efficient Online Scalar Annotation with Bounded Support

Keisuke Sakaguchi, Benjamin Van Durme


Abstract
We describe a novel method for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments. We contrast direct assessment (annotators assign scores to items directly), online pairwise ranking aggregation (scores derive from annotator comparison of items), and a hybrid approach (EASL: Efficient Annotation of Scalar Labels) proposed here. Our proposal leads to increased correlation with ground truth, at far greater annotator efficiency, suggesting this strategy as an improved mechanism for dataset creation and manual system evaluation.
Anthology ID:
P18-1020
Volume:
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Editors:
Iryna Gurevych, Yusuke Miyao
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
208–218
Language:
URL:
https://aclanthology.org/P18-1020
DOI:
10.18653/v1/P18-1020
Bibkey:
Cite (ACL):
Keisuke Sakaguchi and Benjamin Van Durme. 2018. Efficient Online Scalar Annotation with Bounded Support. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 208–218, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
Efficient Online Scalar Annotation with Bounded Support (Sakaguchi & Van Durme, ACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/P18-1020.pdf
Video:
 https://aclanthology.org/P18-1020.mp4
Data
WMT 2016