Importance sampling for unbiased on-demand evaluation of knowledge base population

Arun Chaganty, Ashwin Paranjape, Percy Liang, Christopher D. Manning


Abstract
Knowledge base population (KBP) systems take in a large document corpus and extract entities and their relations. Thus far, KBP evaluation has relied on judgements on the pooled predictions of existing systems. We show that this evaluation is problematic: when a new system predicts a previously unseen relation, it is penalized even if it is correct. This leads to significant bias against new systems, which counterproductively discourages innovation in the field. Our first contribution is a new importance-sampling based evaluation which corrects for this bias by annotating a new system’s predictions on-demand via crowdsourcing. We show this eliminates bias and reduces variance using data from the 2015 TAC KBP task. Our second contribution is an implementation of our method made publicly available as an online KBP evaluation service. We pilot the service by testing diverse state-of-the-art systems on the TAC KBP 2016 corpus and obtain accurate scores in a cost effective manner.
Anthology ID:
D17-1109
Volume:
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Month:
September
Year:
2017
Address:
Copenhagen, Denmark
Editors:
Martha Palmer, Rebecca Hwa, Sebastian Riedel
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
1038–1048
Language:
URL:
https://aclanthology.org/D17-1109
DOI:
10.18653/v1/D17-1109
Bibkey:
Cite (ACL):
Arun Chaganty, Ashwin Paranjape, Percy Liang, and Christopher D. Manning. 2017. Importance sampling for unbiased on-demand evaluation of knowledge base population. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1038–1048, Copenhagen, Denmark. Association for Computational Linguistics.
Cite (Informal):
Importance sampling for unbiased on-demand evaluation of knowledge base population (Chaganty et al., EMNLP 2017)
Copy Citation:
PDF:
https://aclanthology.org/D17-1109.pdf
Video:
 https://aclanthology.org/D17-1109.mp4