A Multi-Axis Annotation Scheme for Event Temporal Relations

Qiang Ning, Hao Wu, Dan Roth


Abstract
Existing temporal relation (TempRel) annotation schemes often have low inter-annotator agreements (IAA) even between experts, suggesting that the current annotation task needs a better definition. This paper proposes a new multi-axis modeling to better capture the temporal structure of events. In addition, we identify that event end-points are a major source of confusion in annotation, so we also propose to annotate TempRels based on start-points only. A pilot expert annotation effort using the proposed scheme shows significant improvement in IAA from the conventional 60’s to 80’s (Cohen’s Kappa). This better-defined annotation scheme further enables the use of crowdsourcing to alleviate the labor intensity for each annotator. We hope that this work can foster more interesting studies towards event understanding.
Anthology ID:
P18-1122
Volume:
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Editors:
Iryna Gurevych, Yusuke Miyao
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1318–1328
Language:
URL:
https://aclanthology.org/P18-1122
DOI:
10.18653/v1/P18-1122
Bibkey:
Cite (ACL):
Qiang Ning, Hao Wu, and Dan Roth. 2018. A Multi-Axis Annotation Scheme for Event Temporal Relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1318–1328, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
A Multi-Axis Annotation Scheme for Event Temporal Relations (Ning et al., ACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/P18-1122.pdf
Presentation:
 P18-1122.Presentation.pdf
Notes:
 P18-1122.Notes.pdf
Video:
 https://aclanthology.org/P18-1122.mp4
Data
MATRES