Crowdsourcing StoryLines: Harnessing the Crowd for Causal Relation Annotation

Tommaso Caselli, Oana Inel


Abstract
This paper describes a crowdsourcing experiment on the annotation of plot-like structures in English news articles. CrowdThruth methodology and metrics have been applied to select valid annotations from the crowd. We further run an in-depth analysis of the annotated data by comparing them with available expert data. Our results show a valuable use of crowdsourcing annotations for such complex semantic tasks, and suggest a new annotation approach which combine crowd and experts.
Anthology ID:
W18-4306
Volume:
Proceedings of the Workshop Events and Stories in the News 2018
Month:
August
Year:
2018
Address:
Santa Fe, New Mexico, U.S.A
Editors:
Tommaso Caselli, Ben Miller, Marieke van Erp, Piek Vossen, Martha Palmer, Eduard Hovy, Teruko Mitamura, David Caswell, Susan W. Brown, Claire Bonial
Venue:
EventStory
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
44–54
Language:
URL:
https://aclanthology.org/W18-4306
DOI:
Bibkey:
Cite (ACL):
Tommaso Caselli and Oana Inel. 2018. Crowdsourcing StoryLines: Harnessing the Crowd for Causal Relation Annotation. In Proceedings of the Workshop Events and Stories in the News 2018, pages 44–54, Santa Fe, New Mexico, U.S.A. Association for Computational Linguistics.
Cite (Informal):
Crowdsourcing StoryLines: Harnessing the Crowd for Causal Relation Annotation (Caselli & Inel, EventStory 2018)
Copy Citation:
PDF:
https://aclanthology.org/W18-4306.pdf
Code
 CrowdTruth/Crowdsourcing-StoryLines
Data
FrameNet