A Principled Framework for Evaluating Summarizers: Comparing Models of Summary Quality against Human Judgments

Maxime Peyrard, Judith Eckle-Kohler


Abstract
We present a new framework for evaluating extractive summarizers, which is based on a principled representation as optimization problem. We prove that every extractive summarizer can be decomposed into an objective function and an optimization technique. We perform a comparative analysis and evaluation of several objective functions embedded in well-known summarizers regarding their correlation with human judgments. Our comparison of these correlations across two datasets yields surprising insights into the role and performance of objective functions in the different summarizers.
Anthology ID:
P17-2005
Volume:
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
July
Year:
2017
Address:
Vancouver, Canada
Editors:
Regina Barzilay, Min-Yen Kan
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
26–31
Language:
URL:
https://aclanthology.org/P17-2005
DOI:
10.18653/v1/P17-2005
Bibkey:
Cite (ACL):
Maxime Peyrard and Judith Eckle-Kohler. 2017. A Principled Framework for Evaluating Summarizers: Comparing Models of Summary Quality against Human Judgments. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 26–31, Vancouver, Canada. Association for Computational Linguistics.
Cite (Informal):
A Principled Framework for Evaluating Summarizers: Comparing Models of Summary Quality against Human Judgments (Peyrard & Eckle-Kohler, ACL 2017)
Copy Citation:
PDF:
https://aclanthology.org/P17-2005.pdf
Note:
 P17-2005.Notes.pdf
Video:
 https://aclanthology.org/P17-2005.mp4
Code
 UKPLab/acl2017-theta_evaluation_summarization