Pruning Basic Elements for Better Automatic Evaluation of Summaries

Ukyo Honda, Tsutomu Hirao, Masaaki Nagata


Abstract
We propose a simple but highly effective automatic evaluation measure of summarization, pruned Basic Elements (pBE). Although the BE concept is widely used for the automated evaluation of summaries, its weakness is that it redundantly matches basic elements. To avoid this redundancy, pBE prunes basic elements by (1) disregarding frequency count of basic elements and (2) reducing semantically overlapped basic elements based on word similarity. Even though it is simple, pBE outperforms ROUGE in DUC datasets in most cases and achieves the highest rank correlation coefficient in TAC 2011 AESOP task.
Anthology ID:
N18-2104
Volume:
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Editors:
Marilyn Walker, Heng Ji, Amanda Stent
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
661–666
Language:
URL:
https://aclanthology.org/N18-2104
DOI:
10.18653/v1/N18-2104
Bibkey:
Cite (ACL):
Ukyo Honda, Tsutomu Hirao, and Masaaki Nagata. 2018. Pruning Basic Elements for Better Automatic Evaluation of Summaries. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 661–666, New Orleans, Louisiana. Association for Computational Linguistics.
Cite (Informal):
Pruning Basic Elements for Better Automatic Evaluation of Summaries (Honda et al., NAACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/N18-2104.pdf
Data
Universal Dependencies