Generating and Evaluating Summaries for Partial Email Threads: Conversational Bayesian Surprise and Silver Standards

Jordon Johnson, Vaden Masrani, Giuseppe Carenini, Raymond Ng


Abstract
We define and motivate the problem of summarizing partial email threads. This problem introduces the challenge of generating reference summaries for partial threads when human annotation is only available for the threads as a whole, particularly when the human-selected sentences are not uniformly distributed within the threads. We propose an oracular algorithm for generating these reference summaries with arbitrary length, and we are making the resulting dataset publicly available. In addition, we apply a recent unsupervised method based on Bayesian Surprise that incorporates background knowledge into partial thread summarization, extend it with conversational features, and modify the mechanism by which it handles redundancy. Experiments with our method indicate improved performance over the baseline for shorter partial threads; and our results suggest that the potential benefits of background knowledge to partial thread summarization should be further investigated with larger datasets.
Anthology ID:
W17-5532
Volume:
Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue
Month:
August
Year:
2017
Address:
Saarbrücken, Germany
Editors:
Kristiina Jokinen, Manfred Stede, David DeVault, Annie Louis
Venue:
SIGDIAL
SIG:
SIGDIAL
Publisher:
Association for Computational Linguistics
Note:
Pages:
263–272
Language:
URL:
https://aclanthology.org/W17-5532
DOI:
10.18653/v1/W17-5532
Bibkey:
Cite (ACL):
Jordon Johnson, Vaden Masrani, Giuseppe Carenini, and Raymond Ng. 2017. Generating and Evaluating Summaries for Partial Email Threads: Conversational Bayesian Surprise and Silver Standards. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 263–272, Saarbrücken, Germany. Association for Computational Linguistics.
Cite (Informal):
Generating and Evaluating Summaries for Partial Email Threads: Conversational Bayesian Surprise and Silver Standards (Johnson et al., SIGDIAL 2017)
Copy Citation:
PDF:
https://aclanthology.org/W17-5532.pdf