What Makes a Good Summary? Investigating the Focus of Automatic Summarization in an Educational Context

Maartje ter Hoeve, Julia Kiseleva, Maarten de Rijke

Published 2020-12-14, updated 2021-05-25Version 2

Automatic text summarization has enjoyed great progress over the last years. However, there is little research that investigates whether the current research focus adheres to users' needs. Importantly, these needs are dependent on the envisioned target group of the generated summaries. One such important target group is formed by students, due to their usage of summaries in their study activities. For this reason, we investigate students' needs regarding automatically generated summaries by means of a survey amongst university students and find that the current direction of the field does not fully align with their needs. Motivated by our findings, we formulate three groups of implications that together help us formulate a renewed perspective on future research on automatic summarization. First, the educational domain requires a broader perspective on automatic summarization, beyond the approaches that are currently the standard. We illustrate how we can expand these approaches regarding the input material, the purpose of the summaries and their potential format and we define requirements for datasets that can facilitate these research directions. Second, we propose a methodology to evaluate the usefulness of a summary based on the identified needs of a target group. Third, in more general terms, we hope that our survey will be reused to investigate the needs of different user groups of automatically generated summaries to broaden our perspective even further.