Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation

arXiv:2607.10806v1 Announce Type: cross
Abstract: Quantifying abstractiveness in generated summaries is essential for evaluating summarization models beyond
surface-level metrics like ROUGE. We introduce Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction
Ratio (AR) — a set of principled heuristic metrics that measure how much a summary diverges from extractive copying
of the source text. The formulation uses the harmonic mean of document lengths modulated by a cubic non-overlap
factor, yielding dimensionally consistent, bounded output with non-linear sensitivity to the extractive-abstractive
boundary. Evaluation on 100 XSUM documents across four summarization models (BART-large-cnn, Pegasus-xsum, DistilBart,
MT5-small) demonstrates that the metrics successfully discriminate between extractive models (SA ~ 0.12-0.26) and
abstractive models (SA ~ 0.96-1.77), and that the Abstraction Ratio identifies summaries requiring manual evaluation
for potential hallucination. Code and results are available at https://github.com/katweNLP/AbstractionStudy.