AI
New benchmark pushes medical AI to pinpoint exact phrases behind clinical summaries
A human-annotated dataset called MAPLE reveals that while large language models excel at sentence-level citations, they struggle to ground claims in the precise phrases clinicians need to verify.
Trustworthy summaries of medical research require more than vague pointers—clinicians need to see the exact words that support each claim. A new benchmark published on arXiv tackles this challenge by introducing a more granular approach to attribution in clinical summarization.
The precision problem#
Current systems typically cite entire sentences or documents when summarizing medical research, forcing clinicians to hunt through surrounding text for the specific phrases that actually matter. This gap between what's cited and what's actionable undermines trust and efficiency. Researchers argue that attribution should operate at the same level of precision that clinicians must work at: the phrase itself.
Introducing MAPLE#
The MAPLE dataset (Medical Aspect-Based Summarization with Phrase-Level Evidence) provides a human-annotated benchmark built on 152 randomized controlled trial abstracts spanning 16 clinically motivated aspects. It contains 1,799 aspect-based summaries, each with dual-level evidence: the supporting sentences and the contributory phrases within them. This dual annotation allows systems to be evaluated on whether they can locate both broad context and precise supporting details.
How the evaluation works#
The research team designed a decoupled framework that separately measures three dimensions:
- Content accuracy — whether summaries are factually correct
- Traceability — whether claims are linked to source sentences
- Locatability — whether evidence is pinpointed to the exact relevant phrases
The framework also estimates how much source text clinicians must review to verify each claim, a practical proxy for usability.
What the benchmarks revealed#
Testing eleven large language models showed a stark disparity in performance. Sentence-level citation was consistently strong, with the best models reaching 90.9% on content F1 scores. Phrase-level grounding proved far more uneven, ranging from 66.1% to 84.5%—and emerged as the most discriminative measure across all models tested. This suggests that generating accurate summaries is not the central bottleneck; the real challenge is localizing supporting evidence with enough precision for clinicians to efficiently verify claims.
