New Evaluation Framework Tests How Well AI Handles Long Legal Documents

Modern language models can process documents stretching to 1 million tokens, yet researchers still lack clear tools to measure how well they handle real-world long-context tasks. A new evaluation framework called Gavel tackles this challenge by examining how effectively language models summarize multi-document legal cases—a domain where single cases routinely exceed 100,000 tokens.

Two-Part Evaluation Approach#

Gavel splits evaluation into two complementary components. Gavel-Ref uses a reference-based method that checks summaries against established criteria, examining whether key facts are present, analyzing remaining information gaps, and assessing writing quality. Meanwhile, Gavel-Agent operates reference-free, directly cross-checking summaries against source documents to evaluate factual accuracy without relying on gold-standard summaries.

What the Testing Revealed#

Testing across 12 advanced language models uncovered consistent patterns in model behavior. Models tend to leave out important details more often than they invent false ones. While straightforward information like filing dates rarely gets missed, complex or infrequent details—such as settlement terms—frequently go unmentioned. Performance also deteriorates as case documents grow longer, suggesting that even cutting-edge systems struggle with scale.

The Gavel-Agent approach demonstrated efficiency gains, using at least 36% fewer tokens than baseline evaluation methods while maintaining competitive accuracy. This efficiency extends beyond law: when applied to medical document summarization, the agent-based approach performed best while reducing token consumption by at least 77%.

Grounding in Human Judgment#

The researchers validated Gavel's reliability by collecting 160 hours of human annotations, ensuring the framework reflects genuine evaluation standards rather than automated assumptions alone. This meta-evaluation step strengthens confidence that the results capture meaningful distinctions in model performance.

The framework is available on arXiv.