New Framework Reveals Measurement Gaps Between LLM Safety Scores and Real-World Robustness

A new research paper introduces EvalSafetyGap, a framework that diagnoses why large language model (LLM) safety evaluations can mislead. The core problem: test scores and safety metrics often improve without corresponding gains in the actual robustness that matters—leaving a hidden gap between what we measure and what we need to measure.

The Measurement Problem#

Language models are evaluated using benchmarks, reward signals, and safety tests. Yet there's a persistent issue: these metrics can climb higher while the underlying capabilities they're supposed to reflect remain unverified or even worsening. This mirrors a broader challenge in AI: Goodhart's Law, which states that when a measurement becomes a target, it ceases to be a good measure.

The research team conducted a hybrid survey spanning 2018–2026, synthesizing evidence across eight streams:

• Benchmark validity and drift
• Dynamic evaluation methods
• LLM-as-judge reliability
• Safety evaluation protocols
• Jailbreak and refusal robustness
• Reward hacking
• Mechanistic interpretability
• Governance and auditability

Audit Findings: What the Data Actually Shows#

The authors audited ten language models to test their framework. Key results challenge common assumptions:

• Capability and robustness diverge: The expected relationship between raw capability and sustained adversarial robustness was statistically indeterminate (r = +0.232, p = 0.520), meaning a more powerful model is not reliably a safer one.
• Open-closed gap is modest: The apparent safety difference between open and closed models proved modest and driven mainly by governance and transparency practices—not behavioral robustness improvements.
• Protocol matters: Safety measurements vary significantly depending on how attempts are counted, making comparisons unreliable across studies.

A Vocabulary for Better Safety Work#

Rather than ranking models, the paper offers a diagnostic vocabulary and evidence map. The authors introduce two new conceptual tools:

1. : Breaking down how evaluation metrics behave under optimization pressure.