Researchers develop new metric to measure safety margins in AI alignment

Researchers have introduced a technique to measure the robustness of safety mechanisms in large language models trained using reinforcement learning from human feedback (RLHF). The work raises questions about how much protection current alignment strategies actually provide against adversarial attacks.

The Safety Margin Problem#

When models are aligned to refuse harmful requests, the question becomes: how much wiggle room remains before they comply? Traditional evaluation methods struggle to capture this operational margin. The researchers tackle this by focusing on a single moment in time—the first decoding step—where the model's confidence in refusing versus complying becomes visible. They define the refusal-affirmation logit gap: the difference between the model's top score for tokens that would refuse and the top score for tokens that would affirm a harmful request.

This metric offers a single number that quantifies per-prompt safety. Across three model families, alignment widened this gap on 97.5% to 99.8% of toxic prompts, suggesting defenses are working. However, the metric also revealed something concerning: the median gap closure tracked closely with attack success rates across different suffix strategies, indicating the safety margin can be meaningfully eroded.

A Fast, Efficient Attack Method#

To validate whether their metric captures something real, the team developed logit-gap steering—a method that finds short text strings (under 10 tokens) to append to prompts that close the gap without requiring backpropagation. Running it on a single A100 GPU for approximately 2 minutes generates effective attack strings, using roughly 125 times less computation than existing GCG-based attacks.

Remarkably, suffixes discovered on smaller 0.5B to 2B parameter models transfer directly to 72B models within the same family. An ensemble of 8 such suffixes achieved attack success rates of 38% to 96% across 13 models tested on standard benchmarks (AdvBench and HarmBench).

Why Perplexity Defenses Fall Short#

A critical finding concerns existing defense strategies. Many recent papers have proposed filtering out adversarial suffixes by penalizing high perplexity (unusual word sequences). These defenses reduced traditional GCG attacks from a 64.7% success rate to just 1.0%, seemingly a major victory. Yet the in-distribution suffixes discovered by logit-gap steering—which have 1,000 to 10,000 times lower perplexity than GCG attacks—remained largely unaffected, slipping through defenses at 76.0% of their original effectiveness.