New benchmark reveals steering methods destabilize LLM safety across multiple dimensions

Researchers have introduced SteeringSafety, an evaluation framework designed to assess how well representation steering methods—techniques that aim to guide language model behavior—perform when tested across multiple safety dimensions simultaneously.

What the benchmark covers#

The benchmark evaluates five leading steering approaches (DIM, ACE, CAA, PCA, and LAT) on three models: Gemma-2-2B, Llama-3.1-8B, and Qwen-2.5-7B. Rather than testing a single safety goal, SteeringSafety examines nine distinct safety perspectives using 18 datasets. These include refusal capabilities, bias reduction, hallucination control, social behaviors, reasoning quality, epistemic integrity, and normative judgment.

Key findings on method-safety tradeoffs#

The results reveal a critical problem: steering methods exhibit substantial entanglement, where improvement on one safety dimension frequently degrades performance elsewhere. The consequences vary dramatically across specific pairs of concerns:

• Social behaviors are most vulnerable, with degradation reaching 76%
• Jailbreaking attempts that weaken refusal protections often undermine commonsense morality by up to 26%
• Hallucination steering produces unpredictable political shifts, ranging from a 25% rightward lean to a 28% leftward lean depending on the model

Why this matters#

While representation steering shows promise for targeted behavior modification, the benchmark demonstrates that these methods cannot be evaluated in isolation. The cross-cutting impacts suggest that modifying language models through steering requires careful attention to unintended downstream effects. A technique that successfully prevents harmful outputs may inadvertently amplify other vulnerabilities or shift model behavior in unexpected ways.