New method trains AI models to steer clear of safety classifier boundaries

Most large language models rely on safety classifiers to block harmful outputs. But a common training approach—pushing models to reduce their likelihood of triggering filters—often backfires: it nudges generation toward the classifier's decision boundary, where both false alarms and slip-throughs become more likely.

The Problem with Naive Safety Fine-Tuning#

When developers fine-tune generative models to avoid safety filters, they typically optimize for lower filtering probability. This strategy, while intuitive, creates a counterintuitive outcome: the model learns to produce text that sits precariously close to the boundary between allowed and blocked outputs. This margin-hugging behavior increases errors on both sides—legitimate requests get wrongly rejected, and problematic outputs occasionally slip through.

Boundary Guidance: A New Approach#

Researchers propose Boundary Guidance, a reinforcement learning method that explicitly steers generation away from the classifier's margin. Rather than simply minimizing filter activation, the approach maintains a safety buffer by directing models toward the interior of the "safe" region, further from the decision boundary itself.

Testing and Results#

The team evaluated their method on challenging prompts including jailbreak attempts, ambiguous requests, and long-context inputs. Using LLM-as-a-Judge evaluations, they found that Boundary Guidance improved both dimensions: generated outputs were safer while remaining more useful and coherent. Ablation studies across different model scales and reward function designs confirmed the robustness of the approach.

Why It Matters#

This work highlights a fundamental tension in AI safety: naive optimization can create brittle systems that fail in ways hard to predict. By explicitly modeling the geometry of the decision boundary, Boundary Guidance offers a more principled path toward models that are both reliably safe and practically useful.