AI Safety
New Attack Breaks Safety Defenses in Open-Source AI Models—But Researchers Have a Fix
A machine learning team demonstrates how a refined prefilling attack circumvents recent safety improvements in large language models and introduces a novel technique to strengthen alignment.
Researchers have identified a vulnerability in a popular safety defense mechanism for open-source large language models and proposed a stronger alternative to protect against misuse.
The Problem: A More Sophisticated Attack#
Prefilling attacks—where users inject affirmative text before a model generates its response—have long threatened the safety guardrails of open-source LLMs. A recent defense strategy attempted to address this by augmenting training data so models would learn to refuse harmful requests even when presented with affirmative prefills. However, scientists at the intersection of AI security have now shown this defense is insufficient.
The team developed what they call the Rank-Assisted Prefilling (RAP) attack, which works by scanning the model's top 20 predicted tokens at each generation step and strategically selecting low-probability harmful tokens while bypassing the high-probability refusal responses the model learned to output. This simple but effective technique allows attackers to extract harmful content from models that seemed defensively stronger.
PRESTO: A New Path Forward#
Rather than simply pushing models to output refusals with higher probability, the researchers propose PRESTO (PRefill attEntion STOpping), which takes a fundamentally different approach. The method matches the rank distribution of tokens generated by the safety-aligned target, rather than just their probability scores. This involves regularizing how much attention the model places on harmful prefill tokens during generation.
The results are substantial: PRESTO delivers up to 4.7x improvement in safety against RAP attacks across three widely-used open-source models compared to the basic data augmentation defense alone.
Why This Matters#
As open-source models become increasingly capable, the ease with which they can be misused grows alongside their utility. Defenses that work in principle but fail in practice offer only false security. By identifying a practical attack vector and offering a measurably stronger countermeasure, this work moves the needle on practical safety for models deployed in the wild. The authors have released accompanying code to enable broader adoption and further research.
