Researchers Expose Novel Prompt Injection Attack that Deceives Leading AI Models

Security researchers have identified a significant vulnerability in large language models: excessive training on safety data can actually make them more susceptible to sophisticated attacks. A new study introduces FATS (Feign Agent Attack with Toxic-shots), a prompt injection technique that manipulates this weakness to coerce harmful outputs from models that are otherwise designed to refuse them.

The Attack Method#

FATS works by disguising malicious instructions as benign security checks. The approach combines three key tactics: obscuring how it extracts system preferences, poisoning the safety examples used during training, and gradually inducing the model to behave in ways that violate its guidelines. Rather than relying on brute-force jailbreaking, this method exploits the model's learned tendency to trust security-related signals.

Real-World Vulnerability#

The researchers tested their attack against major production models using the FAQuery dataset alongside established benchmarks like Advbench. The results proved sobering: GPT-4.1 fell to the attack 61.6% of the time, while Deepseek-R1 was compromised in 99.3% of test cases. These success rates demonstrate that even advanced models with state-of-the-art safety alignment are vulnerable when attackers target the safety training process itself.

Implications for Model Development#

The findings highlight a critical tension in modern language model training: while techniques like Reinforcement Learning with Human Feedback (RLHF) improve ethical behavior, models may become overconfident in security-related data. This false trust creates an opening for attackers who can craft convincing false signals. The research underscores the urgent need to validate and secure every component of training data, particularly those labeled as safety-critical, rather than assuming such information is inherently trustworthy.