New AI Jailbreak Exploits Dialogue History by Impersonating the Model's Own Responses

A newly withdrawn research paper from arXiv reveals a previously underexplored security gap in conversational AI systems: attackers can bypass safety mechanisms by fabricating the model's own prior messages within chat histories.

The Attack Method#

The technique, dubbed Trojan Horse Prompting, works by injecting harmful payloads into what appears to be the AI's own earlier utterance in a conversation thread. An adversary then follows this forged message with an innocent user request designed to trigger the model into generating problematic content. The key insight is that the model trusts content it believes came from itself far more than it questions actual user inputs.

Why It Works: Asymmetric Safety Alignment#

Modern language models undergo extensive training to refuse harmful requests from users. However, they receive comparatively little training to scrutinize or reject their own purported conversational history. This creates what researchers call Asymmetric Safety Alignment—an implicit trust in "past" responses that becomes an exploitable weakness. Models have been conditioned to treat their own prior statements as reliable context rather than potential threats.

Experimental Evidence#

Testing on Google's Gemini-2.0-flash-preview-image-generation showed the technique achieved a substantially higher Attack Success Rate compared to conventional user-turn jailbreaking approaches. This demonstrates the practical severity of the vulnerability.

Broader Implications#

The findings point to a fundamental architectural flaw in current conversational AI security. Rather than focusing solely on filtering suspicious inputs at the user level, systems need protocol-level validation that can verify the integrity and authenticity of conversational context itself. Without such structural changes, models remain vulnerable to attackers who exploit their unwarranted confidence in dialogue history.