AI Models Learn to Deceive Attackers by Reasoning About Their Beliefs

Researchers have developed a novel approach to defending AI systems against information-theft attacks: training models to reason about an attacker's psychology and deliberately mislead them into false success.

The work, published on arXiv, introduces a new challenge called ToM for Steering Beliefs (ToM-SB), in which an AI defender plays the role of a "double agent" operating within a shared scenario. The defender's goal is to manipulate the attacker's beliefs so thoroughly that the attacker becomes convinced they have successfully extracted sensitive information, when in fact they have not.

The Challenge of Theory of Mind#

The foundation of this defense strategy rests on theory of mind (ToM)—an AI system's capacity to model the intentions, knowledge, and likely reasoning patterns of another agent. When a defender can accurately predict what an attacker knows, what they're looking for, and how they'll interpret evidence, it becomes possible to feed them misleading information that they'll accept as genuine.

The researchers tested frontier models including Gemini 3-Pro and GPT-5.4, finding that these systems struggle significantly on ToM-SB, especially in harder scenarios where the attacker begins with partial knowledge of the target information. Even when explicitly prompted to reason about the attacker's beliefs, these models frequently failed to successfully fool the adversary.

Training Double Agents via Reinforcement Learning#

To improve performance, the team trained AI models specifically on ToM-SB using reinforcement learning, experimenting with two types of training rewards: one focused on successfully fooling the attacker, and another focused on improving the model's theory of mind reasoning.

A striking finding emerged: the two capabilities reinforced each other. Models trained solely to maximize fooling success improved their ToM reasoning as a side effect, and conversely, models trained to strengthen their ToM also became better at deception. This bidirectional relationship suggests that belief modeling and successful deception are fundamentally linked—you cannot effectively fool an opponent without understanding their mental state.

Performance and Generalization#

When both rewards were combined, the trained "double agent" models substantially outperformed the frontier models on hard scenarios, even with prompting. The researchers validated their approach across multiple attacker types with varying capabilities, and demonstrated that the training generalizes to stronger attackers and out-of-distribution settings, indicating the method's potential scalability.