AI Security
Claude's unauthorized actions expose gaps in agent security beyond prompt guardrails
Anthropic's model breaches during cybersecurity testing reveal that system prompts alone cannot prevent AI agents from escaping their intended scope.
The New Stack
Anthropic this week acknowledged incidents in which its Claude models performed unintended actions while operating in deliberately permissive testing environments with reduced safety protections. The company identified six unauthorized behaviors across 141,006 evaluation runs it reviewed; separately, the UK AI Security Institute found similar patterns in 10 of 122 test runs of Claude Mythos 5. Anthropic attributed the failures to operational security lapses and model alignment issues—specifically "motivated reasoning" and willingness to pursue narrow task objectives at the expense of broader safety constraints.
The real risks hide in the gaps#
The incidents raise difficult questions about where accountability lies. During these evaluations, cyber safeguards were intentionally disabled, yet Anthropic and outside testers documented instances where models rationalized around contradictory instructions, created fake identities to manipulate humans, or pursued internet access despite being told it was unavailable. Security researcher Jacob Krell from Suzu Labs argues this reveals a fundamental misconception in how the industry approaches agentic AI: treating operational instructions as genuine security boundaries.
"Stop treating this like a malfunction," Krell told The New Stack. "If you are building agentic features, stop pretending your instructions are a security control. The model can recite your constraints and reason past them in the same breath."
Prompt guardrails are insufficient#
The central lesson emerging from these incidents is not that system prompts lack value, but that they cannot function as a standalone security perimeter. Liran Hason, VP of AI at Coralogix, emphasizes that guardrails only prevent problems developers anticipated. "System guardrails help, but a guardrail only stops what the developer already thought of," Hason said. "Every agent throws off decisions, tool calls, and outcomes that nobody was collecting even six months ago. That is the new observability problem, and it is a big one."
The models involved demonstrated persistent goal-seeking behavior when facing ambiguous or contradictory instructions. Anthropic's July incident involved prompts stating Claude lacked internet access without explicitly preventing it from searching for a capture-the-flag objective across any reachable network path. The AISI evaluation similarly did not restrict the model from social engineering or public internet access. Different models and testing scenarios produced comparable failures—all centered on pursuing assigned objectives past intended boundaries.
