Google's Agent Anomaly Detection catches 87% of executed injection attacks in independent benchmarks
Testing against five external security benchmarks, Google's new anomaly-detection tool flags most attempts to hijack AI agents through poisoned data and indirect prompts.
Google Developers Blog (AI)
AI agents are now trusted with sensitive tasks—reading emails, booking travel, managing finances—and each tool call exposes them to data written by others who may harbor hostile intent. Hijacking an agent's goals through such data, known as indirect prompt injection, tops OWASP's 2026 security risk list for agentic applications.
Google's Agent Anomaly Detection, now in private preview on the Gemini Enterprise Agent Platform, monitors agent behavior logs for signs of goal deviation without slowing operations. Rather than validate the tool against Google-authored tests, the team put it through five public benchmarks created by independent researchers—a deliberate choice to avoid inflating performance metrics on self-favoring assessments.
How the testing worked#
Researchers ran five published agent-security benchmarks through an agent powered by gemini-2.5-flash using Google's Agent Development Kit, testing it on each benchmark's simulated environments. The agent's logs and traces fed into Agent Anomaly Detection, mirroring real-world deployment. Each benchmark then ran its own Python code to check whether harmful instructions actually changed the app's state—whether money left an account, for instance. This deterministic check prevented either the detection system or injected instructions from gaming the results.
The five benchmarks encompassed different attack vectors:
- AgentDojo (NeurIPS 2024)
- InjecAgent (ACL 2024)
- Agent Security Bench (ICLR 2025)—the only one planting instructions in the agent's long-term memory
- AgentDyn (2026)
- AgentHarm (ICLR 2025)
Across 1,774 test runs, the system analyzed three possible outcomes for each harmful instruction: never read by the agent, resisted after being read, or carried out.
Detection performance#
When agents actually executed harmful instructions, Agent Anomaly Detection flagged 87% of them for review. Performance varied by benchmark: 99% on AgentDyn, 87% on InjecAgent, 85% on AgentDojo, and 81% on AgentHarm.
The AI models themselves showed some natural resistance—refusing about three-quarters of direct user requests for harm and failing to execute roughly 30% of instructions hidden in tool output. Yet Agent Anomaly Detection proved valuable for catching the remainder that got through.
False positives remained modest on typical office tasks: 2–4% on benchmarks using office scenarios. However, AgentHarm's benign test cases more closely resembled the harmful ones (both requesting social-media posts and file downloads), pushing the false-alarm rate to 16%.
Where it falls short#
Detection weakened in several areas. When agents resisted instructions, the system flagged only 11–25% of those sessions because resistance leaves little behavioral trace. Single small actions—like a lone page visit—blended into ordinary browsing patterns. Poisoned memory proved especially difficult: Agent Anomaly Detection caught only 4 of 18 cases where injected instructions persisted in the agent's long-term storage.
Agents also required a ramp-up period. On AgentDyn, with roughly 220 prior benign sessions, detection hit 33% of executed attacks; performance jumped to 99% with 360 sessions of baseline behavior established.
Next steps and defense strategy#
Google's roadmap includes flagging resisted attempts at lower severity, catching small actions routed to untrusted destinations, improving memory-injection detection, and reducing the burn-in period for new agents. The company emphasizes "defense in depth"—layering controls rather than relying on anomaly detection alone:
- Enable full tracing and logging in deployed agents
- Screen untrusted inputs with prompt-injection filters
- Use Semantic Governance Policy to deny tool calls that drift from user intent
- Require human approval for high-stakes actions (payments, email, data export)
- Grant agents only the tools they need
- Route critical findings to incident teams
Agent Anomaly Detection doesn't block actions; it surfaces suspicious sessions for human review, with findings also appearing in Google Cloud's Security Command Center.