Agentic AI
New framework exposes hidden security gaps in AI agent defenses
Ajar reveals that current safeguards for language model agents fail to prevent unnecessary access to tools and data even when existing attack tests pass.
Language model agents rely on tools to carry out tasks, and the information they encounter during execution can influence which capabilities they access. Researchers have developed new safeguard techniques that try to control what agents can do—limiting tool access, managing information flow, and isolating risky operations. But existing security tests focus narrowly on stopping prompt injection attacks without measuring whether defenses grant more privileges than necessary.
The measurement gap#
A defense can score well on current benchmarks by blocking attacks and maintaining utility, yet still leave unused capabilities available. These unnecessary permissions represent a security risk that conventional testing does not surface. This gap is what Ajar aims to address—a measurement framework that quantifies unneeded privilege independently from attack resistance or task performance.
How Ajar works#
Rather than creating new attack scenarios, Ajar integrates with existing agent security benchmarks like AgentDojo by reusing their tasks, tool definitions, and success criteria. For each benign task, the framework generates tool calls that the task logic does not require, then presents these to the defense at every decision point. Any unnecessary call that succeeds indicates open privilege the defense failed to restrict.
What the testing revealed#
Researchers evaluated five defenses—Progent, CaMeL, AC4A, Permission Assistant, and Claude Code's Auto mode—and found striking differences. Two defenses leaked similar amounts of unnecessary access yet achieved vastly different success rates on legitimate tasks. One defense appeared tighter partly because it refused access to tools the task should have been able to use. Critically, these gaps in privilege control did not correlate with attack success or utility scores on standard metrics.
Why it matters#
The finding suggests that security for AI agents requires at least three independent measurements: attack resistance, utility preservation, and open privilege. A defense strong on two axes may still expose risky unnecessary permissions, creating a potential avenue for misuse that static benchmarks miss.
