New testing framework reveals hidden failure points in conversational AI agents

Conversational AI agents powered by large language models can pose real dangers when their internal logic fails—completing transactions without user confirmation or bypassing security checks. The challenge lies in testing for these state-dependent failures, since the critical verification steps and guardrails often sit behind layers of multi-turn conversation, hidden from conventional testing approaches.

A new approach to discovering hidden boundaries#

Researchers have introduced AgentEval, a black-box testing framework designed to uncover and stress these deeply buried safety boundaries. Rather than probing an agent blindly with random prompts, the system first mines a model of the agent's behavior by studying its conversational patterns. Specifically, AgentEval constructs a conversational workflow graph that captures how the agent moves through different states and decision points.

How it works in practice#

Once AgentEval has mapped the agent's decision structure, it uses the graph to systematically identify guards and checkpoints as explicit test targets. Instead of starting from scratch for each test, the framework replays the exact sequence of conversational steps needed to reach a specific boundary—then applies a targeted perturbation to see whether the agent fails in an unsafe way. The system evaluates pass-or-fail outcomes using only the conversation itself, without needing to inspect the agent's underlying code.

Benchmark results#

In testing against four benchmark agents, AgentEval successfully uncovered 23–38 distinct boundaries per agent—substantially more than simpler baselines. An ablation study confirmed that the structured graph approach was responsible for this improvement: the system found 23 distinct boundaries compared to just 12 with a prompt-only baseline, while also reducing duplicate and false-alarm rates. The framework was even benchmarked against a privileged white-box auditor that had direct access to source code—a standard the black-box approach came close to matching.

Why it matters#

As conversational agents become more integrated into real-world decision-making, the ability to systematically test their safety boundaries—without needing to reverse-engineer their inner workings—becomes increasingly important. This technique offers a practical path to discovering failure modes that could otherwise go undetected.