Study shows verification tools improve AI code agents only when they match actual failure modes

Researchers tested how much value verification tools add to AI-powered coding agents, and the answer is more nuanced than "more tools, better code." The study reveals that a verification tool's effectiveness hinges on whether it can actually catch the failures that matter in a given context.

The experiment#

The team built a minimal coding agent and systematically varied which tools it had access to—forming the agent's "verification surface." They deployed this agent across six different AI models and eight distinct tool configurations to generate 1,116 web applications. Human graders scored each application against a consistent rubric, and automated tests probed for defects in the API behavior.

Where gains arrive first#

The cheapest win comes from ensuring an application launches at all. Without any tools, roughly one in seven builds fails outright. Adding a single boot probe—a lightweight check that the app starts—eliminates almost all of these crashes for about 35 percent of the token cost that a full shell would consume. A complete shell environment multiplies the cost of the no-tools baseline by 2.35 times.

The screenshot gap#

Screenshots prove most useful when problems are visually apparent—wrong element placement, broken interactions, layout issues. Yet even in those scenarios, the marginal gain over a shell alone is small and disappears when researchers correct for multiple statistical comparisons. Where failures can only be detected through measurement rather than sight—like maintaining smooth scrolling performance on a list with 100,000 rows—screenshots contribute nothing.

The bottom line#

A verification tool's true value emerges only when its reach aligns with how an application actually fails. Blindly expanding a tool kit without considering what each tool can diagnose wastes tokens and adds latency without lifting output quality.