Deep reasoning beats extra tools for AI coding agents' first-try success

Researchers at arXiv cs.SE conducted an observational study to determine what actually drives reliable code generation in agentic AI systems. The findings challenge a common assumption: that giving coding agents more capabilities—like automated testing tools or specialized system prompts—automatically produces better software.

The Study Design#

The team ran 90 independent agent instances to build the same application, a real-time retrospective board, from a single detailed specification. Each run was scored on a 14-criterion functional rubric (up to 42 points) and evaluated for visual quality. The experiments varied across several dimensions: model generations, two different agent harnesses, two levels of reasoning effort, a testing tool, and two design-focused prompts.

What Actually Matters: Frontier Models & Reasoning Effort#

Model capability dominated results. Frontier-tier models clustered near the performance ceiling, while a low-cost local model dropped to 24–37 points. But the most striking finding was the impact of reasoning effort: raising it from High to xHigh lifted first-try perfect runs from 28 percent to 89 percent and reduced corrective prompts needed by roughly five-fold. This improvement came at a cost—9 to 29 percent higher computational expense—but delivered measurably better reliability.

Where Tools and Prompts Fell Short#

The testing tool, despite adding 42 to 68 percent to operational costs, produced no improvement in functional scores, reliability, or even interface-visible criteria. Meanwhile, a design-oriented prompt successfully boosted visual quality (from 3.0 to 4.5 on a 5-point scale) without lifting functional performance. Notably, a one-paragraph paraphrase of the prompt's directive replicated the entire visual lift, suggesting much of its value came from framing rather than sophisticated instruction.

The Container Deployment Problem#

Analysis at the criterion level revealed failures masked by aggregate scores. Container deployment emerged as the dominant defect, failing on first try in 44 percent of runs. Failure rates for this criterion shifted sharply across model generations, even as overall mean scores budged less than a point—underscoring how aggregate metrics can hide critical problem areas.