Agentic AI
Study reveals how AI coding agents struggle with multi-hunk bug repairs
Researchers benchmark LLM-driven agents on fixing bugs spread across multiple code regions, uncovering sharp differences in performance and surprising patterns in how agents fail.
A new study examines how modern large language model coding agents handle a category of software bugs that has been largely ignored by automated repair research: defects scattered across multiple, separated sections of code. Unlike traditional single-location bugs, these "multi-hunk" problems demand coordinated edits that present substantially greater technical difficulty.
The benchmark and key findings#
Researchers evaluated four state-of-the-art agents—Claude Code, Codex, Gemini-cli, and Qwen Code—on 404 multi-hunk bugs from the PolyHunk dataset, generating 1,616 repair attempts for detailed behavioral analysis. The results revealed striking disparities in capability. Localization accuracy (finding the bugs in the first place) ranged from 40.4% for Qwen Code to 75.3% for Codex. Repair success rates were even more variable, spanning from 26.98% to 92.82%, with Claude Code demonstrating the highest proficiency.
How bug characteristics affect performance#
Performance degradation followed a clear pattern: as bugs became more dispersed and complex—measured by hunk divergence and spatial proximity—all agents struggled. The best-performing agents (Claude Code and Codex) maintained semantic consistency and achieved net positive improvements in test outcomes. Lower-performing agents, by contrast, frequently introduced new failures while attempting fixes.
The hidden cost of failure#
Perhaps most strikingly, agents that ultimately failed at repairs did not fail fast. Instead, they consumed substantially more resources—between 33% and 440% additional input tokens—and required 35% to 330% longer execution times before abandoning the task. This inefficiency has practical implications for deployment costs and latency.
Improving agent performance with context#
The researchers developed Maple, a tool that supplies agents with repository-level context to enhance their understanding. When used with Gemini-cli, Maple improved repair accuracy by roughly 21%, primarily through better bug localization. This suggests that augmenting agent capabilities with structured contextual information can meaningfully narrow performance gaps.
