Agentic AI
Researchers pinpoint what makes code-fixing tasks hard for AI agents
A new study reveals that software issue difficulty can be predicted from structural properties—and patch complexity is the primary driver.
Benchmarks for coding AI agents are improving rapidly, but researchers haven't had a clear way to understand why some tasks prove harder than others. A new study from arXiv tackles this gap by systematically analyzing what makes software issue resolution tasks difficult for autonomous agents.
Mapping difficulty to structure#
The research team built a measurement framework to identify which structural properties of coding tasks correlate with agent success or failure. Using CoderForge-Preview—currently the largest dataset of coding agent trajectories—they extracted features from three angles: the changes required to fix a bug (the patch), the codebase being modified (the repository), and the instructions given to the agent (the prompt).
Their analysis shows that task difficulty is substantially predictable from these static features alone, achieving an AUC of 0.863. This suggests that the inherent structure of a problem encodes much of what determines whether an agent will solve it.
Patch complexity dominates#
The study's key finding is that patch fragmentation and repository scale are the dominant drivers of difficulty. Tasks requiring changes scattered across multiple files or within very large codebases present steeper challenges for agents. For tasks in the middle difficulty range, linguistic properties of the prompt prompt become more visible as contributing factors—revealing a layered structure where different properties matter at different difficulty levels.
Why it matters#
These insights create an opportunity for better benchmark design. Rather than relying solely on empirical testing to learn which tasks are hard, researchers can now estimate difficulty before running evaluations. This opens the door to constructing benchmarks with controlled difficulty levels, making it easier to assess and compare agent capabilities on a more systematic footing.
The work underscores that agent performance isn't just about raw intelligence—the shape of the problem matters significantly.
