Agentic AI
New benchmark reveals AI coding agents struggle with long-term software maintenance
Researchers introduce SWE-Milestone, showing that AI agents drop from 80% accuracy on isolated tasks to just 38% when handling continuous software evolution.
A new evaluation framework exposes a significant weakness in today's most advanced AI coding agents: while they excel at solving individual programming challenges, they falter dramatically when tasked with maintaining and evolving software systems over time.
The performance gap#
Researchers from the computer science community have introduced SWE-Milestone, a benchmark designed to test AI agents on realistic, ongoing software development scenarios. The findings are striking: frontier models that achieve success rates exceeding 80% on isolated, one-time coding tasks see their performance plummet to a maximum of 38% when working through sequences of interconnected development milestones. This gap highlights agents' difficulty managing error accumulation and maintaining system integrity across a series of related changes.
How the benchmark works#
The research team developed DeepCommit, an agentic pipeline that reconstructs meaningful development sequences from raw commit history. Rather than treating code changes as independent puzzles, the benchmark organizes them into milestone DAGs—directed acyclic graphs where each milestone represents a functionally cohesive development goal. This structure mirrors real-world software evolution, where changes depend on earlier work and introduce technical debt if not handled carefully.
What's being tested#
The evaluation covered 12 frontier AI models across 4 different agent frameworks, all working through streams of milestone-level tasks. The core challenge is two-fold: sustaining system integrity as changes accumulate and preventing errors from compounding through successive modifications. These dimensions of long-term software stewardship have been largely absent from earlier benchmarks, which focused on disconnected, immediate problem-solving.
Why it matters#
As AI agents become increasingly deployed as long-running systems in production environments, understanding their limits in continuous evolution is critical. The gap between isolated-task and sequential-task performance suggests that current models may not be ready for roles requiring sustained maintenance and architectural awareness over extended periods.
