AI Code Agents Show Uneven Fragility When Facing Rewritten Code

Researchers have uncovered a surprising weakness in AI-powered code agents: their ability to fix real software bugs deteriorates when the surrounding codebase is rewritten in ways that preserve its meaning but change its surface form.

The Test#

The evaluation applied common code transformations—restructured logic, injected unused code, and renamed identifiers—to real codebases while preserving their semantic behavior. Researchers then tested whether agents could still fix existing issues in these altered versions. They benchmarked two different agent systems (mini-SWE agent and OpenCode) paired with four frontier language models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B) on instances from SWE-bench Verified and SWE-bench Pro datasets. Each test ran multiple times to separate the impact of code perturbations from natural model variance.

Findings#

The impact was modest but consistent. The worst-affected configurations saw bug-fix success rates drop by as much as 6.7 percentage points, with six out of 16 model-scaffold-dataset combinations showing statistically meaningful degradation. Notably, no model ranked best across all conditions: Qwen performed robustly under the simpler mini-SWE agent scaffold but proved brittle under OpenCode, creating what the researchers call a "jagged robustness frontier." The simpler scaffold itself showed greater tolerance for perturbation overall.

Why It Matters#

The findings underscore a gap between how AI code agents perform under laboratory conditions versus real-world diversity. Although modern models tolerate these transformations reasonably well, their inconsistent performance raises concerns about reliability when deployed across varied codebases with different coding styles and structures.