New Refactoring Benchmark Exposes Real Limits of AI Coding Agents

Existing benchmarks for evaluating AI coding agents have a blind spot: they largely ignore the messy, complex work of refactoring large codebases. A new multilingual benchmark aims to fill that gap—and the results are sobering. The strongest models achieve barely 41% on SWE-Bench ProMax, a rigorous evaluation developed by researchers at Shanghai Jiao Tong University, Peking University, and other institutions.

The finding underscores a deeper problem in how AI agent capabilities are measured. Many popular coding benchmarks suffer from contamination: solutions have leaked into training data, allowing models to appear competent without truly solving the underlying problems. An audit cited in the research revealed that nearly 60% of supposedly solved instances on SWE-bench Verified contain flawed tests.

Why Standard Benchmarks Fall Short#

Most coding agent benchmarks are optimized for speed and reproducibility. They deliberately sidestep large-scale refactoring tasks because such work resists easy automation and verification. This narrow scope creates a false sense of progress. As Vojtěch Pavlík, senior director of technical strategy at SUSE, observes, the best-performing models still cannot comprehend entire codebases at once—a capability that remains "far out of reach."

The consequences are significant. Because frontier models routinely top existing benchmarks—whether through training data leakage or inherent limitations—developers lack honest signals about what these agents can actually accomplish in production settings.

The Refactoring Challenge#

Refactoring presents unique difficulties for language models. It demands absolute precision: zero tolerance for errors, no behavior changes, and complete reversibility. Unlike feature development, where approximate solutions can work, refactoring is an all-or-nothing proposition.

Core obstacles include:

• Structural understanding: Many engineers treat refactoring as a text-generation task, feeding entire codebases into a context window and requesting multi-file diffs. Shane Warden, principal architect at ActiveState, argues this approach conflates token proximity with genuine understanding. "I don't believe that premise," he says. "Token proximity does not guarantee structural understanding."

• Attention limits: Even with sparse attention algorithms (like Deepseek Sparse Attention), context windows can max out, causing models to miss critical details and become confused in large, complex codebases.