AI
Researchers propose standardized framework for comparing LLM-based code repair experiments
A new study reveals that benchmark names alone don't capture how AI systems repair code—calling for a machine-readable schema to make results truly comparable.
Automated program repair systems powered by large language models can approach the same coding task in vastly different ways, yet current reporting methods mask these differences behind simple benchmark names. A new research paper from arXiv proposes a detailed framework to make experimental settings explicit and reproducible.
The hidden variability problem#
When researchers evaluate LLM-based repair systems on standard benchmarks like Defects4J and SWE-bench, they typically report which bugs were fixed and what tests validated the patches. However, this surface-level reporting obscures critical differences in how each system operates. Two studies might claim to use the same benchmark but actually be solving different problems—one might work from just a bug description and search for a fix locally, while another might explore the entire code repository and iterate through multiple repair attempts based on test feedback.
These variations arise from differences in initial input (what information the system receives about the bug), tool access (which repository and testing operations are permitted), repair-time feedback (how the system learns from failed attempts), and resource constraints (time and compute limits).
Framework for transparency#
The researchers introduce a machine-readable schema that captures each of these dimensions explicitly. The framework characterizes repair tasks along seven key axes:
- Task unit — the scope of what needs fixing
- Fault-localization assumptions — what the system knows about where the bug is
- Initial input — the starting information provided to the system
- Tool access — repository and testing capabilities available during repair
- Repair-time feedback — what the system learns from each failed attempt
- Final validation — how patches are tested
- Resource budget — time and computational limits
By analyzing existing reported results from systems evaluated on Defects4J and SWE-bench, the authors show that benchmark identity alone is too coarse-grained to reconstruct what task was actually performed or to fairly compare reported repair rates across studies.
