LLMs boost fixes for regression bugs 1.6x when given original code changes

Researchers have completed the first large-scale empirical evaluation of how well modern large language models (LLMs) can repair regression bugs—defects introduced when code changes break previously working functionality. The findings suggest LLMs are far more effective than older automated repair techniques, but their performance jumps dramatically when given context about the original problematic change.

The regression bug repair gap#

Automated program repair (APR) has advanced rapidly in recent years, with LLM-based approaches showing particular promise for fixing general software bugs. Yet little was known about how these techniques actually perform on regression bugs specifically, which present a distinct challenge: the bug exists only in relation to a prior working version.

Researchers introduced RegressionBug4APR, a benchmark containing 200 regression bugs from real GitHub repositories across Java and Python, designed to enable systematic testing of APR methods. The benchmark revealed clear patterns about where existing tools fail and where new approaches succeed.

How the techniques compared#

Traditional automated repair tools—the workhorse methods of earlier APR research—proved entirely ineffective, failing to fix any bugs in the benchmark. This finding signals that conventional pattern-matching and search-based approaches are ill-suited to the unique structure of regression problems.

LLM-based APR approaches, by contrast, showed genuine promise. The real breakthrough came when researchers incorporated information about the bug-inducing code change—the original modification that introduced the regression—into the repair process. This context-aware enhancement yielded 1.6 times more successful repairs than LLM approaches without such historical context.

The role of context#

An ablation study examined which pieces of the bug-inducing change information contributed most to improvement. The results confirmed that understanding what changed is essential to understanding how to fix it. This consistency held across both Java and Python benchmarks, suggesting the principle generalizes beyond language boundaries.

The work provides initial evidence that the most effective repair strategy for regression bugs is not just applying general LLM capabilities to code, but equipping those models with concrete knowledge of the changes that preceded the defect.