LLMs Generate and Grade Their Own Training Data to Boost Automated Bug Repair

Automated program repair (APR)—the process of using AI to identify and fix software bugs—has long struggled with a fundamental constraint: insufficient training examples, especially across diverse bug types and programming languages. A new paper presents a self-contained approach in which large language models both generate the training data and assess its quality.

The Data Bottleneck#

APR systems learn to fix code by studying examples of buggy programs and their corrected versions. But real-world bug datasets are limited and often skew toward certain languages and error patterns. This gap has kept repair systems from generalizing well across different codebases and bug categories.

Generating Synthetic Examples with LLMs#

The researchers deployed multiple state-of-the-art LLMs to synthesize roughly 30,000 paired examples—each comprising buggy code and a working fix. These samples spanned 12 programming languages and 13 distinct bug categories, providing breadth that natural datasets rarely offer. Rather than simply accepting whatever the models produced, the team added a critical second step.

Cross-Model Quality Evaluation#

Before using the synthetic data to train APR systems, the researchers had independent LLMs evaluate each example against five dimensions:

• Correctness — Does the fix actually resolve the bug?
• Code quality — Is the fixed code clean and maintainable?
• Security — Does the repair introduce new vulnerabilities?
• Performance — Will the fix slow down or improve execution?
• Completeness — Is the fix thorough and well-formed?

This filtering step removed low-quality samples, ensuring that downstream training benefited from verified examples.

Results: Significant Gains#

When the filtered synthetic dataset was used to train APR models, performance jumped markedly on the standard VulRepair benchmark:

• Top@1 accuracy improved from 11.68% (baseline) to 17.18%—a