New Training Framework Enables Open-Source LLMs to Generate Bug Reproduction Tests

Software developers often struggle to pinpoint the root cause of bugs reported by users. A team of researchers has developed a solution that teaches open-source language models to automatically generate test cases that reproduce software issues from natural language descriptions.

The Problem and Opportunity#

Manually writing tests to confirm and investigate bugs consumes significant developer time and slows down the debugging process. While large language models have shown promise in automating this task, most work in the field relies on expensive, proprietary systems. By contrast, open-source alternatives remain largely unexplored for issue reproduction testing.

Introducing SWE-Tester#

The researchers created SWE-Tester, a training framework specifically designed to equip open-source LLMs with the ability to generate bug reproduction tests. The pipeline includes:

• A curated training dataset of 41,000 examples drawn from 2,600 public GitHub repositories
• Fine-tuning techniques applied to models of varying sizes and architectures
• Systematic evaluation against benchmark datasets

Key Results#

The fine-tuned models demonstrated measurable improvements over baseline performance:

• Up to 10% absolute gain in success rate
• Up to 21% improvement in code coverage on the SWT-Bench Verified benchmark
• Consistent progress with larger models, more training data, and additional computational resources during inference

Broader Impact#

Beyond accelerating manual debugging workflows, this advancement supports test-driven development practices where developers write tests before implementation. The trained models could also enhance automated code repair systems by providing more reliable test coverage. By democratizing this capability in open-source models, the work reduces barriers to adopting AI-assisted development tools.