New benchmark tests AI coding agents on spotting and fixing bugs without bug reports

Most benchmarks that evaluate AI coding assistants assume developers hand off detailed bug reports—a luxury rarely available in practice. A new benchmark flips this model on its head, testing whether large language model-powered coding agents can find and fix bugs on their own.

The challenge of proactive debugging#

Researchers have released Active-SWE, a benchmark designed to evaluate how well AI coding agents perform when operating without the safety net of issue reports. Rather than asking an agent to fix a specific bug described in a ticket, the benchmark challenges agents to scan codebases, identify problems independently, and repair multiple defects in a single codebase—mirroring real-world software maintenance.

The dataset covers 1,663 tasks spanning six bug categories and eight programming languages, providing a comprehensive evaluation ground for this largely unexplored capability.

A dual-track evaluation approach#

To construct the benchmark fairly, the researchers developed a difficulty-aware task formulation pipeline that organizes bugs by complexity. The evaluation framework then assesses agents across multiple scenarios: locating and fixing recorded bugs, handling situations where multiple defects exist in the same code, and even discovering previously unknown potential bugs that haven't been formally documented.

This structure moves the field beyond the narrow focus of fixing one known bug at a time, acknowledging that real engineers often juggle multiple problems and uncover edge cases as they work.

Current limitations of state-of-the-art models#

Early experiments reveal sobering results: even the best-performing coding agents struggle significantly with proactive bug-fixing tasks. They show weak performance in all three core areas—pinpointing bugs without explicit guidance, managing multiple defects in tandem, and spotting latent issues that haven't yet been reported.

The findings suggest that autonomous bug discovery and repair remains a hard problem for current AI systems, and that the jump from "fix this bug" to "find and fix all bugs" is far steeper than existing benchmarks have implied.