AI
One in Seven SWE-Bench Instances Show Misaligned Pull Requests and Issues
A new tool called PAIChecker identifies problems in how code-fixing benchmarks pair pull requests with their corresponding issues, uncovering a widespread data quality gap.
Benchmarks like SWE-Bench—datasets used to measure how well large language models can solve real software engineering problems—rely on pairing code changes with the issues they address. But new research reveals a significant alignment problem: roughly one in seven instances in the verified version of SWE-Bench contain mismatches between the problem statement and the proposed solution.
The Scale of the Problem#
Researchers examining SWE-Bench Verified found that 13.6% of PR-Issue pairings exhibit misalignment across five distinct patterns, spanning eleven specific failure scenarios. The problem stems from the complexity of maintaining large codebases, where pull requests may reference issues imprecisely or address tangential concerns not captured in the original problem description.
How PAIChecker Works#
To address this issue quality gap, the team developed PAIChecker, a multi-agent system that validates PR-Issue alignment through three phases:
- Pattern identification: Detecting specific types of misalignment
- Cross-agent label synthesis: Combining assessments across multiple AI agents to improve accuracy
- Code-level validation: Verifying alignment at the source code level to ensure findings reflect actual implementation details
Validation Results#
Testing on both SWE-Gym and SWE-Bench Multilingual, PAIChecker achieved strong performance across four different LLM backbones, reaching binary accuracy rates of up to 92.12% and 91.67% respectively. These results suggest the approach is both reliable and generalizable across different model types and datasets.
Why It Matters#
As LLMs become more widely used for code generation and bug fixing, the quality of evaluation benchmarks directly affects how well we understand their real-world capabilities. By systematically identifying problematic pairings, PAIChecker helps ensure future benchmark construction is more rigorous, ultimately leading to more trustworthy assessments of AI coding systems.
