Agentic AI
New benchmark tackles contamination crisis in AI code-fixing evaluation
SWE-MERA, a dynamic dataset of real GitHub issues, aims to solve data leakage problems that have inflated performance claims on the widely-trusted SWE-bench.
Researchers have identified a major credibility problem in one of the leading benchmarks for evaluating AI models on software engineering tasks. The popular SWE-bench dataset, used to measure LLM coding capabilities, suffers from substantial data contamination—with recent analysis showing that nearly a third of reported successful fixes either leak the solution directly or rely on inadequate test coverage.
The contamination problem#
A closer look at SWE-bench's results reveals troubling gaps in validity. According to the researchers behind the new benchmark, approximately 32.67% of passing patches involve direct leakage of the correct solution, while an additional 31.08% pass only because the test suite fails to catch errors. This means that many performance numbers credited to modern LLMs may overstate their actual problem-solving ability.
Introducing SWE-MERA#
To address these issues, a team has introduced SWE-MERA, a freshly designed benchmark that takes a different approach:
- Real-world source: Tasks are automatically harvested from actual GitHub issues, ensuring they reflect genuine software engineering challenges
- Continuous updates: Rather than a static dataset that ages and becomes prone to gaming, SWE-MERA is dynamically maintained and refreshed
- Quality validation: A rigorous vetting pipeline screens out contaminated or flawed tasks
The result is approximately 10,000 potential tasks in the pipeline, with 728 currently validated and ready for evaluation. Testing with the Aider coding agent confirms the benchmark can effectively discriminate between stronger and weaker models.
Early results#
The team evaluated a dozen recent LLMs on tasks collected between September 2024 and June 2025, demonstrating that SWE-MERA successfully measures real performance differences across state-of-the-art systems. The work underscores how critical reliable benchmarks are as AI capabilities advance—faulty evaluation data can mislead the field about which techniques actually work.
