Decade-Long Survey Reveals Critical Gaps in Code Benchmark Quality for AI Models

A comprehensive examination of code evaluation standards has exposed a troubling disconnect: despite growing awareness of benchmark quality issues, the practice of building rigorous code benchmarks has not kept pace. Researchers analyzing over 670 code benchmarks across a decade-long period found that careless construction methods—particularly around test coverage—remain widespread.

The Problem with Today's Benchmarks#

The research shows a clear pattern of negligence. In 2025 alone, the number of newly released code benchmarks that fail to include proper code coverage in their test cases nearly equals all similar failures across the previous ten years combined. This deterioration highlights how quickly the field is scaling without safeguards.

The stakes are significant: code benchmarks directly shape how the AI research community understands what large language models can actually do. Poor benchmark design leads to inflated performance claims and misguided research priorities.

HOW2BENCH: A Framework for Better Practices#

To address these issues, the researchers have introduced HOW2BENCH, a practical guideline containing 55 concrete checklists. The framework emphasizes three core principles:

• Rigor in designing and constructing benchmarks
• Reliability in how evaluations are conducted
• Reproducibility when benchmarks are released and shared

Why Quality Standards Lag Behind#

Human studies conducted alongside this research reveal two root causes. First, building robust benchmarks demands substantial time and effort—a barrier for many researchers with limited resources. Second, many practitioners simply underestimate how important these quality measures actually are. Better awareness campaigns and simplified best practices could help close this gap.