New benchmark tests AI models' ability to catch errors in synthetic math data

Synthetic mathematical data plays a crucial role in training and improving the reasoning capabilities of large language models, but errors in the automatically generated questions and solutions can significantly diminish their usefulness. A new research effort tackles this problem by creating a structured way to measure how well AI systems can spot and characterize these mistakes.

The benchmark approach#

MathDebugger provides a curated test set designed to evaluate model performance on error detection and diagnosis tasks. The benchmark includes 2,000 verified correct questions alongside 2,000 flawed ones distributed evenly across four distinct error categories. Additionally, 2,000 annotated solutions were included, with 610 containing errors split across three error types. All instances underwent careful human review and categorization, with disagreement levels low enough to indicate strong consistency—Fleiss kappa scores ranged from 0.69 to 0.91 depending on error type.

What the evaluation revealed#

The researchers tested 14 major language models and three process reward models against the benchmark. Results exposed significant limitations: even the most capable reasoning-focused models fall well short of mastering the task, particularly when required to identify nuanced error subtypes. An unexpected finding emerged around what the authors call the solving-verification gap—models specifically optimized for mathematical problem-solving or extended reasoning did not reliably outperform general-purpose alternatives when asked to audit data quality.

Practical value#

The fine-grained error classifications proved useful in practice. When models received explicit information about error types, they demonstrated measurable improvements in correcting flawed questions and solutions, with these gains reaching statistical significance. This suggests MathDebugger can help guide the development of more robust pipelines for generating and filtering mathematical data at scale.