AI
New benchmark pits AI models against each other to measure true math capabilities
Researchers introduce MathDuels, a self-play benchmark where language models both author and solve math problems, revealing capability gaps invisible in traditional static tests.
Traditional math benchmarks have hit a plateau. As the latest frontier language models score near-perfect marks on fixed problem sets, researchers lose their ability to distinguish between different models' actual capabilities. A new evaluation framework called MathDuels takes a fundamentally different approach.
How MathDuels works#
Instead of treating models as passive problem-solvers, MathDuels casts each participant in a dual role: generating original math problems while simultaneously solving challenges created by competing models. Problems are constructed through three stages—meta-prompting to guide generation, problem creation, and difficulty amplification—then vetted by an independent system that rejects malformed questions.
The benchmark uses a Rasch model (a statistical framework for jointly estimating solver ability and problem difficulty) to measure two distinct dimensions: how well each model solves problems, and how well it authors problems that stump other models. This separation reveals that authoring and solving prowess are only loosely coupled—a model might excel at one without matching strength in the other.
Dynamic difficulty that evolves#
One key innovation distinguishes MathDuels from static benchmarks: the difficulty automatically adjusts as new models join. When stronger models enter the arena and author harder problems that defeat previously dominant solvers, the benchmark's overall challenge level rises. Rather than plateauing at a fixed ceiling, the leaderboard difficulty co-evolves with participant strength.
Capability separations#
Testing across 19 frontier models showed that the dual-role evaluation scheme surfaces meaningful performance gaps that single-role benchmarks miss. The authors have established a public leaderboard that updates as new models are released, allowing ongoing assessment of frontier model capabilities.
The work is hosted on arXiv under the title "MathDuels: A Self-Play Benchmark That Grows."
