New Method Cuts LLM Benchmark Costs by 85% While Predicting Model Performance

Evaluating large language models typically requires running them through thousands of different benchmarks—a process that's expensive, time-consuming, and often redundant. A new approach combines statistical modeling with intelligent test selection to predict model performance on unseen tasks while drastically reducing evaluation costs.

The Problem with Benchmark Proliferation#

The field of AI has created an overwhelming number of benchmarks to measure LLM capability—so many that it becomes impractical for researchers to run every model against every test. Yet prior research has shown that much of what these diverse benchmarks measure can be explained by just a handful of underlying latent abilities, suggesting that benchmarking itself could be streamlined.

The challenge lies in developing methods that are both efficient and principled. How do you know which benchmarking approach best predicts how a model will perform on tasks it hasn't encountered yet?

A Dataset and Framework for Comparison#

To tackle this question, researchers compiled the Wide-scale Item Level Dataset (WILD)—an extensive collection of test item-model response pairs covering 65 different models evaluated on 109,564 unique items across 163 separate tasks from 27 different datasets. This resource enabled the first large-scale analysis of which techniques can accurately forecast performance under various cost constraints.

The team combined two key techniques:

• Modified multidimensional item response theory (IRT): A statistical model borrowed from educational testing that treats benchmark items similarly to how exam questions are analyzed, accounting for both item difficulty and model ability
• Adaptive item selection: Using optimal experimental design principles to strategically choose which items to evaluate next, focusing on the most informative ones

Results and Real-World Impact#

The approach achieved striking results. After observing just 16 carefully selected items, the framework could predict performance on 112 held-out benchmark tasks with a mean absolute error below 7%—roughly the threshold of practical accuracy.

More impressively, when the researchers incorporated cost-aware discount factors to account for token expenditure, they reduced the total tokens needed to reach 7% accuracy from 141,000 to just 22,000 tokens—an in evaluation cost.