AI
New adaptive testing method cuts LLM evaluation costs by up to 90%
Researchers introduce ATLAS, an AI evaluation framework based on psychometric principles that dramatically reduces the number of test items needed to measure model performance.
Evaluating large language models currently demands thousands of test items, a resource-intensive process that becomes increasingly difficult to scale. A new approach borrowed from educational psychology could fundamentally change how the field assesses AI systems.
The Problem with Current Benchmarks#
Standard LLM evaluation relies on running models against fixed sets of test questions and averaging their accuracy. This method treats every item as equally valuable, even though some questions are much better at distinguishing model capabilities than others. The sheer volume of items required makes benchmarking expensive and time-consuming, creating a bottleneck as models grow more capable.
Introducing ATLAS: Adaptive Testing for Language Models#
Researchers have developed ATLAS, an adaptive testing framework grounded in Item Response Theory (IRT), a decades-old psychometric method used in human education and standardized testing. The system uses what's called Fisher information-guided item selection to intelligently choose which test items matter most for accurately measuring a model's true ability.
Rather than exhausting the entire benchmark, ATLAS concentrates on the questions that will most sharply reveal what the model can and cannot do. This approach can reduce the number of required test items by up to 90% while preserving measurement accuracy.
Demonstrated Results Across Benchmarks#
On the HellaSwag benchmark—which normally contains 5,600 items—ATLAS matched the full-bank ability estimates using just 41 questions, achieving a mean absolute error of only 0.157. The framework reconstructs raw accuracy from its ability estimates with high fidelity across five different benchmarks, meaning the global performance rankings remain stable.
At the same time, ATLAS reveals fine-grained distinctions that raw accuracy alone misses. Among over 3,000 models evaluated, 23–31% shifted by more than 10 positions in ranking when assessed with ATLAS versus raw accuracy—many of them having identical accuracy scores on the original benchmarks. This suggests the ability estimates capture genuine performance differences that static metrics overlook.
