Economic AI benchmarks may measure broad capability, not a distinct trait

Leaderboards that rank advanced AI systems on economic benchmarks wield significant influence—they shape purchasing decisions and regulatory focus. But a new analysis raises questions about what these rankings actually measure: a specialized economic capability, or simply a reflection of general AI prowess?

The validity puzzle#

Researchers examined 421 AI model configurations across twelve benchmarks, including four focused on economic tasks, using structural and predictive validity tests. The core question was whether economic benchmarks capture something fundamentally distinct from general test-taking ability. The two testing methods, however, yielded opposite conclusions.

What structural analysis revealed#

A factor analysis found that a single dominant factor accounted for 74.5% of shared variance across all benchmarks. Crucially, when researchers adjusted for release date—a strong correlate (R² = 0.505)—this primary factor's share dropped by nearly 15 percentage points. Under the pre-registered dimensionality threshold, the economic benchmarks did not form their own separate factor. This structural result suggested no distinct economic capability.

Predictive tests tell a different story#

Yet when the team used leave-one-benchmark-out testing to predict held-out economic scores, a more granular multi-factor representation outperformed a single general index under linear models (pooled Delta-MSE 0.037, 95% bootstrap interval [0.019, 0.055]). The same multi-factor structure also predicted the eight non-economic benchmarks better, implying that the full battery captures predictive structure a single index misses.

The advantage reversed for tree-based learners, highlighting that the type of statistical model used matters. Economic benchmarks, the researchers found, share predictive value with the broader set without standing as a wholly isolated capability.

Implications for benchmark design#

The study underscores a methodological tension: structural and predictive tests of validity can diverge. The researchers propose that construct validity assessment should integrate both approaches, not rely on either alone. They release their data, analysis plan, and code to enable further scrutiny of how frontier AI benchmarks truly function.