Study scrutinizes what tool and skill gains for AI agents actually measure

When researchers report that giving language model agents tools and reusable skills improves performance, they are not always measuring the same thing. A new arXiv preprint examines the methodological foundations of these evaluations and finds significant gaps between what studies claim to show and what their designs can actually support.

The problem with comparing studies#

The review, which draws on one hundred cited papers and performs detailed analysis of thirty-five key studies, reveals that published improvements refer to fundamentally different kinds of comparisons. By characterizing each evaluation across dimensions like treatment design, target population, outcome metrics, budget constraints, and underlying assumptions, the authors expose how the same claim—"tools make agents better"—can describe very different phenomena.

The challenge runs deeper than inconsistent reporting. The authors show through analytic reasoning and counterexamples that paired experimental runs on the same task do not automatically isolate a tool's effect when the evaluation conditions on something triggered within the aided run. This suggests many existing comparisons may conflate the tool's contribution with other factors that emerge only when the tool is invoked.

Distinct questions, confused comparisons#

The review identifies several critical distinctions that published work often blurs:

• Curated skill provision versus retriever replacement: providing a hand-selected set of tools differs from replacing a retrieval system
• Population effects versus triggered subsets: improvements on all tasks differ from improvements only on tasks where a tool was actually used
• Preparation versus marginal costs: the initial effort to build and integrate a tool is separate from the cost of using it once deployed

The authors also highlight how total effects of deploying a module answer a different research question than efficiency gains when working under the same computational budget.

What the research recommends#

The study provides both a methodological synthesis and a practical reporting checklist aimed at helping researchers align their claims with what their experiments actually demonstrate. The goal is not to invalidate past work but to clarify which conclusions each evaluation design legitimately supports—and which require additional evidence.