LLM agent performance isn't a single metric—it depends on what you're actually testing

When comparing large language model agents, benchmark scores can be misleading. A new academic position paper argues that what researchers call "LLM agent performance" actually masks several fundamentally different kinds of evaluation, each answering different questions about capability and reliability.

The hidden variables in agent benchmarks#

LLM agent benchmark results are shaped by far more than just the underlying model. The agent framework, the testing environment, who or what evaluates the results, and how much computational budget is allocated all influence the final score. When researchers want a fair model-to-model comparison, they use what the paper calls "unified execution"—running all candidate models through identical configurations so differences reflect the models themselves rather than experimental variation.

But model comparison is only one use case for benchmarks.

Different evaluation questions, same label#

Agent benchmarks also serve to assess complete end-to-end systems, where the agent harness and environment are part of what's being tested, not standardized controls. These two classes of evaluation ask entirely different questions:

• Model comparisons isolate model quality by holding the agent stack constant, answering "which model is better?"
• System comparisons evaluate agents as whole products, answering "does this agent system work?"

A third dimension, robustness testing, asks whether conclusions from either type of evaluation hold up when operating conditions change in predeclared ways.

Why clarity matters for leaderboards and standards#

The paper, submitted to arXiv, argues that blending these categories under the umbrella of "LLM agent performance" obscures what any given score actually claims. The authors propose that declaring the candidate boundary (what's being compared—just the model, or the whole system?) and the condition policy (fixed settings or robustness across variations?) should be mandatory for benchmark reporting.