Benchmark Results Hinge on Evaluation Choices, Not Just Data

Researchers conducting evaluations often assume their findings are robust, but a new analysis suggests that what gets measured—and how—can be just as consequential as the underlying data itself.

The Problem: Hidden Semantics in Evaluation#

When researchers benchmark software systems or algorithms, they make a series of technical choices: which metrics to use, how to score responses, and how to combine results. These evaluation semantics are rarely scrutinized, yet they determine whether a conclusion holds up or collapses. The work examines what happens when these choices shift, even slightly.

A Case Study in Instability#

Using a dataset labeled TraceElephant, researchers tested two different scoring approaches on the same 217-row analysis. One metric configuration (F_asym) produced decisive results, showing adoption effects ranging from -40.76 to -43.03 percentage points. Yet when applied under different aggregation rules (F_cc), the identical data yielded zero effect with confidence intervals of [0, 0]—a complete disagreement with no middle ground.

Merging these two approaches into a unified framework (F_audit) failed to identify the true underlying effect, even though one of its components perfectly identified zero. This is not measurement noise; it is a fundamental instability rooted in how the evaluation contract was structured.

What Drives the Disagreement#

A factorial audit traced the entire source of the contradiction to a single factor: one-sided terminal deletion in data handling. This one binary choice accounted for 100 percent of the disagreement, while all other methodological variations contributed nothing.

Importantly, how results rank against each other—which approach is "better"—also shifted across the two symmetric treatment conditions, showing that even relative conclusions depend heavily on specification.

What This Means#

The authors frame this as a localized non-implication witness: a concrete, reusable audit showing that certain evaluation semantics fail to generalize. It is not a claim about how widespread this problem is, nor a complete taxonomy of semantic fragility. Rather, it is a demonstration that benchmark conclusions are only as stable as the evaluation contract allows—and that contract is often treated as invisible.