How benchmark design choices quietly reshape AI evaluation in finance

As large language models gain credibility for analyzing earnings calls, investor statements, and other financial disclosures, researchers increasingly use standardized benchmarks to pick the best models. But a fresh look at one such benchmark exposes a hidden vulnerability: the measurement tools themselves are unreliable.

The benchmark reliability problem#

Researchers tested the Japanese Financial Implicit-Commitment Recognition (JF-ICR) benchmark, which uses human-labeled examples to evaluate how well language models understand financial language. The problem is that the "gold standard" labels aren't as objective as they appear. When researchers rewrote the instructions for labeling—keeping the same underlying concept but changing wording, examples, or detail level—the model scores changed dramatically.

Specifically, different versions of the labeling instructions disagreed on how to classify examples between 16.6% and 30% of the time. These discrepancies clustered around boundary cases—particularly when deciding whether financial language expressed an implicit commitment. This pattern suggests the issue isn't random noise but systematic: how you phrase a rule shapes what the rule identifies.

Which metrics matter most#

The researchers also discovered that not all performance metrics work equally well on this benchmark. Some metrics were too forgiving: one variant (within-one accuracy) gave credit for near-misses and favored the majority class so heavily that it became uninformative. Another (worst-class accuracy) was too sensitive to noise because the rarest class contained only two examples.

Only three metrics proved robust: exact accuracy, macro-F1, and weighted kappa. These remained stable and informative across different model and temperature settings. The implication is stark: if researchers had reported results using different metric sets, they might have reached conflicting conclusions about which model ranked best.

A path toward better evaluation#

When the researchers applied stricter scrutiny—using only the three reliable metrics—competing ranking methods suddenly agreed on model ordering. But when they included all five metrics, disagreement appeared on the closest competitor pairs. This suggests that rigorous benchmark governance requires explicit choices about which metrics to trust and why.

The contribution here is not a leaderboard revision but a call for discipline: financial NLP benchmarks need documented rules about label construction, metric selection, and how results will be aggregated and reported. Without this governance, benchmark results risk becoming decision evidence built on unstable measurement foundations.