How Small Changes in Testing Methods Can Inflate LLM Performance Claims

Widely-used open-source reasoning models, particularly those in the Deepseek-R1-Distill family, may be reporting inflated performance metrics due to sensitivity in how their abilities are measured, according to a new paper on arXiv.

The Evaluation Problem#

The research shows that seemingly minor differences in evaluation conditions can produce dramatically different results for the same model. This variability undermines confidence in the claimed performance improvements that developers and the community have touted for these systems. The instability affects not only Deepseek-R1-Distill models and their derivatives but also extends to other open-source models like QwQ-32B that are built on similar architectures.

Models at the Center of the Issue#

Deepseek-R1-Distill models have gained significant traction in the open-source community, primarily due to their reported strengths across mathematics, scientific reasoning, programming, and related fields. However, when researchers attempted to independently verify these gains, they encountered substantial reproducibility challenges—suggesting the original benchmark results may be less reliable than presented.

Call for Stricter Standards#

The researchers advocate for establishing more rigorous evaluation frameworks to ensure that performance claims are robust and genuinely reproducible. Without such standards, it becomes difficult to distinguish between real model improvements and artifacts introduced by testing methodology choices. The paper provides empirical assessments of the Deepseek-R1-Distill series to demonstrate these issues in practice.