New benchmark reveals major gaps in AI reasoning models through minimal text variations

Researchers have introduced a new evaluation method that uncovers a troubling vulnerability in today's most capable reasoning models: they stumble when faced with minor rewording of the same logical problem.

The work, presented as the MERGE (Minimal Expression Replacement GEneralization) test, challenges the notion that high scores on existing benchmarks signal genuine reasoning capability. As popular datasets reach saturation, the field has struggled to design meaningful tests that measure whether models truly understand reasoning or merely exploit dataset artifacts.

How the test works#

The MERGE approach begins by taking existing Natural Language Inference (NLI) datasets—tasks where models must determine whether one sentence logically follows from another—and generating minimally altered variants. Rather than hand-crafting adversarial examples, researchers use Masked Language Models to swap out individual words and phrases while preserving the underlying logical structure. Automated safeguards then filter out any problematic variants, ensuring the modified examples remain sound.

The team applied this framework to two well-known NLI datasets and tested multiple leading models, including both traditional NLI specialists and large language models.

The findings: widespread fragility#

The results were striking. Even top-performing models failed to maintain consistent accuracy when presented with these slight reformulations. They could correctly classify the original statement but stumble on semantically equivalent variants that differed only in word choice or phrasing. This suggests that models may be matching surface-level patterns rather than performing robust logical reasoning.

The researchers also examined how different factors influenced this brittleness—specifically, which word classes were replaced (nouns, verbs, adjectives, etc.) and which language models generated the variants. Performance degradation varied across these dimensions, pointing toward systematic weaknesses rather than random failure.

Why it matters#

This finding carries significant implications for deploying reasoning models in real-world settings. In practice, the same logical premise can be expressed countless ways. A model that excels on a static benchmark but collapses under minor reformulations is unreliable for applications requiring genuine comprehension. The work suggests that achieving true generalization in reasoning remains an open challenge, despite recent advances in model scale and capability.