Study reveals large language models struggle with reconstructing scrambled text structure

While large language models demonstrate strong semantic understanding, a new investigation from arXiv researchers reveals significant gaps in their ability to handle structural reordering tasks. The work exposes a fundamental disconnect between these systems' capacity to understand meaning and their capacity to reconstruct precise structural arrangements.

The OrderProbe benchmark#

Researchers created a novel evaluation framework called OrderProbe to test how well leading LLMs can restore the correct sequence of scrambled text. Rather than using full sentences—which can have multiple valid reconstructions and defy automated scoring—the team focused on fixed four-character expressions drawn from Chinese, Japanese, and Korean. These character sets have a single canonical order, making it possible to measure success through exact-match accuracy without ambiguity.

Performance gaps across models#

Testing a dozen widely deployed LLMs, the research uncovered a troubling pattern: even state-of-the-art systems achieved zero-shot recovery rates below 35% in many cases. This suggests that structural reconstruction remains substantially more difficult than semantic tasks these models typically excel at.

Beyond accuracy: a richer evaluation framework#

The researchers developed a diagnostic approach that goes beyond simple recovery metrics. Their assessment includes measurements of semantic accuracy, logical validity, structural consistency, robustness across variations, and information density. This multi-dimensional view reveals whether models preserve meaning while failing at structure, or where other kinds of breakdown occur.

Structural robustness as a distinct capability#

The findings underscore an important distinction: strong semantic understanding does not automatically translate into structural precision. The gap between generation that preserves meaning and generation that achieves exact structural restoration suggests these are separate competencies that require targeted evaluation and development.