LLMs excel at predicting world states but struggle with complex planning, study finds

Researchers have designed a new benchmark that pinpoints exactly where language models fail at planning tasks—and the answer may surprise you: it's not usually perception or understanding, but the hard work of sequencing actions toward a distant goal.

The problem with measuring AI competence#

Typical benchmarks that measure how well language models can tackle interactive tasks report a single success rate, which obscures what actually went wrong. When a model fails, was it because it misunderstood the rules? Couldn't retrieve relevant knowledge? Didn't grasp the consequences of actions? Or simply couldn't figure out an effective plan? A aggregate score tells you none of this.

Isolating planning as the true test#

The new LLM-BabyBench study eliminates every source of failure except planning itself. Researchers adapted the procedurally generated BabyAI gridworld into a purely textual environment where:

• The entire grid is serialized directly into the prompt
• Instructions follow a small, unambiguous formal grammar
• Every object's position is explicitly stated
• All six possible actions and their effects are specified upfront
• A deterministic validator checks answers by running them, not guessing

This removes ambiguity, retrieval challenges, and commonsense gaps—leaving planning as the isolated variable to measure.

Three tasks, three distinct metrics#

The benchmark includes three related challenges:

• Predict: Given a sequence of actions, what state results?
• Plan: Find an action sequence that reaches a goal
• Decompose: Break a mission into intermediate subgoals

Each is scored using metrics that distinguish between understanding a mission and actually sequencing the steps to achieve it.

What the data revealed#

Testing seven frontier and open-source models showed a consistent pattern: simulation vastly outpaced planning. Models reliably predicted what would happen if they took an action, yet struggled to work backward from a goal. The strongest models nearly saturated the prediction task, while weaker ones fell well short on planning.