LLM agents struggle with novel environments—a world-modeling fix shows promise

Large language model agents tend to plateau when deployed in unfamiliar environments, a phenomenon researchers are now working to understand and reverse. A new study from arXiv identifies the culprit: weak grounding in how environment states work and evolve.

The Exploration Collapse Problem#

Researchers observed that when LLM agents train via reinforcement learning in novel settings, their diversity of successful attempts actually declines even as their single best trajectory improves. This paradox—called "exploration collapse"—emerges only in environments distant from the model's pretraining data, suggesting the issue roots in shallow state comprehension rather than general learning incapacity. In contrast, agents facing familiar terrain maintain robust exploration breadth.

A Grounded-Initialization Approach#

The team proposes SPA (Self-Experience Pretraining + Augmented RL), a simple two-phase recipe. First, the agent collects its own interaction trajectories and learns to predict both current states and state transitions through supervised finetuning—essentially building a world model from scratch. Only then does it enter standard reinforcement learning. This grounding phase acts as a sturdy springboard rather than an inference-time planning tool.

Empirical Gains Across Benchmarks#

Testing on Sokoban puzzles, SPA doubled the success rate on smaller models: Qwen2.5-1.5B-Instruct jumped from 25.6% to 59.8%, allowing sub-3B parameter models to outperform a 20-billion-parameter baseline. Ablations confirmed that gains depend on four factors: reliable state representations, explicit transition prediction, diverse self-generated trajectories, and sufficient coverage of the state-action space.

The findings suggest that helping LLM agents understand their world, not just optimize within it, is key to robust generalization to unfamiliar tasks.