DeepPlanner tackles planning bottleneck in AI research agents through entropy-aware training

Researchers have identified a critical weakness in how large language models approach complex research tasks: their planning stages remain significantly under-optimized compared to their action-taking abilities. A new system called DeepPlanner aims to fix this by retraining AI agents to make better long-term decisions when tackling problems that require multiple steps and external tool use.

The Planning Problem#

When LLMs are given reasoning and action-generation capabilities, they can theoretically handle intricate tasks requiring extensive planning. Yet current training methods treat planning implicitly—burying it within general reasoning—or apply rigid explicit planners without properly tuning the planning phase itself. The researchers observed that under standard reinforcement learning, planning tokens (the model's decision points) show substantially higher entropy than other tokens, suggesting these pivotal moments aren't being optimized effectively and represent zones of uncertainty in the model's behavior.

How DeepPlanner Works#

The framework takes an end-to-end reinforcement learning approach with two key innovations. First, it shapes token-level advantage by adding an entropy-based term that directs larger training updates toward high-entropy tokens—effectively telling the model to pay extra attention to its most uncertain decisions. Second, it selectively upweights sample-level advantages for rollouts that demand more planning, ensuring the model learns most from scenarios where planning matters most.

Benchmark Results#

Testing across seven research-focused benchmarks, DeepPlanner improved planning quality and matched state-of-the-art performance while requiring substantially less training compute—a significant efficiency gain for research teams working with limited resources.