AI
Researchers Propose SORL to Stabilize AI Agent Training with Multi-Turn Interactions
A new framework addresses instability in reinforcement learning training for multi-turn language model agents by aligning optimization with conversational structure.
Researchers have identified and tackled a critical challenge in training large language models (LLMs) to act as multi-turn conversational agents. When using off-policy reinforcement learning methods like PPO and GRPO, these systems frequently experience training instability and can collapse into poor performance. The core issue stems from a mismatch between how the models are optimized at the individual token level versus how they actually operate across multi-turn interactions.
The Problem with Current Approaches#
The instability arises from two distinct sources. First, there's a granularity mismatch: token-level policy optimization doesn't naturally align with turn-structured agent interactions. Second, off-policy importance sampling produces high-variance and unreliable gradient updates, compounded by inaccurate advantage estimation that leads to unpredictable learning dynamics. These problems make training fragile and often require careful manual intervention, early stopping, or extensive tuning to avoid collapse.
Introducing SORL#
To address these issues, researchers propose SORL (Stabilizing Off-Policy Reinforcement Learning for Long-Horizon Agent Training). The framework introduces two key mechanisms: turn-level importance sampling that respects the structure of multi-turn conversations, and clipping-triggered normalization that adaptively suppresses unreliable off-policy updates. Together, these innovations promote more conservative and robust learning dynamics without demanding careful early stopping or heuristic tuning.
Within this framework, the team instantiated two stabilized algorithms: SO-PPO and SO-GRPO, which extend the popular PPO and GRPO methods with these stabilization techniques.
Evaluation and Results#
The researchers tested SO-PPO and SO-GRPO across multiple benchmarks, including open-domain question answering, multi-hop reasoning questions, and medical multiple-choice exams. They also demonstrated transfer to asynchronous RL settings for mathematical reasoning, training on DAPO-Math-17k and validating on AIME-2024 problems. Results indicate that SORL provides a practical, scalable foundation for stabilizing RL training in multi-turn LLM agents and extends to mathematical reasoning tasks.
