New clipping technique stabilizes reinforcement learning for large language models

Researchers have identified a structural vulnerability in Group Relative Policy Optimization (GRPO), a widely-used training method for aligning large language models through reinforcement learning. The problem emerges from how the algorithm handles updates across different combinations of likelihood ratios and reward signals, leaving one critical scenario unbounded and prone to destabilization.

The Four-Quadrant Problem#

GRPO improves upon earlier approaches like PPO by using a single sequence-level reward rather than token-level advantages. However, this shift creates an asymmetry in how the algorithm constrains updates. A four-quadrant analysis of the (likelihood-ratio, advantage) space reveals that one region—where negative advantages combine with increased likelihood ratios—remains structurally unbounded. In this quadrant, a handful of high-ratio tokens can trigger severely suppressive updates that collapse model entropy and narrow its reasoning capabilities.

ABC-GRPO: A Fix for All Quadrants#

To address this blind spot, the researchers propose All-Quadrant Bounded Clipping GRPO (ABC-GRPO), which applies consistent clipping boundaries across all four quadrants using sign-dependent thresholds. The method clips likelihood ratios before multiplying by advantage values, establishing both a floor in regions of negative advantage and a cap in the problematic quadrant. This ensures every policy update remains bounded and tractable, whether the model is being encouraged or discouraged to take a particular action.

Empirical Results and Transfer#

Testing on mathematical reasoning tasks with Qwen3 base models, ABC-GRPO achieved the highest performance metrics (Avg@64 and Pass@64) and demonstrated statistical superiority over standard GRPO, SAPO, and dual-clip PPO. Notably, the approach maintained substantially higher entropy than competitors, preserving the model's exploratory capacity. Performance improvements generalized to MATH-500 benchmark problems and transferred to out-of-domain code generation on HumanEval, suggesting the fix addresses a fundamental training instability rather than task-specific noise.