New offline learning technique improves reasoning in large language models with minimal computational cost

Training large language models to perform complex reasoning tasks typically relies on reinforcement learning, but online approaches demand substantial computational resources and can introduce training instability. Offline reinforcement learning sidesteps these issues by separating the generation phase from training, though existing offline methods have generally underperformed compared to their online equivalents.

The gradient entanglement problem#

A fundamental challenge in offline reasoning tasks involves what researchers call gradient entanglement. When solving long-horizon problems—such as multi-step mathematical proofs or code generation—correct and incorrect solution paths often share considerable overlap at the token level. This creates a dilemma: gradient updates derived from incorrect trajectories inadvertently suppress tokens that are equally essential for correct ones, degrading overall performance.

Future Policy Approximation (FPA)#

To address this problem, researchers propose Future Policy Approximation, a policy-gradient method that reweights gradients based on predictions of the model's future performance rather than its current state. The core innovation involves estimating the future policy through extrapolation in logit space, allowing the method to anticipate which tokens will matter as training progresses.

Performance across reasoning tasks#

The approach was tested on three different language models across ten reasoning benchmarks: seven focused on mathematical problem-solving and three on code generation. Results show that FPA consistently outperforms widely-used offline baselines including DPO, RPO, KTO, and vanilla offline RL. Notably, FPA stabilizes training over longer sequences, where standard methods typically degrade in performance. The method achieves accuracy matching state-of-the-art online approaches like GRPO and DAPO while requiring significantly fewer GPU hours.

Implications#

The work suggests that offline policy-gradient approaches warrant renewed attention as practical alternatives for scaling reasoning capabilities in language models without the computational and stability costs of real-time learning.