AI
New Method Cuts Training Cost for AI Language Models by Reusing Internal Layers
Researchers introduce Nested-ReFT, a framework that accelerates reinforcement learning fine-tuning of large language models by leveraging off-policy rollouts.
A new approach to training large language models on complex reasoning tasks could significantly reduce computational overhead during the fine-tuning process. Researchers have developed a framework called Nested-ReFT that makes reinforcement learning-based training more efficient by rethinking how models generate training data.
The Challenge with Current Methods#
Reinforcement learning fine-tuning (ReFT) has emerged as an effective way to boost LLM performance on demanding problems like mathematical reasoning. These methods rely on generating multiple candidate answers per problem, then scoring them with a reward function to guide model improvement. However, this approach demands substantial computing resources because the model must perform many inference passes during training—each generating full completions that get evaluated.
How Nested-ReFT Works#
The new method borrows ideas from both off-policy reinforcement learning and speculative decoding to address this inefficiency. Rather than running the full model as a separate behavior model during training, Nested-ReFT repurposes part of the target model itself—specifically a subset of its internal layers—to generate training completions. By dynamically skipping certain layers on a per-batch basis during training, the framework slashes inference costs compared to standard ReFT approaches.
Theoretical Backing and Practical Results#
Theoretical analysis confirms that Nested-ReFT produces unbiased gradient estimates despite using cheaper off-policy rollouts, while still maintaining controlled variance in the updates. On the empirical side, tests across multiple mathematical reasoning benchmarks and model sizes show measurable improvements in computational efficiency, measured in tokens processed per second. The researchers also developed three variants designed to mitigate the bias that can arise from the off-policy nature of these shortcuts, allowing performance to remain competitive with traditional ReFT baselines.
