Reinforcement Learning Outperforms Fine-Tuning at Preserving Model Knowledge

When language models are adapted to perform new tasks, they often lose mastery of previously learned knowledge—a problem researchers call catastrophic forgetting. A fresh analysis of how different training methods handle this trade-off reveals that reinforcement learning (RL) significantly outperforms standard supervised fine-tuning (SFT).

Comparing Training Approaches#

Researchers systematically evaluated how Llama and Qwen models degrade when learning new capabilities through SFT and RL. The findings held consistent across diverse domains: instruction-following abilities, factual knowledge, and mathematical reasoning. In nearly every case, RL achieved stronger performance on target tasks while incurring notably less damage to prior knowledge than SFT methods.

Understanding the Core Advantage#

To understand why RL proved more resilient, researchers modeled the underlying mechanism: they represented the language model as drawing from two separate distributions—one embodying existing knowledge and one representing the new task. Their analysis pinpointed that RL's effectiveness stems from its mode-seeking behavior, a property that emerges directly from its reliance on on-policy data (samples generated by the model's current policy).

The team verified this insight by isolating on-policy data as the primary driver of RL's forgetting resistance. Other commonly cited factors—including KL divergence regularization and advantage estimation—played supporting but secondary roles.

Practical Implications#

The findings suggest that practitioners need not demand perfectly on-policy data to reap these benefits. Approximately on-policy data, which is far simpler and cheaper to collect, can substantially reduce forgetting while remaining much more efficient than gathering fully on-policy samples. This opens a practical path to safer model adaptation in production settings.