Reinforcement Learning–Trained Rewriting Policy Reduces Catastrophic Forgetting in Fine-Tuned Language Models

When large language models undergo supervised fine-tuning (SFT) to master new tasks, they often suffer from catastrophic forgetting—losing capabilities on unrelated benchmarks. A core culprit is the gap between the distribution of downstream training data and the model's inherent generation patterns. Researchers from arXiv cs.CL now propose tackling this through intelligent data transformation.

The Problem: Distribution Mismatch#

Standard fine-tuning pipelines introduce supervision that pulls the model away from its original distribution, causing performance drops on general benchmarks. Prior work tried fixing this via data rewriting, but existing approaches rely on fixed templates or sample rewrites from conditional distributions that don't necessarily reflect how the model naturally answers questions. This rigidity also limits output diversity.

A Policy-Based Solution#

Rather than applying hand-crafted rewriting rules, the team formulates data transformation as a policy-learning problem. They train a lightweight rewriting policy using reinforcement learning, optimizing for two key objectives:

• Distributional alignment with the backbone's question-answering patterns
• Semantic diversity to preserve variability in the rewritten data

The policy operates under a hard constraint: any rewritten examples must remain semantically consistent with the original task—ensuring verification before they're used for downstream SFT.

Results Across Benchmarks#

Testing on three instruction-tuned model backbones, the RL-rewritten supervision delivered downstream task gains comparable to standard SFT while consistently reducing degradation on non-downstream benchmarks. Additional experiments on logical reasoning and medical question answering suggest the learned rewriting policy can transfer across different domains and datasets for the same backbone model.

Why It Matters#

This data-centric approach offers a middle path between brute-force fine-tuning and complex architectural changes. By shaping supervision to match the model's natural tendencies before training, the method preserves general-purpose knowledge while still enabling effective task adaptation.