Training LLMs on Multiple Goals Often Backfires—New Method Could Fix It

When training large language models to meet multiple objectives simultaneously—such as being helpful, harmless, and honest—a paradox often emerges: optimizing for some goals comes at the cost of performance on others. A new study from researchers at arXiv pinpoints this persistent problem and offers a solution.

The Interference Problem#

The challenge arises during multi-objective alignment, a process where models are trained using scalarization algorithms that combine competing objectives into a single score. While this approach sounds reasonable, it frequently backfires: improving performance on one metric (say, harmlessness) directly degrades performance on another (like helpfulness). The researchers formalize this phenomenon as cross-objective interference and reveal that the problem is widespread across different algorithms, though its severity depends heavily on the specific model being trained.

Understanding Why It Happens#

To explain the mechanics behind interference, the team derived what they call a local covariance law. This principle states that whether an objective improves or worsens depends on the statistical relationship (covariance) between its reward signal and the combined scalarized score. When this relationship is negative, optimization hurts that objective. The law holds even when applied to modern reinforcement learning techniques using clipped surrogate objectives, making it broadly applicable to current training methods.

A New Approach: COVER#

Building on this insight, the researchers propose COVariance-floor Enforced Reweighting (COVER), a targeted controller that adjusts how much weight each objective receives during training. COVER operates asymmetrically: it only increases an objective's weight when the covariance between that objective's reward and the learning signal drops below a threshold. This approach mitigates interference while still allowing objectives to improve together when they naturally align.

Why Interference Varies by Model#

The team also conducted global convergence analysis to explain why interference affects different models differently. Their findings connect this variation to the geometry of each model—its internal structure and how its parameters relate to the training objectives. This connection between model architecture and training dynamics could help practitioners anticipate and manage interference in future alignment efforts.