How Values Get Baked Into Language Models During Training

Researchers investigating how large language models develop ethical alignment have uncovered when this critical learning occurs—and it happens earlier than previously thought. A team examining the training dynamics of models like Llama-3 and Qwen-3 has published findings that shed light on how values become embedded during the post-training phase.

The Core Discovery#

Most value alignment happens during supervised fine-tuning (SFT), the earliest stage of post-training where models learn to follow instructions. Once this initial phase is complete, subsequent preference optimization techniques—methods designed to refine model behavior based on human feedback—rarely shift the values a model has already absorbed. This distinction matters for anyone working to steer AI systems toward desired behaviors.

What the Research Examined#

The investigation dissected the effects of different training approaches, tracking both how much values shift and when those shifts occur. The researchers tested multiple model sizes and compared popular fine-tuning and preference optimization algorithms. They also created synthetic preference datasets that let them control which values were being taught, isolating the impact of algorithmic choices from other variables.

Key Finding: Algorithm Matters#

The results revealed a counterintuitive insight: different preference optimization algorithms produce different alignment outcomes even when trained on identical data. This indicates that the choice of training method itself—not just what data is used—shapes which values a model ultimately expresses. The findings underscore that both data curation and algorithmic selection play distinct roles in developing aligned systems.

Why This Matters#

As language models take on roles that require ethical judgment, understanding exactly when and how they learn values becomes essential. These results offer practical guidance for teams building these systems: focusing on high-quality data in the SFT phase and carefully selecting algorithms for preference optimization can be more effective than relying on later-stage corrections to fix misalignment issues.