New method uses contrastive learning to strengthen LLM training without human feedback

Researchers have developed a new approach to scaling large language models by training stronger models on outputs from weaker ones—a process known as weak-to-strong generalization—without needing human feedback or explicit reward systems.

The Challenge with Weak Model Guidance#

The traditional weak-to-strong method faces a critical weakness: noise and biases in the outputs from weaker models corrupt the training signal for stronger ones, reducing both reliability and practical applicability. This limitation has constrained how effectively the paradigm can work at scale.

Connecting Implicit Rewards and Contrastive Decoding#

The key innovation lies in recognizing that implicit rewards—estimates derived from log-likelihood ratios—are mathematically equivalent to Contrastive Decoding (CD), a generation technique that reduces noisy outputs from language models. This structural connection unlocked a new opportunity.

Contrastive Weak-to-Strong Generalization (ConG)#

Building on this insight, the researchers propose ConG, a framework that pairs contrastive decoding between pre- and post-alignment weak models to produce higher-quality training samples. The approach offers several benefits:

• More reliable transfer of capabilities from weaker to stronger models
• Built-in denoising of weak model outputs
• Improved robustness against inherent biases
• Substantial reduction in typical weak-to-strong limitations

Empirical Results#

Testing across different model families demonstrates consistent gains, indicating that ConG generalizes well and works reliably in practice. The findings suggest this pathway could meaningfully advance weak-to-strong training and contribute progress toward artificial general intelligence.