New approach lets language models handle conflicting user preferences through multiple-choice alignment

A new theoretical framework tackles a central problem in AI alignment: how to make language models serve users whose preferences conflict. Rather than forcing all users to accept a single predetermined response, the approach leverages test-time scaling—having the model generate multiple candidate answers that users can choose from.

The core idea: test-time scaling#

The researchers formalize what they call asymptotic universal alignment (U-alignment). The model generates $k$ different responses for each prompt, and users pick whichever they prefer. The framework proves that if a model can maintain a winning rate of approximately $\frac{k}{k+1}$ against any single-response alternative, it achieves the desired alignment property. Mathematically, this win rate approaches 100% as you request more samples, which is proven to be the fastest rate any method can achieve.

The problem with existing methods#

Current post-training techniques, including Nash learning from human feedback (NLHF), fall short despite being optimal for single-response scenarios. The core issue: these methods often collapse outputs to a single response that appeals to a majority, making extra samples redundant. NLHF and similar approaches thus fail to unlock the benefits of generating multiple candidates—their win rates can't reliably exceed 50% except by tiny margins, even when allowed many samples.

A solution via multi-player game theory#

The researchers propose using symmetric multi-player alignment games to preserve output diversity. They prove that any symmetric Nash equilibrium from these games—where a model competes against multiple copies of itself—achieves the optimal $(k,\frac{k}{k+k+1})$-robust alignment. The framework includes convergence guarantees for self-play learning dynamics and extends to settings where competing models also generate multiple responses.

Why it matters#

The work bridges a gap between theory and practice in AI safety and personalization. By formalizing what optimal alignment looks like under test-time scaling, it reveals fundamental limitations of existing methods and points toward approaches that genuinely honor diverse human preferences rather than imposing a manufactured consensus.