Regularized Self-Play Method Lifts LLM Alignment Benchmark Scores

Researchers have introduced a regularized approach to self-play-based fine-tuning of large language models that substantially improves alignment quality while maintaining theoretical guarantees. The method, called Regularized Self-Play Policy Optimization (RSPO), reformulates preference optimization as a two-player competitive game but adds structured controls to prevent models from diverging too far from their original behavior.

The Challenge with Self-Play Alignment#

Self-play methods have gained traction as an effective way to optimize LLMs by treating preference learning as a game between two model instances. However, prior work has given limited attention to how regularization—the mechanism that prevents models from over-fitting to win the game at the expense of general utility—should be implemented in this setting. RSPO directly tackles this gap by offering a unifying framework that accommodates different regularization strategies while preserving mathematical convergence to a Nash equilibrium.

Strong Empirical Results#

The framework delivered measurable improvements on multiple evaluation benchmarks:

• Mistral-7B: Length-controlled win rate (LCWR) on AlpacaEval-2 improved from 28.5% to 35.4%
• LLaMA-8B: LCWR increased from 38.77% to 43.66%
• Gemma-2B: LCWR rose from 50.54% to 51.83%

Beyond AlpacaEval-2, the approach also achieved consistent gains on Arena-Hard, MT-Bench, ArmoRM, and demonstrated better response diversity compared to unregularized baselines.

Design and Availability#

A key strength of RSPO is its simplicity: it functions as a plug-and-play system that allows different regularization schemes to be swapped in while retaining convergence properties. This modularity makes it practical for researchers exploring variations of regularized self-play. The authors have released their code publicly to facilitate adoption and further experimentation.

The work is available on arXiv.