New approach tackles reward over-optimization in AI model training

When large language models are fine-tuned using human feedback, they're trained against an estimated reward signal rather than actual human preferences—a mismatch that can lead to problematic behavior. Researchers from arXiv have introduced a new technique called distributionally robust regret optimization (DRRO) to address this fundamental challenge.

The Problem: Reward Misalignment#

The gap between a learned reward proxy and true human preferences creates a serious problem in reinforcement learning from human feedback (RLHF). Models can become excessively optimized for the proxy signal at the expense of actual quality—a phenomenon known as reward over-optimization. The model's performance plateaus or declines even as the proxy reward continues climbing, making the training process unreliable for aligning AI systems with human values.

How DRRO Works Differently#

Rather than pessimizing worst-case value like traditional robust optimization approaches, DRRO pessimizes worst-case regret relative to the best policy under plausible reward perturbations. The method uses a Wasserstein ambiguity set with promptwise ℓp distances as transport costs, decomposing the optimization problem into individual prompt-level regret problems.

The framework yields practical advantages: the inner adversary takes a closed-form dual-norm solution, and under ℓ1 transport costs, the optimizer exhibits a water-filling structure—the kind of elegant mathematical property that enables efficient algorithms.

Implementation and Results#

The approach translates into a concrete policy-gradient algorithm that requires minimal modification to existing training methods, adding only a simple sampled bonus term to GRPO-style training. Experiments show that DRRO avoids the excessive pessimism of standard distributionally robust optimization while more effectively controlling reward over-optimization than competing baselines.

The full paper is available on arXiv.