New Generative Approach to Value Models Boosts LLM Reinforcement Learning

Assigning credit accurately—determining which actions lead to which outcomes—remains a fundamental bottleneck in reinforcement learning. While classical methods used learned value functions to solve this problem, modern large language model (LLM) training has largely abandoned them, because traditional scalar-prediction critics are prone to training instability and poor generalization.

The Case Against Standard Value Models#

Conventional discriminative value critics predict a single numerical estimate of value in one shot, without intermediate reasoning. Theory and empirical evidence suggest this approach struggles because value functions can be inherently difficult to approximate under such constraints. Scaling experiments show that simply making these critics larger does not reliably improve their performance, unlike many other neural components in modern AI systems.

A Generative Alternative#

To address this limitation, researchers introduced Generative Actor-Critic (GenAC), which reimagines how critics operate. Instead of outputting a scalar value directly, the critic performs chain-of-thought reasoning—decomposing the problem step-by-step before producing its estimate. This richer representational pathway allows the model to capture the nuances of value estimation more effectively.

The method also introduces In-Context Conditioning, a mechanism that keeps the critic's predictions properly calibrated to the current version of the policy throughout training, preventing the critic from drifting out of sync as the actor evolves.

Measurable Improvements#

GenAC demonstrates gains across multiple dimensions: better value approximation accuracy, more reliable ranking of action quality, and improved performance on out-of-distribution scenarios. These advances translate into meaningfully stronger reinforcement learning outcomes compared to both value-based and value-free baselines on downstream tasks.

Broader Implications#

The findings suggest that credit assignment in LLM reinforcement learning need not be sacrificed for training stability. Rather, stronger—and differently structured—value modeling can become a key lever for improving how language models learn from feedback, rekindling interest in actor-critic frameworks for large-scale language model training.