How Human Beliefs About AI Shape Preference Labels in Reinforcement Learning

Researchers studying reinforcement learning from human feedback (RLHF) have uncovered a previously overlooked factor in how AI systems learn from human guidance: the beliefs labelers hold about what the system can actually do. The work challenges a common assumption in the field and offers practical guidance for improving AI training.

The Role of Belief in Training#

When companies and researchers train large language models and other AI systems, they typically rely on human raters to score or compare different outputs. These preference labels then drive the learning process. Existing approaches usually assume that human preferences flow directly from a rater's reward function—their sense of what makes a good answer. But a new study proposes that human beliefs about the agent's capabilities constitute a separate, equally important ingredient in shaping those judgments.

The researchers framed two central questions: First, do human annotators' assumptions about what an AI system can achieve actually sway the preferences they record? Second, what would an ideal set of beliefs look like, and what resulting preferences should humans express?

Bridging Theory and Practice#

The team developed a fresh preference model that explicitly accounts for human beliefs and derived a normative theory that quantifies how errors compound when human beliefs diverge from the AI system's true capabilities. This framework lets practitioners estimate the downstream cost to final model performance based on the belief mismatch.

To test whether beliefs truly matter, the researchers conducted a human study and found that they do—significantly. They also showed that annotators' assumptions can shift through simple, targeted interventions. In parallel synthetic experiments, they discovered that it is often suboptimal for human raters to assume an AI system is behaving optimally when it is not.

Implications for RLHF Practice#

The findings point toward actionable improvements. By narrowing the gap between how humans perceive an agent's capabilities and its actual capacities, practitioners can yield more effective RLHF workflows. The work suggests rethinking standard best practices, such as how organizations brief annotators and what information they share about model behavior during labeling tasks.