Researchers develop methods to train AI safely with corrupted human feedback

Training AI systems from human feedback is increasingly common, but what happens when that feedback is unreliable, deliberately manipulated, or simply inconsistent? Researchers have now developed the first provably robust approach to offline reinforcement learning with human feedback (RLHF) that handles corrupted data.

The Problem: Messy Feedback in Practice#

In real-world scenarios, datasets used to train AI systems aren't always pristine. A malicious actor might flip labels to poison the training process, or human evaluators might make mistakes when comparing different behavioral options. The challenge intensifies in offline settings, where the system must learn from fixed, pre-collected data without opportunities to gather fresh feedback. Prior theoretical work addressed corruption-robust learning from direct rewards and separate work tackled feedback-based learning without corruption—but combining these challenges had remained largely unsolved.

A New Framework for Robustness#

The researchers propose algorithms that can identify near-optimal policies even when an adversary corrupts some fraction of the preference data. Their method operates in two stages: first, it learns a reward model while estimating confidence boundaries around that model's predictions. Then it trains a pessimistic policy that performs well even within those uncertainty margins—choosing actions robust to worst-case scenarios within the confidence set.

Flexibility for Different Data Conditions#

A key insight is that learning the final policy can leverage different types of corruption-robust reinforcement learning oracles depending on the coverage properties of the training data. Some scenarios call for a zero-order approach, while others benefit from first-order optimization. This flexibility allows the framework to adapt to varying assumptions about how comprehensive the original dataset is.

The work represents a significant step toward making AI systems more resilient to contaminated training data—a pressing concern as these systems move into production environments.