New method improves robot learning by combining prior knowledge with human feedback

Preference-based reinforcement learning offers a practical way to train robots without manually specifying reward functions. However, this approach typically demands substantial human feedback, making it expensive and slow to deploy in real-world settings.

Researchers have now proposed a technique called Residual Reward Models (RRM) that significantly reduces the feedback burden by strategically combining existing knowledge with learning from human preferences.

How It Works#

The core insight behind RRM is decomposing the environment's true reward into two complementary components:

• Prior reward: An initial reward signal available at the start, such as a hand-crafted heuristic, a reward function generated by a language model, or one learned through inverse reinforcement learning
• Learned reward: An additional learned component trained via human preferences, which acts as a residual correction

Rather than switching between different learning approaches midway through training (which often destabilizes neural network optimization), RRM smoothly builds on existing information by treating the learned portion as an offset to what's already known.

Results and Impact#

Testing across simulation environments including Meta-World and DM-Control demonstrated substantial improvements in sample efficiency—meaning fewer human comparisons were needed to achieve strong performance. The researchers also validated the approach on a physical Franka Panda robot arm, showing that the method accelerates policy learning and reaches high task success rates in fewer interaction steps than conventional baselines.

The method's flexibility extends to any type of prior reward, making it adaptable to various sources of initial knowledge and demonstrating practical value for reducing the real-world cost of robot training.