New Reward Model Framework Trains Multimodal AI Across Text, Video, and Audio

Existing reward models that guide multimodal AI training face significant hurdles: they tend to prioritize visual inputs, rely on expensive human evaluation, and produce opaque numerical scores that miss the subtleties of complex reasoning. A new approach aims to solve these problems.

The Core Challenge#

The standard approach to training large language models involves reward models—systems that learn to score outputs and guide AI behavior toward desired outcomes. But for models handling multiple media types (text, images, video, audio), current methods fall short. They struggle with inconsistency, high labeling costs, and inability to surface the reasoning behind their scoring decisions.

Omni-RRM: A Multi-Modal Solution#

Researchers have introduced Omni-RRM, a reward model designed to generate nuanced feedback signals across all major media types simultaneously. Rather than producing a single opaque score, the system outputs multi-dimensional reward signals that capture reasoning complexity.

The breakthrough lies in how the model learns. Instead of relying on costly human annotators to rank every preference pair, the team developed Omni-Preference—a dataset created through automatic synthesis. The process works like this: teacher models examine raw preference signals and convert them into explicit, human-readable justifications. This ensures the training data remains both high-quality and interpretable.

The model itself trains through a two-stage regimen combining SFT (supervised fine-tuning) and GRPO (gradient-based reward preference optimization), with special focus on difficult cases—preference pairs with small margins that are easy to get wrong.

Strong Benchmark Results#

Across five evaluation benchmarks, Omni-RRM achieved 70.4% overall accuracy, representing a 17% relative improvement over its baseline architecture. On specialized tests:

• Video: 80.2% accuracy on ShareGPT-Video
• Audio: 66.8% on Audio-HH-RLHF, 65.0% on TA2T

The model also transfers effectively to text-only alignment tasks and can guide best-of-N selection—a practical technique where multiple outputs are generated and the model picks the best one.