Agentic AI
New AI Model Learns to Guide Multi-Agent Searches Without Human Labels
MASPRM, a process reward model for multi-agent systems, improves reasoning tasks by scoring intermediate agent messages and outperforms standard reward models by significant margins.
Researchers have introduced a new technique for making multi-agent AI systems more efficient during inference by training a model to recognize when agent communication is making genuine progress toward solving problems.
The approach, called MASPRM (Multi-Agent System Process Reward Model), addresses a key inefficiency: during search procedures, it's often unclear which intermediate messages between agents actually contribute to finding solutions. Rather than wasting compute on dead-end paths, MASPRM learns to score the sequences of messages and uses those scores to guide search algorithms toward more promising directions.
How It Works#
MASPRM operates as an inference controller for two common search strategies: step-level beam search and Monte Carlo Tree Search (MCTS). The system is trained on multi-agent MCTS rollouts and requires only final outcome rewards—no human annotations of individual steps. This makes it significantly cheaper to train than models that demand hand-labeled intermediate feedback at every decision point.
The model evaluates ordered sequences of messages exchanged between agents and assigns scores that reflect whether each routing decision improved the system's position.
Performance Gains#
Testing across mathematical and reasoning benchmarks—GSM8K, MATH, MMLU, and LogiQA—MASPRM substantially outperformed baseline outcome reward models (ORMs) of comparable size:
- At 1.5 billion parameters: +2.0 to +3.0 points across all benchmarks
- At 7 billion parameters: +4.1 to +14.5 points across all benchmarks
- Additional scaling benefits over policy-based likelihood scoring: average +13.4 points under MCTS
Beyond raw accuracy, MASPRM improved ranking quality, shrinking the performance gap between top-1 and top-5 selections by as much as 10.3 points—a sign that the model better identifies genuinely strong solutions among multiple candidates.
Why It Matters#
As AI systems become more complex and involve multiple agents working together, the cost of exploring all possible interaction paths grows rapidly. MASPRM's ability to learn process-level guidance without expensive human supervision could make agentic workflows more practical and cost-effective in real-world applications.
