Agentic AI
Two-stage reinforcement learning framework optimizes dynamic selection and fusion of LLM ensembles
Researchers introduce RL-Focal, a system that intelligently routes incoming queries to smaller, task-specific subsets of large language models while learning to resolve their reasoning conflicts.
A new framework called RL-Focal tackles the challenge of efficiently using multiple large language models by dynamically selecting and combining them based on specific tasks. Rather than deploying an entire pool of models, the system learns to identify smaller, high-performing ensembles tailored to each downstream task.
How the system works#
The RL-Focal framework operates in two coordinated stages. The first stage uses a "Decider" RL-agent that learns which subset of models to activate for each incoming query. This agent optimizes for both error diversity—ensuring selected models produce varied outputs that can reveal different perspectives—and reasoning performance, choosing ensemble combinations that collectively solve problems effectively.
The second stage introduces a "Fusion" RL-agent that learns to reconcile conflicting predictions across the selected models. Rather than applying fixed aggregation rules, this agent adapts its reasoning resolution strategies based on which specific ensemble teams are active for each task.
The focal diversity metric#
A key innovation is the introduction of focal diversity—a metric that better captures error correlations among multiple models. By modeling how errors correlate across the ensemble, the system can more intelligently prune inferior combinations and actively select ones with stronger generalization properties. This reward-aware and policy-adaptive approach enhances performance while reducing the number of models needed.
Results and implications#
Evaluation across five benchmarks demonstrated an 8.48% performance improvement compared to the single best individual model in the pool, while using a markedly smaller ensemble. The system also showed stronger robustness than baselines. The approach suggests that intelligent routing and fusion—rather than simply scaling up the number of deployed models—can improve both efficiency and performance in multi-model inference scenarios.
