New Framework Uses Flexible Rewards to Teach Smaller AI Models Tool Use Without Rigid Copying

Getting smaller language models to handle tool-use tasks—like calling APIs or running code—has traditionally relied on having them copy larger models' exact steps. But this rigid approach often fails when models encounter situations outside their training data.

Researchers have introduced MENTOR, a new distillation framework that sidesteps this limitation by letting smaller models learn more flexibly from larger "teacher" models. Rather than enforcing exact replication of every action, the system uses adaptive reward signals that balance alignment with the teacher's general approach against achieving strong downstream performance.

The Core Challenge#

Training smaller models to use tools poses a difficult tradeoff. Supervised fine-tuning (SFT)—the conventional method—makes models slavishly follow teacher demonstrations, which leaves them brittle when facing new situations (poor out-of-domain generalization). Reinforcement learning (RL) offers flexibility, but sparse reward signals alone don't provide enough guidance for smaller models with limited capacity.

How MENTOR Works#

MENTOR takes an on-policy approach, meaning it learns as the model explores actions rather than learning from a fixed set of recorded trajectories. The key innovation is its process-aware reward structure—instead of rewarding only successful outcomes or only perfect imitation, MENTOR guides models toward tool-use behaviors that align with what the teacher would do while still allowing room for adaptation.

This middle ground lets smaller models develop more adaptable strategies without being locked into the teacher's exact steps, making them perform better when they encounter unfamiliar or out-of-domain tasks.

Experimental Results#

Testing on verifiable tool-use benchmarks—environments where outcomes can be checked—showed that MENTOR outperformed both standard supervised fine-tuning and strict RL baselines at generalizing to new situations.

The findings suggest that in tool-use settings where correctness can be verified, flexible behavioral guidance works better than slavish trajectory imitation for building small models that can handle novel scenarios.