New Method Catches Tool-Use Errors Before Execution, Boosting AI Agent Reliability

Tool-use agents—AI systems that call functions or write code to solve problems—struggle with contract violations that slip through undetected. These errors, such as passing data in the wrong format to a tool or misrouting function calls, often run to completion without raising alarms, making them difficult to catch and repair. A new paper from arXiv proposes a solution that intercepts these bugs before execution.

The Problem with Current Approaches#

Most refinement techniques rely on unstructured feedback or real execution errors. Unstructured critique—like generic "fix this"—produces inconsistent results across different models. Even when agents have access to actual runtime feedback, improvement is modest. The core issue is that many contract violations don't crash; they silently produce wrong outputs. By the time an error is visible, the damage is done.

How RubricRefine Works#

The approach, called RubricRefine, generates task- and tool-registry-specific rubrics—explicit checklists of what correct code must satisfy. The system then scores candidate code against these contract checks and iteratively repairs violations before any execution happens. Critically, it requires no training; the rubrics are derived directly from supplied tool documentation.

The method caught inter-tool failures including:

• Incorrect output shapes
• Wrong tool routing
• Broken argument provenance

Performance and Limitations#

On the M3ToolEval benchmark, RubricRefine reached an average score of 0.86 across seven different models, with zero actual code executions—outperforming prior baselines while cutting latency. However, results vary by task structure. On API-Bank, where most tasks are single-step, the gain flattens out. On multi-turn AppWorld, the advantage concentrates in sparse-interaction scenarios and shrinks as the number of turns increases.

The method's effectiveness depends heavily on documentation quality: it works even with incomplete docs but fails when documentation is outright incorrect. A rubric-category ablation revealed which validation rules matter most, and top-bin calibration made early stopping feasible, even in contexts where overall calibration is poor.