Researchers Show Tool Calls in Language Models Can Be Read and Steered Before Generation

Language models deployed as agents often need to call external tools—from APIs to calculators—to complete tasks. Selecting the wrong tool can lead to errors that cascade through downstream operations. Researchers have now developed a method to identify which tool a model will choose before it actually generates the call, and to steer the model toward a different tool if needed.

How the Approach Works#

The technique relies on extracting tool vectors—representations derived from the model's internal hidden states when it processes example requests for each tool. By averaging these hidden states across a few samples, researchers create a fingerprint for each tool. When a new request arrives, the model's current hidden states are compared against these tool vectors to predict which tool will be selected.

To redirect the model toward a desired tool, the researchers compute a steering direction—the vector difference between two tool representations. Applying this direction to the model's hidden states nudges the model toward generating a call to the target tool, all without fine-tuning or retraining.

Testing Across Models#

The method was evaluated on eight instruction-tuned language models ranging from 4 billion to 27 billion parameters. In tests with held-out tool pairs, steering successfully changed the generated call to a target tool between 56% and 78% of the time, depending on the model size. Performance declined when tested against real APIs from benchmarks like τ-bench and ToolBench, though the technique remained effective.

Examining the steering mechanism layer-by-layer revealed that the tool choice isn't solely determined at the final layer. In the middle layers of three deeply analyzed models, the steering direction outperformed a simpler approach that only boosted the score of the target tool name—suggesting the tool-selection signal is encoded more broadly throughout the model's computation.

Early Error Detection#

Beyond steering, the tool vectors also identify when a model might make a mistake. Tool-selection errors become more likely when a request sits roughly equidistant from two tool vectors, creating ambiguity. Across four models tested, this signal achieved an area under the receiver operating characteristic curve (AUROC) of 0.61–0.78, outperforming a baseline that relied on confidence in the first generated token on three of the four models.

Why It Matters#