AI
Kev cuts AI agent token waste by skipping text generation on routine decisions
A new open-source model family handles discrete choices and scoring tasks without autoregressive decoding, freeing larger language models to focus on reasoning.
The New Stack
AI agents frequently burn expensive tokens generating text for intermediate decisions that require no written response. As OpenAI disclosed recently, agent workloads can cost thousands of dollars per day. Kev, a fresh family of decision-only models built on Qwen 3.5, offers a remedy by replacing text generation with direct probability scoring for bounded choices.
How Kev sidesteps generation#
Developer Jared Palmer released three Kev variants (0.8B, 4B, and 9B parameters) on Sunday designed for making discrete decisions without autoregressive loops. The approach uses a "pointer head" that reads decision probabilities directly from a single forward pass, processing state and candidate options together without generating intermediate text.
Kev handles three decision primitives:
- Noul — yes/no choices
- Choice — selection among candidates
- Score — ranked levels
When routing a query to tools, Kev outputs normalized probabilities:
search: 0.82
database: 0.13
calculator: 0.05
Since the model only scores options it's given, it cannot hallucinate choices outside the candidate set—a meaningful safety property for agent pipelines.
Multiple decisions in one pass#
The architecture supports evaluating several decisions against the same input state simultaneously. A block-causal attention mask isolates each question while the pointer head independently scores each set of candidates, allowing routing, safety checks, escalation, and ranking to happen in a single inference step.
Palmer's benchmark shows the 4B model processing three questions in 277 milliseconds on an M5 machine. However, without controlled comparison against equivalent text generation on the same hardware, the actual speedup versus conventional generation remains unclear.
Tradeoffs and calibration gaps#
The largest model, Kev-9B, achieved 83.7% accuracy on Palmer's locked out-of-domain test set. Yet probabilities returned by Kev don't always reflect genuine confidence—temperature calibration can drift on unfamiliar data distributions, a problem for agents that use probability thresholds to decide whether to escalate decisions rather than execute them.
