Researchers argue smaller language models are better suited for AI agents than large models

Proponents of increasingly specialized AI systems are making a counterintuitive case: when it comes to building autonomous agents that perform focused tasks, smaller language models outperform their larger counterparts on the basis of capability, cost, and appropriateness.

While large language models (LLMs) have earned recognition for their ability to handle diverse problems and engage in general conversation, they may not represent the optimal choice for the emerging class of agentic AI—systems designed to handle repetitive, narrowly defined operations with minimal variation, according to a position paper submitted to arXiv.

Why smaller is smarter for agents#

The argument rests on three pillars: capability, economics, and design fit. Small language models (SLMs) already possess sufficient performance for the kinds of specialized reasoning and execution that agentic systems require. The efficiency advantage becomes particularly stark when you factor in deployment costs—running dozens or hundreds of inexpensive SLMs for targeted tasks costs substantially less than invoking a general-purpose LLM for each operation. Beyond dollars, there's an architectural alignment: agentic systems don't need the conversational breadth that large models provide, making SLMs a natural structural match.

The hybrid approach for general intelligence#

The authors acknowledge that some scenarios demand the flexible reasoning abilities of large models. Their solution: heterogeneous agentic systems that deploy multiple different models together. An agent system could rely on lightweight SLMs for routine operations while calling on a larger model only when general-purpose problem-solving becomes necessary. This mixed strategy captures the best of both approaches without the waste of running a heavyweight system for every task.

Barriers and technical solutions#

Adoption won't be automatic. The paper identifies practical obstacles to transitioning existing systems from LLMs to SLMs—including entrenched infrastructure and workflows built around larger models. To help overcome these hurdles, the authors sketch out a general algorithm for converting LLM-based agents to SLM-based ones, aiming to reduce friction for organizations considering the shift.

The researchers frame their position as a statement about impact: even a partial industry-wide move from LLMs to SLMs would reshape operational efficiency and cost structures across AI deployment. They invite both supporting evidence and critical feedback, committing to publish responses publicly as part of the ongoing conversation.