New Survey Maps the Landscape of AI Agents Operating Through Command-Line Interfaces

A new survey published on arXiv takes stock of terminal agents—AI systems designed to work through command-line interfaces. Researchers argue that this category of systems deserves focused study, separate from broader discussions of software engineering and tool-using AI.

What Terminal Agents Are#

Terminal agents are systems where the central loop of progress involves executing commands at a terminal, receiving textual feedback, and adapting based on how the environment responds. Rather than treating this behavior as incidental to other research areas, the authors propose organizing a whole research domain around this specific mode of interaction.

A Framework for Understanding Performance#

The survey introduces a seven-dimensional terminal competence profile that bridges system design, how agents learn and improve, and how well they perform. This framework accounts for how different factors shape real-world behavior:

• The underlying language model
• The interface through which commands are issued
• The software harness managing execution
• The runtime environment itself
• The broader computing context

Evaluation Gaps and Implications#

The analysis reveals significant blind spots in how terminal agents are currently tested. Most evaluations focus narrowly on whether tasks succeed or fail in the end, but they obscure important details about process quality, how agents recover from errors, and governance safeguards.

The researchers show that executable trajectories—records of what agents tried and how the environment responded—anchor learning in real feedback and verification. Yet this evidence is rarely captured or reported systematically in research.

What Matters for Future Work#

The study demonstrates that different benchmarks expose different strengths and weaknesses, and that comparing components in isolation can mislead—performance depends heavily on how everything fits together. To advance the field, the authors recommend:

• Explicit documentation of system and runtime setup