Agentic AI
New Dataset Trains AI Terminal Agents with 50K Verified Docker-Based Trajectories
Researchers introduce TerminalTraj, a scalable pipeline for generating high-quality training data for command-line task agents across diverse software domains.
Developing AI agents that can reliably execute terminal commands and complete complex system tasks requires massive amounts of training data—but creating that data poses two significant obstacles. The new TerminalTraj framework tackles these challenges by automating the construction of realistic, verifiable agent trajectories at scale.
The Core Problem#
Building training datasets for terminal-based agents demands both executability and verifiability. Each task instance needs its own Docker environment configured correctly, and because different domains produce vastly different outputs, there's no single way to check whether an agent succeeded. These constraints have made it impractical to generate large, diverse training datasets.
TerminalTraj's Approach#
Researchers propose a three-stage pipeline that:
- Filters high-quality code repositories and constructs isolated Docker environments suitable for execution
- Generates task instances tailored to work within these containerized environments
- Synthesizes agent trajectories complete with automated validation code to confirm correct execution
Using this method, the team curated 32,000 Docker images and generated over 50,700 verified terminal trajectories spanning eight different domains.
Performance Gains#
Models built on the Qwen2.5-Coder foundation and trained on TerminalTraj data showed substantial improvements on TerminalBench, a standard evaluation suite:
- Up to 20% improvement on TerminalBench 1.0
- Up to 10% improvement on TerminalBench 2.0
The resulting TerminalTraj-32B model—with 32 billion parameters—achieved strong performance among models under 100B parameters, reaching 35.30% on TB 1.0 and 22.00% on TB 2.0, while also demonstrating improved scaling behavior at test time.
Why It Matters#
This work addresses a critical gap in agentic AI development: the ability to create realistic, large-scale training data for real-world command-line tasks. The open release of code and data may accelerate progress in building more capable terminal agents across research and industry.
