New Method Uses Existing Scientific Software to Train AI Agents Without Manual Verification

Terminal agents — AI systems that interact with command-line environments — are moving beyond software engineering into specialized scientific applications. A key bottleneck in scaling these agents has been the need to hand-author training environments and verification systems for each new domain.

Researchers from arXiv propose a novel approach called software-in-the-loop reconstruction (SWR) that sidesteps this manual work by mining training data directly from existing scientific software. Rather than building verifiers from scratch, the framework repurposes real software workflows to generate both reference outputs and correctness checks.

How It Works#

The method operates on executable programs that map structured inputs to outputs. For each workflow, multiple input configurations are generated. The resulting cases are then split into two sets: publicly observable input-output pairs and hidden test cases. An AI agent receives only the task instruction, input schema, and public examples, then must construct a working program without seeing the original source code. Its solution gets evaluated against hidden inputs to determine correctness.

A hierarchical verifier judges success across three dimensions: whether outputs match domain-specific semantics, whether the generated program maintains valid structure, and whether the agent avoided taking shortcuts. Public feedback from test cases supports iterative refinement, letting agents revise and resubmit solutions.

Results at Scale#

The team instantiated this framework with 500 workflows spanning 46 different software families across six scientific domains. Using the Qwen3.8-Max model, the system successfully solved 838 tasks across three attempts, generating 1,422 verified trajectories. These were oversampled to create 3,000 training examples.

Fine-tuning a smaller 27-billion-parameter variant of Qwen on this synthetic data raised performance on Terminal-Bench 2 from 47.94% to 53.56% on average. The approach outperformed four baseline models trained on comparable amounts of data, suggesting the quality of self-supervised examples matters significantly.

Why It Matters#

This work demonstrates that AI agents for science and specialized domains don't need hand-curated training sets. Existing domain software itself encodes the knowledge needed to verify correctness. By leveraging these artifacts, researchers can scale agent training across new domains with minimal human engineering — adding new workflows and test cases without building reference solutions or custom verifiers each time.