AI Research
New Benchmark Reveals How Far LLM Agents Still Are from Autonomously Deploying Research Code
DeployBench tests AI agents on a challenge they often fail: getting research artifacts from academic papers to actually run on a fresh machine.
The Problem with Reproducibility#
When machine learning researchers publish new methods or systems, they often release code alongside their papers. But there's a hidden hurdle: setting up that code in a clean environment — installing dependencies, configuring system-level tools, and patching legacy code — remains tedious and error-prone, even for skilled engineers. Large language model (LLM) agents that excel at other software tasks often stumble here, and until now, no systematic benchmark has measured just how badly.
Introducing DeployBench#
Researchers have created DeployBench, a benchmark consisting of 51 real research-artifact deployment tasks spanning AI/ML, computer systems, and scientific computing. Rather than testing agents on abstract toy problems, each task involves getting actual published research code to run from scratch, complete with system dependencies like GPU drivers, CUDA configurations, and kernel-level setups that can't be containerized away. A hidden verification pipeline then executes each paper's designated experiment and checks whether the outputs match the expected results.
How Well Do Today's Agents Perform?#
The team evaluated five leading LLM models using an open-source agent framework called OpenHands. Results were sobering: pass rates ranged from just 7.8% to 51.0% across the model-task combinations. Yet the failures revealed something more interesting than simple incompetence. The dominant failure mode — accounting for 102 of 181 failures — involves agents that prematurely declare victory. These agents terminate early after running some checks that look successful to them, but which actually validate a weaker or different target than what the paper actually requires.
Why This Matters#
The findings expose a critical gap: while LLM agents show promise on software engineering tasks, they often lack the judgment to distinguish genuine success from plausible-sounding false success. This "completion-judgment problem" suggests that better monitoring, clearer success criteria, and more rigorous validation loops will be essential before agents can truly operate autonomously in research artifact deployment. DeployBench now offers the research community a realistic testing ground to measure and improve on these capabilities.