AI
AI coding agents struggle to replicate materials science research, new benchmark finds
A new evaluation tool reveals that large language models have significant gaps when tasked with reproducing computational science workflows.
Researchers have developed AutoMat, a benchmark to test whether AI-powered coding agents can successfully reproduce research findings in computational materials science. The results expose substantial limitations in current systems: even the highest-performing setup achieved only a 53% success rate.
The Challenge of Scientific Reproducibility#
The research addresses a critical gap in AI evaluation. While large language models have demonstrated strong performance on standard software engineering benchmarks, their ability to handle scientific computational workflows remains unclear. These workflows demand more than just coding proficiency—they require agents to decode underspecified procedures buried in research papers, work with specialized and complex toolchains, and correctly interpret whether results actually support the scientific claims being tested.
How AutoMat Works#
The benchmark was developed in collaboration with domain experts who curated a collection of genuine claims drawn from published materials science papers. AutoMat tests whether coding agents can:
- Recover incomplete computational procedures from academic text
- Navigate specialized scientific software and dependencies
- Determine whether results support or refute the original claims
- Execute complex, end-to-end workflows from start to finish
Researchers evaluated multiple coding agent configurations across several leading foundation models to measure their performance.
Where Agents Fail#
The error analysis reveals a consistent pattern of weaknesses. Agents performed worst when forced to reconstruct workflows directly from paper text alone—a realistic scenario in reproducibility work. The most common failure modes included:
- Incomplete or partial procedures
- Methodological deviations from the original approach
- Execution fragility and runtime errors
These limitations suggest that current LLM-based agents lack the robustness needed for reliable scientific reproducibility tasks.
Why This Matters#
As AI systems are increasingly deployed in scientific research settings, understanding their actual capabilities is essential. AutoMat serves a dual purpose: it functions both as a reproducibility benchmark and as a diagnostic tool for pinpointing where agentic AI systems fall short in applied science contexts. The findings underscore that real-world scientific work requires capabilities beyond what current models offer.
