New AI agent Artisan automatically reproduces research results through code generation

Verifying whether published research findings hold up under scrutiny remains a challenge in computer science. A team has introduced Artisan, an AI agent designed to automatically regenerate the numerical tables from research papers by producing executable code that can be independently audited and re-run.

The reproducibility problem#

Scientific integrity depends on the ability to independently verify results. While large language model agents have begun tackling this task, existing approaches typically fail to deliver reproducible code and don't account for how results should be generated—limiting their practical value. Researchers needed a system that could not only reproduce findings but produce transparent, executable evidence of the process.

How Artisan works#

Artisan tackles reproduction as a code generation challenge rather than a black-box result-matching exercise. Given a research paper and its associated artifacts (code, data, or other resources), the system generates scripts that users can inspect, understand, and run themselves. This transparency ensures reproduction is verifiable and independent of the agent's decisions.

The agent relies on automated judging mechanisms that guide it toward correct numerical outcomes without directly revealing the answers. These safeguards prevent shortcuts such as hard-coding pre-computed values and instead encourage the agent to genuinely reconstruct the computational workflow.

Benchmark and evaluation#

To measure Artisan's capabilities, the team developed Artisan-Bench, the first benchmark specifically designed to assess code generation for result reproduction. The benchmark draws from 23 software engineering papers and includes 60 distinct tasks. Testing shows Artisan performs effectively and efficiently, while also surfacing 20 previously unreported errors—some in papers themselves, others in the associated artifacts.

Why it matters#

Success here has implications beyond a single tool. Automating table reproduction could accelerate how the research community validates prior work, builds upon existing findings, and maintains standards of scientific integrity. The approach of turning reproduction into auditable code offers a reusable model for other verification challenges across computational research.