New benchmark tests how well AI agents can replicate real-world research findings

A new benchmark is pushing beyond basic code execution to evaluate whether AI agents can actually verify scientific claims the way human researchers do. Researchers have introduced ReplicatorBench, a framework designed to test large language model agents on their ability to replicate research in social and behavioral sciences—not just reproduce it with existing code.

The gap between reproduction and replication#

Most existing benchmarks focus on reproduction: running published code with available data to confirm computational results. But replication is harder. It requires finding new data, redesigning experiments, and interpreting whether a claim genuinely holds in different contexts. Current benchmarks often gloss over this distinction and rarely assess whether agents can identify research that cannot be replicated.

ReplicatorBench changes this by including both replicable and non-replicable research claims, allowing it to test agents' capability to recognize when a finding doesn't hold up under scrutiny.

How the benchmark works#

The evaluation spans three stages that mirror real scientific work:

• Data discovery and retrieval — finding resources needed to test a claim
• Experiment design and execution — running computational analyses in sandboxed environments
• Result interpretation — drawing conclusions from the evidence

This multi-stage approach evaluates the full workflow rather than just final outcomes.

What the tests revealed#

The researchers built ReplicatorAgent, an agentic framework with tools like web search and sandboxed code execution, and tested it across four different large language models. Key findings:

• Agents performed well at designing and running computational experiments
• They struggled significantly with retrieving new data sources required for replication
• Performance varied based on programming language choice and code access levels

The work underscores that while LLM agents can handle computational rigor, they still fall short at the data discovery and resource-finding tasks that are central to real-world replication efforts. Code and data are publicly available, enabling further research into this emerging challenge.