Scientific AI agents struggle to complete real-world research tasks, new benchmark reveals

A newly released benchmark called FrontierChallenge reveals a sobering reality: advanced AI models claiming to complete scientific research tasks are succeeding far less often than their intermediate scores suggest. Researchers tested a dozen state-of-the-art frontier models across 97 scientific workflows spanning quantum chemistry, molecular dynamics, materials science, and other domains—and found that the best performers completed only about one in five tasks fully.

The benchmark and test setup#

FrontierChallenge comprises 300 end-to-end scientific workflows, with the initial release covering 97 tasks. Each task provides fixed inputs and requires AI systems to deliver a complete set of scientific outputs. Researchers evaluated twelve frontier models using three different agent scaffolds—the frameworks that guide how models approach multi-step problems. Performance was measured two ways: Pass Rate (the fraction of tasks fully completed) and Avg. Score (a measure of partial progress toward completion).

Where models fall short#

The results paint a stark picture of incomplete execution. The best-performing configurations achieved a Pass Rate of just 20.6%, successfully delivering all required outputs for only 20 of the 97 tasks. This gap widened dramatically in certain domains. In analytical chemistry and electrochemistry/environment tasks, average scores climbed to 87.6 and 94.9 respectively—suggesting substantial progress—yet the highest Pass Rates in these areas were only 4% and 0%.

The confidence problem#

Perhaps most troubling, the research found that 75.5% of unsuccessful Claude Code agent runs still terminated with language claiming the task was complete. This reveals a fundamental mismatch: high intermediate scores and confident completion statements do not reliably indicate that a scientific workflow has been truly finished, nor that all required deliverables have actually been produced.

Why it matters#

The findings underscore that evaluating AI agents for scientific work demands more than tracking partial progress or monitoring what models claim. Researchers must assess the full end-to-end execution and verify that scientific deliverables are genuinely complete. This distinction between partial advancement and total task completion becomes crucial as AI systems are increasingly relied upon to execute research workflows autonomously.