AI
New benchmark tests AI's ability to reason through complex biological protocols
Researchers release BioProBench, a large-scale dataset revealing that language models struggle with the procedural precision required for autonomous lab work.
Large language models show surprisingly weak reasoning abilities when asked to follow strict biological laboratory protocols, according to a newly released benchmark study. Even as general language comprehension has advanced rapidly, models falter on the intricate procedural logic, numerical accuracy, and safety considerations embedded in real experimental workflows.
The Resource: BioProCorpus and BioProBench#
Researchers have constructed BioProCorpus, a foundational dataset of over 22,000 human-authored biological protocols drawn from scientific literature and laboratory manuals. From this corpus, they systematically generated a comprehensive benchmark called BioProBench containing 523,784 distinct task instances—providing both a substantial training resource and a rigorous evaluation framework.
The benchmark introduces novel metrics designed to assess not just whether models can recall protocol steps, but whether they can reason about them accurately and safely.
Testing Current LLMs#
An evaluation of 10 major language models revealed a consistent pattern: general comprehension performs adequately, but drops sharply when tasks require deep procedural reasoning, quantitative accuracy, or safety-critical awareness. This gap underscores a fundamental limitation in deploying these systems for autonomous scientific work, where precision and safety are non-negotiable.
ProAgent: A Step Forward#
To demonstrate how domain-specific training could address these gaps, the team developed ProAgent, a system grounded in BioProCorpus. This prototype substantially outperforms baseline approaches, suggesting that targeted resources and training strategies can meaningfully improve LLM performance on protocol-following tasks.
The findings highlight both the opportunities and challenges in automating laboratory work: while models can grasp basic scientific language, truly autonomous experimentation requires far more rigorous reasoning about step sequences, measurements, and potential hazards.
