Agentic AI
New benchmark tests AI agents' ability to curate biology knowledge bases from scientific papers
FlyAOC evaluates how well AI systems can extract and structure information about genes from a large corpus of research literature—a real-world task that curators perform daily.
Maintaining scientific knowledge bases that organize research findings into searchable, structured form is labor-intensive work. Curators must hunt through published papers, cross-reference evidence, and annotate findings using standardized formats—a process that could potentially be accelerated by AI systems. A new benchmark called FlyAOC tests how well autonomous agents can handle this end-to-end workflow.
The gap in AI evaluation#
Most existing AI benchmarks focus on isolated tasks like extracting named entities (gene names, proteins) or identifying relationships between concepts in text. These tests don't reflect the actual complexity of real-world curation, which requires combining multiple search and retrieval steps to build a complete picture of what is known about a given subject.
How FlyAOC works#
The benchmark gives AI agents a gene symbol and a brief gene description, then asks them to search through a corpus of roughly 16,900 scientific papers about Drosophila (fruit flies) and recover as much curator-relevant information as possible. Agents must identify standardized function terms, note expression patterns, and find historical synonyms that connect decades of nomenclature changes. The test draws on over 7,300 expert-curated annotations from FlyBase, a widely used Drosophila knowledge base.
What the results show#
Researchers tested four different agent designs—ranging from simple memorization to coordinated multi-agent systems—and found that performance varied significantly based on how the system was structured, which AI model powered it, and how reliably it used available tools. The benchmark uncovered system-level failure modes that evaluating language models in isolation would miss, suggesting that the architecture of the agent itself matters as much as the underlying model's capabilities.
FlyAOC provides researchers with a reproducible testing ground for improving AI-assisted scientific curation workflows, which increasingly rely on retrieval-augmented systems that search and synthesize information on demand.
