Research Agents Struggle to Find and Organize Essential Papers, Study Shows

Automated research agents that promise to streamline survey writing fall significantly short of expert performance when tasked with finding and organizing scientific literature, according to a comprehensive evaluation presented in a new paper.

The Benchmark#

Researchers built TaxoBench, a rigorous evaluation framework grounded in 72 highly cited surveys on large language models, encompassing 3,815 papers and their expert-authored taxonomies. The benchmark tests systems in two complementary modes: Deep Research measures a system's ability to both locate and structure papers from scratch given only a topic, while Bottom-Up mode provides the correct papers upfront to isolate organizational capability alone.

The evaluation employs multiple metrics suited to different aspects of the task—leaf-level accuracy using ARI and V-Measure, and structural hierarchy quality measured through US-TED, US-NTED, and Sem-Path.

How Well Do Current Agents Perform?#

The results expose two distinct bottlenecks. Across seven Deep Research Agents and sixteen different LLM configurations, the strongest performer retrieved just over 20% of the papers that domain experts actually cited as essential. In the Bottom-Up scenario, where retrieval is not a constraint, none of the 70 standard runs reached the average organizational depth of 4.86 levels that human experts achieved.

When systems were forced to match expert depth, they did so by fracturing the taxonomy structure itself—preserving quantity at the cost of coherence. Even when newer model generations improved on one metric (ARI) by 3.68 percentage points, performance on the Sem-Path metric remained near baseline, suggesting minimal genuine organizational sophistication.

The Organization Gap#

On the ten surveys where direct human-system comparison was possible after depth adjustment, human experts led by 13.27 percentage points across all ten metrics combined. The findings underscore that hierarchical organization and accurate retrieval represent separate, compounding failure modes rather than a single problem.

The research highlights a critical methodological finding: hierarchy metrics require calibration relative to a reference standard before they can reliably compare models. Without this precaution, reported improvements may reflect gaming the evaluation rather than genuine progress.