WebArxiv Benchmark Tests AI Agents on Scholarly Research Tasks

Foundation models have unlocked the ability for autonomous agents to browse and interact with websites, yet most existing benchmarks focus on general-purpose web navigation rather than research-centric work. The research community has largely been underrepresented in agent evaluation frameworks, and reproducibility suffers when benchmarks rely on live websites that change over time.

A Static Testbed for Reproducible Evaluation#

WebArxiv addresses these gaps by using arXiv as a fixed, snapshot-based evaluation environment. The platform offers several advantages: its content remains stable, its hierarchical structure mirrors real scholarly workflows, and it avoids privacy complications. The benchmark comprises 510 carefully designed tasks, each with a deterministic ground truth that doesn't shift between test runs. This stability is crucial for reliably measuring agent performance across different models and iterations.

Beyond Simple Lookup Tasks#

The benchmark goes well beyond basic information retrieval. Rather than asking agents to fetch straightforward facts, WebArxiv emphasizes more realistic scholarly challenges—retrieving papers that satisfy multiple constraints simultaneously, extracting specific details from research documents, and comparing findings across related works. These multi-step, nuanced tasks more closely mirror how researchers actually use discovery platforms.

Performance Gaps and a Memory Solution#

Testing various foundation-model-based agents against the benchmark revealed significant room for improvement. Behavioral analysis uncovered a recurring weakness: agents tend to get stuck in fixed interaction loops, relying too heavily on the same retrieval patterns even when those patterns fail to capture the full picture. This rigidity leads to incomplete or circular reasoning.

To address this limitation, researchers equipped agents with a lightweight dynamic-memory system that allows them to adapt their retrieval and reasoning strategies in response to context. This enhancement helps agents break out of repetitive patterns and adjust their approach based on what they've already learned.

The benchmark and supporting code are publicly available, offering the research community a shared standard for measuring progress in multimodal web agent capabilities.