AI
New benchmark challenges AI systems to reason across scientific documents and images
Researchers introduce DocHop-QA, a dataset designed to test how well language models can synthesize evidence from multiple papers, tables, and figures to answer complex questions.
Despite advances in large language models, current question-answering systems struggle with what researchers describe as the central bottleneck of real scientific research: piecing together information scattered across different documents, formats, and media types.
A new benchmark called DocHop-QA aims to address this gap. Researchers have constructed a dataset of over 11,000 question-answer pairs grounded in PubMed research articles, designed to push AI systems beyond simple text-based lookup and toward genuine multi-document, multi-modal reasoning.
Why existing benchmarks fall short#
Today's QA datasets typically focus on single documents and rely primarily on text. They measure the ability to find and extract short passages, but they sidestep the messier, more realistic challenge: synthesizing evidence when tables, figures, and conclusions span multiple papers without explicit links between them. DocHop-QA is built to capture this complexity.
What makes DocHop-QA different#
The benchmark incorporates three forms of evidence: textual passages, tables, and layout cues from scientific papers. Questions require drawing inferences across documents in ways that mirror how researchers actually work—connecting findings from one paper to data presented in another, then reaching a conclusion that combines both.
To scale the dataset's construction, the team used a large language model to generate question-answer pairs guided by 11 scientific reasoning concepts, ensuring diversity and coherence.
A test of current capabilities#
The researchers evaluated existing models on DocHop-QA using four different task settings, including open-ended answering, identifying multimodal evidence, and predicting document structure. Results show that current systems struggle with the combination of long context windows and multi-evidence integration demands.
This establishes DocHop-QA as a rigorous testing ground for next-generation scientific QA systems that must handle the full complexity of real information seeking.
