Researchers teach AI to ask the right questions about scientific figures

Scientists are addressing a fundamental gap in how AI models understand research papers: while current vision-language models can describe what they see in figures, they struggle to ask the kinds of deeper, context-aware questions that drive scientific discovery.

The core problem#

Multimodal understanding of academic papers requires more than image recognition. Figures in scientific literature serve as anchors for broader arguments—they raise implicit questions that the surrounding text then answers. However, existing AI models typically generate questions that could be resolved by looking at the figure alone, missing the paper's larger research narrative.

Introducing MQUD#

Researchers have created MQUD, a dataset of 1,250 figure-evoked questions sourced from 56 scientific papers. Notably, 708 of these questions were directly annotated by the papers' original authors—experts who understand exactly what each figure is meant to communicate within the research's argument.

The dataset focuses on three critical properties:

• Inquisitive: Questions that aren't already answered by prior context in the paper
• Salient: Inquiries that connect to the paper's main research claims and receive answers later
• Visually grounded: Questions that stem from visual insights in the figure

Testing the approach#

Experiments with open-source vision-language models like Qwen 3.5 revealed a consistent pattern: without guidance, these models ask questions that can be answered directly from the image itself. When fine-tuned on MQUD, however, the models learn to generate questions that align with how figures function within scientific arguments—asking about the implications and connections that drive research forward rather than simply describing visual elements.

This work extends the linguistic framework of Questions Under Discussion (QUD) from text-only discourse into multimodal scientific contexts, pointing toward AI systems that better comprehend how knowledge builds within complex research documents.