Study Questions Whether AI Models Describe Charts Accurately or Add Unsupported Interpretation

Multimodal large language models (MLLMs) like Gemini and GPT can analyze charts and connect them to broader context, but a new study raises questions about whether their interpretations rest on solid evidence or speculation.

Researchers examined over a thousand AI-generated descriptions of charts from 102 visualizations, testing how three different AI models responded to various conditions. The goal was to determine whether the models' claims were grounded in what the charts actually showed or represented assumptions the models themselves introduced.

How the study worked#

The investigation tested four input scenarios: providing only the chart image, supplying accessible chart descriptions alongside the image, withholding context to potentially prompt cautious language, and other variations. The team then categorized each model statement into three types: DIRECT claims (supported directly by the visualization), DERIVED claims (logical inferences from the data), and SPECULATIVE claims (interpretations added by the model). An automated check also assessed whether numeric details in descriptions matched the source data.

Key findings#

When researchers provided accessible chart context—structured descriptions of the data—Gemini and GPT became more likely to make direct, chart-supported claims and improved their numeric accuracy in some cases. However, simply adding the image to the full context did not consistently improve numeric agreement across models.

A noteworthy result emerged around prompts designed to encourage caution: instructing models to withhold context did not reliably push them toward more reserved language. Perhaps most striking, sections where models discussed real-world significance remained predominantly speculative—driven by model inference rather than chart evidence.

Why it matters#

As AI systems become more widely used to generate descriptions for accessibility purposes, the distinction between factual description and model-supplied interpretation takes on practical importance. The findings suggest that accessible description systems need to be designed to separate claims the model can verify from claims it is generating on its own, helping users—especially those relying on these descriptions for access—distinguish between reliable facts and educated guesses.