Most Video QA Benchmarks Don't Actually Need Temporal Grounding, Study Shows

A new research paper challenges the temporal validity of widely-used video question-answering (QA) benchmarks, revealing that most existing evaluation datasets don't genuinely require models to identify and process the most informative video frames.

The Frame Selection Problem#

Vision-language models that process video have a practical constraint: they can only examine a limited number of frames from potentially lengthy videos. This makes frame selection a critical challenge in the field. The research team set out to determine whether current benchmarks actually test a model's ability to select temporally relevant frames—or whether answers can be retrieved regardless of which frames are shown.

Measuring Temporal Sensitivity#

To assess this, the researchers introduced Frame Selection Sensitivity (FSS), a diagnostic metric that compares how a model performs when shown the most relevant frames versus the least relevant ones. A large FSS score indicates genuine temporal sensitivity; a low score suggests the model answers questions equally well (or poorly) regardless of frame choice.

Evaluating six benchmark datasets across eight different vision-language models, the team discovered that most samples are frame-agnostic—the model's accuracy barely changes when the most useful frames are swapped for unhelpful ones. To isolate questions that truly demand temporal reasoning, they combined FSS with a Language Independence Score (LIS), which filters out questions answerable from text cues alone.

The TempCore Solution#

The analysis revealed that only 5.5 to 31 percent of samples across tested benchmarks qualify as genuinely temporally sensitive. In response, the researchers constructed TempCore, a curated subset of existing benchmarks containing only those samples that authentically test temporal understanding. The team plans to release code and per-sample annotations to support more rigorous evaluation of video QA systems.