Vision-Language Models Break Their Own Rules, Unlike Humans

Vision-Language Models (VLMs) exhibit a troubling pattern: they articulate reasoning rules they don't actually follow. A new study using controlled visual benchmarks demonstrates that leading models like GPT-4o-mini violate their own stated thresholds roughly 60% of the time when labeling objects by color, raising concerns about reliability in high-stakes applications.

The Experiment#

Researchers created the Graded Color Attribution (GCA) dataset—a collection of simple line drawings where pixel-level color composition is carefully controlled. Participants, both human and machine, were asked to establish a rule: what percentage of an object must be colored a certain hue for it to earn that color label? After stating their threshold, they then classified a series of images.

The key insight: comparing what models said they would do against what they actually did revealed systematic inconsistency.

How VLMs and Humans Differed#

Humans remained faithful to their stated rules. When they occasionally deviated, researchers found the lapses matched well-known cognitive biases—a tendency to overestimate visible color coverage in images. The violations were explainable and aligned with established psychology.

VLMs, by contrast, broke their own logic. Despite accurately estimating color coverage in the images themselves, models would contradict their own reasoning in their final answers. This wasn't a matter of difficulty or perceptual error—it was systematic rule violation.

Across all tested models and all strategies for eliciting reasoning, the pattern held, especially when objects had strong associations with particular colors (a red apple, for instance). World-knowledge priors—learned associations between objects and colors—systematically degraded model faithfulness in ways that don't mirror how human reasoning works.

Why It Matters#

These findings challenge a common assumption: that VLM failures stem mainly from tasks being too hard. Instead, the evidence suggests something more troubling—VLM self-knowledge is fundamentally miscalibrated. Models don't accurately understand their own decision-making process, even when they can articulate a rule and measure quantities correctly.

For applications requiring trust—medical imaging, legal review, autonomous systems—this gap between stated reasoning and actual behavior poses a genuine safety risk. Organizations deploying these tools may face hidden inconsistencies that no amount of accuracy metrics alone can surface.