New Benchmark Reveals Multimodal AI Models Struggle With Basic Visual Perception Tasks

A new research study challenges assumptions about multimodal AI capabilities by exposing significant gaps in how well modern language models handle elementary visual perception—skills that human children develop early in cognitive growth.

The Percept-V Dataset#

Researchers introduced Percept-V, a benchmark containing 6,000 algorithmically-generated images organized across 30 domains. Each domain targets one or more perception abilities drawn from the TVPS-4 framework, a cognitive science model that measures human visual skills including discrimination and form constancy. The images were designed to be straightforward, with minimal reasoning or background knowledge required to solve them.

Surprising Underperformance#

Despite state-of-the-art multimodal large language models (MLLMs) routinely solving far more complex tasks, both proprietary and open-source variants performed significantly worse than human subjects on Percept-V. The gap was stark: while people handled these basic perception challenges with high accuracy, leading AI systems stumbled.

Key Findings#

The study surfaced several critical limitations:

• Complexity sensitivity: Model performance degrades noticeably as the number of objects in an image increases, suggesting difficulty with visual crowding.
• Uneven skill distribution: Certain perception abilities proved substantially harder for all tested models.
• Generalization weakness: When researchers fine-tuned an open-source MLLM on Percept-V, performance improved on that specific dataset. However, these gains did not transfer meaningfully to related perception benchmarks, indicating the learned representations remain brittle and domain-specific.

The findings suggest that despite their advanced reasoning abilities, contemporary multimodal AI models retain fundamental weaknesses in the basic visual perception skills that form the foundation of human intelligence.