Newer Vision-Language Models Don't Always Drive Better, Study Finds

Researchers have challenged a widespread assumption in autonomous driving research: that newer and larger vision-language models (VLMs) automatically deliver superior performance behind the wheel.

The Study#

Researchers introduced LightEMMA, a standardized framework designed to evaluate how well VLMs perform on driving tasks. The team tested 15 models spanning five major VLM families against the nuScenes prediction benchmark—a widely-used autonomous driving dataset. Crucially, their evaluation protocol relied on the inherent capabilities of each model without special fine-tuning, custom architectures, or tailored prompt engineering, allowing for fair cross-model comparisons.

Key Findings#

The results contradicted conventional wisdom. Despite each new generation of VLMs bringing larger model sizes and stronger general reasoning abilities, successive iterations did not consistently outperform their predecessors in driving scenarios. The researchers also identified systematic failure patterns across models:

• Overreliance on recent actions: Models tended to simply repeat prior movements rather than reasoning about the current scene
• Difficulty resolving visual conflicts: When multiple visual cues pointed in different directions, models struggled to integrate them coherently

Implications#

These findings underscore that raw model capability does not automatically translate to safe autonomous driving. The authors argue that domain-specific adaptation and training—tailored specifically for driving contexts—will be essential before VLM-based systems can match or exceed existing autonomous driving methods. The work highlights a gap between general-purpose AI advancement and specialized robotic control tasks.