AI
New framework bridges vision-language models and autonomous driving for rare, high-risk scenarios
HERMES uses semantic knowledge from foundation models to help self-driving cars navigate unpredictable, safety-critical situations involving mixed traffic.
Researchers have introduced HERMES, an end-to-end autonomous driving system that combines large vision-language models with risk-aware planning to better handle edge-case driving scenarios that pose significant safety challenges.
The challenge of rare, critical moments#
Autonomous vehicles trained with vision-language models excel at understanding the general scene around them, but struggles persist when the driving environment becomes unpredictable. The toughest situations involve mixed-traffic environments—areas with pedestrians, cyclists, cars, and other heterogeneous road users—where unusual interactions occur rarely but carry high stakes. These "long-tail" scenarios are exactly where today's self-driving systems stumble most.
How HERMES works#
The framework takes a multimodal approach: it fuses multi-view camera observations, historical movement data from the vehicle itself, and semantic guidance derived from long-tail driving scenarios. A foundation-model-assisted pipeline annotates two kinds of structured context. First, "Long-Tail Scene Context" captures hazard-centric information about the environment. Second, "Long-Tail Planning Context" records maneuver intent and risk-aware guidance for trajectory generation.
A Tri-Modal Driving Module then integrates all these signals—visual, motion-based, and semantic—through intentional and risk-focused conditioning. This conditioning helps the system generate safer driving trajectories.
Validation and results#
Extensive testing on a large-scale real-world long-tail driving benchmark showed that HERMES consistently outperformed representative recent baselines across overall planning performance and in various safety-critical scenarios. Ablation studies verified that each major component of the system contributes meaningfully to the improvements.
The work opens a path toward more reliable autonomous driving in the complex, unpredictable situations that matter most for real-world safety.
