New Benchmark Tests AI's Ability to Spot Home Safety Hazards During Autonomous Exploration

A team of researchers has unveiled HomeSafeBench, a new evaluation framework designed to assess how well artificial intelligence systems can identify domestic safety hazards by autonomously exploring a virtual home environment. Since preventable injuries at home represent a major public health concern, automating hazard detection through intelligent inspection could help protect residents before accidents occur.

The Benchmark and Its Scope#

HomeSafeBench operates within the VirtualHome simulator, enabling an embodied agent to move freely through a fully interactive 3D home, change its viewing angle, and detect unsafe conditions based solely on first-person visual input. The benchmark encompasses 1,000 human-validated inspection tasks across five categories of typical household hazards, establishing a standardized way to measure how well different AI systems perform at this safety-critical task.

Performance Gap Reveals AI Limitations#

When researchers tested leading vision-language models (VLMs) against the benchmark, the results exposed a substantial shortfall in machine capability. The strongest performing AI system achieved approximately 34.7% F1 score—a metric combining precision and recall—whereas human inspectors reached 98.0%. A particularly revealing pattern emerged: across all tested models, precision significantly outpaced recall, indicating that AI systems tend to underreport hazards rather than falsely flag safe situations. This systematic conservatism in risk detection suggests a shared weakness in how current models recognize and communicate potential dangers.

A Data-Driven Solution#

To address this performance gap affordably, the researchers developed CueBack, a training method that builds supervisory signals from unlabeled video. The approach leverages the natural structure of inspections—where clues about a hazard typically appear before confirmation—by tracing backward along recorded trajectories to find the earliest moment when warning signs become visible, then converting these frames into training examples. After fine-tuning a lightweight 4 billion-parameter VLM on CueBack-generated data, performance on out-of-distribution test cases improved markedly, rising from 18.7% to 45.3% average F1, surpassing the strongest existing closed-source model.