New Benchmark Datasets Reveal How Occlusions Undermine Video Action Detection Models

Computer vision models that detect actions in videos often perform poorly when actors or objects are blocked from view. A new study published on arXiv systematically explores this weakness by introducing five benchmark datasets designed to test how well action detection systems handle occlusions—both synthetic and realistic.

The Challenge of Hidden Actions#

Occlusions are a fundamental challenge in video understanding. When a person performing an action becomes partially obscured—whether by a static object or a moving obstacle—today's best models struggle. The research confirms what intuition suggests: as occlusion severity increases, performance drops significantly. Importantly, the team discovered that models behave differently depending on whether occluders are stationary or moving, indicating the systems rely on patterns that don't generalize across these scenarios.

Five New Benchmark Datasets#

The researchers created five datasets to measure this problem:

• O-UCF and O-JHMDB: Contain synthetically controlled static and dynamic occlusions applied to existing action recognition benchmarks
• OVIS-UCF and OVIS-JHMDB: Add realistic motion patterns to occluders rather than simple blocking
• Real-OUCF: Tests models on occlusions occurring naturally in real-world video footage

These datasets enable reproducible measurement of how different occlusion types affect model accuracy.

Surprising Capabilities of Transformers and Capsules#

The study uncovered three unexpected findings. First, transformer-based models consistently outperformed convolutional neural networks on occluded videos, even when those CNNs had been trained with occlusion as a data augmentation technique. Second, adding symbolic components like capsule networks to model architectures allowed them to recognize occluders they had never encountered during training. Third, neural networks spontaneously developed agreement about occluders in realistic images without explicit supervision, distillation, or contrastive learning objectives.

Training Recipes That Improve Robustness#

Drawing on these discoveries, the researchers developed practical training strategies that significantly boost performance under occlusion. Models using these recipes achieved improvements of 32.3% on O-UCF, 32.7% on O-JHMDB, and 2.6% on Real-OUCF when measured by video mean Average Precision (vMAP) compared to existing action detectors. The code and models have been made publicly available to support further research.