State-Space Models Outpace Attention for Efficient Long-Form Video Generation

Diffusion models have transformed image generation, but scaling them to produce longer video sequences remains computationally prohibitive. A new research effort tackles this bottleneck by swapping attention-based temporal processing for state-space models, achieving more efficient generation of videos extending to hundreds of frames.

The Attention Bottleneck#

Current video diffusion models rely on attention layers to extract temporal information from video frames. While effective, attention mechanisms carry a steep computational penalty: their memory and processing costs grow quadratically as sequence length increases. This becomes especially problematic when attempting to generate extended video sequences, making the approach impractical for realistic long-form synthesis tasks.

State-Space Models as an Alternative#

Rather than accept these constraints, the researchers propose leveraging state-space models—mathematical structures like Mamba—as temporal feature extractors. These models process sequential data with linear-time memory consumption relative to sequence length, a dramatic improvement over the quadratic scaling of attention. Initial work suggested that bidirectional SSMs (which process information in both forward and backward directions) outperform traditional unidirectional variants for spatial feature extraction in image generation.

Key Findings#

The team evaluated their SSM-based approach on long-sequence video benchmarks, including the MineRL Navigate dataset, across multiple model sizes. The results demonstrate a clear advantage: for videos up to 256 frames, SSM models consume significantly less memory than attention-based alternatives while maintaining equivalent video quality metrics (FVD scores). In several cases, SSM variants achieved superior performance within the same GPU memory budget.

Critically, the researchers confirmed that bidirectionality benefits temporal modeling too—not just spatial understanding. This insight guided their architecture design and contributed to the performance improvements observed.

Implications#

The work suggests a practical path forward for scaling video generation systems. By trading attention's quadratic cost for the linear scaling of structured state spaces, researchers can generate longer, higher-quality videos without prohibitive hardware requirements. This could democratize video synthesis tasks and open new applications in media, simulation, and beyond.