Mamba-Based Model Challenges Transformers for Time-Series Forecasting

Transformers have dominated long-term time series forecasting for years, but their reliance on self-attention mechanisms comes with significant drawbacks: quadratic computational complexity and inherent permutation invariance that can hamper temporal modeling. Researchers have now introduced MambaTS, an alternative architecture built on the Mamba model that questions whether self-attention is truly necessary for capturing long-range dependencies across time and variables.

The Problem with Transformers#

Self-attention in Transformer models, while effective at modeling relationships across long sequences, scales inefficiently as sequence length grows. Additionally, attention's permutation-invariant nature—treating all input orderings the same—creates a fundamental mismatch with time-series data, where temporal ordering carries crucial information. This motivates the search for more efficient alternatives.

How MambaTS Works#

The framework uses Variable-Aware Scan along Time (VAST), a mechanism that learns relationships between different variables during training and automatically determines an optimal scan order at inference time through a shortest-path decoding strategy. Rather than relying on learned attention weights, this approach models dependencies through structured, linear-complexity operations.

At its core lies the Temporal Mamba Block (TMB), which adapts the vanilla Mamba architecture by removing unnecessary causal convolutions—a simplification justified by the presence of independent input variables. The authors further reduce overfitting risk by applying selective dropout to specific parameters within the block.

Validation and Performance#

Testing across eight public datasets shows that MambaTS achieves competitive or state-of-the-art results on most benchmarks, demonstrating that the linear-scan paradigm can match or exceed Transformer performance without the computational overhead. The approach opens a new direction for time-series modeling that prioritizes efficiency alongside accuracy.

Code for the project has been made publicly available for reproducibility and further development.