Vision Transformers Trained on Images Now Outperform Time Series Models

Researchers have discovered a promising bridge between computer vision and time series analysis: a new framework called Time Vision Transformer (TiViT) that repurposes pretrained image recognition models to classify temporal sequences—without requiring any fine-tuning of the underlying model.

The Core Breakthrough#

The key insight is that Vision Transformers (ViTs) trained on massive image datasets contain representational knowledge that transfers effectively to time series problems. Rather than starting from scratch or adapting vision models through intensive retraining, the TiViT approach freezes the pretrained weights and leverages the hidden layer representations from OpenCLIP models. This strategy achieves competitive performance with state-of-the-art dedicated time series foundation models (TSFMs), which were specifically built for this task.

How It Works#

Not all layers in a vision transformer contribute equally. The researchers found that intermediate ViT layers exhibiting high intrinsic dimensionality—a measure of how much information they capture—are most useful for time series classification. This discovery reveals that the hierarchical structure of image-trained models encodes features applicable to sequential data in surprising ways.

The approach also explores complementarity between vision-based and time-series-specific representations. Combining features from both sources through concatenation yields additional performance gains, suggesting the models capture different aspects of temporal patterns.

Unlocking Further Gains#

While frozen models perform well, the framework can be enhanced further. By unfreezing the ViT backbone for continued pretraining and aligning it with TSFM representations using contrastive learning on time series data, lighter variants of TiViT achieve even better results. This hybrid strategy balances computational efficiency with predictive power.

Implications#

The findings suggest a new pathway for adapting large vision foundation models to new domains and tasks. Rather than redesigning models from the ground up, researchers can strategically repurpose pretrained representations, reducing development time and computational cost while maintaining competitive performance.