Audio-JEPA: Efficient Self-Supervised Learning Framework Predicts Masked Sound Representations

Researchers have introduced Audio-JEPA, a self-supervised learning framework that learns audio representations by predicting masked regions in spectrogram data. The approach adapts the Joint-Embedding Predictive Architecture (JEPA) paradigm—which has shown promise in computer vision—to handle audio effectively.

How It Works#

Rather than attempting to reconstruct raw audio waveforms, Audio-JEPA operates in a higher-level feature space. The system uses a Vision Transformer backbone to process mel-spectrograms, the input audio first converted into visual-like spectral representations. During training, the model masks random patches of these spectrograms and learns to predict the latent representations of the masked regions without seeing the actual content.

Training and Efficiency#

The team pre-trained their model on unlabeled clips from AudioSet, a large-scale audio dataset, using 10-second audio clips at 32kHz sampling rate. A key strength of the approach is its data efficiency: Audio-JEPA achieves performance comparable to established models like wav2vec 2.0 and data2vec while requiring less than one-fifth of their training data. The implementation required no extensive hyperparameter tuning, suggesting the method is relatively straightforward to apply.

Evaluation and Results#

The researchers evaluated Audio-JEPA using the X-ARES benchmark suite, which tests performance across diverse audio tasks spanning speech recognition, music analysis, and environmental sound classification. The framework held its own against these stronger baselines despite the simpler training recipe and data constraints.

The team has released both the code and pre-trained model checkpoints publicly on GitHub, making the approach accessible for further research and application.