Getting L0 Wrong Breaks Sparse Autoencoders' Feature Learning, Researchers Find

Sparse Autoencoders (SAEs) are a key tool for understanding what features large language models learn from their training data, but researchers have discovered a critical problem: the sparsity level — how many features activate per token — must be tuned precisely or the entire approach falls apart.

In work posted on arXiv, David Chanin and colleagues show that when the L0 parameter (the average number of active features per token) is not set correctly, SAEs fail to discover the underlying, interpretable concepts they're meant to extract.

The Problem with Mistuned Sparsity#

When L0 is set too low, the autoencoder skimps on active features and responds by mixing correlated concepts together to maintain reconstruction quality. Set it too high, and the model finds degenerate solutions that also blend features together. In both cases, researchers end up with an SAE that appears to work — it reconstructs activations well — but actually learns muddled representations that combine multiple semantic ideas.

The researchers note that standard practice treats L0 as a tuning knob with no objectively "correct" value, only a trade-off between sparsity and how well the model reconstructs the original data. This assumption turns out to be wrong.

Finding the Right Sparsity Level#

The team developed a proxy metric to identify the appropriate L0 for a given training distribution. Their method successfully pinpoints the correct sparsity level in toy models and aligns with the sparsity that maximizes performance on sparse probing tasks when applied to real language models.

A major finding: most SAEs in widespread use today employ an L0 that is too low. This means existing interpretability research based on these models may be working with noisy, entangled feature representations rather than clean, monosemantic ones.

Why This Matters#

The study underscores that practitioners cannot treat L0 as a free parameter if they want SAEs to extract genuine, disentangled features from neural networks. Getting the sparsity right is not optional — it is fundamental to training interpretable autoencoders.