Orthogonal Features Could Make AI Interventions Safer and More Predictable

Researchers working in mechanistic interpretability have long pursued a deceptively simple goal: understand how neural networks actually work by identifying the concepts they learn. A foundational assumption in this field is that meaningful semantic ideas — like mathematical reasoning or factual knowledge — get encoded as linear combinations of activations within the model's hidden layers. But this assumption carries a practical problem: when you tweak one concept, others often get scrambled as a side effect.

The Feature Entanglement Problem#

In principle, if we could identify a single feature that represents a concept, we could surgically alter it to study or modify the model's behavior. In reality, features overlap and interfere with each other. This feature entanglement means that an intervention targeting one concept bleeds into others, muddying the causal picture and making it hard to predict what will happen when you edit the model.

The researchers frame this challenge through the lens of the Independent Causal Mechanisms principle, which suggests that modular, separable representations should lead to more reliable causal edits. Their core insight: pushing features toward orthogonality — making them nearly perpendicular to one another in activation space — could reduce interference and enable cleaner interventions.

How the Approach Works#

The team formalizes the problem by measuring how much unintended downstream effects arise when you change a single feature. They show that this "feature interference" can be bounded in terms of the self-coherence of the dictionary of learned features, and connect this bound directly to an orthogonality regularization penalty applied during training.

The mechanism is elegant: by adding a regularization term that encourages the learned feature dictionary to have orthogonal columns, the model learns representations where changing one feature does less collateral damage to others. This doesn't require changing the model's core architecture — it's a training-time constraint.

Empirical Validation#

In experiments, the researchers demonstrate that this orthogonality regularization lets them perform more isolated interventions on mathematical reasoning concepts while keeping overall model performance intact. The results suggest that the method works as predicted: more orthogonal features genuinely lead to less feature interference and more predictable edits.