Geometry-Aware Adaptation Boosts Pretrained Model Performance Without Retraining

Pretrained machine learning models often face a fundamental challenge: they are trained on only a fraction of the label space they may eventually need to predict. A new approach from computer science researchers addresses this gap by leveraging the geometric structure of label spaces—the distances and relationships between different classes—to boost performance without requiring additional training.

The Core Technique#

The key innovation involves swapping the traditional argmax prediction rule with the Fréchet mean, a geometric averaging method that accounts for the structure of the label space. This simple substitution acts as a drop-in replacement for standard inference, making it practical to integrate into existing systems. The method, called Loki, exploits metric information about how classes relate to one another—such as similarity or distance measures—to make more informed predictions about classes the model has never explicitly encountered.

Empirical Results#

Testing demonstrates substantial gains across multiple benchmarks. On ImageNet, Loki achieves up to 29.7% relative improvement over SimCLR when paired with external metrics. The approach also scales effectively to scenarios with hundreds of thousands of classes. For zero-shot models like CLIP that lack explicit external metrics, Loki self-derives metrics from class embeddings and obtains a 10.5% improvement in performance.

Theoretical Foundations#

The researchers provide rigorous analysis of their framework, establishing three key contributions: learning-theoretic bounds that trade off label space diameter, sample complexity, and model dimensionality; complete characterizations of which scenarios allow prediction of any unobserved class; and an active learning-inspired procedure for optimally selecting training classes when full coverage is infeasible.

Broader Implications#

By transforming how predictions are made from a trained model without requiring retraining, this work opens a practical path for adapting existing models to broader label spaces and improving zero-shot prediction capabilities in real-world settings.