Optimizing Model Predictions by Focusing on Top Classes Improves Accuracy at Test Time

A new study challenges conventional thinking about test-time optimization, demonstrating that refining predictions to emphasize the most probable categories of a sample can deliver meaningful accuracy improvements across both large language models and image recognition systems.

The Core Finding#

The research posed a straightforward question: can adjusting a model's focus toward likely classes during inference—using only the test sample itself—yield better predictions? Previous work had suggested this approach would not help. This new investigation provides evidence to the contrary, with gains averaging around 0.6% across 26 large language models tested on 12 text datasets and 18 image recognition models trained on ImageNet.

How the Optimization Works#

The team explored several approaches, with entropy minimization—a technique that increases predictions for likely classes while reducing those for unlikely ones—producing the strongest results. Notably, they discovered that simply maximizing the logits (raw scores) of the two most probable classes captured most of entropy minimization's benefits in many cases, even occasionally surpassing it in performance.

This finding suggests that the secondary mechanism of suppressing unlikely classes matters less than previously assumed. The core benefit comes from amplifying the top candidates themselves.

Gradient-Ray Ensembling#

The researchers also introduced gradient-ray ensembling, a method that applies the same gradient direction across multiple learning rates and aggregates the resulting probability predictions. This approach improved accuracy further to 0.75% on average while reducing the sensitivity to choosing a precise learning rate—a practical advantage that trades slightly more computation for more robust performance.

What Moderates Success#

The investigation uncovered several factors influencing whether this optimization helps:

• Prediction uncertainty emerges as the strongest moderator—optimization works best when the model is genuinely unsure
• Model architecture choice plays a meaningful secondary role
• Dataset characteristics exert a smaller but measurable effect