AI
Perplexity Misleads: New Distillation Method Shows Log-Likelihood Hides Real Generation Quality Gaps
Researchers reveal that standard metrics can mask significant gaps in language model performance when generating text autoregressively rather than ranking answers.
A new study from researchers working with arXiv cs.CL exposes a critical disconnect in how distilled language models are evaluated: metrics that look good on paper often fail to predict real-world generation quality, sometimes hiding performance gaps as large as 20 percentage points.
The Perplexity Blind Spot#
When researchers compress large Transformers into smaller, faster hybrid models through distillation—a process of training a smaller student model to imitate a larger teacher—they typically measure success using log-likelihood scoring. This approach ranks candidate answers by their statistical likelihood under the model, rather than asking the model to generate answers from scratch. The team demonstrates that this method systematically underestimates performance degradation. In one striking example, a 7-billion-parameter distilled model that appeared nearly equivalent to its teacher (within 0.2 percentage points) actually fell 20.8 percentage points behind when forced to generate text autoregressively—a vastly different outcome that standard benchmarking would have missed entirely.
GenDistill: A Controlled Comparison Framework#
To investigate this phenomenon systematically, the researchers introduced GenDistill, a multi-stage pipeline for converting pretrained Transformers into efficient hybrid models called Hybrid Kimi Delta Attention (Hybrid-KDA). Using this controlled testbed on the Qwen 3 model, they methodically tested six critical design decisions: training objectives, loss masking strategies, training duration, dataset selection, parameter freezing, and architectural choices. Each variant was evaluated under both log-likelihood and generation-based metrics.
What Actually Matters for Generation#
The findings reveal that log-likelihood-based evaluation not only underestimates gaps—it can actually reverse the ranking of design choices, potentially leading researchers down misleading paths. Among the factors studied, dataset selection, completion-only masking (ignoring loss on certain tokens), and freezing attention layers during post-training emerged as the most consequential for preserving generation quality.
