AI
Fine-tuning improves how well language models match target probability distributions
Researchers show that language models can learn to generate outputs that follow intended distributions through targeted training on synthetic tasks.
Language models are frequently deployed in applications that demand outputs aligned with specific probability distributions—yet their generated tokens often fail to match those intended distributions in practice. When a system needs to sample randomly from a distribution that can't be pre-enumerated, passing the job to a separate randomizer isn't possible. A new study explores whether models can acquire this calibration ability directly via fine-tuning.
The approach#
Researchers trained language models on synthetic tasks requiring sampling from mathematical distributions, testing two distinct fine-tuning strategies. The soft-target method converts desired output distributions into next-token probabilities using trie-derived targets, while the hard-target approach trains on actual samples drawn from the target distribution itself. Both represent different angles on the same core question: can models learn to produce properly calibrated outputs?
What improved#
Across a suite of 12 models spanning four different model families, both training methods delivered substantial gains. Models improved at sampling from held-out distributions they hadn't encountered and at parameter settings outside their training regime—suggesting the learned skill generalizes beyond the narrow synthetic examples. The gains extended beyond mathematical distributions into real-world stochastic generation: fine-tuned models dramatically expanded their first-token support in open-ended random generation tasks by one to two orders of magnitude, and showed improvements on NoveltyBench.
Trade-offs observed#
The improvement didn't come free. Some downstream capabilities suffered, particularly arithmetic reasoning, with the magnitude of degradation varying across different model architectures. These costs merit careful consideration when deciding whether the calibration gains justify the performance trade-offs for a given application.
Broader implications#
The findings demonstrate that probabilistic calibration represents a learnable skill, not a fixed constraint of model architecture. More importantly, the capability learned on mathematical tasks—where distributions are precise and testable—actually transfers to natural-language stochastic generation, where distributions are fuzzier and less well-defined. This suggests that focused training on well-controlled synthetic tasks can produce meaningful improvements in more ambiguous real-world scenarios.

