Aligned Language Models Narrow Their Output Diversity Through Probability Concentration

Despite strong performance on many tasks, aligned large language models tend to produce remarkably repetitive and predictable text. Researchers have now identified a concrete mechanism behind this phenomenon: the concentration of probability mass in the model's output distribution.

How Probability Concentration Works#

The study introduces the Branching Factor (BF) as a diagnostic measure—it captures the effective number of plausible next words the model could choose at each step, calculated from the entropy of the output distribution. A high branching factor indicates genuine diversity in generation, while a low one signals that the model heavily favors a narrow set of tokens.

The team's empirical findings establish two complementary patterns:

• BF declines naturally during generation, making models progressively more deterministic as they produce text. This decline occurs across tasks and appears to stem from autoregressive self-conditioning—where earlier tokens constrain later ones—rather than alignment design.
• Alignment training compresses BF dramatically from the outset, reducing it by factors of 2 to 5 overall, and sometimes by an order of magnitude at the start of generation. For example, base models might begin with a branching factor of 12, while aligned versions drop to 1.2.

Why Alignment Matters for Diversity#

This compression explains a long-observed phenomenon: aligned models are far less sensitive to decoding strategies. When the distribution is already sharply peaked, sampling temperature and other generation tricks cannot recover true diversity that was never available in the first place.

The results also illuminate how chain-of-thought prompting works in aligned models. When prompted to reason step-by-step, these models generate longer reasoning chains that eventually reach highly deterministic stages (low BF). This trajectory yields more stable outputs, suggesting that CoT's value lies partly in steering toward naturally low-entropy regions of the model's learned representations.

The Base Model Hypothesis#

Rather than fundamentally rewiring how models generate text, the research suggests alignment operates through a more elegant mechanism: it nudges models toward specific stylistic tokens—phrases like "Sure" or formal openings—that unlock already-existing low-entropy pathways in the base model. Nudging experiments support this view; prompting unaligned models with the same tokens produces comparable branching factor reductions.