Cognitive Bias in Training Data Drives LLM Collapse—and a Simple Prompt Fix Could Help

The Diversity Problem#

When large language models undergo post-training alignment—the process that makes them safer and more useful—they often become less diverse and creative. This phenomenon, known as mode collapse, causes models to generate repetitive or overly cautious responses even when variety would be more valuable. Researchers have typically blamed algorithmic limitations in training methods, but a new study reveals a deeper root cause that has been overlooked.

A Human Bias in the Data#

The core culprit is typicality bias: a well-documented cognitive pattern where annotators systematically favor familiar, conventional text when rating model outputs during preference data collection. Rather than stemming from the training algorithms themselves, this bias is embedded in the preference datasets used to fine-tune models. The researchers formalize this bias theoretically, confirm its presence empirically across real preference datasets, and demonstrate that it plays a central role in driving mode collapse.

Verbalized Sampling as a Training-Free Fix#

To sidestep this problem without retraining, the team introduces Verbalized Sampling (VS)—a simple inference-time prompting strategy. Instead of asking a model to generate a single response, VS instructs it to generate multiple responses simultaneously and to assign a probability to each one. For example, rather than "Write a joke about coffee," the prompt becomes "Generate 5 jokes about coffee and their corresponding probabilities."

By explicitly asking models to produce and weigh multiple options, VS taps into the latent diversity already encoded in pre-trained weights, essentially circumventing the alignment-induced collapse.

Broad Gains Across Creative Tasks#

Extensive testing shows that VS delivers significant improvements across diverse applications:

• Creative writing: Poetry, stories, and jokes see 1.6–2.1× boosts in diversity
• Dialogue simulation: More varied and natural conversations
• Open-ended question-answering: Richer response variety