Four Rules for Understanding How Language Models Actually Behave

Language models don't work the way many people assume. According to researcher Naomi Saphra, understanding how LLMs actually function requires accepting some counterintuitive truths about their capabilities and limitations.

Language Models Prefer to Memorize#

The first principle: LMs will memorize whenever they can, because it's easier than learning concepts. If training data is repetitive—like reusing the same test questions year after year—a model will simply memorize the answers rather than grasp the underlying principles. This means a model getting something right doesn't prove it understands the concept. To force genuine learning, training data must be diverse enough that memorization becomes less efficient than actual comprehension.

Testing reveals this matters for what counts as "unseen" data. A model might generate exact Bible verses verbatim through pure recall, yet also complete counting sequences (28, 29, 30...) through learned pattern recognition. The key difference: diversity of examples. When a model encounters concepts like "leopard-print" across many contexts—not just on leopards, but on chairs and other objects—it learns to generalize the pattern rather than memorize every instance.

They Act Like Populations, Not Individuals#

The second rule flips a common misconception: language models don't behave like single people; they behave like populations. This becomes obvious when using temperature settings. At temperature 1, an LLM samples from its learned distribution of training data. At lower temperatures, you're essentially taking a "vote" across that population. Chess models trained on human players perfectly illustrate this—at temperature 1, they make the same mistakes individual human players make, but with temperature near 0, the ensemble outperforms any single human.

This also explains why diverse, less-expert populations can outperform homogeneous expert ones. Training on 100 experts with diverse specialties—each knowing about different domains—actually produces better results than training on 10 super-experts with overlapping knowledge, even when total knowledge coverage is identical. The diversity forces the model to learn distinct concepts rather than shared misconceptions.

They Only Learn What's Written Down#

The third principle: language models cannot learn what isn't in their training text. This has serious consequences. Companies hoping to interview LLMs instead of surveying actual people miss crucial ground truth—models describe groups as describe them, not as those groups would describe themselves. They lack exposure to human uncertainty and disagreement in training data, which increases hallucination risk. Models trained only on confident responses will confidently generate false information (like the documented cases of AI-generated fake court cases submitted to real courts).