AI Research
Transformers Make Fast, Irreversible Decisions Through Task-Specific Commitment Heads
New research reveals that small language models commit to outputs early in processing and rarely correct course—a phenomenon called prolepsis that repeats across diverse tasks.
Researchers have identified a repeating pattern in how small transformers make decisions: they lock in their outputs early and then stick to them, using specialized attention heads to route that commitment through the network. The finding, demonstrated on models like Gemma 2 2B and Llama 3.2 1B, suggests a fundamental architectural constraint in how decoder-only language models operate.
The Prolepsis Pattern#
The team introduced prolepsis to describe this behavior—a three-stage process where a transformer first commits to a decision at an early point, then task-specific attention heads preserve and route that commitment forward, and finally no downstream layer reverses it. This occurs despite the models having theoretically many opportunities to reconsider.
Replicating earlier findings by Lindsey et al. (2025) on open-source models, the researchers asked five key questions about how and where this commitment happens.
Where Planning Remains Hidden#
Planning activity proved remarkably difficult to locate using conventional analysis methods. Six different residual-stream-based approaches failed to surface evidence of where decisions form, yet CLT-based steering successfully revealed the signal. The commitment spike manifests at a single point—the final prompt token—with consistent shape across models tested, despite prior work suggesting the signal might appear at a newline character.
Routing Through Specific Heads#
The missing piece in prior analysis was identifying which attention heads route the committed decision to the model's output. This resolves a long-standing gap in attribution graphs that were unable to trace decision pathways. The routing substrates differ between models and tasks, but the template repeats.
Depth Constraints and Two-Model Hypothesis#
Evidence supports a two-stage division: search operations fit within at most 16 layers, while commitment appears irreversible beyond that threshold. The same motif recurs in factual recall tasks at different network depths, though with zero overlap between heads active during planning and those handling top-10 factual information.