How Language Models Learn from Examples: Information Filtering as the Core Mechanism

Researchers at arXiv have uncovered a key mechanism explaining how large language models leverage few-shot examples to solve new tasks—a phenomenon known as in-context learning (ICL). The work shifts focus from how models acquire knowledge to how they filter out noise to focus on what matters.

The Problem with Zero-Shot Predictions#

When models encounter a task without any examples, they generate nearly random outputs. The researchers demonstrate that without guidance, language models encode queries into broad representations containing signals for all possible tasks simultaneously. This unfocused state leads to poor accuracy because the model has no clear signal about which task to prioritize.

How Few-Shot Examples Trigger Selective Filtering#

The breakthrough finding is that providing just a few labeled examples catalyzes a specific computational process: the model systematically removes irrelevant task information from its internal representations. Using carefully designed metrics, the team observed that few-shot ICL operates like a low-rank filter—effectively pruning away the tangled, multi-task signals embedded in the model's hidden states and redirecting attention toward the intended task.

This selective information removal transforms the broad, unfocused representations into task-specific ones, substantially improving accuracy on the demonstrated examples.

Identifying the Mechanisms at Work#

The researchers pinpointed specific neural components called Denoising Heads—particular attention heads responsible for performing this filtering operation. By ablating these components during inference, they confirmed their critical role: blocking the denoising process caused in-context learning accuracy to collapse, particularly when the correct label was missing from the few-shot set.

This finding validates the hypothesis that information removal, not information addition, is central to how few-shot learning works in large language models.