Study reveals how transformers learn logical reasoning through two distinct pathways

Researchers studying how language models perform logical reasoning have identified two fundamentally different computational strategies that transformers use to solve multi-step deductive problems. The findings shed light on the low-level mechanisms underlying what appears to be abstract reasoning in neural networks.

Two modes of deductive reasoning#

The distinction centers on timing: in horizontal reasoning, models generate intermediate steps one after another (autoregressively), explicitly spelling out each logical move. In vertical reasoning, by contrast, inference occurs implicitly across the transformer's layers before any output is produced. Both pathways can solve the same logical task, but they rely on different internal structures.

To study this, researchers trained small transformer models from scratch on synthetic tasks designed to test deductive logic—specifically, following chains of symbolic implications and navigating trees from root to leaf. Through mechanistic interpretability (a technique that reverse-engineers neural network decision-making), they could observe exactly how the model's components implement reasoning.

Chain-of-Thought as curriculum, not just guidance#

A key finding involves Chain-of-Thought prompting, where models are trained to show their work step-by-step. In the horizontal setting (where steps are explicit), Chain-of-Thought supervision leads models to learn genuine rule-based inference rather than merely exploiting statistical shortcuts in the training data. The resulting circuits are interpretable, built from induction-head-like mechanisms that handle rule completion, rule chaining, and final decision-making.

For vertical reasoning, however, Chain-of-Thought appears to function more like curriculum learning—gradually exposing the model to increasingly complex reasoning patterns—rather than as explicit step-by-step guidance. Without it, models tend to memorize or latch onto dataset biases instead of learning generalizable logic.

Decoding the transformer's internal representations#

The researchers also introduced a technical innovation: a truncated pseudoinverse method that can decode what information flows through the transformer's query, key, and value vectors—the internal components responsible for attention. This tool may help future work further unpack how transformers encode and manipulate logical information.