Transformers Learn Analogical Reasoning Through Intermediate Copy Tasks

Analogical reasoning—the capacity to recognize patterns and apply learned solutions to unfamiliar problems—is central to human intelligence. Teaching artificial systems to replicate this cognitive skill at scale remains a significant challenge.

Researchers have now demonstrated that transformers can successfully learn analogical reasoning on letter-string analogy tasks when trained with a carefully designed curriculum. The breakthrough comes from an unexpected source: including copy tasks in the training data acts as a crucial intermediate step, guiding models to focus on the most relevant information within each problem.

How the Training Works#

The team employed Meta-Learning for Compositionality (MLC) to train 3-layer encoder-decoder transformer models. Their key finding was that models struggled with letter-string analogies until they were exposed to simpler copying tasks during training. This intermediate step helped the networks learn to attend selectively to problem elements, effectively unlocking their ability to solve more complex analogies.

The researchers also discovered that training on diverse, heterogeneous datasets improved the models' ability to generalize to entirely new alphabets—a valuable indicator that the learned reasoning could transfer across novel contexts.

Understanding the Model's Logic#

To peek inside the "black box," the team reverse-engineered the algorithm the transformer was using to solve particular analogy tasks. Using interpretability techniques, they verified that specific attention heads directly influence the model's outputs and can be used to steer it between different analogy types. This suggests the model develops interpretable, algorithm-like reasoning strategies rather than learning task-specific tricks.

Scope and Limitations#

On their letter-string analogy benchmarks, the best models matched the performance of state-of-the-art systems. However, the approach showed mixed results when tested on more ambitious generalization goals: the models could compose learned transformations to some degree, but failed to handle entirely novel transformations unseen during training.

Implications#

The findings highlight how the path to learning—not just the final training set—shapes what neural networks can accomplish. By drawing parallels to human learning (where intermediate skills often pave the way for advanced reasoning), the work offers lessons for scaling analogical reasoning to larger models and deeper reasoning tasks.