Perfect OOD accuracy doesn't guarantee compositional learning, new study shows

A new arXiv paper challenges a widely held assumption in machine learning: that strong out-of-distribution (OOD) performance indicates a model has learned genuinely compositional features capable of systematic generalisation.

The gap between performance and understanding#

Researchers from the study designed experiments where they compared three standard architectures—multilayer perceptrons, convolutional neural networks, and Transformers—alongside newer object-centric, slot-based models. All of these systems initially failed to generalise beyond their training data when tested on tasks with clear compositional structure.

To probe deeper, the team introduced two custom attention-based architectures with interpretable final hidden layers designed to reveal whether models actually develop compositional representations. The key insight: even models that achieved near-perfect OOD accuracy failed to discover the underlying compositional rules.

What the interpretable layer revealed#

By making the hidden layer transparent and inspectable, the researchers uncovered a striking disconnect. High OOD performance, even near-perfect scores, masked the real problem: models were relying on task-specific biases rather than discovering reusable compositional primitives. In other words, they were pattern-matching rather than learning the underlying logic.

Why this matters#

The findings suggest that current evaluation frameworks for compositional generalisation are incomplete. Testing only whether models succeed on out-of-distribution tasks misses whether they've truly grasped the compositional structure. A model can pass the test while learning through shortcuts—sophisticated but ultimately brittle solutions that won't transfer to genuinely novel scenarios.

This work underscores a growing recognition in AI research: performance alone is not a reliable proxy for the kind of understanding needed for robust, generalising systems.