Knowledge-Augmented Reasoning Pushes Language Models Forward on Abstract Problem-Solving

Large language models optimized for reasoning have shown impressive results on math and science exams, yet their grasp of abstract reasoning and the ability to generalize remain limited. A new study evaluates how these models tackle the Abstraction and Reasoning Corpus (ARC) benchmark, a test explicitly designed to measure human-like abstraction and transfer learning.

Reimagining Problem-Solving as Code Generation#

Researchers framed the ARC challenge as a program synthesis problem and tested nine different solver approaches. The strongest performer was repeated-sampling planning-aided code generation (RSPC), which generated multiple solution attempts and consistently generalized well across different LLMs. This foundation set the stage for further improvements.

Structured Knowledge Improves Reasoning#

The team introduced Knowledge Augmentation for Abstract Reasoning (KAAR), a new solver that takes RSPC further by encoding domain knowledge into an organized hierarchy. Rather than flooding the model with all available information at once, KAAR structures knowledge into three levels based on dependency relationships. The system then feeds priors to the LLM incrementally, with RSPC generating candidate solutions after each augmentation stage.

This staged approach avoids cognitive overload—by progressively introducing relevant knowledge rather than presenting everything together, the model encounters less interference from irrelevant priors and maintains clearer reasoning paths.

Measurable Gains Across Models#

Empirical testing shows KAAR achieved approximately 5% absolute improvement over standard RSPC, with some configurations yielding relative gains as high as 64.52%. Critically, the method preserved strong generalization, performing consistently well across multiple LLM architectures tested.

Remaining Challenges#

Despite these advances, ARC remains stubbornly difficult for reasoning-focused LLMs, signaling that abstract generalization remains a frontier for AI development. The researchers have made their code publicly available to support future work.