Reverse Language Models Learn to Reason Backwards, Unlocking New AI Capabilities

A team of researchers has developed LEDOM, a purely reverse-direction language model that processes text from right to left rather than the standard left-to-right approach. The open-source models (available in 2B and 7B parameter versions) were trained on 435 billion tokens and reveal unexpected cognitive abilities that emerge when machines learn to predict the past based on future context.

How Reverse Models Work#

Unlike conventional autoregressive language models that generate text sequentially from left to right, LEDOM conditions on later context to reconstruct earlier content. This inverted factorization creates a fundamentally different learning dynamic, allowing the model to develop reasoning patterns that don't naturally arise in forward-only training. The researchers found that the model exhibits abductive inference (reasoning backward from effects to causes), question synthesis, and a natural resolution to the "reversal curse"—a phenomenon where language models fail when exposed to reversed training facts.

Boosting Math Performance with Bidirectional Scoring#

The team explored combining forward and reverse models through a technique called Reverse Reward, which leverages the mathematical duality between forward likelihood and reverse posterior probability. Rather than relying solely on a standard model's confidence, Reverse Reward reranks the model's outputs by checking whether the generated solution can be reliably reconstructed in reverse. The researchers prove that this bidirectional approach penalizes hallucinated reasoning chains—incorrect steps that don't hold up under backward scrutiny.

On challenging mathematics benchmarks, the method achieved gains of up to 6.6% on AIME 2024 and 15% on AMC 2023 across multiple state-of-the-art baseline models, suggesting that reasoning becomes more robust when validated from both temporal directions.

Open-Source Release#

The researchers have released all models, training code, and datasets publicly, making their findings accessible for further research and application development.