LLMs Learn to Reason Better by Training on Their Own Uncertainty

A new post-training method helps large language models generate more reliable answers by tapping into their own internal confidence signals rather than requiring external human feedback.

The technique, called Reinforcement Learning from Self-Feedback (RLSF), addresses a core challenge in LLM deployment: models often sound confident while being wrong. Even when a model possesses the reasoning capability to solve a problem, its probability estimates may be poorly calibrated, limiting trustworthiness on demanding analytical tasks.

How the approach works#

The method leverages an insight from recent research: chains of thought reasoning are already latent in pre-trained language models. Rather than treating this as fixed, RLSF extracts these reasoning paths and measures them strategically.

The process unfolds in three steps:

• Generate multiple chain-of-thought decoding pathways from a frozen model
• Compute the confidence score for each final answer
• Rank the resulting traces to create synthetic preference pairs

These preference signals then feed into standard preference optimisation to fine-tune the model—all without requiring human annotations, ground-truth answers, or hand-crafted reward functions.

Results and implications#

Testing on arithmetic reasoning and multiple-choice question answering tasks demonstrated that RLSF achieves two simultaneous improvements: it restores well-calibrated probability estimates and strengthens step-by-step reasoning quality. The model learns not just to arrive at correct answers, but to express appropriate confidence in its outputs.

By converting a model's uncertainty into actionable feedback, RLSF establishes reinforcement learning on intrinsic model behavior as a principled, data-efficient stage of the post-training pipeline. The approach sidesteps the need for manual prompt engineering or external supervision, relying instead on capabilities the model already possesses.