New framework trains AI to explore multiple reasoning paths before choosing the best solution

Researchers have introduced a novel approach to enhance how large language models tackle complex reasoning tasks—by first asking them to outline multiple potential solution strategies before committing to one. The method, called Hint-Guided Diversified Policy Optimization (HDPO), mirrors the way humans often work through difficult problems by considering different angles.

The Challenge with Current Methods#

Today's most advanced training approaches for reasoning tasks, such as reinforcement learning with verifiable rewards, typically focus only on whether a final answer is correct or incorrect. This binary feedback signal doesn't guide models toward exploring alternative approaches or help them recognize when multiple valid solutions exist. The methods also don't explicitly reward a model for understanding why one solution path might be more trustworthy than another—a critical aspect of human problem-solving that involves evaluating trade-offs and reliability.

How HDPO Works#

The proposed framework operates in two stages. First, a "Cold Start for Structured Reasoning" phase trains the model to think in a more organized way. Second, "Hint-Guided Diversified Reinforcement Learning" encourages the model to follow a "propose-select-think" workflow: it generates multiple candidate solution outlines as hints, then selects the most dependable approach, and finally executes detailed reasoning based on that choice.

This structure gives the model explicit incentives to generate variety and to develop better judgment about which solutions are likely to succeed.

Results and Implications#

Early experiments indicate that HDPO successfully strengthens a model's reasoning performance and produces a wider range of candidate solutions. The approach also appears to improve the model's ability to identify which solutions are genuinely reliable, rather than just correct by chance. This technique could help make AI systems more robust and transparent in domains like mathematics, logic, and scientific reasoning where multiple valid pathways often exist.