AI
New Training Method Teaches AI Models to Explore Multiple Plausible Next Words
Researchers introduce BOW, a reinforcement learning framework that trains language models to reason about the full spectrum of contextually valid continuations rather than committing to a single path.
Language models are typically trained on next-word prediction tasks where they learn from a single correct continuation, even when many different words could plausibly follow a given context. Researchers have now proposed a new approach that explicitly addresses this limitation.
The Problem with Standard Training#
Conventional next-word prediction forces models to pick one observed continuation during training. Yet in reality, most contexts have multiple valid continuations that make sense. Recent reinforcement learning methods tried to tackle this by asking models to generate reasoning that justifies their prediction, but this approach can inadvertently entrench whatever preference the model already had.
How BOW Works#
The new framework, called BOW (reasoning over plausible next words), takes a different path. Rather than rewarding models for justifying a single continuation, it trains them to generate balanced, impartial descriptions of the entire space of plausible next words. The method works by having the model produce a reasoning trajectory that the scorer evaluates—but crucially, the scoring mechanism only sees the generated trajectory, not the original context. This architectural bottleneck forces the trajectory itself to contain all necessary information about why multiple continuations are valid. A variant called BOW-Reg adds a regularization term that actively discourages the model from prematurely collapsing its reasoning onto just one option.
Performance and Findings#
Testing across ten general reasoning benchmarks revealed that BOW remains competitive with standard instruction-tuned models and often exceeds trained baselines. The regularized version achieved the best performance on correctness metrics while also producing the least single-word collapse on ambiguous tasks. Human evaluators confirmed that BOW-Reg elicits richer, broader reasoning about plausible continuations, and the trajectories remain predictive of actual next words in the data.
