LLMs Learn to Sample From Hard-to-Query Distributions Using Reinforcement Learning

Autoregressive language models store learned patterns from training data as conditional probability distributions over the next token. This architecture works well for straightforward generation tasks but creates a fundamental limitation: extracting knowledge for more complex inference problems—such as filling in missing text, completing partial sequences under constraints, or generating explanations—requires sampling from probability distributions that standard models cannot efficiently compute.

The Core Challenge#

Most large language models operate through a unidirectional, step-by-step generation process. While this is efficient, it constrains what kinds of inference the model can perform. Many practical applications need the model to generate outputs that satisfy specific conditions or constraints, essentially drawing from a posterior distribution rather than just predicting what comes next. These posterior distributions are generally computationally intractable for standard LLMs to sample from directly.

A Solution via Amortized Inference#

Researchers addressed this by treating the problem as one of amortized Bayesian inference. The approach involves fine-tuning LLMs using reinforcement learning algorithms specifically designed to match target probability distributions—particularly generative flow networks (GFlowNets). Rather than maximizing likelihood for single correct answers or optimizing for reward like typical RL methods, this technique trains models to generate diverse samples that collectively represent the full posterior distribution.

Applications to Reasoning and Tool Use#

An especially promising direction interprets chain-of-thought reasoning—where models break down problems into intermediate steps—as a latent variable modeling challenge. The method demonstrates that models fine-tuned this way can adapt more data-efficiently to tasks requiring multi-step explanations and external tool integration, suggesting the approach offers practical benefits beyond pure inference theory.

This distribution-matching perspective on fine-tuning presents an alternative to both standard maximum-likelihood training and conventional reward-optimization approaches used in recent LLM alignment work.