New decoding approach separates reasoning from formatting to boost LLM performance

Large language models struggle when asked to solve complex problems while adhering to strict formatting rules at the same time. This dual demand creates conflicting objectives that undermine the model's reasoning ability. A new approach aims to fix this by cleanly separating the two tasks.

The core insight#

When instructions bundle together task goals with formatting requirements, models must juggle both simultaneously. This creates what researchers call an "entanglement" that degrades performance on downstream tasks like math reasoning or evaluation.

How Deco-G works#

The proposed Deco-G framework introduces a dedicated Format Estimation Module (FEM) that operates independently during decoding. This module handles three key responsibilities:

• Performs probabilistic lookahead to predict whether the model will meet format requirements
• Reweights token probabilities based on those predictions
• Ensures the LLM can concentrate entirely on solving the underlying problem

To make this practical, the researchers developed three supporting innovations: instruction-aware distillation (training the format module efficiently), a flexible trie-building algorithm (for efficient constraint representation), and HMM state pruning (to reduce computational overhead).

Performance and guarantees#

Testing across mathematical reasoning, information extraction, and judge-based evaluation tasks showed consistent improvements over standard prompting and other structured generation baselines. Critically, Deco-G provides guaranteed format compliance—outputs always conform to the required structure, not just probabilistically.

The research was published on arXiv, with code made available to the community.