Direct Preference Optimization Tackles Text Degeneration Beyond Chat Alignment

Hugging Face and Dharma AI researchers have demonstrated a novel application of Direct Preference Optimization (DPO) that goes beyond chat alignment — using the technique to systematically reduce text degeneration in structured document OCR tasks.

The Degeneration Problem#

When vision-language models perform Optical Character Recognition, they sometimes enter repetition loops, where the same token gets sampled repeatedly until the output hits the token limit. Vanilla degeneration rates in tested open-source models ranged from below 1% to above 33%. Supervised fine-tuning (SFT) reduced these rates but rarely to production-acceptable levels, suggesting a structural ceiling.

The issue stems from how SFT works: it optimizes token-by-token likelihood of observed sequences, never explicitly penalizing completion-level failures like repetition loops.

How DPO Addressed It#

The researchers applied DPO after the SFT stage using the same training documents. The critical methodological innovation: they deliberately retained degenerate outputs from the SFT model as rejection examples in preference pairs, rather than filtering them as noise.

"Preference-guided implicit unlikelihood" is how the paper frames it — the model learns toward better outputs and away from a specific failure class. Across five model families tested:

• Average degeneration reduction: 59.4%
• Peak reduction: 87.6% (Nanonets-OCR2–3B: 1.61% to 0.20%)
• Consistency: every single model family showed improvement

Key Findings#

The Qwen2.5-VL-3B result provides the clearest evidence of mechanism: the model's vanilla degeneration rate was 0.60% (too generic to attempt the task seriously), rose to 3.23% after SFT (now capable but exposed to failure modes), then fell to 1.41% after DPO. This shows SFT and DPO target different failure dimensions — capability and degeneration resistance can move independently.

Generalizable Principles?#

The approach requires three structural conditions:

1. Identifiable failure modes — degeneration is categorically distinct (a repetition loop vs. a missed character), enabling coherent rejection examples