Machine translation evaluation gets smarter by pinpointing errors instead of just scoring

Error Span Detection (ESD) aims to improve how machines automatically evaluate translations by not only rating quality but also identifying where problems occur and measuring their severity. Unlike simpler scoring approaches, ESD pinpoints the exact problematic text and classifies how serious each issue is.

The decoding problem#

Most generative ESD systems use Maximum a Posteriori (MAP) decoding, which picks the highest-probability output from a language model. This strategy assumes the model's confidence scores reliably match how well outputs align with human judgment. In practice, however, models often assign higher likelihood to incorrect answers than to what human annotators actually chose—a systematic mismatch that limits accuracy.

A probabilistic alternative#

Researchers addressed this gap by switching to Minimum Bayes Risk (MBR) decoding, a technique that selects outputs based on their expected similarity to reference annotations rather than raw probability. By measuring similarity at the sentence or span level, MBR decoding steers the model away from overconfident but incorrect hypotheses toward options that genuinely resemble human feedback.

Validation and efficiency gains#

Testing on the WMT24 Metrics Shared Task revealed that MBR substantially boosted span-level performance and met or exceeded MAP's results at system and sentence levels. Since MBR involves sampling and scoring multiple candidates—a computationally expensive process—the team distilled the learned preference rankings into a simpler model using greedy search. This condensed approach eliminates runtime latency without sacrificing performance gains, making the technique practical for production deployment.

The work suggests that aligning model decoding strategies with annotation structure, rather than raw confidence alone, offers a clearer path to reliable automatic translation evaluation.