New QA-Based Method Evaluates Machine Translation Without Knowing the Target Language

The challenge#

When a monolingual English speaker receives an automatically translated document in French or another language, how can they verify whether the translation is reliable? Current machine translation error detection and quality estimation tools fail to address this practical, real-world need. Users without knowledge of the target language have limited ways to judge whether a machine-generated output is trustworthy enough to share or act upon.

AskQE: A new framework#

Researchers have developed AskQE, a system that pairs question generation with question answering to flag significant translation errors and offer actionable guidance. The approach allows users to evaluate whether to accept or reject machine-translated outputs even if they cannot read the target language themselves.

The team trained and refined their method using ContraTICO, a curated dataset of synthetic, deliberately varied translation errors in the COVID-19 domain. The optimized version leverages LLaMA-3 70B and uses entailed facts to steer question generation toward the most informative queries.

Stronger performance#

When tested against the BioMQM dataset—a collection of real-world translation mistakes annotated by human judges—AskQE showed superior results compared to existing quality estimation metrics. The system achieved higher Kendall's Tau correlation and decision accuracy in matching human ratings of translation quality.

Background#

The research was submitted to arXiv on April 15, 2025, and has undergone revisions since then. This work opens a pathway for non-experts to use language models to validate machine translations in specialized domains like healthcare and medicine, where accuracy is critical.