How Well Do AI Judges Evaluate Summaries in Other Languages? New Multilingual Study Reveals Significant Gaps

Automatic evaluation is central to developing better text summarization systems. However, the tools used to judge summary quality—whether traditional metrics or AI-powered judges—are often tested only in English. A new study challenges whether these evaluation methods actually work across languages.

The Problem with Monolingual Evaluation#

Text summarization research has grown increasingly reliant on automated evaluation metrics and large language models acting as judges to assess summary quality. This approach speeds up model development by eliminating the need for human evaluation at every step. Yet these evaluation tools themselves need scrutiny: do they accurately capture what humans think makes a good summary, especially beyond English?

Previous research on this question has been sparse and limited mostly to English-language systems. The gap left researchers without clear guidance on which evaluation approaches work reliably in multilingual settings.

A New Multilingual Benchmark#

To address this, researchers created BASSE, a new meta-evaluation dataset specifically designed for multiple languages. The dataset contains human-annotated judgments on 2,040 abstractive summaries. These summaries were produced either by human writers or by five different large language models using four distinct prompting strategies.

Annotators rated each summary across five criteria on a 5-point scale:

• Coherence (logical flow and clarity)
• Consistency (accuracy relative to source material)
• Fluency (natural, readable language)
• Relevance (whether content matters to the source)
• 5W1H (coverage of who, what, when, where, why, how)

How Evaluators Compared#

The team then tested how well automatic metrics and LLM-as-a-Judge systems correlated with the human judgments. The findings revealed a clear hierarchy of performance:

Proprietary judge LLMs (commercial systems from major AI companies) showed the strongest alignment with human evaluators. Criteria-specific automatic metrics came in second, capturing certain dimensions of summary quality reliably. Open-source judge LLMs, by contrast, performed poorly overall, suggesting they were not yet calibrated well enough for reliable evaluation work.