Large language models struggle to preserve urgency in crisis translation across languages

Researchers have identified critical vulnerabilities in using large language models for multilingual crisis communication, raising serious concerns about deploying these tools for emergency triage and disaster response.

The Problem#

While LLMs are increasingly proposed for crisis preparedness and response, especially for breaking language barriers, their actual effectiveness in high-stakes emergency contexts has rarely been rigorously tested. A new study from arXiv examines whether state-of-the-art systems can handle the demands of crisis translation—particularly the need to preserve urgency, the essential signal that tells responders and patients which cases demand immediate attention.

What the Research Found#

Using multilingual crisis datasets across 30 languages (including TICO-19) plus a newly created urgency-annotated dataset in 29 languages covering 100 scenarios, the researchers discovered that both dedicated translation models and LLMs show substantial quality loss, especially for low-resource languages.

The most striking finding emerged from human evaluation: while human raters consistently judged urgency levels the same way regardless of which language they read, LLMs produced wildly divergent urgency classifications across languages for identical crisis scenarios. The variance was so extreme that the same situation would be rated "Not Urgent" in one language but "Critical" in another—spanning the entire spectrum of triage priority.

Why This Matters#

The inconsistency poses genuine safety risks. In a real crisis, a patient whose condition is correctly flagged as high-priority in English might be deprioritized as "non-urgent" if the same information is processed in another language through an LLM. This could directly compromise care decisions in multilingual disaster zones or cross-border emergency responses.

The Path Forward#

The findings underscore an urgent need for multilingual, human-centered evaluation frameworks before deploying general-purpose language technologies in life-or-death contexts. Organizations considering LLM-based crisis response systems should first validate performance across all relevant languages and urgency levels, not assume that strong English-language performance translates to other tongues.