LLMs shift their political interpretations based on prompt language, not just content

When large language models analyze politically sensitive documents, their conclusions may depend less on what they're analyzing and more on the language in which the analysis is requested. A new arXiv paper reveals this troubling pattern across multiple frontier models.

The experiment#

Researchers tested ChatGPT 5.2 and Claude Opus 4.5 on a single, contentious Ukrainian civil-society document. The researchers posed semantically equivalent prompts in both Russian and Ukrainian, keeping the source material identical. The results were striking: both models shifted their interpretations along the same axis depending on the prompt language, without any change to the underlying text.

Conflicting frames#

Russian-language prompts led models to produce readings that delegitimized the document's authors, emphasizing criticism or undermining their credibility. Ukrainian-language prompts, by contrast, elicited interpretations that affirmed and legitimized those same authors. The magnitude of the shift varied between models, but neither system demonstrated neutrality—each consistently adopted a stance keyed to the language of the question.

What this means#

Since contested political questions lack a single correct answer (only competing valid interpretations), the researchers interpret this behavior as language-conditioned activation of different ideological frameworks. The models neither hold a consistent position across languages nor surface the plurality of reasonable readings available. Instead, they silently default to whatever interpretive tradition dominates in the prompt's linguistic context.

Implications for AI alignment#

The findings underscore two urgent challenges: evaluation of multilingual systems must test the same content across different languages to catch these hidden shifts, and pluralistic alignment—ensuring models can acknowledge competing legitimate viewpoints—remains an unsolved problem in multilingual contexts.