Research Benchmark Tests Large Language Models' Grasp of Urdu Idioms and Figurative Language

Researchers have developed the first comprehensive benchmark for evaluating how well large language models understand and translate Urdu idioms—expressions whose meanings cannot be derived from the literal words alone. The work highlights a significant gap in NLP research, where low-resource languages like Urdu have received far less attention than major European languages, despite the complex cultural knowledge required to handle figurative speech.

The Challenge of Idioms in Translation#

Idioms pose an unusual problem for language models because they rely on shared cultural understanding and historical context rather than grammatical rules. A phrase like "raining cats and dogs" means heavy rain, not literal precipitation of animals. Urdu contains thousands of such expressions rooted in Persian, Arabic, and local traditions, making them particularly difficult for AI systems trained primarily on English-dominant data. The gap widens when considering that Urdu can be written in both the native Perso-Arabic script and in Romanized form—neither of which has been standardized for idiom translation work until now.

Dataset and Evaluation Framework#

The researchers created a dataset of 4,000 manually verified Urdu-to-English idiom sentence pairs, available in both Perso-Arabic and Romanized versions. They tested the performance of frontier LLMs using multiple strategies: direct translation, paraphrasing, detecting which words form the idiom within a sentence, and translating back to Urdu to check consistency. The team also experimented with targeted prompting techniques—such as explicitly asking models to consider cultural context, emphasizing idiomatic meanings, and providing a few examples to guide the model.

Key Findings#

Frontier LLMs substantially outperformed traditional neural machine translation systems at preserving figurative meaning while generating accurate translations. However, the results revealed a critical weakness: models showed relatively consistent performance when processing Urdu in its native script, but stumbled significantly when encountering Romanized versions. The lack of a standardized orthography for Romanized Urdu—meaning writers spell words differently without agreed-upon rules—created substantial obstacles, particularly for tasks requiring the model to identify exactly which words in a sentence constitute the idiom.

Implications for Low-Resource Languages