New Fact-Checking Framework Tackles LLM Reliability in Urdu

Large language models are generating increasingly realistic text, but their accuracy—especially in languages beyond English—remains a critical concern. A new research effort addresses a particular gap: there were virtually no automated fact-checking tools designed for Urdu, despite the language being spoken by more than 200 million people worldwide.

The Problem: LLM Reliability in Low-Resource Languages#

Existing fact-checking systems have been built almost exclusively for English and other high-resource languages. This leaves speakers of languages like Urdu vulnerable to misinformation generated by LLMs, with no reliable automated way to identify false claims. The challenge is compounded by the scarcity of fact-checking training data and verified evidence sources in Urdu.

A Multi-Part Solution#

Researchers have introduced three complementary components:

• UrduFactBench: A hand-annotated benchmark dataset for claim verification in Urdu, created by native speakers through a multi-stage annotation process
• UrduFactQA: A second benchmark specifically designed to evaluate whether language models produce factually correct answers to questions in Urdu
• UrduFactCheck: A modular framework that performs fact-checking using both monolingual evidence retrieval (searching within Urdu) and translation-based strategies (converting claims to other languages to find supporting evidence)

Key Findings#

The researchers evaluated twelve different LLMs using their new benchmarks. The results showed that models using translation-augmented pipelines—which leverage evidence from multiple languages—consistently outperformed those relying solely on Urdu-language sources. However, open-source models still struggled with Urdu fact-checking, underscoring the need for continued resource development in the language.

Why It Matters#

This work marks the first systematic effort to benchmark and improve LLM factuality in Urdu. By publishing their code and datasets publicly, the researchers have created a foundation for further development. As LLMs become more widely deployed globally, ensuring their reliability across diverse languages is essential for combating misinformation at scale.