Why large language models struggle with Bengali: tokenizer design, not script complexity

The Bengali LLM Gap#

Despite Bengali being a first language for over 230 million people, large language models consistently underperform on Bengali compared to English. A new study evaluating ten open-source LLMs from four different model families found that English-language tasks achieve a macro LLM-judge score of 0.79, while equivalent Bengali tasks score only 0.63—a substantial gap that has gone largely unmeasured in the broader LLM benchmarking landscape.

But the apparent gap is partly deceptive. The researchers discovered that evaluation methodology itself inflates the difference, and that the core challenge lies not in how models process language, but in how their tokenizers break down the script.

Two Sources of the Gap: Measurement and Tokenization#

The measurement problem emerges from reliance on exact-match scoring, which demands that answers conform to a specific format regardless of whether the model actually understood the question. Format failures happen unequally across English and Bengali, distorting the apparent performance gap by two to five times for some models—and even flipping the sign for one reasoning-focused variant.

On the technical side, the real bottleneck is script fragmentation. Bengali uses an alphasyllabary where written units (grapheme clusters) span multiple Unicode code points. Subword tokenizers designed for English carve these units into fragments smaller than a single character. The cost varies dramatically: Bengali requires roughly five times more tokens per word than English. Across the ten models tested, this token cost ranges from 4.0 tokens per word (using the Tekken tokenizer) to 7.9 (under Llama 3), and across three of four tokenizers, the average Bengali token spans fewer bytes than a single Unicode code point.

What the Data Reveals#

The researchers translated eight English benchmarks into Bengali using a single consistent pipeline, then evaluated the models on paired inputs in both languages. The key finding: the vocabulary design, not the script itself, sets the cost. All ten models share only four distinct vocabularies, and how each tokenizer handles Bengali varies twofold. Token count and sequence length show only weak correlation with final scores (r = -0.23), suggesting that while tokenization imposes a real processing cost, it is not the direct driver of the performance gap.

The researchers have released the datasets, evaluation pipeline, and code to allow the community to investigate further and develop better multilingual tokenization approaches.