New Benchmark Exposes Fragility in Multilingual Math Reasoning Across Language Models

Researchers have introduced MGSM-Pro, a multilingual dataset designed to uncover hidden vulnerabilities in how well language models perform mathematical reasoning across different languages. The work extends prior English-only research and demonstrates that model robustness varies dramatically depending on the language and how test problems are phrased.

The Problem: Inconsistent Performance#

While large language models have improved significantly at solving math problems, benchmarks for non-English languages have remained limited and often lag behind English in both challenge level and currency. A key insight from earlier research showed that models produce highly variable results when solving the same question with different numerical values—but this discovery had only been tested in English.

MGSM-Pro addresses this gap by creating five different versions of each problem from the original MGSM dataset. These variations alter names, numbers, and contextual details while preserving the underlying mathematical challenge. Across nine languages, the results painted a troubling picture.

Key Findings#

Evaluating multiple models revealed stark disparities in robustness:

• Low-resource languages suffer the most: When numerical values shifted from the original test set, many languages saw dramatic performance drops—a problem less visible in high-resource languages.
• High-resource resilience doesn't transfer: Models that show strong robustness in high-resource language settings don't necessarily maintain that stability in low-resource ones.
• Proprietary models show mixed results: Google's Gemini 2.5 Flash and OpenAI's GPT-4.1 proved vulnerable to changes in numerical values, while Gemini 3.0 Pro demonstrated greater consistency. Open-source models like GPT-OSS 120B and DeepSeek v3 exhibited stronger robustness overall.

Implications for Evaluation#

The research argues that single-shot problem evaluation—using just one version of a math problem—provides an incomplete and potentially misleading picture of model capability. The authors recommend that mathematical reasoning assessments use at least five digit-varying instantiations per problem to produce more reliable and representative performance metrics.

This finding underscores a broader challenge in AI evaluation: surface-level benchmark scores may mask significant fragilities, particularly in underrepresented languages and contexts.