New benchmark tests how well AI models generate ML pipelines across languages

A newly released benchmark called ML2B fills a critical gap in evaluating how well large language models can tackle machine learning tasks when those tasks are described in languages other than English. Despite the global expansion of AI systems, most evaluation of LLMs has focused narrowly on English-language prompts.

What ML2B includes#

The benchmark comprises 35 Kaggle competitions spanning tabular data, text analysis, and image tasks. Researchers translated these challenges into 14 different languages, with each translation completed by native speakers who also have machine learning expertise. This careful approach produced 490 unique task-language pairs for evaluation.

To maintain rigorous evaluation standards, the benchmark incorporates 10 private competitions—competitions whose solutions are not publicly available—and uses isolated evaluation infrastructure that restricts runtime access to only the essential machine learning resources needed. This design prevents models from gaming results through shortcuts.

How models performed#

Testing with frontier LLMs including GPT-4.1-mini, GPT-OSS-120b, and Gemini-2.5-Flash revealed surprising findings. Performance degradation across different languages is not uniform or predictable. Rather than following traditional assumptions about which languages have more or fewer training resources available, the models showed task-dependent patterns—sometimes performing better in non-English languages and sometimes performing substantially worse, depending on the specific competition characteristics.

Why this matters#

The results challenge conventional wisdom about how multilingual language models work and demonstrate why language-specific evaluation is essential for understanding true model capabilities. The researchers have open-sourced the complete benchmark, baseline algorithms (including AutoGluon), and evaluation infrastructure to support future research in this area.