AI
New Dataset Tests How Well AI Handles Real Computer Science Competitions
Researchers evaluate large language models on bilingual, multimodal problems from competitive programming, revealing where AI succeeds and where it struggles.
A team of researchers has created a new benchmark to measure how well large language models perform on authentic computer science competition problems, uncovering both the promise and pitfalls of using AI in educational settings.
The Dataset and Challenge#
The researchers developed a bilingual dataset (English and Romanian) containing multiple-choice questions drawn from actual high-level competitive programming contests. What makes this benchmark distinctive is its intentional design: some problems are more naturally solved through pen-and-paper reasoning, while others benefit from writing and testing code. This mix reflects real-world problem-solving strategies in computer science.
How AI Models Fared#
The team systematically tested current state-of-the-art large language models against the dataset to assess their capabilities in theoretical programming tasks. A key finding was that language choice matters—the same models performed differently when evaluated in English versus Romanian, highlighting how language can affect AI reasoning on technical content.
The results mapped both the strengths of modern LLMs in handling complex CS problems and their concrete limitations, particularly in competitive contexts where precision and depth of understanding are critical.
Educational and Ethical Implications#
Beyond raw performance metrics, the research examined important questions about fairness and integrity. As AI tools become more prevalent in student learning and assessment, the authors addressed concerns about academic integrity and whether traditional exams remain equitable when students can access powerful AI assistants. These findings aim to guide future decisions about LLM use in educational policy and practice.
Making Tools Available#
To support continued research and learning, the team plans to release the full dataset publicly in both languages. They are also launching an interactive educational application for Romanian students, allowing learners to use the problems for self-assessment and skill-building in a practice-oriented format.
