New benchmark reveals LLMs still lag elite human coders despite rapid gains

Researchers have created LiveOIBench, a comprehensive evaluation framework designed to rigorously test whether large language models can match the coding prowess of world-class competitive programmers. The study reveals that even the most advanced AI systems currently underperform relative to elite human contestants.

The benchmark and its scale#

LiveOIBench draws from 72 official programming contests spanning 14 Informatics Olympiads held between 2023 and 2025, compiling a dataset of 403 carefully vetted problems. Each problem includes roughly 60 test cases on average, providing thorough coverage of edge cases and solution approaches. The problems come from expert-curated sources and include detailed rubrics for partial credit, mirroring the structure of real olympiad competitions.

The benchmark was designed to overcome limitations in existing coding evaluation frameworks, which often lack sufficient problem complexity, rely on online platforms with access restrictions, and risk contamination through data leakage. LiveOIBench operates as a fully offline system with built-in safeguards against data contamination.

How LLMs performed#

Testing 34 different general-purpose and reasoning-focused language models revealed a striking gap between artificial and human capability. GPT-5 achieved the 81.76th percentile, meaning it solved problems at a level exceeding roughly four out of five participants—but still fell short of the top tier. Among open-source models, GPT-OSS-120B reached only the 60th percentile, indicating significantly weaker performance.

Analyzing the reasoning processes of more advanced models showed that stronger performers tend to focus on careful problem decomposition rather than exhaustive exploration of multiple approaches. This suggests that systematic analysis of what a problem demands proves more valuable than brute-force trial-and-error.

Contamination safeguards and continuous refinement#

The researchers implemented multiple checks to ensure the benchmark remains reliable and resistant to data leakage. They analyzed code similarity between model outputs, examined task familiarity patterns, and tracked temporal relationships between model release dates and contest dates. These analyses found minimal signs that models had been inadvertently trained on benchmark problems.

LiveOIBench also features continuous updates, with new problems added regularly from ongoing competitions. This rolling refresh reduces the risk that any future model iterations could memorize its contents.