New Dataset Reveals How AI Models Diagnose Student Code Errors—With a Twist

Researchers have created a new benchmark for evaluating how well large language models can identify bugs in student-written code. The resulting insights reveal a counterintuitive finding: depending on how you measure success, different AI approaches can appear dramatically more or less capable than they really are.

The PyMETA Dataset#

The study introduces PyMETA, a collection of nearly 50,000 Python student submissions spanning 155 programming problems. Each submission is labeled according to the first error that would surface when the code runs, or marked as error-free if it passes all automated tests. A smaller subset of 97 submissions received additional expert review, where specialists identified all errors present in the code through cycles of testing and repair.

The labeling system operates at three levels of granularity, with the most detailed containing 14 distinct error categories: standard Python exceptions, Logic Error, No Error, and an "Other Errors" catch-all.

When Recent Models Shine—and Stumble#

When judged against simple first-execution errors, the performance gap between approaches becomes stark. Modern prompted LLMs achieved macro F1 scores between 87.5% and 93.8%—surpassing the strongest finetuned baseline at 80.6%. This result flips the expected narrative compared to how older prompted models performed.

Yet the picture darkens when researchers examined the harder expert-labeled subset. Exact matching across all identified errors dropped to just 43–49%, although individual sample F1 held steady around 80%. The discrepancy highlights a crucial distinction: the models often spot some errors correctly but miss multiple problems that humans identify in the same code.

Format Matters More Than Expected#

The choice of prompt structure proved surprisingly influential. When asked to identify a single error, none of the four recent models flagged Logic Errors in 45 audited submissions where experts found none. Yet when allowed to return multiple possible errors, these same models included Logic Error between 47% and 58% of the time—usually after identifying a more obvious syntax or runtime fault.

Why This Matters#

The results underscore how evaluation methodology shapes conclusions about model capability. The same AI system can appear competent or flawed depending on whether researchers measure it against only the first detectable error, all latent problems, or some middle ground. This finding has implications for how educators and developers assess whether language models are truly ready to assist with code review and debugging.