AI & Education
Machine Learning Beats Deep Models at Predicting Programming Success
A new study compares six different approaches to forecasting whether students' coding attempts will be accepted, finding simpler models outperform complex neural architectures.
Predicting whether a student's programming submission will be accepted is crucial for building smarter coding tutors and triggering timely help. Yet the messy reality of how learners code—making many small fixes, hitting different types of errors, and progressing unevenly—makes the task surprisingly tricky.
A new analysis from arXiv tested six machine learning and deep learning models on three related prediction challenges: whether the current attempt passes, whether the next one will, and whether acceptance comes within three attempts.
The Models and Tasks#
The researchers tested linear SVM and XGBoost (traditional ML), BiGRU and BiLSTM (deep learning), and two large pretrained language models—GraphCodeBERT and CodeT5+—designed specifically to understand code structure.
Three prediction scenarios were examined, varying how much of a student's prior attempt history the model could see: just the current submission, pairs of attempts, or longer sequences.
Surprising Winner: Simplicity#
The results defied expectations about what works best in machine learning. Traditional ML models, especially XGBoost, proved most reliable overall and remained stable across different prediction tasks. In the hardest scenario (predicting three-attempt success using only current data), XGBoost achieved a 99.09% area under the precision-recall curve and an MCC of 0.63.
The pretrained code models showed strong performance on the three-attempt task (GraphCodeBERT and CodeT5+ hitting F1 scores of 80% and 73.68% respectively), but their gains were inconsistent and task-dependent.
Counterintuitive Insight#
One of the study's most striking findings: using historical attempt data did not consistently improve predictions. Pairwise and multi-step histories provided no reliable edge over looking at the current attempt alone. This suggests that the current state of a submission may carry most of the predictive signal, and earlier revisions add noise rather than signal.
