Software Engineering
Comparing code snippets beats predicting how easy individual code is to understand
A new approach to machine learning for code comprehensibility shows that asking which of two code samples is easier to read yields far better results than trying to measure how readable a single piece of code is.
Researchers are rethinking how machine learning models should evaluate code readability. Instead of trying to score how understandable a single code snippet is in isolation, they propose asking a simpler question: given two pieces of code, which one is easier to comprehend?
The Problem with Absolute Measurement#
Traditional approaches attempt absolute comprehensibility (AC) — assigning a readability score to an individual code sample. However, this method performs poorly because comprehensibility is fundamentally subjective and tied to complex cognitive processes. Existing metrics correlate weakly with how developers actually understand code, leading researchers to explore machine learning models that learn directly from code features and developer characteristics. Yet even these models struggle, with simple baseline approaches matching or beating them in most cases.
A Relative Alternative#
The team proposes relative comprehensibility (RC) as a better framing: given two snippets, predict which is more understandable or whether they're similarly difficult. This shift in perspective requires models to only identify distinguishing features between code samples rather than estimating absolute values — a theoretically easier learning task.
Experimental Results#
Using 150 Java code snippets and over 12,000 human comprehensibility assessments gathered from prior studies, researchers evaluated their approach across classical machine learning models, convolutional neural networks, and two large language models. They tested two prediction modes: snippet-wise (aggregating judgments across multiple evaluators) and developer-wise (individual judgments).
The results strongly favored RC:
- Absolute comprehensibility models rarely outperformed baselines, achieving at most 33.4% improvement
- Relative comprehensibility models beat baselines in 96.8% of configurations, with gains reaching 159.8%
- These RC improvements remained consistent across different model architectures
- Developer-wise results proved more variable, though snippet-wise metrics showed clear gains