AI
When code comments work: study reveals comments only help AI coding when they encode correct solutions
Researchers find that comments boost code generation performance only when they describe working solutions—not because they're comments themselves.
Large language models frequently write comments as they generate code, incorporating those annotations into the context that guides subsequent output. Yet the mechanics of why and when comments actually improve code-generation results have remained opaque. A new study from arXiv examines this question through both observational work and targeted experiments, revealing that comments help only when they encode correct solutions.
What the research found#
Researchers tested whether simple metrics—like how often comments appeared or their general purpose—could predict better code generation outcomes. Using the LiveCodeBench evaluation suite, they discovered that neither comment frequency nor broad intent reliably correlated with successful output. The real driver turned out to be more subtle.
The team then ran controlled tests by having weaker models generate code after receiving comment blocks authored by stronger models. By separating the comment surface form from the actual solution content, they could isolate what actually mattered. Comments sourced from solutions that passed tests boosted pass@1 scores by an average of 17.2%. By contrast, comments describing failed solutions provided no reliable improvement, and comments written for unrelated problems actually harmed performance, reducing pass@1 by 20.8%.
Why this matters#
The findings challenge a common assumption that comments—purely as expository text—help models reason through coding problems. Instead, the evidence points to a different mechanism: comments appear to succeed because they embed correct algorithmic or structural insights that direct a model's solution path. When a comment describes a failing approach or addresses a different problem entirely, it misleads rather than guides.
Across various recipient models and prompt variants tested, most showed no significant ability to recover the external-comment benefit, with the best case recouping only 24% of the gain. This suggests that models don't deeply internalize comment patterns across different contexts—they leverage the solution content they contain.
Implications for code generation#
The work underscores that improving AI code generation is not simply a matter of adding more documentation. Rather, the quality and correctness of context matters profoundly, and surface features like comment presence alone cannot substitute for guidance rooted in working solutions.
