Code Summaries Improve LLM Bug Repair, but Modest Gains Persist

Researchers at arXiv have examined whether asking language models to write natural-language summaries of code before attempting repairs could help fix bugs more effectively. The findings suggest the technique works modestly but falls short as a complete solution.

The Repair Strategy#

The team tested a straightforward approach: have an LLM first summarize a piece of buggy code in natural language, then use that summary to guide the generation of a repair. This "summary-mediated repair" works as a prompt-only method—no additional tools or external data needed—making it simple to implement and deploy across different models.

The researchers evaluated the approach across eight different large language models using two distinct test sets: real bugs from HumanEvalPack and errors that arose when the models themselves failed to generate correct MBPP solutions.

Results and Performance#

Diagnostic, error-aware summaries—those that specifically called out problematic aspects of the code—performed best among the variants tested. This approach successfully repaired up to 65% of previously unseen bugs and improved performance by an average of 5% compared to direct repair without summaries.

However, the gains prove modest and vary significantly depending on the model. The technique does little to address the fundamental challenge of asking an LLM to fix its own code through prompting alone, leaving room for more robust repair methods.

Key Takeaway#

Code summaries emerge as a lightweight, practical diagnostic layer for LLM-based program repair but cannot stand alone as a complete solution. The modest improvements suggest that while natural-language context helps, addressing deep implementation-level bugs in generated code will likely require additional strategies beyond summary-mediated prompting.