LLMs Disappoint as Program Repair Tools in First Controlled Test

Researchers investigating whether large language models could serve as mutation operators in automated program repair have found sobering results: LLMs substantially underperformed hand-crafted approaches in a controlled benchmark.

The Experiment#

Scientists extended an established repair framework called FlowRepair by swapping some of its traditional mutation operators with LLM-generated fixes. They then tested this hybrid system against a collection of 19 real-world faulty Simulink-Stateflow models—software used to design cyber-physical systems—running both approaches under identical computational budgets for a fair comparison.

The results were stark. The LLM-augmented approach produced working patches for only 4 out of 19 models, compared to 16 for the original system. When counting "plausible" candidate patches (repairs that compile but may not fully solve the problem), the LLM variant generated them for 4-6 models versus 18 with the traditional operator.

Why LLMs Struggled#

The research identified three core obstacles. First, LLMs were unable to handle the precise, syntactically correct edits that Simulink requires—making small symbolic changes without introducing errors proved difficult. Second, generative models operate without behavioral feedback during the repair search process, limiting their ability to learn what works. Third, the mutations LLMs produced were often noisy and incoherent, cluttering the search space rather than narrowing it toward a solution.

A Path Forward#

Rather than dismissing LLMs for program repair entirely, the authors argue that the mismatch stems from how the tools were integrated. Simply dropping a generative model into a structured search framework was insufficient. They propose that hybrid strategies combining the precision of symbolic mutation operators with generative guidance may better leverage what LLMs can offer—a direction for future work in this field.