Large Language Models Can Catch Hidden Refactoring Bugs That IDE Tools Miss

Refactoring tools built into popular code editors—like IntelliJ-IDEA, Eclipse, and NetBeans—sometimes introduce bugs that break code functionality or fail to compile, even when developers expect the transformation to preserve behavior. A new research paper examines whether large language models can step in as a verification layer to catch these mistakes before they cause problems.

The Problem with Automated Refactoring#

Developers rely on IDE refactoring tools to cleanly restructure code without accidentally changing what it does. However, traditional verification approaches—which use handcrafted rules, static analysis, and runtime testing—often struggle to spot subtle correctness issues. These gaps create a trust problem: developers become hesitant to use automated tools, slowing down code quality improvements.

Using Foundation Models as Validators#

Researchers tested whether zero-shot prompting of large language models (LLMs) could detect refactoring errors without any task-specific training. They evaluated the approach against a dataset of 226 real refactoring bugs collected over a decade from three major Java IDEs, covering 47 different refactoring types.

The results were encouraging:

• GPT-5.4 achieved 93.8% accuracy in one-shot evaluation
• GPT-OSS-20B reached 80.5% accuracy
• Gemma-4-31B performed best among open-weight models
• Gemini-3.1-Pro-Preview delivered the strongest overall performance

Beyond Simple Detection#

Foundation models offer benefits that go further than traditional checking systems. They can flag suspicious transformations without needing explicit rules for each refactoring type, provide brief explanations to help developers review changes, and operate as lightweight triage tools integrated into development workflows. The models remain consistent when tested against semantics-preserving variations—though the researchers note this should be viewed as robustness evidence rather than proof against memorization.

Next Steps#

The findings suggest LLMs could complement existing refactoring safeguards, catching edge cases and tricky behaviors that conventional static and dynamic analysis might overlook. Rather than replacing traditional checks, foundation models appear poised to work alongside them as an additional verification layer that makes developers more confident in automated code transformations.