Study Evaluates How Well Large Language Models Can Pinpoint Software Bugs

Large language models have shown impressive capabilities in code generation and repair tasks, but a new empirical study reveals significant gaps in their ability to accurately identify where bugs exist—a foundational skill for automated debugging.

Researchers conducted a systematic evaluation of four major LLMs across different testing scenarios to understand their statement-level fault localization performance—the ability to pinpoint the exact line or section of code where a defect occurs. The study examined both open-weight models, such as Qwen2.5-Coder-32B-Instruct and DeepSeek-V3, alongside closed-source systems like GPT-4o mini and Gemini-2.5-Flash.

Testing Methodology and Scope#

The evaluation was conducted on two benchmark datasets: HumanEval-Java and Defects4J, using multiple prompting approaches to assess performance. Researchers tested Zero-shot prompting (no examples), Few-shot prompting (with code examples), and Chain-of-Thought reasoning (step-by-step problem-solving) to understand how different input strategies affect outcomes. Performance was measured through three complementary lenses: exact matches, partial matches, and consistency across multiple runs.

The team also compared LLM results against established non-neural baselines like PMD and LineDef under identical source-code-only conditions, providing a reality check against conventional static analysis tools.

Key Findings#

Several patterns emerged from the analysis. Bug report context—additional information about the expected behavior—notably improved fault localization performance when working with real-world defects in Defects4J. However, Few-shot prompting produced inconsistent gains across models and datasets, suggesting this technique is not universally beneficial. Chain-of-Thought reasoning showed mixed results depending on which model and dataset was tested, indicating the approach does not reliably enhance localization accuracy.

Practical Implications#

Beyond accuracy metrics, the study examined real-world constraints. Researchers measured end-to-end response time and API token costs across models, revealing critical efficiency and financial trade-offs. These findings highlight that selecting an LLM for fault localization requires balancing localization accuracy against deployment cost and latency.

The research demonstrates that while LLMs excel at certain code tasks, fault localization remains an area where these models have notable limitations and where conventional tools continue to play a valuable role in software engineering workflows.