AI
New Test Method Reveals LLMs Struggle to Capture Actual Code Behavior
Researchers use mutation analysis to show that AI-generated code summaries often describe what programs should do rather than what they actually do.
Large language models are increasingly trusted to produce code summaries for documentation, testing, and review. But a new study raises concerns about whether these summaries actually capture what the code does—or just what it appears to do.
The Problem with Intent vs. Behavior#
Researchers have identified a critical gap in LLM-generated summaries: they frequently describe what code is intended to do while glossing over the subtle logic changes and edge cases that determine what it actually does. This distinction matters. A confident but inaccurate summary can mislead developers relying on it for understanding, testing, or modifying code.
Testing with Mutations#
To measure this gap, the team developed a mutation-based evaluation methodology. The approach works by:
- Generating an initial summary of a code sample
- Injecting a targeted mutation (a small, intentional change) into the code
- Checking whether the LLM updates its summary to reflect the altered behavior
This method directly tests whether a summary is truly grounded in the code's logic, rather than just matching surface-level patterns.
What the Experiments Found#
The researchers ran 624 mutation-summary evaluations across 62 programs in three phases:
Synthetic Programs: Testing 12 controlled programs with 324 mutations revealed sharp accuracy declines as complexity increased—from 76.5% accuracy for single functions down to 17.3% for multi-threaded systems. Mutation type and location had weaker effects on performance.
Real-World Code: On 150 mutated samples from 50 human-written programs in the Less Basic Python Problems dataset, accuracy dropped to 49.3%, confirming that failure patterns persist with genuine code. Models tended to describe the algorithmic intent rather than the actual mutated behavior.
Model Comparison: GPT-5.2 showed substantial improvement over GPT-4 (85.3% vs. 49.3% accuracy) and improved ability to flag mutations as "bugs." However, both models continued to conflate implementation details with standard algorithmic patterns.
