AI
New Framework Maps How AI Models Generate Fake Code Instead of Refusing Impossible Tasks
Researchers create a taxonomy and benchmark to understand why large language models confidently produce non-functional code rather than admitting when a problem cannot be solved.
Large language models tasked with impossible coding problems frequently produce code that appears syntactically correct but is fundamentally ungrounded in reality—importing packages that don't exist, claiming to implement mathematically impossible algorithms, or fabricating entire ecosystems of nonexistent tools. A new research effort isolates this behavior from ordinary bugs, proposing structured categories to understand and measure when models hallucinate rather than admit defeat.
Understanding Code Hallucination#
The researchers distinguish between standard programming errors (bugs in otherwise functional code) and code hallucination—the generation of plausible-looking but entirely fictitious code. This ungrounded generation happens even when the correct response would be to refuse the task. The key insight is that hallucinated code isn't simply broken; it's anchored to nothing real.
A Three-Dimensional Taxonomy#
The framework organizes hallucinations across three dimensions:
- Groundedness: Whether violations represent absolute, universal impossibilities versus contingent or ecosystem-specific fabrications
- Manifestation level: Whether problems appear at the syntactic (code structure), semantic (logic), or factual (domain knowledge) layers
- Behavior: Ranging from confident false claims through to outputs that degenerate into gibberish
These dimensions are arranged in a severity hierarchy to help researchers assess the gravity of different types of hallucination.
The Benchmark and Its Findings#
The research team built an adversarial evaluation suite containing 270 prompts spanning six programming languages, organized across 24 subcategories. Each impossible task was paired with 91 matched solvable controls—similar problems with valid solutions. Two independent evaluators judged responses using a validated protocol, achieving 82% agreement with strong inter-rater reliability.
Testing twelve open-weight models for code generation and reasoning revealed a stark pattern: these models produced ungrounded code on roughly 60% of unsatisfiable prompts while refusing only 27% of the time. Reassuringly, they never wrongly refused the solvable control tasks, suggesting the models could theoretically learn to distinguish impossible from difficult problems.
