AI
LLM-generated code has 63% more quality issues than human-written alternatives
A new study comparing code smells finds that ChatGPT, Gemini Pro, Codex, and Falcon all fall short of professional coding standards, particularly on complex tasks.
Researchers have conducted a systematic analysis of code quality produced by leading large language models, uncovering a significant gap between AI-generated and professionally written solutions.
The Study#
An academic team evaluated Java code generated by four state-of-the-art LLMs—Gemini Pro, ChatGPT, Codex, and Falcon—by measuring code smells, a standard software engineering metric for identifying maintainability and design problems. They compared the AI output against reference solutions written by human programmers, testing the models across different coding scenarios that varied by topic and difficulty level.
Key Findings#
The results reveal consistent quality shortfalls across all tested models. LLM-generated code contained an average of 63.34% more code smells than human-written baselines. Performance varied by model:
- Falcon performed best, with a 42.28% increase in code smells
- Gemini Pro showed a 62.07% increase
- ChatGPT exhibited a 65.05% increase
- Codex ranked worst at 84.97% more smells
The research also found that quality problems concentrated differently depending on smell category—implementation-level issues were more prevalent (73.35% higher), while design-level problems increased by 21.42%.
Where Models Struggle Most#
Task complexity proved critical. Code generated for advanced topics and object-oriented programming tasks showed markedly worse quality than simple assignments. The authors note that LLM performance on harder problems correlates closely with how difficult those same problems are for human programmers—but at a consistently lower absolute standard.
Why It Matters#
While much research has focused on whether LLM-generated code works, this study highlights that functional correctness does not guarantee code quality. Poor code smell profiles suggest that AI-generated solutions may be harder to maintain, debug, and extend—practical concerns for teams relying on these tools in production environments.
