Vibe Coding's Mixed Track Record: Gains on Fresh Code, Questions Remain on Long-Term Value

What Is Vibe Coding?#

Vibe coding—a term coined by AI researcher Andrej Karpathy in February 2025—refers to a development approach in which programmers describe their intent in natural language and then test the resulting code by running it rather than reviewing it line by line. This practice leverages large language models to generate code automatically, fundamentally shifting how developers interact with AI tools.

A Fragmented Evidence Base#

A new state-of-the-art review synthesizes the first body of empirical evidence on vibe coding across software engineering, human-computer interaction, labour economics, security, and education. The findings paint a contradictory picture. Researchers found that while AI models can reliably generate working code in many cases, they struggle with fault detection and produce documentation that is difficult to audit. Early performance benchmarks appear to have reached saturation, yet capability varies dramatically depending on the task.

The productivity story is particularly mixed. Peer-reviewed field experiments report developers completing 26% more tasks per week with AI assistance, yet independent randomised trials have measured a 19% slowdown. Meanwhile, analysis of actual team telemetry shows code-review time increased by 441%—suggesting that generated code may require more scrutiny, not less.

Reconciling the Contradictions#

The authors argue these conflicting results are not mutually exclusive once you account for how studies differed in their methods, scope, and measurement timeline. They identify six recurring patterns that explain the dispersion:

• Effect shrinkage when researchers measure more broadly
• Self-report diverging from independent measurement, meaning developers perceive greater gains than external observers measure
• Output volume being confused with productivity, where more code written does not necessarily mean more value delivered
• Bold claims retreating when tested over longer time periods

Documented Risks and Downsides#

The review documents concrete harms: security vulnerabilities in production systems, visible degradation in code quality across large datasets of deployed applications, unresolved copyright questions around training data, and evidence that developer skills may atrophy with heavy reliance on AI assistance.