AI
New dataset reveals how developers really use AI coding agents—and where they fall short
A large-scale analysis of real-world coding agent sessions shows that while AI generates significant portions of code, it remains inefficient, introduces security risks, and meets developer pushback in half of all interactions.
Researchers have released SWE-chat, a groundbreaking dataset capturing nearly 18,000 real coding agent sessions from open-source developers. The compilation includes over 229,000 user prompts and 2 million tool calls—the first empirical look at how AI coding agents perform outside of controlled lab settings.
Understanding Real-World Usage Patterns#
Development workflows split into distinctly different modes. In roughly 41% of sessions, AI agents write virtually all the committed code, a pattern termed "vibe coding." Meanwhile, about 25% of sessions show the opposite extreme: humans handle all code authorship themselves. This bimodal distribution suggests that developers adopt AI agents in highly varied ways, from near-total reliance to selective use alongside traditional coding.
Quality and Reliability Concerns#
Despite recent improvements in AI capabilities, the results in the wild reveal persistent limitations. Only 59% of agent-generated code makes it into user commits—meaning that nearly half of what AI produces gets discarded or reworked. More troubling, code written by agents introduces more security vulnerabilities than code authored by human developers, indicating that AI-assisted development requires stronger oversight.
Developer skepticism shows up in the interaction patterns: users actively push back against agent outputs in fully 50% of all turns, through corrections, failure reports, and interruptions. This resistance underscores that many developers view AI assistance as unreliable without human verification.
Moving Beyond Benchmarks#
The dataset represents a shift toward understanding AI agents through real-world context rather than curated performance benchmarks. By tracking complete interaction traces and attributing code authorship between humans and machines, SWE-chat provides evidence for how AI actually performs in production workflows, setting the stage for more grounded evaluation and improvement of coding agents.
