Study reveals how developers use LLMs for coding—and where AI-generated code stumbles

Researchers have analyzed nearly 600,000 real-world conversations between developers and large language models to understand what programmers ask for, how their interactions evolve, and whether the quality of AI-generated code improves over time. The findings paint a nuanced picture: while LLMs are widely used for coding help, the code they produce often contains language-specific defects that persist even in extended conversations.

The Dataset: A Window into Developer-LLM Interactions#

The study created CodeChat, a dataset sourced from WildChat containing 587,568 developer-LLM conversations and 1.7 million code snippets spanning more than 20 programming languages. This scale provides rare empirical insight into how developers actually interact with AI assistants, moving beyond lab experiments to capture authentic usage patterns.

What Developers Ask For#

The most common topics developers bring to LLMs reveal clear priorities:

• Web design dominates, accounting for 9.6% of all conversations
• Machine learning ranks second at 8.7% of discussions
• Developers frequently extend conversations by shifting between different use cases, which explains why chats often grow longer over time

The Quality Problem Across Languages#

Researchers evaluated generated code quality across five major languages—Python, JavaScript, C++, Java, and C#—and uncovered a troubling pattern: language-specific issues appear regularly and their prevalence does not consistently decline as conversations continue beyond five turns. This suggests that simply keeping a conversation going does not automatically lead to better code.

Among instances where developers successfully resolved syntax errors, the most effective approach was explicitly identifying the error in a new prompt and requesting a corrected version. This points to a design opportunity: conversational coding assistants need to be more proactive about tracking shifts in developer intent and continuously monitoring code quality as discussions evolve.

Implications for Tool Design#

The research suggests that current conversational assistants fall short in two key ways: they don't adequately track when a developer pivots to a new task or use case, and they lack systematic mechanisms to catch and prevent recurring code quality problems. Better context management and explicit quality checks across conversational turns could meaningfully improve the developer experience.