AI
Dataset of 53.6K Developer Edits Reveals How Programmers Fix AI-Generated Code
A new collection of real-world code modifications shows that most edits to AI completions happen within 15 minutes and can teach smaller models to match frontier LLM performance.
Researchers have created a dataset documenting how developers actually modify code produced by AI assistants, offering a window into the gap between what AI systems generate and what production-ready code looks like.
The collection, called DECODE (Developer Edits of Code Dataset), contains more than 53,000 in-IDE edits captured from over 1,000 developers working with AI-generated Python, TypeScript, and JavaScript code. By tracking these real modifications rather than relying on final Git commits, the dataset provides a granular record of developer behavior that wasn't previously available.
Why This Data Matters#
Prior research on code has largely depended on publicly available Git repositories, which only show the final state of accepted code. This misses crucial information about how developers actually interact with AI output—the small refinements, deletions, and rewrites they make along the way. Manual edits reveal the specifics of what doesn't work and how developers fix it, offering insights that Git data alone cannot capture.
Key Findings on Developer Behavior#
Analysis of the dataset uncovered patterns in when and how developers modify AI completions:
- Most edits occur early: The majority of modifications happen within the first 15 minutes after accepting an AI suggestion
- Rejection is common: About 31% of edit sequences result in the developer ultimately rejecting or removing the AI completion entirely
- Editing varies by context: The dataset shows distinct patterns for when developers choose to refine AI output versus discarding it
Improving Smaller Models Through Real Data#
The researchers used DECODE to train language models on code editing tasks. The results showed that open-source models with just 3 billion parameters could match or exceed the performance of much larger frontier LLMs when fine-tuned on this realistic editing data. This suggests that training approaches grounded in actual developer behavior may yield better results than simply scaling up model size.
Looking Forward#
The work underscores the value of developer-centric approaches to building AI coding assistants. Rather than optimizing solely for code generation quality, future tools might benefit from understanding and predicting how experienced developers refine and correct automated suggestions. The dataset is .
