AI
Study maps how developers use AI chatbots for software design tasks
New research combines analysis of real GitHub conversations and practitioner interviews to reveal what benefits—and limitations—LLMs bring to the design phase of software development.
A new empirical investigation sheds light on how software engineers are putting large language models like ChatGPT to work during the design phase of software projects. Rather than assuming LLM capabilities, researchers conducted a hands-on study examining both real-world developer-AI interactions and direct feedback from practitioners.
Study Design#
The research combined two complementary approaches: a mining study of 291 public ChatGPT conversations shared on GitHub, paired with surveys from 65 software practitioners. This mixed-methods approach captured both observable patterns in how developers actually use AI tools and the insights practitioners gained from their experiences.
Design Tasks LLMs Can Support#
Researchers identified nine categories of design work where developers called on ChatGPT, ranging from high-level architecture design to lower-level data model design and the selection of design patterns. Within these conversations, developers most commonly sought help at the detailed design level—working out the specifics of their systems rather than sketching the big picture.
What Developers Use AI For#
Analysis of the GitHub conversations revealed that developers primarily turned to ChatGPT for two purposes: knowledge acquisition (learning about technologies and approaches) and design-related code generation (getting runnable or template code based on design decisions).
Reported Benefits and Limitations#
Practitioners identified several concrete advantages. The survey uncovered seven key benefits, including better guidance on technology choices and earlier detection of design flaws before implementation.
However, the same practitioners also noted six significant limitations. These ranged from practical issues—such as overly lengthy or repetitive outputs—to more fundamental problems like code that is either incorrect or impossible to run. Developers also flagged that LLMs struggle when they lack project-specific context, often generating suggestions misaligned with particular system requirements.
