AI and Software Engineering
Study reveals ChatGPT's mixed impact on how developers solve unfamiliar code problems
A laboratory study of student programmers finds that AI assistance changes problem-solving tactics but doesn't guarantee better outcomes when developers get stuck.
Researchers have conducted a small controlled study examining whether ChatGPT genuinely helps developers understand and modify complex code they've never seen before. The findings suggest the answer is more nuanced than simple productivity gains.
The Study Design#
Ten advanced student developers participated in a laboratory experiment where they had to implement a meaningful change to a large, unfamiliar software system. Half worked with AI assistance; the other half did not. Researchers analyzed their work using Pólya's classic four-phase problem-solving framework and tracked 25 distinct behaviors that emerged during coding sessions. Data collection spanned think-aloud protocols, code modifications, web searches, and the prompts developers sent to ChatGPT.
How AI Changed the Approach#
The developers using AI assistance regularly leaned on the tool to delegate specific steps in their workflow. However, a counterintuitive finding emerged: both groups—with and without AI—ultimately employed the same core problem-solving behaviors. Rather than fundamentally transforming how developers think through code challenges, AI appeared to shift where and when they applied their strategies.
Stuck and Unstuck#
Nearly all participants (nine out of ten) hit a wall at some point. What differed was the path they took to arrive at that moment and, crucially, how they escaped it. The researchers identified seven distinct reasons developers got stuck, ranging from logic errors to architectural misunderstandings. Critically, AI assistance proved to be a double-edged sword: in some instances it helped participants break through impasses, while in others it either prolonged confusion or offered incomplete guidance that led them further astray.
Why This Matters#
As large language models become routine tools in development workflows, understanding their cognitive effects—not just their speed-up metrics—matters for training and team practices. This work signals that AI adoption may reorganize how developers allocate mental effort without necessarily changing the underlying reasoning required to truly comprehend and modify unfamiliar code.
