Study reveals which ChatGPT prompting strategies boost productivity in real code generation tasks

Researchers have conducted one of the first systematic investigations into how developers can most effectively collaborate with ChatGPT for complex, real-world coding projects. Rather than testing simple one-off function prompts, the study examined interaction strategies across multi-file codebases that more closely mirror actual software engineering work.

Why this gap matters#

Most prior research on large language models and code generation has focused narrowly on isolated function-level tasks—often using textbook examples that don't capture the complexities developers encounter in production. This work addresses that limitation by designing benchmark tasks at the project level, where multiple classes interact, dependencies sprawl across files, and context becomes harder to manage.

The experiment and findings#

A team conducted a user study with 36 participants of varying backgrounds who tackled assigned tasks by prompting GPT with specific interaction patterns. Researchers then analyzed behavioral data by reviewing screen recordings and chat logs. The empirical analysis, guided by statistical methods, surfaced three concrete interaction features that reliably boosted productivity across participants. Beyond that narrow set, the team distilled five overarching principles for enhancing the human-LLM collaboration process.

A practical taxonomy of errors#

The research also cataloged 29 distinct runtime and logic errors that can emerge during code generation workflows with LLMs. For each class of error, the researchers outlined mitigation strategies to help developers anticipate and prevent pitfalls.

The work represents a step toward moving beyond generic prompting advice, grounding best practices in empirical evidence from realistic development scenarios.