Study reveals mixed picture on AI coding assistants' impact on developer productivity

A new systematic review examining nearly four decades of research has found that large language model coding assistants deliver tangible benefits for software developers—but with important caveats around code quality and collaboration risks.

What the research shows#

Researchers analyzed 39 peer-reviewed studies published between early 2014 and late 2024 to assess how LLM-based tools (such as GitHub Copilot and similar assistants) affect programmer productivity. The findings paint a nuanced picture. A strong majority of studies reported measurable advantages: faster development cycles, reduced time spent searching through existing code, and automation of routine or mechanical tasks. Yet a meaningful subset of research also surfaced potential downsides, including concerns about developers relying too heavily on suggestions without critical review and reduced cross-team communication.

The code quality puzzle#

One key unresolved question emerged: whether these assistants actually improve or harm code quality. Different studies reached contradictory conclusions depending on their evaluation methods and specific contexts. This variability suggests the relationship between AI assistance and software quality is more complex than simple cause-and-effect, shaped by how teams integrate the tools and what measures they prioritize.

Research gaps and dimensions#

The review noted that while 90% of studies adopted a multi-dimensional approach—examining at least two aspects of the SPACE framework (Satisfaction, Performance, Efficiency, Collaboration, Activity)—only 15% explored beyond three dimensions. Satisfaction, Performance, and Efficiency commanded the most attention, while Communication and Activity received far less scrutiny. Most research was exploratory rather than longitudinal, and few studies evaluated team-based scenarios, leaving substantial gaps in understanding how these tools affect group dynamics over time.

The full study is publicly available on arXiv, with supporting materials linked in the paper.