GitHub Copilot Speeds Up Code Tasks But Doesn't Boost Understanding, Study Finds

Generative AI tools like GitHub Copilot can make programmers faster at completing tasks, but a new study suggests they may not deepen understanding of the code they're working with—a critical concern for software engineering education.

The Study#

Computer science researchers at an unnamed institution conducted a controlled experiment with 15 graduate students tasked with implementing features in unfamiliar codebases. Each participant completed feature-implementation work both with and without GitHub Copilot's assistance. The researchers measured not just task performance, but also how well participants understood the code they modified.

The results revealed a stark disconnect: while Copilot significantly improved task completion speed and accuracy, participants showed no meaningful improvement in overall code comprehension ($p=0.59$). In fact, faster completion did not correlate reliably with better understanding.

What Worked, What Didn't#

The picture became more nuanced when researchers examined specific types of comprehension. Students using Copilot did show some gains in identifying where and what to modify within a codebase ($\rho=0.50$). However, they struggled comparatively when asked to explain how existing code functioned or predict the consequences of code changes ($\rho=-0.57$)—though these category-level findings did not survive statistical correction for multiple comparisons.

The Verification Loop Effect#

Perhaps most revealing was the behavioral analysis. Students who achieved higher comprehension spent more time verifying and inspecting code. They performed "write-then-view" cycles—generating code and then examining it closely—4.7 times more frequently than students with lower comprehension ($p=0.001$). This suggests that passive acceptance of AI-generated code may undermine learning, while active inspection and revision foster genuine understanding.

Implications for Education#

The researchers argue that programming education needs to rethink how it assesses student learning. Currently, correctness—whether code works—is often conflated with comprehension. Their findings suggest these should be evaluated separately. They also recommend that educators teach students to actively inspect and explain AI-generated suggestions, and that tool designers create features encouraging verification rather than mere acceptance of automated suggestions.