Software Engineering
Eye-Tracking Study Reveals How Developers Review LLM-Generated Code Differently
A Wizard-of-Oz experiment finds that labeling code as LLM-generated increases attention and changes review strategies, though it doesn't improve thoroughness.
Researchers conducted an eye-tracking study to understand how software developers actually review code when it's labeled as being generated by large language models. Despite widespread recognition that LLMs are error-prone and require careful inspection, little was known about developers' real-world review behavior.
Study Design#
The team ran a Wizard-of-Oz experiment with software engineers performing code review tasks. They collected both behavioral data via eye-tracking and qualitative feedback through exit interviews, then combined Bayesian statistical analysis with qualitative findings.
Key Findings#
The research revealed a surprising gap between intentions and practice:
- Attention shifts: Developers spent significantly more time visually fixating on code explicitly labeled as LLM-generated, showing that the label itself influences where reviewers focus their attention.
- No change in thoroughness: Despite spending more time on LLM code, the overall thoroughness of code reviews didn't actually improve.
- Strategy adaptation: Practitioners adapted their review approach for labeled LLM code by assessing logical correctness and using the code's generation prompt as a guide for inspection.
Implications#
The findings suggest a mismatch between what developers believe they should do and what they actually do when reviewing AI-generated code. The study recommends that tool designers incorporate labeling and prompts as software artifacts, and calls for companies to revisit their policies around LLM-assisted development to better support developers in this increasingly common task.