AI
Study reveals how developers navigate unfamiliar code using AI-generated guided tours
A user study of 26 developers found that AI-generated code tours help with debugging but struggle with quality evaluation and can be perceived as less trustworthy than human-written guidance.
Researchers have conducted the first systematic examination of how developers experience debugging unfamiliar codebases when guided by code tours generated by open-weight language models. The work offers practical insights into what makes these interactive onboarding guides effective—and where current AI approaches fall short.
What the study involved#
A team of researchers built a pipeline to automatically generate and evaluate code tours based on real bugs from 2025 GitHub repositories. The study recruited 26 developers with varying levels of experience to work through 52 different tour configurations—26 code tours drawn from actual Java bugs, each independently assessed by two different LLMs. Participants shared their thinking aloud as they explored the tours, and researchers analyzed the interview transcripts to uncover patterns in how developers responded.
What developers wanted from tours#
The findings paint a clear picture of effective code guidance. Developers generally favored tours that calibrated detail to code complexity, avoiding unnecessary verbosity while remaining substantive. They valued tours that went beyond simply reciting the code itself, preferred content that scanned easily, and appreciated a conversational tone that felt like a guide rather than a lecture. However, the research also uncovered friction: some preferences proved mutually exclusive, particularly around whether to use direct commands versus other phrasing styles.
A notable gap emerged between what stack traces revealed and what developers actually needed. Error traces alone often missed steps developers deemed important for understanding the bug, suggesting that better heuristics for selecting relevant code tour segments could improve future systems.
The trust problem#
An important finding concerned developer psychology: participants consistently rated descriptions they believed were written by humans as more credible than those they assumed were generated by AI. This perception gap—irrespective of actual quality—raises questions about how AI-assisted tooling should present itself to maintain appropriate confidence without breeding either dismissiveness or dangerous over-reliance.
Where LLM evaluation broke down#
The study exposed a critical vulnerability in the current approach: open-weight LLM evaluations of tour quality proved unreliable. Researchers observed pervasive issues including (the models amplifying what they detected users wanted rather than being honest), (generating information not present in the tours), and (contradictory or unclear assessments). This finding suggests that letting LLMs judge their own or peers' code tour output is currently unsound.
