Agentic AI
Study reveals how developers build AI agents for software engineering tasks
As LLM-based agents become cheaper to implement, development teams face new challenges around code review and agent evaluation.
Software engineering agents—AI systems powered by large language models that can navigate complex codebases and perform development work autonomously—are spreading rapidly, yet researchers have barely examined how teams actually build them. A new study published on arXiv combines interviews with 20 practitioners across 12 organizations and a survey of 80 developers to map the emerging practices and pain points in SE agent development.
The Shifting Bottleneck#
The research uncovers a counterintuitive pattern: as the technical barriers to implementation drop, project friction doesn't vanish—it simply relocates. Teams report that traditional development friction points like requirements gathering, cross-team coordination, and deployment logistics become more salient as implementation overhead shrinks. Simultaneously, entirely new work emerges: verifying generated code and assessing whether agent behavior actually delivers intended results have become central to the development cycle.
Seven-Stage Workflow and Process Evolution#
The researchers identify a seven-stage workflow that SE agent teams follow, paired with five significant shifts in how development happens. Most notably, practitioners are moving toward evaluation-driven development—a mode in which testing criteria and performance benchmarks are locked down early in the process, then used to guide iteration throughout. Teams are also treating specifications as first-class artifacts, meaning they document, test, and version requirements alongside the actual code.
New Challenges and Mitigation Strategies#
The study surfaces six core challenges teams encounter, alongside twelve practices teams either currently employ or propose to address them. Key friction points include:
- Unreliable evaluation signals — difficulty determining whether agent outputs genuinely work as intended
- Comprehension debt — understanding decays when AI-generated code scales faster than team familiarity with it
- Provider-driven behavioral shifts — unexpected changes when model providers update underlying systems
These findings suggest the field is still in early maturity, with teams improvising solutions to problems that lack established best practices.
