Study Reveals Developers Struggle to Balance Autonomy and Safety in Agentic AI Systems

Agentic AI systems operate with a degree of autonomy that traditional software cannot match—they make decisions independently, interact with external tools, and respond dynamically to their environments. Yet this very flexibility that makes them powerful also introduces distinctive hazards, according to a new study of how industry developers think about and manage risk in these emerging products.

Researchers interviewed 35 software engineers building agentic AI systems and discovered a fundamental misalignment between what risks they perceive, what they prioritize, and the tools they have available to address those risks.

Risks Flow from Agency#

The study found that the characteristics developers value most in agentic systems—autonomy, tool integration, and real-world deployment—are the same features that generate novel safety concerns. Developers recognized this connection: their risk perception directly mirrored the technical properties that enable agency itself. The problem, however, lay in what came next.

Business Before Broader Impact#

When forced to rank competing concerns, developers consistently elevated product-level and business risks—such as system reliability and customer satisfaction—above downstream societal harms like workforce displacement and user privacy violations. This prioritization had downstream consequences: teams that focused first on immediate commercial risks often lacked the bandwidth or motivation to seriously tackle the broader societal dimensions of agentic risk.

The Control Gap#

Perhaps most critically, developers reported having few mature mechanisms for managing agentic risks effectively. The safeguards they relied on most often involved restricting the very capabilities that justified building autonomous systems in the first place. Constraining autonomy, limiting tool access, or reducing goal complexity all reduce risk—but they also diminish the systems' practical utility.

This creates what the researchers call a capability-versus-control tension: the tools developers need to make agentic systems safer threaten to render them less capable and less appealing to users and business stakeholders alike.

The findings suggest that without new approaches to safety-preserving control mechanisms, the agentic AI industry may face pressure to either accept greater risks or sacrifice the features that make these systems distinctive.