New Framework Proposes Limited Autonomy for AI in Software Development

Generative AI models have begun handling the full lifecycle of software development—writing application code, securing it, monitoring it, and searching for vulnerabilities. A new research paper challenges the assumption that these systems should operate with increasing autonomy, instead proposing a framework that limits their authority while keeping humans in critical control positions.

The Problem With Shared Substrates#

The research identifies a key vulnerability when a single family of AI models performs multiple roles in development. All these roles rely on overlapping upstream dependencies—the training corpus, the model itself, alignment procedures, vendor choices, and development tools. When these foundations are shared, the AI systems may fail in correlated ways across different tasks, undermining the independence that makes verification reliable.

Classical approaches to software safety have relied on organisational independence: different teams building, defending, and testing systems separately. But this traditional guarantee breaks down once all three roles share the same AI substrate. A flaw in the underlying model can cascade across building, defense, and attack functions simultaneously.

A Five-Level Autonomy Framework#

The framework proposes that humans retain authority in three specific areas:

• Authority over specification: defining what software must accomplish
• Accountability: holding the system and its operators responsible, using governance structures that cannot be fully delegated
• Halt authority: the power to stop operations as a last resort

Beyond these boundaries, autonomy levels range from minimal (requiring human approval at each step) to higher degrees of independence, depending on five factors:

• Consequence severity
• Reversibility of decisions
• Time criticality
• Verifiability of outcomes
• Exposure to adversarial risk

Restoring Independence#

The research argues that heterogeneous models (different AI systems for building, defending, and testing), deterministic analysers, and formal verification can restore statistical independence for the fault classes they address. This diversity makes correlated failures less likely across lifecycle roles.