Agentic AI
Researchers propose classification framework for multi-agent software delivery systems
A new study offers clear boundaries for separating concerns across layers of automated software factories orchestrated by AI agents.
Automated software delivery pipelines increasingly layer orchestration logic on top of individual coding agents. Yet the systems lack clear criteria for determining which layer—the agent harness, factory harness, or domain layer—should handle different responsibilities and failures.
The Problem#
When multiple agents run in sequence, deciding where mechanisms belong and where defects originate becomes ambiguous. Without shared definitions, teams struggle to classify whether a failure stems from an individual agent's logic, the coordination layer managing multiple runs, or domain-specific requirements. This ambiguity makes it harder to debug systems and understand architectural boundaries.
A New Framework#
Researchers propose a set of boundary criteria grounded in a precise operational definition of a "run." The classification works as follows:
- Agent harness: Mechanisms confined to a single run
- Factory harness: Logic that coordinates runs, passes state between them, or adapts based on run outcomes
- Domain layer: Rules that must shift as application requirements change
For failures, the framework tracks three separate dimensions: where the defect originated, which control measure failed to catch it, and where it was ultimately fixed. The scheme requires positive evidence before blaming environmental causes.
Testing and Results#
The researchers developed these criteria using documented failures from a single operator managing a software pipeline across six projects, plus 26 components from three published systems. Two independent coders then applied the frozen criteria to 46 held-out cases.
The coders achieved strong agreement on mechanism layers (100%) and good agreement on incident fields (18–19 of 20, with kappa 0.86–0.93). However, both flagged about half the cases as ambiguous, usually due to incomplete information, and marked 8–13 of 20 incident fields as undetermined.
Open Questions#
Testing revealed tensions in the rules. A revised approach separating mechanism classification from incident attribution resolves some conflicts but shows slightly lower agreement (kappa 0.78–0.93) on sensitivity re-coding. The authors note that agreement metrics alone neither measure accuracy nor prove practical utility. Human coder performance remains untested, and seven hypotheses from the case evidence are presented as preliminary findings awaiting further validation.

