AI Security
AI-powered coding is outpacing security infrastructure, forcing a platform rethink
Platform engineers are rapidly adopting AI to write code but lack the security guardrails needed to protect against its novel risks, sparking urgent questions about how to secure autonomous systems.
The New Stack
At a summer conference for platform engineers, a striking pattern emerged: hundreds of developers had begun using AI to generate code, but nearly all lacked confidence in their security systems to safely manage it. The gap between adoption speed and security readiness is widening, raising a fundamental question: can organizations move at the velocity of AI while maintaining safe practices?
The unpredictability problem#
Machine learning systems don't behave deterministically—their outputs are not fully predictable. This fundamentally changes how organizations must think about risk. According to Liz Rice, chief open source officer at Isovalent, "The threat landscape has really changed a lot because finding vulnerabilities is almost wasted work now. What we're doing in the AI world is non-deterministic, so having guardrails around what systems can do becomes ever more important because we can't predict what models are going to do next."
The risk spreads beyond traditional development teams. When non-technical workers across a company can now generate working code through AI assistance—whether HR staff writing agents or finance teams automating processes—the number of people creating potential security vulnerabilities multiplies. These workers often lack the training to recognize or avoid insecure patterns. As Rice noted, "However good your model is, it probably is not going to write absolutely secure code any more than a human does."
The governance gap: AI-BOMs and sprawl#
Shadow AI—the uncontrolled deployment of AI tools across an organization—presents risks comparable to shadow IT but with higher stakes. One critical emerging need is tracking what goes into AI systems themselves through an AI bill of materials (AI-BOM): documenting model weights, training datasets, third-party APIs, and other non-deterministic components.
Joe Baguley, CTO of EMEA at Broadcom, warned that supply chain visibility has become urgent. "There is no such thing as a zero-day anymore," he said, pointing to the need for strict understanding of dependencies and components. Without oversight, organizations risk duplicating effort—Baguley cited an HR department where 17 professionals each built separate AI agents to solve overlapping problems, fragmenting security responsibility and multiplying maintenance burden.
The alternative platforms must now weigh is whether to onboard all workers—human and agent alike—to a centralized internal developer platform, or resort to traditional IT command-and-control structures. Neither feels entirely adequate for the current moment.
