How AI is forcing software engineers to rethink assurance and correctness

Software engineering has long pursued the ideal of formal correctness—the notion that code can be rigorously proven to work as specified. But in practice, teams accept imperfect solutions informed by tacit knowledge, domain expertise, and implicit contextual understanding that rarely makes it into documentation or code comments.

Generative AI has exposed a critical gap in this approach. Unlike traditional software systems grounded in formal logic, AI models produce statistically probable outputs—educated guesses rather than logically derived certainties. This shift forces a reckoning: our existing assurance methods, which rely on explicit specifications and expert reasoning, were never equipped to handle systems whose foundations are inherently probabilistic.

The Hidden Context Problem#

Much of what guides sound software judgment is invisible. Domain knowledge, organizational norms, and unspoken assumptions about how a system should behave shape every decision, yet they rarely appear in code repositories or specifications. These shadows of understanding can be hard to capture and nearly impossible to encode into AI models. When generative AI learns from incomplete records of human expertise, it inherits not just knowledge but also gaps—places where human judgment filled in the blanks in ways the data alone cannot reveal.

Beyond Code to Purpose#

Traditional assurance frameworks focus narrowly on whether code meets its specifications. Yet true software engineering concerns itself with a wider question: does the system actually serve its intended purpose? The gap between what is formally specified and what the system should genuinely do remains largely unaddressed. Adding AI to the mix compounds this challenge, since statistical models excel at pattern matching but struggle with the messy real-world constraints and contextual nuances that define fitness for purpose.

A Case for Systematic Thinking#

The field lacks a coherent framework for choosing among assurance techniques. Cost-benefit tradeoffs between different verification approaches—testing, formal methods, code review, simulation—are rarely made explicit or systematic. Rather than allowing AI to force us further into ad-hoc assurance choices, the author argues engineers should embrace a more deliberate approach: understanding the strengths and limits of each technique and combining them intelligently based on the actual risks and requirements at hand.

The name "software engineer" implies engineering discipline. Meeting that standard requires moving beyond intuition toward reasoned, documented decisions about how much confidence is enough—and how to achieve it cost-effectively.