AI Engineering
Engineering teams need an AI slop registry to catch subtle code mistakes at scale
AI coding tools generate plausible but subtly flawed code that passes basic checks—an invariants-based verification system catches systematic mistakes where instruction files alone fail.
The New Stack
AI coding assistants make engineers faster—at making the same mistakes faster. The problem isn't obviously broken code; it's subtly wrong implementations that compile and pass initial review: overengineered abstractions, API calls that don't exist, naming violations, or misapplied patterns.
These errors are systematic and preventable, but instruction files like CLAUDE.md alone aren't enough. The New Stack reports on a shift toward "intent-driven verification"—a two-layer approach that treats code review like building inspection rather than architectural oversight.
How the two-layer system works#
The model separates user criteria (acceptance criteria specific to a single PR) from invariant criteria (org-wide rules that apply to matching changes automatically). When verification runs, both layers combine into a unified checklist.
A developer specifies what a change should do; invariants enforce what every change must respect—for instance, that all HTTP handlers call authentication middleware before business logic, or that migrations declare a down block.
The key difference: the same non-deterministic agent that wrote the code doesn't verify it. A separate verification agent checks the output with no shared context or blind spots, producing structured reports per criterion rather than subjective opinions.
Mining patterns from review comments#
The invariants registry grows from recurring feedback. Any review comment left more than twice becomes a candidate for codification—turning comments like "use UserRepository.UpdateProfile instead of direct writes" into automated checks applied across the codebase.
One experiment showed that a second agent verified 6,000 lines of code against 65 user-defined criteria in six minutes, passing 60, failing 4, and partially passing 1. Design-level decisions were verified before code generation; org invariants enforced automatically throughout.
The shift moves human judgment upstream—reviewers evaluate specs and constraints, not 500-line diffs. Quality control becomes part of the machine that builds software.
