Why AI adoption metrics mask what really matters: measuring impact, not just usage

Engineering teams are filling dashboards with impressive-looking numbers—rising token consumption, higher seat activation rates, climbing code-generation metrics. Yet when you walk into their planning meetings, little has actually shifted about how they build software. That disconnect is the real problem, obscured by metrics designed to measure the wrong things.

The wrong metrics, the right incentives#

Traditional adoption measures—token spend, pull request volume, percentage of code generated by AI—capture activity but not value. A token count reflects someone typing into a tool. A code-generation percentage says an autocomplete suggestion got accepted. Neither reveals whether the work improved.

This echoes Goodhart's Law: once a metric becomes the target, it stops describing reality and starts describing what the incentive rewards. Teams can trivially inflate these numbers without changing anything meaningful. The honest diagnostic question isn't "how much are people using this?" but rather "if we removed these tools tomorrow, what would actually break?"—not what people would miss, but what would genuinely stop working. If the answer is nothing, you're paying for a subscription, not adoption.

Three flavors of adoption—only one sticks#

Genuine adoption takes three distinct forms, with wildly different lifespans.

Individual workflow shifts happen when someone integrates a tool into their personal process—drafting commit messages faster, or navigating unfamiliar code more quickly. These are real and useful, but fragile. Under deadline pressure, people revert to familiar methods; a new workflow is inherently high-risk, so it's abandoned when certainty matters most.

Permanent delegation means handing off a recurring task for good. Someone decides they no longer do a specific job; a workflow handles it and they verify the output. It survives pressure because reverting would create extra work. Yet it remains dependent on that individual—when they leave, the practice often walks out the door with them.

Structural change is different. The team deliberately rewrites a process, documents it, and enforces it regardless of individual enthusiasm. Only this tier survives turnover and budget pressure. But it doesn't live in tools. It requires the harder conversation: explicitly saying the old way was worse, then weathering the inevitable failures when the new approach breaks something in week three.

Most organizations celebrate the first, occasionally achieve the second, and rarely reach the third. The barrier isn't technical—it's that structural change demands public commitment and accountability.