How AI Evals Are Replacing Product Requirements at Anthropic

Inside Anthropic, the old playbook for building products is disappearing. Instead of writing product requirement documents, teams are running evaluations to define what their AI models should do—and detecting capabilities that emerge suddenly across model upgrades.

According to Dianne Penn, Anthropic's Head of Product for AI Research and Labs, speaking on Lenny's Podcast, the shift reflects a core tension: traditional software is deterministic and predictable, but frontier AI is probabilistic. That unpredictability makes old-fashioned requirements obsolete.

Evals as the source of truth#

The company's product teams now treat evaluation suites as their primary artifact. For each major feature, Penn's team creates 30 to 40 representative test cases—prompts paired with expected outputs that define correct behavior. These evals run automatically against new model versions, letting teams track whether a capability improves incrementally or suddenly.

The stakes of missing these capability jumps are high. Model improvements often come in sudden leaps: a skill like reliable math reasoning can appear to materialize overnight. Without rigorous evals and testing infrastructure in place, teams might not notice these advances at all, leaving what Penn calls "product overhang"—capabilities that exist but remain unknown.

A different kind of debugging#

This also transforms how teams find and fix bugs. Instead of tracing code execution, engineers read through conversations to understand why a model made a particular choice. A hallucination, overconfidence, or failed tool call can all look similar but require entirely different solutions. Penn highlighted sycophancy—where a model reinforces a user's incorrect assumption rather than correcting it—as a particularly sneaky issue that can slip past testing and reach production.

Leading by doing#

The new approach demands hands-on leadership. Managers who don't regularly use the models they oversee risk losing the intuition needed for good architectural decisions, according to Penn: "If you're not building yourself, you're not gonna make it."

The coding pivot#

This engineering mindset also shaped one of Anthropic's biggest strategic shifts. In 2023, the company had no association with coding tools. But as competitors' models struggled with programming tasks, Anthropic doubled down on coding as a core competency for Claude 3 Opus (launched March 2024) and later Claude Code. The move reflected Anthropic's vision: models needed to evolve from conversational partners into agents that could actually execute tasks.