AI
Anthropic proposes mandatory safety tests, not bans, for AI models amid industry rift
As OpenAI and Google join a coalition backing open-weight AI models, Anthropic charts a different course with a testing regime—but critics say the proposal leaves critical questions unanswered.
The New Stack
Anthropic's CEO Dario Amodei has rejected calls for restricting open-weight AI models, instead proposing a framework of mandatory safety evaluations before any sufficiently capable system reaches the public. The move positions the company differently from recent signatories to a major industry letter supporting open-weight development, including OpenAI and Google.
Three pillars, one undefined threshold#
Amodei's plan rests on three elements: tightening U.S. export controls on advanced semiconductors and manufacturing gear, blocking large-scale model distillation by competitors, and requiring pre-release safety testing for models deemed sufficiently capable. The first two target policy makers in Washington; the third would reshape release practices across the AI industry.
Central to the proposal is the notion that testing regimes enjoy emerging consensus—even the Trump administration, Amodei notes, has signaled support. Recent industry proposals would apply evaluations universally while carving out exemptions for "less capable models, such as those from startups and academia."
The exemption reveals the scheme's core tension: nobody has specified where the capability line sits. That threshold determines which models get released and which face testing delays. Whether capability is measured by benchmark performance, training compute, or risk-category scoring matters enormously—the chosen metric effectively sets release timelines for every lab publishing weights.
The broader coalition and notable absences#
The landscape shifted rapidly this week. An industry letter titled Open Weights and American AI Leadership has gathered 50 signatories—including Microsoft, Nvidia, Meta, GitHub, Ollama, and now OpenAI and Google. Anthropic and Amazon remain conspicuous holdouts.
Monday brought another development: Nvidia announced the Open Secure AI Alliance, enlisting over 35 founding members like IBM, Hugging Face, Palantir, and the Linux Foundation. Nvidia's rationale stresses that open frontier models enable better incident investigation—a point underscored by a recent Hugging Face security breach that closed tooling failed to analyze, prompting researchers to turn to an open-weight alternative.
