Anthropic researchers go public: AI safety gap widening as companies race toward superintelligence

A pretraining researcher at Anthropic publicly resigned this week, sparking candid admissions from two of his colleagues still at the company about the state of artificial intelligence safety. The three researchers agree on a core concern: the technical challenge of aligning superintelligent AI systems remains unsolved, yet companies are racing to build them anyway.

Insiders voice private doubts publicly#

Jacob Coxon, who spent three years building machine learning systems at OpenAI and Anthropic, announced his departure on social media, framing the decision as a broader indictment of how the industry prioritizes capability growth over safety. "The people building AI earnestly believe that it could kill us all by the end of the decade," Coxon stated, adding that while executives often downplay such concerns in public, many privately express genuine fear.

His comments prompted direct responses from colleagues. Evan Hubinger, who leads Anthropic's alignment science efforts, confirmed the concern is widespread: "We really do earnestly believe AI could kill all humans." Hubinger—whose team evaluates vulnerabilities in Anthropic's own safety techniques—estimated a greater than 10% chance this could occur within a decade and acknowledged that Anthropic "do not yet have a plan to solve alignment for superintelligence."

Samuel Marks, who oversees scalable oversight research at the company, laid out the technical specificity. Current alignment methods can influence model behavior but cannot guarantee it reliably. The industry's emerging fallback plan relies on using AI systems themselves to align their more powerful successors—essentially betting that future systems will be better at safety than humans, despite unproven track records.

The dependency trap#

This creates what researchers call a recursive vulnerability: developers will eventually need to trust AI systems whose safety they cannot fully verify to help oversee even more capable systems. The gap between what these models can do and what humans can understand about their reasoning is expanding faster than safety methods can keep up.

Marks noted that many developers want to slow progress to address safety, but fear that less cautious competitors will forge ahead first. Money and competitive pressure override caution.

Acceleration already underway#

Anthropic disclosed that Claude now writes more than 80% of code merged into the company's codebase, up from single digits before early 2025. The average engineer is merging eight times as much code per day compared to 2024—a pace driven by AI-assisted development that underscores the feedback loop Coxon flagged: AI is already accelerating the creation of its own successors.