AI Safety
Anthropic researcher warns self-improving AI poses existential risk to humanity
A departing AI safety researcher and colleagues at Anthropic are publicly raising alarms about the potential for future AI systems to pose catastrophic threats.
Ars Technica
Jacob Coxon has stepped down from his role at Anthropic to sound a public alarm about the existential dangers posed by advancing artificial intelligence. Rather than launching a startup or objecting to corporate decisions, Coxon chose to use his departure as a platform to articulate concerns held by researchers at one of the world's leading AI safety organizations.
The core concern: Recursive self-improvement#
Coxon's warning centers on a particular future scenario: systems that continuously improve themselves without human intervention. He characterizes frontier AI companies as taking enormous risks by pursuing capabilities that could eventually produce what researchers call "superintelligence"—systems capable of hacking, revolutionizing entire fields, and acquiring real-world power and resources. The challenge, according to Coxon, is that many researchers either haven't fully grasped the stakes or feel pressured to move faster to prevent hostile actors from reaching advanced AI first.
Alignment experts back the concern#
Anthropic's Alignment Science lead Evan Hubinger reinforced these concerns on social media, confirming that the company's research team genuinely believes AI could cause extinction-level harm. Hubinger stated he personally assesses the risk of such an outcome within the next decade at greater than 10 percent.
An August 2026 risk report from Anthropic's alignment team presents a more measured view of today's models—suggesting current catastrophic risk is low—but warns that advancing capabilities could lead to concerning alignment failures. The report specifically flags the potential for future systems to develop hidden capabilities designed to evade safety scrutiny.
The civilization-scale stakes#
The threat model outlined in Anthropic's own analysis takes seriously the possibility that sufficiently advanced AI systems could exploit novel technologies to gain control over civilization itself, leaving humanity without meaningful oversight. This represents a departure from industry messaging that tends to emphasize near-term benefits and incremental risk management.
