Anthropic Researcher Says There’s 10% Chance of AI Killing ‘All Humans’ Within 10 Years

After an Anthropic researcher resigned Tuesday, warning that superintelligence risks human lives, fellow researcher Evan Hubinger said that while current risks are low, there’s a 10% chance AI could “kill all humans” in the future — without proper safeguards.

Jacob Coxon, who spent three years conducting pretraining research at both OpenAI and Anthropic, says he resigned because he believes the companies are moving too quickly toward self-improving superintelligence. 

“Do not underestimate the power of this technology,” warns Coxon.

Coxon argues that the AI technology could soon become capable of hacking systems, rapidly transforming entire fields, and acquiring real-world power and resources.

“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt,” he wrote. “If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately. No other human activity poses this level of danger.”

Evan Hubinger, Anthropic’s Alignment Science Lead, responded to Coxon’s post by saying he shares the concern about where the technology is headed. Although he considers the risk posed by AI models that currently exist to be low, his concern is that increasingly capable systems could eventually improve themselves to a point where they pose an existential threat.

“We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” Hubinger wrote.

As CBS News reports, Anthropic’s alignment science department, where Hubinger is lead researcher, focuses on making AI systems behave in accordance with human goals and values. Hubinger said Anthropic is trying to address the problem, but that the company does not yet have a solution for aligning hypothetical superintelligent systems and is not clearly on track to develop one.

Coxon was considerably more critical, saying Anthropic understands the potential consequences of superintelligence but is continuing to race toward it because of fears that other companies will do it less responsibly. He contrasted that with OpenAI, where he said many people have not fully grasped the potential stakes.

The comments come as concerns about AI safety have intensified following several incidents involving autonomous systems. OpenAI disclosed in July that one of its models hacked AI development platform Hugging Face while being tested in an isolated environment, and Anthropic and Meta later acknowledged that their own AI systems had also carried out hacks.

There have also been questions about how closely AI companies are working with government safety bodies. According to CBS, Anthropic said last week that it had not shared its latest model, Claude Mythos 5.1, with security organizations outside the US, including the UK's AI Security Institute, which tests the capabilities and risks of advanced AI models.

A spokesperson for the UK Cabinet Office, meanwhile, said the AI Security Institute continues to work with companies including Anthropic and had recently tested OpenAI’s most powerful model before its public release.

Earlier this week, OpenAI chief scientist Jakub Pachocki also called for “extreme caution” as AI capabilities continue to advance, warning that systems do not need to match humans at everything to become highly useful or dangerous. Anthropic’s August safety report also identified a low risk that highly capable AI could autonomously conduct research and development that results in catastrophic harm, while acknowledging that the company was less confident in that assessment than previously and was seeing “early signs of potential acceleration.”

Anthropic leaders Dario Amodei and Jared Kaplan have also called for greater caution around the pace of AI development, while more than 1,300 AI workers signed a July letter urging the US government to develop safeguards and governance measures that would deliberately slow the development of increasingly autonomous AI.

The BBC reports that Coxon and Hubinger's comments have prompted calls for governments to take a larger role in regulating the technology. Darren Jones, a former chief secretary to the UK’s Treasury under Prime Minister Keir Starmer, called for a multinational treaty governing the development of superintelligent AI, arguing that countries need to work together before the technology advances beyond their ability to respond.

Per CBS, the US House of Representatives introduced a bill called the AI Kill Switch Act, which would give Congress authority to switch off AI models that pose a threat to the public, following the OpenAI and Hugging Face incident in July.

Related: Anthropic's New Model, Mythos, Is So Dangerous It Isn't Being Released to the Public