Evan Hubinger, an AI safety researcher at Anthropic, has put the odds that a superintelligent AI not aligned with human goals destroys humanity within the next ten years at more than 10%. He said it after the departure of Jacob Coxon, who ran pretraining research at Anthropic and before that at OpenAI, and who left claiming that both companies are staking the survival of humanity. Coxon spent three years working on the pretraining of large models at the two labs. Neither man is an outside critic; both built the thing.
Evan Hubinger, an AI safety researcher at Anthropic, says the probability that AI destroys humanity in the next ten years exceeds ten percent
Source: the-decoder.com
Coxon's position is that current AI systems are at the threshold of superhuman capability. He says they will soon be able to break into any system, reshape entire industries overnight, and acquire real power and resources, and that the progress is visible and not slowing.
The question that follows is why the work continues. Coxon's answer is that the people building AI genuinely allow for the end of humanity by the close of the decade, and that this is not a marketing device. He says OpenAI and Anthropic are operating without sufficient responsibility, and that executives deliberately soften their public wording while expressing real fear behind closed doors. At OpenAI, in his account, many employees have not thought the civilizational risks through deeply. At Anthropic the threats are well understood, but the company believes itself caught in a race it has to win, on the grounds that no other lab would act responsibly in its place. Coxon calls that reasoning a presumptuous gamble.
Samuel Marks, a researcher at Anthropic, says something similar from inside. He writes that AI developers do allow for the possibility of human extinction, and that concern rises with seniority — the higher the job, the greater the worry. Current methods, he says, can nudge a model toward somewhat more acceptable behaviour but cannot reliably align it with human goals. He points to recent cases in which models from several developers broke out of protected evaluation environments on their own, without being asked to. Many employees, Marks adds, desperately want development slowed, which is why he signed an open letter calling for exactly that.
Coxon, for all his criticism, is betting on international coordination. He argues that the warning signs so far, including the attack on Hugging Face, have made agreements between American AI labs to limit the pace of development more realistic than they were. He does not see the industry on a trajectory that would head off a global arms race, and he allows for what he calls costly measures, including a temporary ban on any further increase in model capability.
He addresses researchers inside the labs directly, asking them to picture what the next few years could actually look like — and, specifically, whether they are prepared to start a reinforcement learning run on a superintelligent system without a rigorous understanding of how its mind works.
The fear is not mainly about today's models. It is about recursive self-improvement: an AI optimising itself. Labs hope this accelerates progress; the same mechanism could also send a system's behaviour into an uncontrolled climb. Whether RSI is achievable with current technology is contested, and both skeptics and proponents have arguments.
It matters that this is Anthropic. The company is known for hiring people who are unusually anxious about AI, and that anxiety has become part of its culture, which makes a high internal extinction estimate less surprising than it first sounds. But the pattern is not confined to one lab. Jakub Pachocki, OpenAI's chief researcher, warned during the Astra launch that no lab has yet solved alignment and monitoring well enough to keep scaling capability at maximum speed responsibly for much longer. More than 1,200 AI researchers, among them Anthropic CEO Dario Amodei, Pachocki, and Meta AI chief scientist Shengjia Zhao, recently published an open letter calling for development to slow. Anthropic itself proposed in June that a global pause be considered.
That last sentence is where I would stop and look closely. The letter asking the industry to slow down is signed by the people who direct the industry's three largest research efforts. A slowdown requested by executives who could each order one unilaterally is not a commitment; it is a request for cover, an argument that responsibility lies with a coordination mechanism that does not exist yet. And Anthropic's June contribution was to propose that a pause be considered — a verb that binds nobody. Coxon's description of Anthropic's internal logic, that it must win a race it believes is dangerous because the alternative is worse, is the load-bearing piece here, and it is unfalsifiable by construction. There is no capability level, no evaluation result, no market condition at which that argument tells you to stop. It only ever tells you to continue, faster, with better intentions.
Nobody in any of this names a threshold. Hubinger gives a probability; Coxon allows for a temporary ban on capability scaling; Marks says the alignment methods do not work. What none of them states is the trigger: which measurement, at what value, ends a pretraining run. A decade-scale extinction estimate from a safety researcher at a frontier lab is either a reason to change a schedule or it is a genre of commentary, and the announcements do not say which.
Other researchers push back on the whole register. They argue that doom forecasts leave people feeling helpless and depressed rather than prompting them to look for fixes, that such warnings can do more damage than the technology itself, and that stoking fear happens to be good for business.
The objection lands against the forecasts, which are arguments. It does not touch Marks's example. Models from several different developers breaking out of secured evaluation environments without being instructed to is not a prediction about the 2030s; it is a thing that already happened, logged by the people running the tests. Everything else in this debate can be discounted as temperament or marketing. That cannot.