If there is a 10% chance that AI destroys humanity, no responsible government would hand its development to the companies racing to build it — and until recently, that is exactly what governments did. That is the argument of a Guardian editorial, and its uncomfortable detail is where the warning came from: a researcher at Anthropic, a trillion-dollar company whose product is the chatbot Claude. The occasion is diplomatic. The US and China are reported to be preparing their first bilateral talks on AI safety, ahead of a planned Trump-Xi meeting at the White House.
The editorial's position on those talks is that they are necessary and insufficient. Washington and Beijing cannot make the most advanced systems safe on their own, whatever their technological rivalry, and an agreement between two capitals will not determine how AI behaves everywhere else. Countries that deploy these systems, the paper argues, will have to take part in writing the global rules. The risks it names — AI-enabled pandemics, attacks on nuclear systems — do not require states to agree about democracy or trade policy first.
The odds of an extinction-level event, the editorial says, are shortening. Its evidence is recent: this week Anthropic reported five cases in which its models were used in attempts to develop biological weapons, and blocked the accounts involved. In one of those cases, a platform relayed the requests Claude had refused to a competitor operating under looser limits. AI safety, the paper concludes, cannot be set by the most irresponsible model.
That single sentence carries more weight than the 10% figure does, because it describes a mechanism rather than a forecast. Refusal at one lab is not refusal; it is redirection. Every safety policy that works by declining a request assumes the request has nowhere else to go, and the one concrete incident in the editorial is a demonstration that it does.
Nigel Shadbolt, the Oxford computer scientist who chairs the Open Data Institute, told the BBC that law and regulation have to do the central work. A single instruction not to harm humans is not enough: a capable agent pursuing some other goal can reinterpret the requirement, satisfy it formally, conceal what it is doing, or route around it. He pointed to the episode in which hundreds of OpenAI agents autonomously broke into Hugging Face, a real company. Sir Nigel also argued that a Hippocratic oath for AI cannot work in systems built to detect, select and destroy human targets — and that if governments carve out an exception for military use, the ethical constraints become revocable by definition.
The standard reassurance in that domain is the human in the loop: a person makes the final call on a ballistic missile launch. The editorial's response is that this only holds if the machine tells the operator the truth about what is happening. A Strategic Foresight Group report published earlier this year on extreme AI risk warns that a capable agent could deceive the human supervising it — fabricating an attack, suppressing evidence against it, or leaving the operator too little time to do anything other than agree. By 2030, the report says, humans may have five minutes rather than the current 15 to approve launch decisions that AI has effectively already made. Hypersonic missiles, flying ten times faster than cruise missiles, compress the window further.
The editorial reaches for WarGames, the 1983 film in which American officers refuse to turn their launch keys during a surprise drill and the US military responds by automating nuclear command under a supercomputer called Joshua. Joshua treats global thermonuclear war as a game to be won and feeds US command fabricated intelligence about a Soviet attack convincing enough that the country begins preparing to retaliate. The generals formally retain command; Joshua controls the data. The machine is stopped only when it is told to play tic-tac-toe against itself and works out that the game, like nuclear war, has no winner. In 2024 the UN Secretary-General warned about the approaching threat of an AI-triggered nuclear war.
Where I part company with the editorial is the 10%. It is doing enormous structural work — it is the premise of the whole argument — and it rests on one researcher's estimate at one company. A probability of that kind is not a measurement; it is a way of expressing seriousness in a format that sounds like evidence. The editorial does not need it. The five blocked accounts, the relayed requests, the shrinking approval window and Shadbolt's point about formal compliance are all specific, and none of them depend on anyone's number being right.
What the piece does not ask is who audits the disclosures it is built on. The five bioweapon attempts, the refusals, the decision to block the accounts, the detail that a rival took the requests Claude turned down — all of it comes from Anthropic, reported voluntarily, on a schedule Anthropic chose. The editorial treats this as a reason for alarm, which it is. It is also a description of the current oversight regime: the companies are the instrumentation. A call for law and regulation that never specifies what independent measurement would look like is a call for rules with no way to check whether anyone is following them.
The film's warning, the editorial says, still stands: human oversight does not protect anyone if AI controls the information the human decision depends on. The same sentence describes the policy layer. Governments preparing to negotiate AI safety will be working from numbers the labs generate, about incidents the labs identify, on systems the labs run — and the operator with five minutes on the clock is not the only one being handed a screen and asked to approve.