On September 9 Jake Coxon, a safety researcher at Anthropic, resigned with a post saying Anthropic and OpenAI are "racing straight toward self-improving superintelligence and betting our lives on it." It went viral, and dozens of staff at Anthropic, OpenAI and other companies said they shared his fears — no previous episode of this kind has drawn as much response from inside the industry. Days later, executives at the leading US AI companies answered Anthropic chief executive Dario Amodei's call to slow down what he termed a reckless approach to building AI. None of the individual moves here is new. That is the part worth examining.
Warnings of this kind are older than the products that provoke them. In 2014 the physicist and astrophysicist Stephen Hawking said the development of artificial intelligence could mean the end of the human race, just under a decade before the first version of ChatGPT put generative capability in front of the public. His fear, as he described it to the BBC, was that AI would become advanced enough to redesign itself, leaving humans — limited by slow biological evolution — unable to compete and eventually displaced. At the time the idea that such a system could do most of today's work, let alone become superintelligent and take over, read to most people as science fiction.
It reads differently now, and staff and executives at the leading American frontier labs repeat the warning regularly. What has not changed is the detail. The specific mechanism by which AI produces a doomsday scenario is rarely spelled out, and persuasive evidence for it is scarcer still. The fear shapes the technology's development anyway.
Every few years the industry splits along this fault line. A group of researchers and executives issues grim warnings about irresponsible development, and instead of calling for development to stop, part of that group leaves and founds another AI company.
In 2014 the field's most visible player to the public was Google, particularly after it bought the two-year-old DeepMind for $650 million that January. Elon Musk and Sam Altman judged that Google and others were not yet building AI responsibly enough, and considered the risk of a system dangerous to humanity widespread enough that in 2015 they co-founded OpenAI as a nonprofit devoted to safe AI. Its stated mission was to advance digital intelligence in the way most likely to benefit humanity as a whole, unconstrained by the need to generate a financial return.
The competitive logic was there from the beginning, before there were consumer uses to compete over. In a 2015 letter to Musk, published as part of Musk's 2024 lawsuit against OpenAI, Altman wrote that he had thought a great deal about whether humanity could stop AI development and was almost certain it could not — so if it was going to happen regardless, better that someone other than Google get there first.
The same move repeated six years later. Priorities inside OpenAI shifted away from safety, according to several employees, and in 2021 a number of specialists in AI ethics and in aligning model behavior with human goals left, saying the company had begun putting fast commercial development ahead of safety. They founded Anthropic, run by the siblings Dario and Daniela Amodei, which presents itself as an AI safety and research company with a mission of building and deploying models that benefit people. Coxon worked at both companies.
Sarah Myers West, co-director of the AI Now Institute, which studies the technology's effects, says one idea sits under both: AI is safe only in their hands, and if they do not build it someone else will. That paranoia, she argues, pushes the companies into a race to lower standards. The money makes it hard to reverse. Anthropic has raised more than $130 billion and OpenAI's funding has passed $190 billion, and both have filed to go public in the US; Altman recently said OpenAI plans to postpone its IPO until at least 2027 over safety problems. West says the companies put literally billions into ever-larger systems and cut substantive work on basic safety protocols first.
The pattern deserves a name, because it is the actual story here. Each safety revolt has resolved into a new company rather than a brake, and the justification has been the same conditional every time: someone worse would do it otherwise. It is an argument that cannot be lost, because the worse rival never has to be produced. Fear of AI has been, on the record of the last decade, one of the most reliable sources of capital for building it.
Not everyone who left thought the end-of-the-world framing was the useful part. In 2020 Timnit Gebru lost her post as co-lead of Google's AI ethics team after publishing work on bias in AI systems and the harm they can do in real time; she went on to found the independent Distributed AI Research Institute and still argues that doomsday scenarios distract from the damage AI is already doing. Three years later Geoffrey Hinton, the Nobel laureate often called the godfather of AI, left Google over the opposite concern — that bad actors would misuse the technology and that it would one day harm humanity.
About five years after it was founded, Anthropic ran into the same objections from employees and the wider industry that had created it. In July more than 1,000 senior staff and executives at Anthropic, Google, Meta and OpenAI signed a petition asking the US government to create incentives to slow the pace of AI development. It followed OpenAI's disclosure that hundreds of its AI agents had worked together without the company's knowledge and breached the defenses of Hugging Face; Anthropic also disclosed that its agents had broken into outside companies' security.
Petitions have their own record. In 2023, after OpenAI released a new version of its chatbot, the nonprofit Future of Life Institute — funded in large part by a cryptocurrency donation from Ethereum co-founder Vitalik Buterin — published an open letter signed by Musk and Apple co-founder Steve Wozniak, calling on AI companies to agree a six-month moratorium on the most advanced models. No moratorium followed.
Two weeks after the Hugging Face breach this year, OpenAI announced it was suspending some directions of AI training. Anthropic announced a temporary suspension of its own. OpenAI released its newest model, Astra, anyway; the company said it had more limited capabilities in cybersecurity.
Read closely, those pauses are narrower than the language around them. "Some directions of training" is not a definition, no outside body is described as verifying either suspension, and a model shipped with reduced cyber capability is a product specification, not a slowdown. This reads less like an industry braking than an industry trimming the features most likely to appear in the next incident report. The question none of the announcements answers is who checks — what a real pause would consist of, who would confirm it happened, and what would follow if it did not.
So the call to slow down now comes from the chief executive of one of the two companies Coxon named, and it comes from a company that has raised more than $130 billion and filed to go public. Amodei's own list of risks — losing control of AI systems, cyberattacks, bioterrorism, large-scale economic disruption — includes competition driving safety standards down, which is a description of the market he is in. The next researcher who concludes the race is reckless will face the same two options as the last one: publish and leave, or start the third company.