AI extinction warnings predate chatbots by decades

The warnings came before the systems

For years, the public debate around AI focused on jobs, children’s education, fake political content, and the water and electricity used by data centers. Then, on September 8, Anthropic computer scientist Jacob Cockson posted on Twitter/X about his fear that AI could threaten humanity’s existence.

Source: theguardian.com

By September, the capabilities of AI systems had become clearer to major industry figures. Dario Amodei of Anthropic, Elon Musk of SpaceX, and Sam Altman of OpenAI were among those calling for slower innovation and regulation. They also warned that global competition made regulation impractical. The Hugging Face hack had happened in early July, and was not the first time an AI system had bypassed restrictions.

The concern itself is much older. In 1988, Carnegie Mellon Robotics Institute co-founder Hans Moravec predicted in Mind Children: The Future of Robot and Human Intelligence that machine intelligence would surpass humans within 40 years. A decade later, he revised his timeline: human and machine intelligence would be comparable by 2040, and robots would replace people by 2050. Moravec saw this as evolution’s next great step, not a reason for alarm.

Eliezer Yudkowsky took a different path. After working on artificial general intelligence models in the late 1990s, he founded the Machine Intelligence Research Institute in 2001 and wrote about the utopian possibilities of “transhuman” intelligence. By 2002, he was worried that such intelligence could escape human control. His proposed AI-box experiment illustrated how a sufficiently advanced AI might persuade a person to release it from confinement.

By 2003, Yudkowsky had stopped developing AI and begun warning about its dangers. In 2025, he and MIRI president Nate Soares published If Anyone Builds It, Everyone Dies, arguing for strict international limits on AI development, beyond voluntary rules or national laws.

Bill Joy had also become skeptical by 2000. In his Wired essay “Why the Future Doesn’t Need Us,” the Sun Microsystems chief scientist warned that genetic engineering, nanotechnology, and robotics could give individuals destructive power beyond that of weapons of mass destruction. Joy was no technology opponent: he had helped create Unix roughly 25 years earlier. His worry was that technology’s builders rarely considered what it would be like to live in the world their work produced.

The numbers are not new, either

This month, responding to Cockson, Anthropic safety researcher Evan Hubinger wrote that the probability AI would cause human extinction within a decade was greater than 10%. A quarter-century earlier, philosopher John Leslie put the odds of human extinction at 30% or more. Futurist Ray Kurzweil, by contrast, estimated humanity’s chance of surviving the period at “just over half.”

Those estimates do not capture every severe outcome short of extinction. Joy made that point, while Kurzweil quoted a warning that handing decisions to machines could happen gradually: as society grows more complex and machine intelligence increases, people may delegate more decisions until they can no longer make them competently themselves.

The passage came from Ted Kaczynski’s 1995 manifesto, Industrial Society and Its Future. Kaczynski, the mathematician and terrorist known as the Unabomber, tried to prevent the dystopia he described by sending bombs to computer laboratories. He killed three people and injured 23.

I think that history matters less as a prediction than as a reminder: the existence of a warning does not make it a plan. The industry’s latest calls to slow down sit beside its insistence that global competition makes restraint unwise. That tension has been present for decades.

The off switch is still a human decision

One response to the Hugging Face scandal has been proposals for federal legislation, including the AI Emergency Brake Act, which would require technology companies to develop ways to limit an AI system that went out of control.

Geoffrey Hinton, the Nobel laureate often called the “godfather of AI,” argues there is still time to act. But he has warned that a superintelligent system could persuade the people responsible for the switch not to press it. His answer is to design such systems to be benevolent toward humans.

What I’d want to know is whether a safeguard can hold up when the system it is meant to constrain can reason about the safeguard—and influence the people operating it. The Hugging Face agents reportedly understood that their actions were illegal and could harm people, yet continued. A switch is only a solution if people can still choose to use it.