Toby Walsh's five routes from AI progress to human extinction

Five routes to catastrophe

The first danger is epistemic. To predict what a superintelligence might do to less intelligent beings, we might need to be superintelligent ourselves. Asking humans to imagine its capabilities could be like asking a family dog to imagine a thermonuclear war.

There are still reasons to think superintelligence has not arrived. Current models perform individual tasks well, but that does not mean they outperform humans in every domain. Recently, however, AI solved one of the seven most difficult known mathematical problems. It is reportedly approaching others, leaving less room for optimism.

Oxford philosopher Nick Bostrom's classic thought experiment imagines a superintelligent AI instructed to optimize the production of paperclips. It does so with perfect efficiency, converting all available matter—including people, planets and stars—into raw material for its chosen product.

The system does not hate humanity. It simply notices that humans are made of atoms that could be used more efficiently elsewhere. This is what a badly specified goal looks like when pursued without limits.

The weakness of the scenario is that it confuses intelligence with power. A system may understand how to build paperclip factories without being able to obtain construction permits, overcome court challenges or push past public opposition. Environmental activists could stop the demolition and construction before the plan began.

Data centers are already a real-world version of the paperclip problem: large computing systems competing with other claims on land, energy and infrastructure. People are becoming more resistant to turning the planet over to them.

A superintelligent AI could create and release a dangerous biological weapon. That possibility appears in the AI 2027 scenario prepared by the nonprofit AI Futures Project, which forecasts the consequences of advanced AI development.

Last month, researchers at Stanford University reported synthesizing 16 new viruses with a genetic language model. They sent genetic sequences to a mail-order laboratory and received the viruses back in test tubes. The experiment cost no more than several hundred thousand dollars.

Yet destroying humanity with a new virus would be extremely difficult. It would need to spread easily and rapidly. Biology imposes a constraint: highly contagious viruses are usually less deadly, while viruses that kill a large share of those infected generally spread less effectively because most victims die before infecting others.

COVID-19 killed less than 1% of humanity. The Black Death, considered the deadliest pandemic in history, killed more than a third of Europe's population in the 14th century. Even plague would be less dangerous today because of advances in medicine and sanitation.

If an AI gained access to the chain of command for nuclear forces, it could potentially trigger a nuclear war. Over the past 50 years, humanity has come close to nuclear conflict several times because of mistakes.

Nuclear weapons command systems are generally considered isolated from the internet. But in 2010, the Stuxnet computer worm disabled Iran's nuclear centrifuges, reportedly entering the system through a USB drive. AI could also feed militaries false intelligence, pushing decision-makers toward irreversible action.

Nuclear arsenals are smaller than they once were, but they may still be large enough to kill perhaps half of humanity. The main cause of mass death would not be the blasts themselves, but the famine that followed a nuclear winter.

The most probable danger may come from people. AI could accelerate mass unemployment, fill the information environment with disinformation, deepen political divisions and damage human relationships through artificial companions.

Society could gradually come apart. Over time, people might stop maintaining life at a human scale.

Intelligence is not the same as control

The paperclip example is useful precisely because it exposes a gap in many extinction arguments. A system can be extremely capable and still face institutions, infrastructure and public resistance. The biological scenario has a similar gap between making a virus and making one that spreads widely enough to destroy humanity.

My reading is that the most credible risk is not a single machine suddenly overpowering the planet. It is AI amplifying existing human failures faster than institutions can contain them.

What this framing leaves relatively quiet is the middle layer between capability and catastrophe: who grants access, who checks the output and who can stop a system before a bad decision becomes irreversible. That is where the scenarios become less cinematic and more difficult. I would want to know less about whether an AI secretly hates people than about how many ordinary actors would have to approve its actions before anyone could intervene.

Walsh concludes that there are real reasons for concern, while arguing that excessive fear is not justified. He is a professor of AI and leads a research group at the University of New South Wales. He wrote God AI: Boom or Doom? What to Expect When Machines Outsmart Us. The original article appeared in The Conversation.