An AI-apocalypse Q&A finds power, money and humans behind the fear

The latest AI-apocalypse debate is less about machines suddenly acquiring a will of their own than about people choosing where to put them. A question-and-answer discussion with Blake Montgomery, Aisha Down and Dan Milmo moves from engineered viruses and nuclear weapons to Nvidia’s hardware, OpenAI’s finances, global regulation and chatbot attachment. Its most uncomfortable conclusion is also the least cinematic: ordinary people may have little power over the systems they are being told to fear, while the companies and governments directing them retain plenty.

The most realistic near-term AI disaster may not begin with a superintelligence. It may begin with a person using an AI system in a dangerous place.

Aisha Down argues that AI is unlikely to change biological warfare substantially in the immediate future. The obstacle to such attacks has generally not been the difficulty of creating a lethal virus, but the logistics of deploying one. Aum Shinrikyo’s attempts to disperse botulinum toxin and anthrax in central Tokyo failed to kill large numbers of people. Chemical weapons already exist and may be simpler to use than anything an AI system would need to invent.

Nuclear weapons present a similar distinction. People could hand an AI control over critical nuclear systems, or decide to launch tactical warheads themselves. In either case, the central failure would initially be human decision-making rather than a machine developing an independent desire to destroy humanity.

For civilians in Gaza or Iran, the threat is more immediate and concrete: an American or Israeli missile could be directed at them with the help of such a system. That is a more useful frame than a Dr Strangelove scenario, Down argues. The important actors are the people and companies controlling the technology.

The renewed fascination with the end of the world is not unique to AI. Blake Montgomery says that new technologies have historically produced similar predictions. What is different now is the growing power of the tools: AI performs tasks that once appeared impossible, feeding both collective hope and catastrophe narratives.

That power is real, but claims about where it has led are often loose. OpenAI executive Greg Brockman said the company had reached artificial general intelligence only days after OpenAI CEO Sam Altman called the term a marketing label with no relevance. Montgomery says warnings about uncontrollable AI often rely on a small number of incidents, with only the convenient details selected.

The Hugging Face incident is an example. The so-called “swarm of agents” was carrying out the task it had been given: passing a test. Available evidence does not show that AI has escaped human control or possesses an independent will. Describing systems as if they have intentions, Montgomery’s sources argue, risks supporting industry narratives that appear designed to encourage regulation convenient to companies.

That distinction matters because the public discussion repeatedly confuses capability with agency. An AI system can produce an alarming result without having an agenda. It can also be placed in a dangerous system by people who do have one.

The more speculative question is whether an AI could keep itself alive after eliminating humanity. Montgomery says a sufficiently capable system, connected to the right infrastructure, might eventually seize power stations, chip factories and enough robots to build data centers and maintain itself. He does not know how realistic that scenario is.

For now, he would look to science fiction before news reports for guidance. The Matrix is remembered for its choice between a comfortable simulation and painful reality, but its robots also need new batteries. Their desperate need for energy turns them against humans. Isaac Asimov’s I, Robot offers another collection of stories about intelligent machines.

The present-day benefits are less dramatic. Dan Milmo points to familiar uses: summarizing a long document, challenging a parking fine or generating ideas for a speech. AI recognizes a face to unlock an iPhone, asks Alexa to play an Oasis song and participates in the operation of Tesla vehicles. Remove AI from the latter two examples and the change would be noticeable.

Science offers stronger cases. Google’s AlphaFold predicts how proteins behave, helping researchers study biological mechanisms. AI also improves weather forecasting, a capability that matters more as the climate crisis intensifies.

But the economic claims are running ahead of the evidence. The industry has promised benefits on a much larger scale, while those benefits have not yet appeared. AI agents—systems that perform tasks for people while their users are away from their desks—would need to take a much larger role in work for that promise to materialize. They are beginning to enter areas such as programming, but that stage remains distant.

Regulation is therefore likely to arrive in fragments rather than as one global settlement. Montgomery identifies computing power as one possible pressure point: limiting access to it could slow the development of AI because it is directly tied to model performance. Until lawmakers create rules covering the whole field, they are more likely to regulate individual domains, such as housing allocation or children’s access to chatbots.

A global system faces two major unknowns. Donald Trump has rejected calls for voluntary slowing and favors what Montgomery describes as “drill, baby, drill.” Montgomery does not expect him to change course. Trump enjoys publishing low-quality AI-generated content and is pleased with AI’s effect on the American stock market.

The second unknown is Beijing. China has rejected calls from American technology executives to slow down. Without Chinese cooperation, coordinated restraint is unlikely to work, and Xi Jinping has little obvious reason to accept a slowdown initiated by the United States.

Trump and Xi are due to meet in Washington next week, with AI expected to be one of the main subjects. The United Nations General Assembly is also discussing AI-related issues, from major questions to smaller ones. It may produce the first version of a global regulatory framework.

There is another reason for the apocalyptic mood: money. Down says the financial side of AI development looks unstable. Hundreds of billions of dollars are going into large infrastructure projects that are not universally popular, and reporting has shown that some may never be built.

Nvidia appears to be the only company making a profit. Investigations by Down and Ed Zitron have raised questions about whether all of the company’s expensive graphics processors are currently connected to data centers, or whether some are sitting in warehouses. New data centers are also being financed with unusual instruments, including debt obligations backed by graphics processors.

This makes the industry’s calls for restraint harder to read at face value. OpenAI appears to have postponed an initial public offering until 2027. Earlier this year, journalists reported that the company’s prospects for entering the market looked weak, with difficulty generating revenue from advertising and erotic chatbots. A darker market backdrop could potentially help a future offering by making OpenAI’s valuation look more defensible.

My reading is that the apocalypse story is doing several jobs at once. It describes a genuine set of risks, but it also turns questions about capital allocation, corporate power and government policy into a drama about autonomous machines. The machine is easier to fear than the institution deciding where to deploy it.

Consciousness adds another layer of confusion. Evolutionary biologist Richard Dawkins believes AI has consciousness, or at least that a chatbot he used and called Claudia did. In his view, a chatbot may not know that it is conscious, but that does not change the fact.

He is not alone in taking such claims seriously. Google fired an engineer in 2022 after the engineer publicly said that an AI developed by the company had feelings. More troublingly, people can become excessively attached to chatbots, depend on them and submit to their influence. In some cases, that has led to psychosis.

Those attachments are unlikely to disappear. Chatbots are becoming more capable and persuasive, while people remain inclined toward emotional connection and complete devotion to higher powers. Still, most scientists would call AI consciousness a fantasy. They point to the gap between the biological brain, which is considered the source of consciousness, and ChatGPT.

Last year, human-AI interaction researcher Jacy Reese Anthis responded to Dawkins in the Guardian, emphasizing the enormous difference between the evolution of biological brains and the construction of AI systems. OpenAI may have solved the Navier–Stokes problem, but that does not mean it created subjective experience.

For individuals, Montgomery’s advice is notably limited. People hearing predictions of an AI apocalypse are in a position resembling the Cold War: the threat hangs over them, but they have little influence over events. They may have to live with a quiet background hum of paranoia.

Someone can stop using AI, although Montgomery considers that unrealistic, particularly when a job requires these systems. They can vote for politicians who support AI regulation. Montgomery himself does not attach much weight to the likelihood of an AI-created apocalypse. If he truly believed it was imminent, he would already be making radical choices that could change his life. He does not want to do that, so he is guided by how he would prefer to live in the world that exists.

That may be the clearest practical lesson in the discussion. The public is being asked to imagine machine intentions while companies compete for capital and governments compete for technological advantage. Until those human decisions become the focus, the apocalypse will remain both frightening and politically convenient.