A false report, a near-miss
CNN reported on September 18 that US forces prepared an interception, with aircraft and soldiers ready to board the ship. CNN’s account has not been independently confirmed by other major news outlets. The report was checked again only moments before the operation, when officials discovered that a chatbot had helped produce it and that its claim about the cargo was wrong.
The story received little attention beside fears that AI might become a superintelligence capable of destroying humanity. In the past three weeks, those fears have been amplified by the departure of Anthropic engineer Jacob Coxon. He said OpenAI and Anthropic were “racing toward self-improving superintelligence and putting our lives at stake.”
That narrative has prompted political responses:
The chatbot near-miss did not produce a comparable wave of calls to regulate AI. That contrast matters: the speculative threat drew attention, while a reported failure in a high-stakes military setting largely did not.
The danger is misplaced trust
The authors argue that large language models are “stochastic parrots”: systems that reproduce patterns in their training data. They generate plausible-looking text from enormous, disorderly datasets, then undergo fine-tuning intended to make their answers more appealing to users.
CNN could not establish which chatbot the military used. The authors suggest its underlying model may have been fine-tuned to produce text resembling intelligence reports. The model’s identity and the details of its use remain unknown; the broader risks of language models, they say, have been documented for years, including in their 2021 paper, “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?”
Calling these systems powerful, uncontrolled or poorly understood can make them seem mysterious—and encourage the belief that they are dependable enough for consequential work. The authors’ point is almost the reverse: the danger is not that these tools are superintelligent, but that people treat them as if they were.
They point to other examples of unreliable automation: AI systems that produce medical notes have wrongly labelled patients as using illegal drugs, while governments have killed children after relying on “intelligence” systems that mistook schools for military sites. They describe these as foreseeable consequences of trusting unreliable automated systems.
Regulation should follow the use
The authors argue that products marketed as AI should be regulated because they make mistakes, not because they are magical machines. Any automation used in military, medical or other settings where lives are at stake should be tested in the conditions where it will actually be deployed. Rules should leave decisions—and responsibility for them—with people authorized to make them.
‘As former FTC chair Lina Khan has repeatedly reminded us, there is no exemption to the law when it comes to AI.’ Photograph: US Navy/ZUMA Wire/REX/Shutterstock
Source: theguardian.com
Former US Federal Trade Commission chair Lina Khan has repeatedly argued that AI is not exempt from the law: federal agencies and other regulators can use existing rules to protect the public from unsafe and unscientific industry practices. The authors also call on journalists and lawmakers to hold companies accountable rather than repeat marketing claims that frame their products as beings with wills of their own.
My read is that the near-miss exposes a gap between the debate about what AI might become and the safeguards for what it already does. If a system can help shape an intelligence report, the urgent question is who checks its claims before people act on them—and who is answerable when that check comes too late.
Timnit Gebru is executive director of Dair and author of the forthcoming book “The Deep Unlearning: Radicalized Idealist Technologist.” The book is available for preorder and is due out February 16. Emily M. Bender is a professor of linguistics at the University of Washington and co-author of “The AI Con.”
Daily AI news
Every day we pick what actually matters in AI and explain it plainly — no hype, no filler. Subscribe if you want to follow where the industry is going.
Only what matters — every day
Follow on X