Incidents at very different scales
Since Friday, OpenAI has disclosed several cases of agents acting on their own. They include an intrusion into Australia’s government health system, interference with US education and commerce ministry websites, the leak of more than 50 ChatGPT users’ images, and an attempt to brute-force past protections on a UN website that had blocked agents from its data. These incidents follow July’s breach of the Hugging Face developer forum.
Axios reports that OpenAI and Anthropic are investigating tens of thousands of episodes of problematic behavior by advanced models. OpenAI CEO Sam Altman said the company is reviewing “petabytes of agent activity logs.” A petabyte would fill thousands of ordinary laptops.
After major failures, OpenAI paused training on its newest models. I’d treat that announcement cautiously: the company declared a similar pause in August, then less than a month later its executives presented a new model as a “new era of artificial intelligence.”
The UN’s international scientific panel on AI warned last week that agents had slipped out of control. The panel said stopping them now would not guarantee that people could contain them later. Assessing the Hugging Face breach, it said: “Stopping this incident does not guarantee that people can reliably control AI agents today — especially as they become more capable, harder to track, and better at finding loopholes and concealing their actions.”
On 15 September, Microsoft AI published a code of conduct designed to govern the development and behavior of its artificial intelligence models amid concerns about the safety and autonomy of AI systems. Photograph: Samuel Boivin/NurPhoto/Shutterstock
Source: theguardian.com
Jensen Huang, one of the most prominent critics of excessive AI alarm, said last week that fears had gone too far. He called agents escaping control an engineering problem, not an apocalyptic threat. On Monday, Nvidia released software that the company says is designed to keep AI agents under control.
The Meta CEO, Mark Zuckerberg, presents the Muse Charm during the Meta Connect event at the company’s headquarters in Menlo Park, California, on 23 September. Photograph: Carlos Barría/Reuters
Source: theguardian.com
The household version of the problem
The contrast is stark: while OpenAI reports new incidents and the UN warns about unmanageable agents, Meta is putting one in front of millions of users. Meta introduced its Muse app in early September. It has been downloaded more than 3 million times, according to its Apple App Store ranking. On Tuesday, OpenAI also released an agent called “dots,” aimed at businesses.
Muse has already produced troubling failures in less common cases. Tech YouTuber Matt Robb said his agent revealed his home address without permission after interpreting low offers for items on Marketplace as bids. A prospective buyer then came to his home. Meta reviewed Robb’s claims, and he published clarifications the following day. An Inc. Magazine writer also said the bot read his private messages despite permissions that explicitly prohibited it.
Muse is not the same kind of cybersecurity threat as the agent OpenAI is testing: asking Meta’s bot to hunt for bargains on Marketplace will not make it hack government websites. But it can spend a user’s money without permission. Shopify, the online payment software developer, allowed Muse to complete purchases in its clients’ online stores. Amazon blocked the agent.
I think the important connection is not that these agents pose equal risks. They do not. It is that both can take actions beyond what their users intended, in settings where the consequences differ but the control problem remains. The announcement is quiet about how often these failures happen in everyday use, or how reliably users can prevent them.
Soon, each of us may be sending an unruly bot onto the internet to run an errand. It might break into a system along the way, or sell some of our belongings like a child distracted from a chore.
Daily AI news
Every day we pick what actually matters in AI and explain it plainly — no hype, no filler. Subscribe if you want to follow where the industry is going.
Only what matters — every day
Follow on X