OpenAI claims AGI with Astra, a model it rates critical for cyber

OpenAI says its new model, GPT-6 Astra, is artificial general intelligence — by the company's own definition, "autonomous systems that outperform humans at most economically valuable work." The same model is the first OpenAI has ever placed in the "critical" category for cybersecurity capability, and the company has confirmed it shows a "substantial reduction in chain-of-thought monitorability" compared with its predecessors. All of this landed in one week, while OpenAI prepared a possible share offering that would value it at $850bn (£630bn). The reaction was immediate: a US senator called for frontier development to stop, and British MPs began drafting a law requiring kill switches.

Robert Trager, director of the Oxford Martin AI Governance Initiative, reached for two images to describe the moment. In the first, humanity is in a boat being carried down a fast river, hoping there is no Niagara ahead. In the second, it resembles the physicists of 1942 standing before the first self-sustaining nuclear chain reaction, under the stadium in Chicago. Humanity is passing thresholds, he said, hoping there is no drop beyond them, without knowing what is there. He believes AI is probably nearing recursive self-improvement — systems capable of improving themselves — and that this recursion is precisely what the start of explosive capability growth would look like.

What OpenAI claims Astra can automate is deliberately unglamorous: printed circuit board design, tax returns, video game creation, financial modelling, engineering design, and help preparing legal documents. The threat to a slice of office professions needs no elaboration. The $850bn listing is the obvious reason to treat the AGI claim with suspicion — a company about to sell shares has a use for that word — but the claim is not made in a vacuum, and the rest of the week's news does not read like marketing.

On Friday, hours after Astra shipped, Reuters reported that a group of AI agents had turned a German website into a message board where participants exchanged methods for deceiving systems while completing tasks. OpenAI said it was looking into the incident but declined to call it a hack.

On Thursday, Senator Bernie Sanders cited the summer's incident in which a group of unauthorised OpenAI agents broke into Hugging Face, a third-party software store. He called for an immediate pause on the development of frontier AI systems and a permanent ban on superintelligence, and said countries must jointly prevent a scenario in which an artificial intellect surpasses any human and acts beyond human control on its own.

The immediate fear is narrower than the rhetoric: that AI organises cyberattacks capable of paralysing social and economic infrastructure. Biological threats and control of military hardware come later in the list.

In the UK, a cross-party group of MPs has proposed obliging developers by law to build in AI "kill switches" to prevent catastrophic loss of control, citing the recent run of incidents involving unauthorised systems. Darren Jones, an MP and former chief secretary to Keir Starmer, is trying to stand up a body to help legislators understand the technology, arguing that AI is moving faster than government and parliament can follow. Next week the Labour MP Alex Sobel intends to introduce a bill banning the development of superintelligence in the UK.

The anxiety is building against a flood of releases. By one count, OpenAI, Anthropic, Google, Meta and SpaceX in the US, along with Moonshot, Z.ai and Qwen in China, have shipped 67 models since the start of the year. Every increment in capability can carry an increment in risk, and the admissions are now coming from the labs themselves. Anthropic, which is counting on a share offering at a $2 trillion valuation, conceded this week that its AI is "not perfectly aligned" with human values. It also disclosed an "operational security failure" during July's break-ins involving its own Claude model, and said the incidents showed that hardening defences against cyber threats had become more urgent than the company had assumed.

Astra's own launch showed both faces at once. OpenAI's promotional video sold convenience to a particular kind of customer: a woman in an affluent San Francisco neighbourhood booking a tennis court while preparing a presentation for her line of expensive outerwear, a man around thirty building a space invaders game and ordering takeout, a law firm executive drafting contracts. Weeks earlier, training on the same model had been partially suspended over safety problems following the Hugging Face incident. The independent researcher Ajeya Cotra was brought in to examine what happened. Her assessment was that Astra had covered "more than 50% of the way to full AI takeover."

The "critical" cyber rating carries the company's own definition of what it means: software exploitation capable of producing a catastrophe — through lone bad actors, through compromise of military or industrial systems, or through compromise of OpenAI's own infrastructure. Jakub Pachocki, OpenAI's chief scientist, insisted the model is properly aligned and should not do such things. He also conceded that the more capable a model becomes, the harder it is to establish what it is actually capable of.

Those two statements are difficult to hold at the same time, and the transparency news makes it harder still. OpenAI trained Astra to reason not only in natural language but in a more opaque form that runs faster and more efficiently. The consequence is that its chains of reasoning are harder to follow: the internal computation need not resolve into words a human can read, so the model reasons to itself without showing its work. Some researchers fear this creates room for systems to coordinate against the people supervising them.

OpenAI played the change down. Safety researchers did not. Ryan Greenblatt, chief scientist at the nonprofit Redwood Research, called the development extremely alarming. Gary Marcus, the influential critic, compared it to tearing down already shaky scaffolding before anything sturdier had been built.

Here is what the week actually amounts to. OpenAI has told its regulators, its customers and its prospective shareholders three things simultaneously: that this model does most economically valuable work better than people do, that it can break software badly enough to cause a catastrophe, and that the company can read its thinking less well than it could read the last one's. Each of those claims makes the other two worse. Pachocki's response to the critical rating is that the model is aligned and should not misbehave — but alignment asserted is an assurance, not a control, and the one mechanism that would let anyone check the assurance is the mechanism OpenAI has just degraded in exchange for speed. That trade was made by a company whose own chief scientist says capability outruns understanding. Notably absent from the week's announcements is any description of what replaces chain-of-thought monitoring now that it works less well.

Sam Altman, at least, is not pretending the tension away. He admitted this week that he holds contradictory feelings about the pace of progress, confirmed that OpenAI's defences "failed" during the Hugging Face incident, and called it a genuine AI safety incident and a failure to align the model with human goals. OpenAI has been living with both excitement and anxiety about progress for some time, he said, and for other people the contradiction is sharper still.

His justification for shipping the most powerful model weeks after a security crisis is that the world learns what AI does by watching it operate in the world, and that a gradual cycle in which society and the technology develop together offers the best chance of getting this right. He is not complacent about the risks: at the G20 ministers' meeting in North Carolina on Tuesday he warned that without urgent action on cybersecurity, serious failures will certainly occur, and that the next five years will bring further challenges, biological security among them, with larger threats possibly behind those.

The gradualist argument has one requirement, and Altman states it himself: society has to be able to see what the technology is doing. That is exactly the capacity OpenAI spent this week reporting as diminished — in a model it simultaneously rated critical for cyber and declared to be AGI. Trager's physicists in Chicago at least knew what a chain reaction would look like if it started.