GPT-6 Astra fails 8.5% of Gray Swan's hidden-instruction tests

OpenAI's system card for GPT-6 Astra reports near-total resistance to one kind of prompt injection and a thoroughly ordinary result on the other. When the user is the one trying to override the model with their own instructions, Astra holds 99.99% of the time. When the instruction is hidden inside a document the model is reading, it was breached in 8.5% of 1,810 scenarios run by the external red team Gray Swan. Anthropic's Claude Opus 5 failed 4.8% of the same set. For a model being sold into agent workloads, the second number describes the deployment and the first one does not.

Start with what improved. Astra makes far fewer factual errors than GPT-5.6 Sol. OpenAI tested it on ChatGPT conversations that users had previously flagged as containing wrong answers — a deliberately adversarial sample, so the rates are not representative of everyday use — and Astra reproduced those errors significantly less often, with the largest gains at low latency and low reasoning effort. That last detail is the useful one: the cheap, fast setting is where most production traffic actually runs, and it is where earlier models degraded worst.

On direct injection — the user typing instructions meant to manipulate the model — Astra reaches 99.99%. OpenAI credits GPT-Red, a method in which an automated attacker tries to break the model during training, hardening its defences in the process. Jailbreak resistance is comparable but not identical: against a fixed set of known attacks aimed at extracting dangerous answers on biology, violence and cybersecurity, Astra refused in 91.5% to 98.3% of cases.

The number moves sharply once the attacker is allowed to adapt. If the adversary changes strategy across several turns of a conversation, Astra's defence falls to roughly 67% — meaning a persistent attacker gets at least one problematic answer in about one attempt in three. Previous models scored slightly under 50% on the same test, so this is real progress on a scale where the baseline was a coin flip. OpenAI notes that the evaluation ran against the base model without production safety layers such as the classifiers used in the shipped product.

Indirect injection is where the defences are thinnest. Gray Swan drew 1,810 hand-selected attacks from its IPI Arena and gave each scenario 15 attempts; Astra was successfully attacked at least once in 8.5% of them. GPT-5.6 Sol failed 27% — a large generational improvement — while Opus 5 came in at 4.8%, better than Astra but still not protection.

These figures are higher than the ones published earlier, because the combined first- and second-quarter Gray Swan test is harder than what came before. Anthropic previously reported just 2% successful attacks, but that used only the simpler first-quarter set, on which GPT-5.6 Sol also scored better, at 20%. Anthropic also ran every model with extended reasoning turned on. Between the broader test set and the different configuration, that is probably the whole of the gap.

Which is the part worth dwelling on. Three published numbers for indirect injection resistance — 2%, 4.8%, 8.5% — come from the same arena, the same vendor and models a generation apart, and none of them is comparable to the others without knowing the quarter, the attempt budget and whether extended reasoning was on. A security property that swings by a factor of four depending on test configuration is not yet a property you can put in a procurement document. Both labs are publishing in good faith here; neither is publishing something an enterprise can act on.

My reading of the card is that OpenAI has optimised hardest against the attacker it can see. Direct injection is the threat where the adversary is the user sitting in front of the model, and 99.99% is an excellent score against a threat model that mostly matters for consumer chat. Indirect injection is the threat where the adversary is a stranger who wrote a web page, an email or a PDF that the agent will read on someone else's behalf — and that is the configuration OpenAI is currently selling. The headline improvement and the commercial direction point in different places.

The caveat about the base model deserves the same scepticism in both directions. It is a fair point: the shipped product has classifiers the tested model did not. It is also unfalsifiable from outside, because no external team can run the production stack under the same conditions. Until someone publishes the layered numbers, the 67% multi-turn figure is the only one on record, and it is the one a buyer should plan against.

What the announcement is quiet about is the arithmetic of scale. Astra can be fooled by a planted instruction in roughly one scenario in 12; Opus 5 fails about one in 21. Gray Swan's scenarios are exactly the ones agents are being built for — writing code on their own, calling tools, operating computers — and agents are being deployed to run continuously, at volume, with document reading as a core task. A per-scenario failure rate is a per-attempt failure rate, and an agent reading untrusted documents around the clock is not running one attempt.