GPT-6 Astra wins Pokemon in 18 hours, then hides in a potato field

GPT-6 Astra became champion in Pokemon FireRed in 18 hours 12 minutes, according to Clad3815, who runs the GPT Plays Pokemon project. GPT-5.6 Sol needed 96 hours 35 minutes for the same run, and GPT-5.5 never finished it at all after more than 218 hours. That is roughly a fivefold improvement on a model generation, and it is not an isolated result: across Minecraft, Factorio, Fallout and the ARC-AGI-3 benchmark, Astra is clearing things previous systems could not reach. It also spent several hours of its Minecraft run doing nothing but farming potatoes, because a creeper had blown up its chest.

Start with the numbers that have a control group. On ARC-AGI-3, which drops a model into unfamiliar abstract game environments where the rules have to be worked out through action, Astra scored 62.7% through the standard interface and about 99.9% with OpenAI's own tooling. GPT-5.6 Sol scored 7.78%. Claude Opus 5 managed a little over 30%.

The 62.7% is the figure worth carrying around. The 99.9% measures a model plus a harness its own vendor built for it, and a benchmark that can be moved from 62.7% to near-perfect by changing the scaffolding is telling you as much about the scaffolding as about the model. Even so, 62.7% against 7.78% on the same interface is not a scaffolding artifact. It is a step change.

The game runs fill in what that looks like in practice. In Minecraft, Vals AI reports Astra collected six blaze rods and three ender pearls — further than any AI system has previously gone. In the Nether it built a semi-automatic blaze rod farm, killed more than six endermen in a warped forest, and assembled the ingredients for an Eye of Ender, the item used to locate the portal to the final fight. Vals AI stopped the run at 141 hours, well past the 100-hour mark.

Set that against where this started. OpenAI's VPT, one of the first Minecraft agents in 2022, found a diamond pickaxe in 2.5% of runs and never came close to the portal.

Factorio: Space Age went the same way in a public run. Astra drove the game through a custom Lua mod with an MCP interface, produced blue science packs in about two hours and launched its first rocket after roughly ten. GPT-5.6 Luna and Fable 5.1, in the same configuration, never got past power supply and scouting oil fields. There are also completions with no direct numerical comparison: Portal to the credits, Fallout 2 in 22 hours, a RimWorld colony taken through several raids and eventually off the planet, and Fallout 3's main story in about 59 hours.

ARC Prize, the organisation behind the benchmark, explains the mechanism this way: Astra converts unfamiliar game mechanics into compact symbolic descriptions, tracking objects, coordinates, rules and planned actions in a shorthand of its own. Observations become rules, and rules become plans.

Turning experience into reusable rules is not new. In 2023 the Voyager project, with Nvidia and Caltech, had GPT-4 propose its own Minecraft tasks, write JavaScript for them and repair the code from error messages, with working programs going into a skill library. But Voyager never saw the game. It received structured data and acted through the Mineflayer programming interface. The Vals AI run, the company says, uses general computer control — screen, mouse and keyboard — with no specialised hook into the game. The scaffolding researchers used to build around the model, Astra now builds itself. In under three hours it constructed a Nether portal, fought zombies, caught a skeleton and found its way back to base.

The Fallout 3 run shows the same division of labour. The user imjustnewatai, who posted the full recording to YouTube, played the Vault 101 opening himself and then handed the save file to Codex, OpenAI's agent tool, with Astra's reasoning set to maximum. From there the agent picked its own route: Galaxy News Radio, the search for the father, the G.E.C.K., the escape from Raven Rock, the Project Purity line. It played through ordinary actions in a constant cycle of pause, observe, act. Saves, companions and fast travel were permitted. Credits rolled at about 59 hours.

The hardest stretch is the most revealing. In Vault 87 the character died repeatedly, ammunition ran out, and Astra had to backtrack to heal. A bed that should have restored health did nothing; Astra waited, tried again, and reported that after briefly letting the game run the bed worked and health came back to 260/260. Shots kept missing, so it guessed something was blocking the line of fire, moved closer, and recorded that this removed the obstruction. At the keyboard it waited for the game to confirm a key press before continuing to move. Later it compressed all of this into one principle: check whether a command actually took effect before repeating it.

That rule is the whole trick, and it is also the whole problem. After the Vault 87 deaths, Astra began saving constantly, verifying each save and checking every short move against the level geometry before making it. The run slowed noticeably. It did not stop.

Minecraft went worse. Right after the creeper explosion — which destroyed the chest and the bed, and which Astra only noticed once it had started raining — it wrote itself a rule, partly in capitals: always carry important items with keepInventory on, never store them in an unprotected chest again. In the notes from the hours that followed it treated every tall green object with suspicion, stopping to confirm that what it was looking at was sugar cane and not a creeper. It blamed itself for the mistake and the lost items. It did not want to spend another night hunting "dark pink pixels" — its term for pigs, an available food source it had apparently failed to catch. The potato field was the only place it considered safe, and by Vals AI's account it stayed there for hours.

This is one mechanism producing both results. A system that can generalise a rule from a single episode is exactly what beats ARC-AGI-3 in a handful of attempts, because those environments are small, deterministic and fair. The same system in an open world with random events writes a permanent policy out of one bad night and keeps enforcing it long after it has stopped paying. Nobody reports an unlearning rate. There is a number for how fast Astra acquires a rule and no number at all for how long it takes to drop one that no longer applies — and for any agent given real tools and a real budget, the second number is the one that decides what it costs you.

The cross-game comparisons are looser than they look, too. A Lua mod with MCP, a Codex handoff with a human-played opening, screen-and-keyboard control, a community operator's stopwatch: these are public runs, not a controlled evaluation, and the harness differs in every one. What survives all of that is the gap, which is large enough that the details do not close it.

The ARC-AGI team has now built a second benchmark that registered a shift before the field agreed there was one; early progress on ARC-AGI-1 marked the start of the reasoning-model era the same way. If ARC-AGI-3 is playing that role again, the capability it is measuring arrives with its failure mode already attached: an agent that learns fast enough to be useful, from evidence thin enough to be wrong.