An AI agent built a playable shooter over 70 autonomous iterations

When one pass isn't enough

One of the main problems with coding agents has been obvious to anyone who has tried handing them something bigger than a 50-line function. On a short task everything looks brisk. On a long one the agent tangles itself in its own steps, fixes one thing and breaks another, loses sight of the original requirements, and declares the work finished far too early.

The authors of Harness-of-Harness propose a new harness for long-horizon autonomous development. The idea is simple: don't expect one agent run to assemble a whole product, make it work in cycles instead — plan, write, verify, move on.

If you want an AI agent not just to add code but to build a program from scratch with no human nearby, it needs more than access to a terminal and an editor. It needs a process that holds the project in memory and keeps it from degrading from one iteration to the next.

The conversation about autonomous development moved past tidy demos long ago. The question is no longer whether an LLM can write a chunk of code. The question is whether it can carry a project for weeks without losing the thread.

Two modes of development: an agent under constant human supervision, and fully autonomous development from top-level requirements.

What Harness-of-Harness is

The term in the title sounds heavy at first. The meaning behind it is clear enough.

Today's coding agents already come with a harness of their own: access to files, a terminal, build commands, sometimes tests and the repository. The authors don't rewrite that harness. They build a harness over the harness — an outer layer that organizes the agent's work over a long stretch.

Instead of one big run, the system repeats a cycle built from three roles:

🟠 The project planner decides what to do in the next iteration, working from the requirements and the results of earlier checks

🟠 The developer makes changes to the project and runs local tests along the way

🟠 The tester independently checks what came out and records what actually works and what doesn't

The key part here isn't the roles themselves. Schemes like this have been tried before. What matters more is that between cycles the system carries forward not only the code but the state of knowledge about the project.

The authors separate two things:

🟣 The project artifact — the current code, assets, configs, everything already built

🟣 Execution evidence — what has been checked, which errors turned up, what broke, what needs to be kept for the next iteration

That is the central idea of the paper: an agent has to remember not only what it wrote but what has already been verified. Without that, every new attempt is close to guessing the state of the project from the code alone.

The Harness-of-Harness scheme: planner, developer and tester work around a growing project, and the system preserves both the code and the verified evidence between iterations.

Why an ordinary agent fails at long tasks

The authors describe the problem of long autonomous development quite precisely.

As the project grows, the agent starts failing in familiar ways:

🟠 Early decisions get lost: why the architecture is built this way and not another, which constraints were already accepted

🟠 Local fixes cause regressions: the menu gets fixed and saving breaks, a mechanic gets added and input is ruined

🟠 A "look, fix, look again" loop sets in with no visible progress in the product

🟠 Gaps get dressed up as completion: the code is there, coherent behavior isn't

Put simply, an agent has trouble holding a long project history in its head. The context window is finite. The conversation history runs out. And code by itself is a poor place to keep the explanation of what a check has confirmed and what only looks like it works.

So the authors don't bet on memory as a separate module. They store documents, reports and the history of iterations, and let the agent retrieve what it needs by index rather than drag the whole project into every prompt.

This is an engineering answer. Not "let the model hold it all," but "let the system keep the important parts outside the model, carefully."

How the work cycle is built

Every HoH iteration is capped in scope. The planner isn't allowed to say "build the whole game." It picks a small but complete increment.

For example:

🟣 fix one specific build blocker and add a basic results screen

🟣 preserve the input handling that already works and finish the collision mechanic

🟣 don't rewrite half the project, close one observable scenario

That solves two problems at once. First, the change surface is smaller, so the source of a break is easier to find. Second, the result is easier to check.

One more move: testing sits both inside development and after it.

The developer runs local checks while the code is changing, to catch its own mistakes fast. But the final call on whether the system works belongs to a separate tester, working on a frozen version of the project. That tester can't quietly write code on the developer's behalf. It can only check and record facts.

This removes a common failure of multi-agent setups, where the agent writes the code and then immediately talks itself into believing it's done.

What the benchmarks showed

The authors ran HoH on three different task sets:

🟠 GameCraft-Bench — building playable games in Godot from a text description

🟠 FrontierSWE — hard tasks in building and optimizing software systems

🟠 ProgramBench — reconstructing a program from scratch given an executable and documentation

Three harness-and-model pairings were tested:

🟣 Codex with GPT-5.5

🟣 OpenCode with DeepSeek-V4-Pro

🟣 Pi with MiniMax-M3

In every case HoH was compared against a baseline where the same agent simply makes one ordinary pass with no outer cycle.

The iterative scheme gives a large gain almost everywhere.

The numbers, briefly:

🟠 On GameCraft-Bench the average final score rose by 16.6–22.1 points depending on the pairing

🟠 On FrontierSWE the gain in dominance came to 19–29 percentage points

🟠 On ProgramBench the improvement reached 16.85 points on the share of hidden tests passed

🟠 The authors put the average relative gain across all experiments at 52.25%, with a maximum of 82.86% after three iterations

The results in short:

🟣 HoH improves the result on all three benchmarks

🟣 The gain shows up across different models, not only in the strongest pairing

🟣 What produces the win is the organization of the cycle, not one extra pass

Baseline against HoH after three iterations on four measures of game quality: mechanics, content depth, visual functionality and presentation.

It's also clear that the gain isn't confined to the strongest pairing. Even weaker baseline agents improve noticeably once they're given the right work cycle.

That's a practical conclusion. Progress here comes not only from "get a better model" but from organizing the process properly.

Improvement builds from iteration to iteration

One of the most useful parts of the paper is the check on whether this is just an artifact of the first two extra attempts. The authors look separately at how quality changes as the number of cycles grows.

On the game benchmark, scores for all three pairings rose from HoH@1 to HoH@3. On FrontierSWE the Codex + GPT-5.5 pairing was run out to ten cycles. Dominance there climbed from 27.33% in the baseline to 72.67% at the tenth iteration, and the best checkpoint, at cycle nine, hit 76%.

On FrontierSWE quality rises as the number of cycles grows: HoH keeps improving the result well past three iterations.

So this isn't a case of "the agent got another shot." The system really does accumulate useful state between passes.

The authors checked another question too: maybe the whole effect is that HoH simply burns more tokens? To find out, they compared HoH against a mode where the baseline agent is just told to "keep developing" for one more pass, and then another.

The result is telling. At an equal number of passes HoH is still ahead. More than that, HoH after two cycles beats the baseline after three, and it spends fewer tokens doing it.

What follows from this:

🟠 This doesn't come down to brute force

🟠 Extra tokens on their own don't explain the gain

🟠 What works is the structure of the cycle: plan, implementation, independent verification, evidence carried forward

What actually helps

The authors also ran an ablation, pulling out the key pieces of the scheme one at a time.

If you:

🟣 don't update the plan between iterations, quality drops noticeably

🟣 don't pass the verification results into the next cycle, it drops as well

🟣 don't continue from the previous state of the project and start nearly from scratch each time, quality drops and token spend rises

Which gives a simple conclusion: HoH doesn't work because "three agents beat one" on its own. It works because three mechanisms run end to end:

🟠 a plan that gets updated

🟠 feedback from verification

🟠 continuity of the project artifact

Remove any one of those supports and the system gets worse at holding a long task together.

A shooter built over many days

The most striking part of the paper isn't the benchmarks but a long case study in autonomously developing a first-person shooter. The system was handed only a product requirements document and then ran the project on its own for more than 70 iterations.

Across dozens of iterations HoH assembled a playable first-person shooter: story, combat, weapons, enemies, interface, menus, animation and sound.

The finished project included:

🟠 a coherent storyline

🟠 combat mechanics

🟠 weapons and enemy interaction

🟠 player navigation

🟠 interface and menus

🟠 animation, visual presentation and sound

What matters isn't only that a game appeared at all. The dynamics are more interesting. Over 70 cycles the system opened 81 tasks, closed 65, and later had to reopen 17 of them because of regressions. That looks a lot like ordinary life on a software project. The product grows, bugs surface, some fixes break later, and then get fixed again.

This is where it becomes especially clear why version history and verification evidence are needed. Without them, long autonomous development turns into chaos fast, with the agent no longer able to tell what was working last week or why it stopped working now.

The case study in brief:

🟣 More than 70 iterations with no human running the project

🟣 81 tasks opened, 65 closed, 17 reopened

🟣 Regressions show up even in a successful long cycle

🟣 Project history and verification results matter as much as the code itself

What this changes

Autonomous development is a process problem, not only a model problem.

If you're building a coding agent, good tool access and a large LLM aren't enough. You also need:

🟣 short verifiable steps instead of one giant run

🟣 a split of roles between planning, implementation and verification

🟣 project history preserved as both code and evidence

🟣 independent verification that can't quietly fix the result itself

Which points to something broader. The talk around LLMs keeps circling world models, reasoning, agency. But in programming work the deciding factor is often more mundane: whether the system can organize its own work over a long stretch.

Takeaway

Harness-of-Harness shows that progress in autonomous development can come without a new model and without fine-tuning. It's enough to do what ordinary engineering settled on long ago: break the work into a multi-step cycle, keep a record of decisions, separate writing code from checking it, and carry forward not just files but the knowledge of what has already been confirmed.

For the industry that means something simple. The next step up for coding agents probably won't come from more powerful LLMs alone. It will come from a better harness, where the agent improvises blind less and works more like a disciplined team, even one assembled around a single model.