The best AI agent solves just 54.5% of game development tasks

We are used to measuring agent progress on tasks like fixing bugs in GitHub repositories, writing Python scripts, or building a front end from a mockup. Real development — game development especially — is far messier and far more interesting than that. Generating a function is not enough here: you have to understand scenes, object hierarchies, sprites, shaders, animations, interfaces, and how all of it looks and moves on screen.

When AI can write code but can't assemble a game

That sore but highly revealing spot is exactly where a new paper lands: GameDevBench: Evaluating Agentic Capabilities Through Game Development. Researchers from Carnegie Mellon, Princeton and other labs propose the first benchmark that asks not whether an agent can code, but whether it can complete full game development tasks in a modern engine.

The paper's conclusion is sobering and promising at the same time: even the best models are still far from working confidently in that world. But give them the simplest form of visual feedback and the scores climb noticeably.

Why this matters

Game development is not an arbitrary choice here. The authors argue it is close to an ideal proving ground for multimodal agentic systems.

First, games are complex projects with many files, dependencies and data types. Second, they are multimodal by nature: the model has to account not only for code but for images, animation, where objects sit in a scene, effects and camera behavior. Third, unlike most visual tasks, this one supports deterministic checking: instead of asking another LLM whether it likes the result, a test can verify that the required node exists, that the signal is connected, that the object is in frame, that the collision fires correctly.

That puts GameDevBench somewhere between SWE-bench, computer use tasks and multimodal software testing. The stakes are higher, though: an agent that learns to work confidently inside a game engine would mark progress not only in code generation but in a more general understanding of digital environments.

What GameDevBench is

The benchmark is built on 132 tasks for the Godot 4 engine. They were sourced from written and video tutorials covering typical game scenarios: character animation and collisions, UI, 2D and 3D graphics, shaders and effects.

The authors picked Godot for pragmatic reasons. It is open source, popular enough, works much like Unity, and its projects map cleanly onto files and code. That last point matters: the researchers did not have to invent an exotic API for actions in the editor — many tasks can be solved by editing scene files and scripts, and then checked automatically.

The tasks themselves are markedly harder than the usual software engineering benchmarks. On average the reference solution touches about 5 files and 106 lines of edits. By volume of change that is more than three times heavier than tasks in SWE-Bench, the yardstick the industry likes to quote.

The range of media types matters too. Most tasks contain not just scripts and scenes but PNGs, fonts, shaders, audio and engine resources. The agent is not navigating a purely textual world; it works in an environment where visual structure is part of the task itself.

How they built it

The methodology is careful and quite practical. Rather than inventing a hundred tasks by hand from scratch, the authors assembled them from real teaching material produced by the Godot community.

First they selected tutorials: YouTube videos and written guides, all with open GitHub repositories and compatible licenses. Transcripts and repository code then served as raw material for generating tasks automatically. An agent was told to turn each tutorial into a set of independent, checkable subtasks — animation broken out as one, collisions as another, UI as a third.

Then came a multi-stage cleanup. The automatically generated tasks were checked against checklists and then validated by hand. Eight annotators, some with game development experience, fixed ambiguous instructions, overly strict tests and other familiar problems. The result does not read like a raw dataset scraped off the internet; it reads like a seriously vetted set of engineering tasks.

The verification system deserves separate mention. Instead of a vague "looks about right", the benchmark uses Godot's test framework. This is one of the paper's real strengths: a multimodal task gets strict programmatic verification.

How game dev tasks differ from ordinary coding

The paper makes a good case that "write the code" and "build the game feature" are not the same job. A task that sounds simple may require all of the following at once:

— working out which sprite or set of frames the animation needs;

— adding nodes at the right place in the scene tree;

— wiring up physics, collisions and signals;

— fixing the code;

— confirming the result is visible to the camera and behaves correctly over time.

One clear example is the minimap UI task. From above it looks like a visual scene with objects of interest highlighted; from below, like the code representation of those same entities. The agent can solve it either through the editor or through the files, but either way it has to line up the visual and the structural world.

The authors also split tasks by skill: gameplay logic, 2D graphics and animation, 3D graphics and animation, user interface. And by editor type in Godot: scene editor, script editor, and contextual editors such as animation, shader or tilemap. That breakdown matters for analysis, because it shows not merely which model is better but where exactly it breaks.

The headline results: even the leaders stall

The paper's strongest number reads like this: the best agent solves only 54.5% of the tasks. For a field where some models are already closing in on impressive scores on other benchmarks, that is a very sobering result.

The leader is Gemini 3 Pro, running in its own native agentic environment with multimodal support. Behind it: Gemini 3 Flash, Claude Opus 4.5 and Claude Sonnet 4.5. But none of them gives any sense of settled maturity: nearly half the tasks still go unsolved.

The distribution across task types is especially interesting. Agents do best on gameplay-oriented scenarios, where the average success rate is around 46.9%. They do worst on 2D graphics and animation, where success drops to 31.6%. That tracks: once you have to pick the right sprites, parse a spritesheet and work out which frame does what, textual intelligence is no longer enough.

Another important finding: quality falls off very sharply as soon as you step away from frontier models. Qwen3-VL-235B, which holds its own on some visual benchmarks, is nearly helpless here. That is a useful signal for the market: success at tasks like "build this UI from a picture" transfers poorly to dense multimodal engineering environments.

Simple visual feedback helps far more than expected

Perhaps the most practical insight in the paper is that agents gain even from very crude forms of multimodal support.

The authors tested two mechanisms:

— an editor screenshot served through an MCP server, which gives the model a picture of Godot's current state;

— a video or recording of a playthrough, so the agent can see the dynamics over time and the final camera view.

This is not a decorative option. Claude Sonnet 4.5, with no extra visual help, solved 33.3% of tasks; with video, 47.7%. That is an enormous jump for what amounts to a single added feedback loop. For Gemini 3 Flash the best mode took it from 47.0% to 52.3%.

Curiously, which kind of help works best depends on the model. Some gain more from editor screenshots, others from execution video. Combining both rarely produces a dramatically larger gain, but it almost always beats the text-only mode.

The finding fits neatly into a broader trend of recent months: agents get noticeably smarter when they can not only act but see the consequences of their actions. In game development that is especially obvious, because the mistake is often not in the syntax — the object ended up outside the frame, the wrong sprite got attached, a node was nested in the wrong place, or the animation simply looks wrong.

Where the models actually go wrong

The authors also ran a qualitative error analysis, and it is arguably no less interesting than the metrics table.

The first big problem is weak multimodal understanding. Models pick the wrong images, confuse animation frames, misread visual assets. These are not silly bugs but a systemic weakness: the agent sees a file without always truly understanding how that file relates to the game result it is after.

The second problem is not knowing game development's domain patterns. Godot, like any engine, carries a mass of unwritten rules: where a given node belongs, which object type a property applies to, where signals get connected and where in the scene tree that actually makes sense.

This is exactly where the characteristic failures appear, like the one in the authors' case study: the model sets the correct sub_emitter property on the wrong object. Formally it knows the right words, but it does not understand the engine's structure at the level of engineering practice. That is a pointed lesson for anyone who believes training an LLM on GitHub code is enough to turn it into a working game developer.

The price tag: quality versus cost

The paper analyzes the cost-quality trade-off separately. The overall picture is what you would expect: multimodal feedback raises the cost per task, but it usually raises the odds of success too. The best value for money turned out to be Gemini 3 Flash.

Cost, interestingly, is not always predictable from per-token price or model size. The agent framework matters a great deal as well. The same model can score noticeably better or worse depending on whether it runs in its own native CLI or through OpenHands. Another reminder that in agentic AI what counts is not only the model but the scaffolding — the tools, the observation loop, and the way the environment is edited.

What follows from this

GameDevBench is not just one more benchmark for the pile. It shows something important: the moment we step outside the world of text in, text out, many of AI's current achievements start to look far less convincing.

Yes, today's models can already help with code. But game development exposes how far off genuinely multimodal engineering agents still are. They lack visual understanding, they lack domain patterns, and they lack any reliable ability to check their own actions against the state of the environment.

The paper is not pessimistic about this. If anything it lays out a fairly concrete path forward: even simple visual feedback — an editor screenshot or a short video — already lifts quality appreciably. Which means the problem is not that agents are incapable of this kind of work, but that they are still missing the right sensory loop and better tuning for the domain.

The takeaway

The paper's main result comes down to this: game development turns out to be an excellent stress test for AI agents. It demands programming discipline, visual understanding, and the ability to work inside a complex structured environment. On that test even the leading models are scraping a pass rather than earning top marks.

For researchers that is an important signal: if we want genuinely useful autonomous developers, code benchmarks alone are no longer enough. For the industry it is the same signal: "AI writes code" does not yet mean "AI can build a product". And for anyone following the progress of agents, GameDevBench offers one of the more honest snapshots of where the field stands today.

And perhaps the most interesting question after this paper is no longer whether AI can make games, but what exactly keeps it from seeing a game as a coherent system rather than a pile of files. The next big step probably lies in the answer.