ChatGPT Atlas solves Sudoku in 2:28 and scores zero in Flappy Bird

What happens if you give an agent eyes and hands inside the browser — not just the context of the page but an intent, plus the ability to click and press keys on purpose? Researchers decided to find out by running one through several browser games. You can probably guess the answer already: Atlas has real strengths in turn-based logic, and real-time control is its Achilles heel.

Why test an agent on games

Why games? Because a game is a small model of the real world, and it gives agents a rich spread of tasks with clear measurements, varied behavior and fast feedback. The researchers ran a baseline evaluation of Atlas as a browser agent on the games below, looking only at whether it extracts rules and goals from the content of the page rather than from explicit instructions; how accurately it turns purposeful actions into mouse and keyboard input; whether it can change its behavior when it gets stuck; and whether it can grasp the game's context, its story and its multi-step goals as a whole.

How the experiment was set up

The authors ran ChatGPT Atlas in Agent Mode on each game, with memory, code and the file system turned off, and with the same prompt every time — one that offered no hints during play. They did ten independent runs, starting each agent from the game's standard opening, and compared the results against simple human baselines: a medium-difficulty Sudoku normally takes a person 10–12 minutes, and for the two real arcade games they used scores from first human attempts.

What happened

Sudoku is where Atlas did well, solving the puzzle in every single run in 2 min 28 s with 100% accuracy, roughly 4.5× faster than a person. In 2048 it averaged 2,242 points and never lost, but a human scored 3,463 on a first attempt. The agent made progress without showing any grasp of a long-term strategy.

In T-Rex, Atlas scored 45.5 against 388.9 for a human — 11.7% of the human level. In 9 of 10 runs it failed to clear even the first obstacle, often trying to jump but doing it too late (which meant a collision) or too early (losing height before the obstacle arrived).

In Flappy Bird, Atlas scored 0 in every run, while humans on their first attempts averaged around 2.9. Both arcade games demand fast repeated presses of the same key at precise intervals, and that is plainly what Atlas could not manage.

Across runs it did change its behavior — varying the press rate, or deliberately slowing its start — and it searched the interface for a way to lower the difficulty or slow the opening down, but neither game offered anything of the kind.

In 2048 the agent spends a good deal of time exploring the interface, first trying clicks on the tiles, then the arrow keys, then WASD, all while working from the current game context. But too many actions pile up: cycles of presses in all four directions, pauses, the same positions repeated with no clear exit condition — often getting stuck among three possible moves. It sometimes reaches the 512 tile, but it frequently stalls at 64.

Stein.world needs a long-term strategic plan, and its interface requires different keys for movement. The agents could not get out of the starting room, so the authors began feeding them hints — first mixing clicks and keys, with no success. Then two separate hints: one about WASD for movement, another about F for interaction. With those, Atlas got out quickly and interacted with an NPC, completing the first quest.

Broadly, the browser agent is bad at anything that demands a fast real-time reaction, which will clearly be a bottleneck in plenty of real applications too. On analytical tasks, though, it behaves with confidence. It is good at pulling information about the interface and the game's goals out of implicit context, and weak wherever there is no clear instruction about what to do. It tries to work out the next step, often with decent reasoning behind it.

Why this matters

The overall conclusion is that a browser agent is worth using to retrieve information and to handle tasks that are well specified and have a proven path to a solution. But a real browser has forms, floating windows, animations, latency and the rest of it, all of which can get badly in the way outside the lab. Whether this kind of interface is needed at all is an open question. On anything harder, the agent badly lacks low-level control and any real ability to set goals in an unpredictable environment.

The authors are careful to say that this is an observation rather than a systematic benchmark, that the sample is small and the scenarios limited. They call the results early, but the pattern is consistent enough to say something about the limits of Atlas as a web agent. They call for the task list to be extended beyond games — dynamic forms, live visualizations, complex web interfaces — and compared against other agents. They also see value in behavioral protocols that separate out the different components of behavior.

Takeaways

The ChatGPT Atlas experiment is a clean illustration of the border between "understanding" and "acting". Atlas can solve a Sudoku, work out the rules of 2048, even reason its way to a strategy — but where events have to be answered in milliseconds, its actions arrive late. That is not a bug, it is a fundamental property of the architecture: large language models are built to reason, not to have reflexes.

And that is exactly why experiments like this matter: they show where the line falls between an AI's cognitive ability and real interaction. Those abilities will most likely be split apart in the future, with coordinated multi-agent systems pairing “thinkers” with “doers”. One agent would hold the context and the goals, another would drive actions in real time, and a manager would coordinate the two.

A game is only a small model of the world, but it is in toy sandboxes like these that the next step in AI's evolution takes shape: from thinking to doing.