How Virtual Worlds Help Agents Learn Faster

Worlds You Can Build with Code

An AI agent needs more than a goal and a list of available actions. It needs somewhere to experiment, make mistakes, and see the consequences of its decisions. But conventional simulators require models, textures, and complex graphics. Video models can generate beautiful scenes, yet they struggle to enforce precise rules: a door should open when pressed, an object should stay put, and collisions should happen predictably.

This article proposes combining the two approaches. AgentGarten stores the world’s rules and state in code, while a neural renderer creates the view seen by the agent. The result is an environment where the physics stay under control, but the scene looks like ordinary video. Agents can learn to act in this environment, then pass what they’ve learned on to others in short notes.

Code worlds can be rendered in different styles and for different tasks, and agents learn in them from notes left by earlier rounds.

State and Images, Kept Separate

In AgentGarten, a simulator or game engine controls what happens. It stores the objects’ positions and properties, along with the rules for how they interact. When an agent moves or picks something up, the engine updates the world state and sends the renderer a description of the visible geometry.

The renderer turns that description into frames. It receives a depth map or surface normals—the shape and position of objects—along with a reference image, a text description, and previous frames. It produces a visual observation that the agent uses to choose its next action.

This separation matters for long interactions. If an image is inaccurate, the simulator’s state remains unchanged. An object won’t teleport just because the video model drew it slightly out of place. The same sequence of actions can also be shown from a different viewpoint or in a different visual style: the events stay the same; only the image changes.

The engine updates the world state, while a neural renderer turns geometric descriptions into frames for the agent.

A world can be built from a text description or a single image. In the latter case, separate models help identify objects and estimate depth. A coding agent then creates the scene, specifying object positions, physical properties, and rules. It checks its work by viewing frames from different angles and running trial actions. If the setup is unstable or the movement looks implausible, the code is revised.

There’s no need to build detailed 3D models of every object. The program defines the basic geometry and behavior; the renderer handles the appearance. For example, a living-room photo could be turned into a scene containing the room in view, with additional rooms added behind it—even though they weren’t in the original image.

A scene built from a single photograph combines estimated geometry with code, then gets tested in trial runs.

Teaching the Renderer to Work Fast

For an interactive game, it’s not enough to render individual frames convincingly. The renderer must take previous observations into account and show the result of each action quickly. The authors started with a pretrained video model, then trained it to work with geometry and generate video in short blocks.

Training has two parts. First, the model learns to continue a sequence of frames based on the history so far. Then it’s accelerated: instead of taking many steps to generate video, it takes just four. This lets the agent act quickly and see the consequences right away.

The authors call their method Adversarial Forcing. It combines training on the model’s own video continuations with an additional quality check against real video data. That check addresses a common problem in long generations: as a video continues, textures may repeat and scene details may disappear.

There’s also a technical challenge. When training on a sequence of frames, errors in later frames should help the model handle earlier frames—the ones included in its history—more effectively. But storing the entire computation graph takes too much memory. Instead, the authors first record the result of a regular run, then repeat the computations block by block and recalculate the history so that gradients can flow through it. They call this technique exact replay.

“Exact” here means more than “close enough.” Compared with recalculating the entire sequence in a single call, this method reproduces the run bit for bit. It also doesn’t increase peak memory use. On a single H100 GPU, the renderer produces more than 35 frames per second at a resolution of 480 × 832.

Five worlds at five moments of one rollout: untextured scene geometry on top, the frame the renderer draws from it below.

Agents Learn from One Another

The authors tested the environment on several tasks. The most vivid example is a game of hide-and-seek. One agent hides while another searches. They see only first-person frames, act by running short Python programs, and don’t receive object coordinates or the simulator’s full state.

After each round, an agent reviews its actions and records what it learned in a “skill book”—a collection of files containing observations and advice. Each new agent starts with no conversational history, but gets access to these notes and can use them in a new game.

The hiding agent, for example, learned to move a panel to block an entrance. The searching agent learned to carry a ramp to a wall and climb it. If its first attempt failed, it could move the ramp closer and try again. According to the authors, hiding strategies emerged by round four, and ramp use by round ten.

For comparison, in an earlier hide-and-seek study, agents trained from scratch with reinforcement learning learned to hide after about 25 million episodes and to use ramps after about 100 million. This isn’t a direct comparison between the two methods. The new agents already have general knowledge about objects and space from pretraining; their task is to apply that knowledge in a specific environment and correct mistakes in their actions.

In hide-and-seek, agents see the world from a first-person perspective and learn to use hiding places and ramps, recording their experience for future rounds.

The same approach was tested in four other worlds: one with a dog the agent must keep in contact with; one with a narrow bridge where two cars must pass each other; a herding task; and a quarry with a loader. In each case, the agents received a description of the goal, available actions, and constraints—but not a solution.

Results varied by task. On the bridge, crossing time fell from 71 to 41 seconds. In the herding task, agents learned to get all four sheep into the pen. Progress in the quarry was less consistent: the loader scored no points in round two, but by round four it delivered a rock to the hopper and finished the task just short of the time limit.

After each attempt, the agent records lessons for participants in the next round.

Where the Approach Falls Short

AgentGarten combines the control of a simulator with the flexibility of video generation. But the neural renderer doesn’t convey everything the engine knows. Depth and surface normals describe an object’s shape and position well, but they don’t always reveal what it’s doing. For example, geometry alone may not show that a symmetrical object is rotating if its outline stays the same. Color, materials, and small details can also get lost, especially after an object has been out of frame for a long time.

There are limits to the evaluation of agent learning, too. The work shows that agents can handle several purpose-built worlds and improve over a few rounds. But that doesn’t yet prove they can learn just as well in unfamiliar environments or reliably transfer skills between tasks. For now, the notes are tied to individual worlds. As the number of environments grows, agents will need to decide which notes are relevant to the situation and which are out of date.

Finally, the environment has to be created in the first place. The authors want coding agents to handle that work and automatically check whether a world is understandable, solvable, and useful for learning. Otherwise, it would be easy to generate lots of scenes with nothing to learn—or rules that don’t match what the agent sees.

Conclusion

AgentGarten offers a way to expand the environments available for training AI agents: the rules stay in executable code, while a general-purpose video model generates the images. In this environment, agents can see the consequences of their actions, try again, and pass on what they’ve learned through skill notes.

So far, the work demonstrates a handful of tasks, and the comparison between a few rounds and millions of episodes involves different training setups. But the core idea is clear: if new worlds can be created faster and shown to agents in real time, researchers can more often test which skills agents have actually learned—and whether they can use them in the next task.