Game worlds can be turned into testable learning tasks, and playthroughs into training data for visual AI agents. In Game2Player, players themselves test the games: their reports help fix the environments, and every action is tied to measurable progress. A model trained on this data is more likely to move toward its goal, even in unfamiliar games.
The main idea
Training AI to play takes more than video. It takes records with clear goals, actions, and reliable measures of success. Game2Player automatically prepares this data from web games and uses in-game progress to identify useful actions.
🟠 A collector adapts each game and creates tasks whose progress and success are measured using the game’s internal state.
🟠 Two players test the environment in different ways and report problems.
🟠 Once fixed, games yield trajectories: sequences of screens, actions, and progress scores.
🟠 Training gives more weight to actions that move the player toward the goal.
What is Game2Player?
Game2Player is a pipeline for finding, adapting, and testing games, collecting gameplay records, and training a model. The environment and the players improve each other: players find errors the collector missed, and the collector checks their reports against the playthrough record.
Game2Player prepares a game, tests it with two players, fixes reported problems, and collects training data.
🟠 Game finder selects games that can run offline in a browser, be played solo, and be restarted reliably.
🟠 Collector packages each game in a reproducible environment, sets the starting conditions, and creates tasks. For each task, it defines how to measure intermediate progress and what counts as success.
🟠 Image-based player sees a screenshot and chooses an action; the game then pauses until the next decision. It plays in a way that closely resembles how the trained model will act.
🟠 Programmatic player writes a separate program for each game. The program reads information about visible objects that is available to the player and acts in real time, without calling a model at every step.
🟠 Report review: Players report broken controls, incorrect progress scores, or insufficient observable data. The collector checks each report against the recording and the game state, then fixes confirmed problems.
🟠 Data selection gives more weight to actions followed by noticeable progress and less to repeated actions and final moves in failed attempts.
How it works: a task in a web game
Suppose the task is to deliver iron and buy a level-one drill. The system needs to record button presses and distinguish progress toward the goal from aimless play.
The programmatic player acts on observable data in real time, while the image-based player chooses an action from a screenshot; their records are converted to a common format.
First, the collector finds the game’s data for iron supplies and drill purchases. It uses these to define a progress scale: each delivery of iron raises the score, and buying a drill counts as success. The score is based on the game state, not on another model’s judgment.
The players then attempt the task. The image-based player sees the screen, reads the control instructions, and chooses an action, such as pressing a movement key. The programmatic player gets information about visible objects and can calculate more precisely when to press or release a key.
If iron is delivered but the progress score does not change, the player reports a possible error. The collector reviews the recording and checks whether it linked the progress metric to the right in-game event. If the review confirms a problem, the collector fixes the measurement and has the players try the task again. Things can still go wrong: a report may be mistaken, the game state may be misread, or the task itself may turn out to be impossible. So the collector must verify every report and check that the fix worked.
Once the environment is approved, playthroughs become training examples. The system pairs each screen and the preceding history with the action to take. Examples followed by progress are sampled more often during training; unhelpful repetitions get less weight. This teaches the model to choose actions that move the task forward.
How the results are evaluated
The authors assess both the quality of the game environments and the value of the data they produce. In the first test, they compare the full pipeline with a version that has no player reports and with a standard coding agent that lacks specialized tools. The tasks are set in advance, and the environments are checked against the original games; people also review the progress scores.
The trained model is evaluated on two benchmarks: Player Bench, with 50 games held out from training, and the external GameWorld benchmark, with 34 games not included in the training collection. Both measure progress—the average share of the path to the goal completed—and success—the share of tasks completed in full. The training games do not overlap with Player Bench, and GameWorld is also separate from the collection.
The comparison includes base models, models trained with Game2Player, and other vision models. Participants and models receive task and control descriptions, and each attempt is capped at 100 steps. One caveat: the game pauses while the models make each decision, so the test does not show how they would handle real-time delays.
In a study of 20 games, player reports increased the number of correctly implemented tasks and the number of tasks completed.
The numbers
The model improved substantially on both evaluations, including on games it had not seen during training. In a separate test of 20 games, player reports raised the share of correctly implemented tasks from 65% to 72% and increased the number of tasks completed by programmatic players from 34 to 46. Having players check the environments together helps collect more records and fix the conditions those records depend on.
Conclusion
Game2Player automates the collection of game data with verifiable goals. A programmatic player can run many times without a separate model call at every step, while the image-based player adds decision-making examples that resemble how the future model will perceive the game. Progress-based weighting helps the model learn from actions that actually move the task forward.
Testing does not guarantee that every environment is error-free: some tasks in the study were implemented incorrectly after collection. The dataset is limited to solo games that run offline in a browser; it does not cover online games, multiplayer modes, or large commercial titles. Models also help build the game worlds and review reports, and individual versions of those models change over time.
For now, the results suggest this approach is useful for training and testing visual agents in games. The next step is to find out whether the gains hold in more complex environments, where agents must make decisions without pausing the game and account for real time.
Read next
How to teach an AI agent to develop an app from scratch
InclusionMed: Which AI should I choose for medical tasks?
Why Is It Difficult for an AI Agent to Act in the Real World?
WorldFoundry: How can you distinguish video generation from world
How AI Can Enhance an Individual’s Intelligence
FrontierPhysics: Let’s see whether AI agents can make discoveries
How AI Chooses the Right Model for Each Task
CaPtain: How to control a robot using a small AI model
WorkFactory: Hundreds of thousands of professional tasks helped train
How AI Agents Learn General Skills Across Different Tasks
How can you tell that an AI agent is doing the task incorrectly?
How an AI agent can master a new skill through practice using the Socratic method
AI reviews in simple way
Every day we read fresh AI papers and retell the essentials in plain human language — no hype, no fluff. If you want to see where AI agents are heading before everyone else, subscribe.
New reviews — every day
Follow on X