Lumine plays Genshin Impact for hours and transfers to other games

General-purpose agents are back at the center of the conversation. The team behind Lumine offers a concrete recipe for an agent that holds up for hours on hard tasks — 3D navigation, puzzles and dialogue — inside the open world of Genshin Impact, and then carries over to other games with no additional training.

Why this is hard

Closed environments gave us plenty of good algorithms and taught us very little about surviving in an open world, where an agent has to read a 3D scene and a 2D GUI at the same time, plan over long horizons, handle keyboard and mouse precisely, hold context in memory — all under hard timing constraints. Genshin Impact is a brutal testbed: enormous, full of dialogue, side quests, distinct mechanics, physics and weather, and the interface exposes only what is on screen, with no access to any internal game state.

How the agent works

At its core is an open-architecture vision-language model, Qwen2-VL-7B-Base. The agent sees the world the way a person does — through its eyes, taking in a 1280×720 frame every 200 ms. It acts like a person too, through keyboard and mouse, with one important caveat: those are encoded as text descriptions that emulate every keypress and micro-movement. At each nominal step the agent emits six chunks (one per 33 ms) carrying compressed information about what was on screen and exactly how the player moved the mouse and pressed keys. The six chunks are then turned into a virtual K&M action covering 200 ms of game time. This is what makes click positioning accurate in the GUI, supports mouse aiming in combat, and cuts errors in dialogue and menu choices.

The key idea is hybrid reasoning. At every step the agent decides whether to think right now — to generate a short plan in words — or simply emit the next useful K&M chunk. Thoughts are not needed in most situations; they tend to appear at turning points in the story and serve as long-horizon notes. The last 20 steps of history are kept separately. Together that is enough to make the agent tolerant of latency and consistent in how it behaves.

How it was trained

The researchers propose three stages, each with its own logic. First, imitation learning. That takes a lot of varied human data: video and K&M logs from playing Genshin from scratch — 2,424 hours in total. They filter for clean K&M traces with no idle time or camera jitter, which leaves 1,731 hours in which players make good use of every game mechanic. At this stage the model learns more than walking forward and pressing buttons: it interacts with objects, fights, handles the GUI reliably, switches quests quickly and navigates the map.

Second, imitation data alone is not enough. The agent has to get from "complete the quest" to "get closer" or "change your route" — it needs robustness and a grasp of how to follow the instruction it was given. For that they use 165 hours of short human episodes with instructions drawn from three different tasks, then expand that with a helper model and filtering into 200 hours of episodes in which the agent has to follow instructions across varied situations.

Third, clearing missions that run for hours — combat, NPC dialogue, puzzles and the rest — takes reasoning. The authors carefully annotate first-person inner thoughts on every key frame of one act of a story mission: 15 hours in all. The checkpoint thoughts come out short, so the agent gets used to reaching for them rarely.

Speed matters

Everyone likes peak performance, but the authors had to sweat for it — 25.3× faster inference once the optimizations are stacked. The model keeps its KV cache in FP16, quantizes Qwen2-VL to AWQ4, uses speculative decoding for the thought and K&M separators, and streams K&M from every frame. Preprocessing and visual-token packing are pushed onto the GPU. The result: the first K&M chunk (when no reasoning is needed) arrives in about 110 milliseconds, and inference latency for an arbitrary K&M is around 3 milliseconds. The game stays smooth even when it is demanding the GPU and the AI is generating thoughts frequently on hard missions.

What it does in practice

First, the instruction-following numbers: 141 tasks across all four categories — collection, combat, NPC interaction and puzzle solving. After the instruction-tuning stage the agent clears more than 80% of the easy tasks and behaves consistently on the hard ones. History helps most on collection and puzzles, where order matters.

Second, the long story mission. Lumine clears it in roughly 56 minutes, on par with a strong human player. The reasoning stage is what visibly improves robustness: the agent stays on purpose, wanders outside the task less often, makes fewer routing mistakes and handles the game's GUI more carefully.

Third, transfer. With no additional training the agent handles missions that take over an hour of real time in Wuthering Waves, and finishes the first chapter of Honkai: Star Rail in about five hours. It is always good to see an agent that does not fall apart outside the environment it was trained in.

How this differs from earlier approaches

A short memory is not a bad thing. Fewer hallucinations, more stability. The authors count two absences as their strongest move. The first is a rigid hierarchy with semantic abstractions for different tasks and skills; that kind of structure cultivates high-level chunks of behavior but is hard to optimize jointly. The second is forcing reasoning at every step, which on long missions badly hurts overall reaction time and fills the context with noise from redundant, hallucinated thoughts. The hybrid strategy looks like a simple and effective compromise.

What is still rough

The paper spells out the limitations in detail. The agent rarely uses fast travel, avoids healing, dies to hard bosses, and struggles to route between large numbers of map markers because its memory is only 20 steps long. All of it looks fixable with better minimap handling, a longer memory, extra data on rare mechanics and a light pass of reinforcement fine-tuning. The authors see the next big step as giving the model a good world representation to plan with.

Why it matters

This work feels encouraging. It shows that a general-purpose game agent driven by vision can be built — one that uses thinking, action and memory to behave reliably across different games, provided all of it runs in a single loop and response time is treated as the first constraint. This could become part of the architecture for a game AI that plays like a live player: watching the screen, reasoning, pressing a key and remembering what it just did. Yes, most of it is described and measured on a single story mission in Genshin. But the intuition is that the next step toward game AI built this way is within reach.