LingBot-World: an open-source world model you can steer in real time

Not long ago, models learned to generate video from text with a few seconds of coherent motion. But ask one of those systems to walk forward, look back and return to a familiar object, and the magic stops. Objects trade places, details drift, and causal logic gives way to statistical coincidence. That gap between a nice-looking picture and an actual simulation of a world is what the authors call one of the central problems on the way to real world models.

LingBot-World is an attempt to move from text-to-video to interactive text-to-world: a system that does not just render a clip but supports controllable, stable and reasonably plausible time spent inside a scene. A big part of the intent is openness — both the code and the weights are published, to close the distance between open-source and closed systems.

Data: where to get experience for an interactive world

The most painful part of any interactive model is data. There is plenty of good-looking video on the internet, but it almost never carries the link between an action and the world's reaction, and without that it is hard to teach a model to act in an environment. The authors go hybrid: ordinary real-world video, plus game recordings where the keypresses and camera motion are known, plus synthetic footage from Unreal Engine. What makes the synthetic part especially valuable is exact camera parameters and the ability to generate specific trajectories — 360-degree turns, say, or returning to an object to probe spatial memory.

Then comes the cleanup: filtering, cutting into clips, semantic scoring for quality and motion, and approximate geometric annotation for ordinary video where the camera parameters are unknown. On top of that sits a hierarchical text annotation: separate captions for the overall storyline, separate ones for the static scene with no action in it, and separate dense temporal descriptions per interval. The reasoning is simple: fold everything into a single caption and the model has trouble telling what in the scene is set dressing and what is the result of an action or a camera move.

How a video generator becomes a controllable world model

The core idea in LingBot-World is not to build a simulator from scratch but to grow one out of a strong video model across several stages. It starts from a powerful open video generator — the paper mentions Wan2.2 image-to-video diffusion at 14B parameters — as the source of visual quality and baseline coherence. The model is then fine-tuned to hold a longer context, forget less of the scene structure and start following user actions. This is where Mixture-of-Experts enters: two experts, one better at global structure, the other at detail. The goal is to keep stability and image quality at the same time.

Actions are fed in explicitly: discrete signals such as W/A/S/D, plus continuous camera rotation encoded through dedicated representations. So that control actually shapes generation, actions are woven into the network's layers through adaptive normalization. At the same time the authors try not to wreck the visual capability the model already has: the base blocks are frozen, and training mostly touches the adapters responsible for control.

There is one more obstacle: speed. Classic diffusion models are heavy for interactive use — waiting seconds, or minutes, for a render is not an option. So the third stage converts the system to a causal, streaming mode and distills it down to a handful of steps. The final LingBot-World-Fast is claimed to run in real time at 16 fps with sub-second latency.

Almost a game: long memory and real control

By the authors' account, the strength of LingBot-World is the combination of three things that rarely appear together in open source: domain diversity, a long horizon, and high dynamics at interactive speed. In the demos the model holds scenes consistent on the scale of minutes: landmarks are not repainted once they leave the frame, and some off-screen processes keep unfolding plausibly even while they cannot be seen.

There is a quantitative side as well: on VBench for long videos, over 30 seconds, the model shows a particularly clear gain in Dynamic Degree — the richness and plausibility of motion. Motion smoothness and flicker scores stay competitive, which matters if the result is to feel like a living world rather than a stack of frames.

The applications are worth a look of their own. First, the world can be branched with prompts: change the weather or the style, add local events like fireworks, without breaking geometry or dynamics. Second, the authors demonstrate a pairing with an action agent: an LLM (Qwen3-VL-2B) predicts a sequence of actions several seconds ahead, and the world model rolls video forward along them. Third, the generated clips can be turned into 3D reconstructions: the point clouds look consistent enough to suggest, indirectly, that the model is not merely painting frames but maintaining a more stable representation of the scene.

Open by default, honest about the limits

What makes the work land is that the authors are not hiding the system: a fully open-source pipeline, weights and inference code. That is a real step for a community that wants more than pretty clips — it wants to build tools on top, from content generation to training environments for embodied AI.

The limits are laid out plainly. Memory comes mostly from the context window, with no dedicated module, so very long sessions can drift apart. Inference is still expensive and needs serious GPUs. The action space is fairly narrow for now: navigation and basic movement, with no precise object manipulation. And this is still a single-agent view, without a proper multi-agent system.

Taken together, LingBot-World reads as a bridge between video generation and an interactive world model: not the dream game, but already something you can experiment with, compare against and improve — and, above all, do it in the open.

💾 Code

🤖 Model