Generating 4D scenes as simulator code drops physics failures to 10%

Generative models have learned to produce striking clips, but that kind of video has a weak spot: nothing forces it to obey physics. An object can drift in mid-air, particles can ignore gravity, rigid bodies can pass through each other. For spatial intelligence that is not enough. What you need is a world model that doesn't just look plausible but behaves plausibly, because a simulation is running underneath.

The authors of Code2Worlds: Empowering Coding LLMs for 4D World Generation propose treating the creation of 4D scenes — 3D plus time — as a code generation problem aimed at a simulator. The point is that the LLM should not improvise motion by eye; it should write an executable script that a render then checks and that gets corrected iteratively.

Two obstacles on the way to living scenes

The first barrier the authors call cross-scale context entanglement. When a model generates a scene monolithically, it struggles to hold on to the large structure of the environment — terrain, light, vegetation density — and the fine geometry of the object that matters — a leaf with its veins, the shape of a cup — at the same time. Usually the wide shot wins and the detail gets simplified, and simplified detail animates badly.

The second problem is the gap between meaning and execution. A phrase like "leaves trembling in the wind" has to become specific parameters: where the object is anchored, how turbulence is set, which collision constraints are switched on. In open-loop mode the LLM writes that code blind, and physical hallucinations follow.

How Code2Worlds works: two streams and a closed loop

The architecture splits the work into two independent parts.

The first is the Object Stream, which builds the main object in high detail. Rather than generating from nothing, the system takes a retrieval-augmented approach: it pulls parameter templates and reference code out of libraries, then fits the parameter values to the text. It resembles a designer working from good pre-made assets — fewer syntax errors, more control, more repeatability.

The second is the Scene Stream, which assembles the environment. Here the hierarchy is what counts: first a plan (weather, season, terrain, object density), then the translation of qualitative words into numbers, then the generation of executable code. That ladder lowers the risk of the LLM mixing levels of abstraction and returning a script that looks good but doesn't hold together.

Then the interesting part kicks in — the closed-loop. First a dedicated PostProcess agent brings the scene to life, adding dynamics and physics settings. Then a VLM-Motion Critic watches the simulation video and returns feedback: what fails to match the text, where the physics breaks, which parameters need tuning. The code is edited, the simulation runs again, and the cycle repeats until the result is acceptable.

What it does in practice

The authors introduce a benchmark called Code4D and compare their approach against a range of code-centric systems and against video diffusion. By their numbers, Code2Worlds wins clearly on semantic consistency and scene richness: +41% SGS and +49% Richness over the baselines. They give separate emphasis to the drop in physical failures: a Failure Rate 10% against substantially higher numbers for several video diffusion models, which show up in the tables at 50–70%.

The qualitative examples show what the whole exercise was for: the system produces not a generic "sense of movement" but simulated events — wind, falling leaves, contact with water, a jellyfish moving through its medium, fire.

To get a feel for the range, it helps to look at key frames from individual scenes: wind, a rainy forest, or a rolling bottle all show how object behavior changes over time.

Why this matters and what to watch next

Code2Worlds pushes a practical idea: a 4D world needs more than generation, it needs execution that can be checked. Code gives you controllability, the simulator gives you causality, and the closed loop with a critic keeps pulling the LLM's invention back down to physics that actually runs. The split between object and environment also looks like a sound way to keep detail where detail really matters.

The limits are readable between the lines: the approach demands infrastructure — rendering, simulation, iteration — and the quality still depends on how rich the libraries of parameters and reference code are. But as a direction for building world models through language and code, it holds up.

💾 Code