An agent recovers physics from video by writing simulator code

Code as a world model

Large multimodal models can already describe what happens in a video: a ball rolls, a cup falls, a car turns. Physics is where the old problem persists. They see the phenomenon, not the mechanism. They can give a tidy account of a clip but don't always understand why the object moved the way it did, how fast it was going, what would change if you pushed it harder, or what is hidden behind the picture at all.

The authors of Code-as-World propose representing the physical world not as pixels, not only as text and not only as a 3D scene, but as executable code — a description you can run in a simulator, check, correct and run again.

If you want a model that doesn't just retell what it saw but can compute velocity, estimate sizes, reason about collisions and answer "what happens if" questions, it needs a world model with objects, their parameters, state and dynamics written out explicitly. Code turns out to be a convenient language for that.

Why pixels and text fall short

The paper opens with an honest account of how the world is usually represented in AI today.

Why code complements pixels, 3D and language: every representation has gaps, and executable code closes some of them.

Pixels preserve an enormous amount of detail. They are poor at showing causes. When something shifts in a video, the image alone doesn't always tell you whether the object moved on its own, was pushed, the camera moved, or one thing simply occluded another.

3D holds geometry better. That still isn't enough. Recovering an object's shape is not the same as understanding its physics. A 3D model alone won't tell you mass, friction, momentum or the rules of interaction.

Language works well as a compact description. It compresses meaning: "a person picked up a cup", "the ball bounced off the wall". But language breaks down where you need exact numbers and continuous states: distances, velocities, accelerations, contacts, angles, scales.

Hence the paper's central claim: code can serve as an executable abstraction of the world. In it you can state explicitly:

🟠 which objects are in the scene

🟣 what sizes and physical parameters they have

🟠 how they move and interact

🟣 how all of it looks from the camera's point of view

This is no longer just a description of a clip. It is a description of a world that can be run through a simulator and checked against data.

What an executable world representation is

The authors introduce their basic object — an executable world representation. It is essentially a program split into three parts.

🟣 Physical composition: which objects exist, their shape, size, mass, friction, gravity, tables, walls, floor and other scene elements

🟣 Dynamics: initial states, events over time, trajectories, collisions, episode length

🟣 Visuals: camera, background, materials, lighting, frame rate, render settings

The split looks practical. You can change the world and the way it looks independently. Keep the same collision physics, for instance, but move the camera. Or change the ball's initial velocity and watch the outcome change.

An example of an executable world representation: the scene is described in structured code and then run in a simulator.

Under the hood this is built on MuJoCo and a structured scene description. "World as code" is not a metaphor here but a concrete format you can run.

How the agent builds a world from video and text

The most interesting part of the work isn't the format itself but how it gets built from the input. Recovering world code from a clip is an inverse problem: you have the observation and need to reconstruct the hidden mechanism.

For this the authors build an agentic search loop. It runs as an iterative hypothesis:

🟠 propose a version of the world

🟠 assemble it in the simulator

🟠 run it

🟠 render the result

🟠 compare it with the source video or text

🟠 correct the hypothesis and repeat

The search loop for an executable world representation: propose, run, render, check and refine.

This resembles how a person explains a physical scene. You watch a video of a billiard ball, form an internal hypothesis about positions and forces, run it forward in your head, then check it against what you see.

Text and video get different input adapters.

With text as input, the system extracts entities, relations, events and the expected outcome. Missing parameters are then filled in with physical assumptions and refined inside the loop.

With video as input, the system first builds auxiliary observations: depth, object masks and their tracks. Then it tries to assemble a 3D scene, assign each object a position, scale and motion, run it all through the simulator and see how closely the render matches the original video.

Here the agent doesn't just produce an answer in one pass. It checks itself against an external environment.

What the approach buys you in practice

The first useful thing is controllable re-simulation. Once the world exists as code, you can change the conditions and get new clips with the same scene logic.

Once the world is recovered, scene parameters can be changed and the event re-simulated from another angle or with different physics.

For example:

🟣 change an object's initial velocity

🟣 change the direction of motion

🟣 move the camera

🟣 get a new realistic clip that preserves the same physical structure

That gives you a way to build data with control over the physics, not just over the image.

The second is a better match to the video across iterations. The authors compare a single pass against the iterative loop and against a "sample five independent candidates and pick the best" mode. The iterative loop wins on several metrics at the same verification budget.

With each round, iterative world search improves the match to the video and beats independent sampling at the same budget.

Put differently, correcting a hypothesis from feedback works better than guessing from scratch many times over.

The results in short:

🟠 visual match improves round by round

🟣 object mask agreement improves as well

🟠 trajectory error drops

🟣 object position accuracy goes up

Executable worlds as training data

The most practical application in the paper is training models for quantitative physical reasoning. These are tasks where the model has to answer with a number rather than say "the object is moving fast": what size, what velocity, what acceleration, what displacement.

That is harder than ordinary video questions. Real clips rarely come with such answers labeled. But with a verified executable world representation, the quantities can be read straight out of the simulation: object dimensions, trajectory, velocity, acceleration, world coordinates.

The authors train in two stages:

🟣 first they fine-tune the model on measurement tasks in image space — sizes, coordinates, velocity in pixels

🟣 then they add training on data from executable worlds, where physical quantities already come in real-world units

Both supervised fine-tuning on labeled data and reinforcement learning are used. The reward accounts for how close the numeric answer is to the correct one, along with correct units and answer format.

The idea: first the model learns to see and measure, then to calibrate those measurements to the physical world.

Results on QuantiPhy

The paper's main benchmark is QuantiPhy, which asks quantitative physics questions about video.

The Code-as-World-VL family posts the best or near-best result among the models compared, closed systems included.

QuantiPhy scores rise with model size, and the 27B version reaches the best result.

The key numbers:

🟠 Code-as-World-VL-4B: average score of 50.6

🟣 Code-as-World-VL-9B: 55.4

🟠 Code-as-World-VL-27B: 58.6

🟣 for comparison, Gemini-3.1 Flash: 54.8

🟠 many open models land well below that — around 30–40

Something else matters here: the 9B model beats a number of larger and more expensive systems. So it isn't scale alone that counts, but the kind of data and representation.

Another interesting detail: adding training on executable worlds on top of pixel-space measurement also improves the model's basic ability to measure directly from an image. Physical supervision helps the "lower-level" visual skill too.

Where the limits are

The approach has clear limitations, and the authors don't hide them.

First, the current version mostly handles rigid bodies and relatively controlled physics. The real world is harder: fluids, cloth, deformation, fire, fine contacts, complex friction, long chains of interaction.

Second, if the simulator can't reproduce a process faithfully, the loop may settle on a plausible but not necessarily correct mechanistic explanation.

Third, the question-answering model itself doesn't yet run the whole world-search loop internally. It learns from the pipeline's output but doesn't inherit the procedure — hypothesis, simulation, check, correction. That logic still lives in the external scaffolding.

Today the system benefits from data produced through a world model, but it hasn't become a physical investigator in its own right.

Why this matters

There is one idea in this work that reaches well beyond video physics. Executable code can be a good intermediate language between observation and reasoning.

Pixels are too raw. Text is too vague. A network's hidden state is too opaque. Code offers a rare combination:

🟣 compactness

🟣 explicit structure

🟣 verifiability

🟣 room to intervene

🟣 compatibility with simulators

That makes code a convenient scaffolding for tasks where a model must not only recognize a scene but also compute, test hypotheses and answer questions about the consequences of actions.

Seen more broadly, this bridges to several directions at once: more physically consistent video generation, AI systems with an explicit world model, robotics that needs scene-level planning, and multimodal agents that can treat the world as an executable system instead of merely talking about it.

The takeaway

Code-as-World proposes treating a physical scene as a program, not just as a stream of pixels or a text description. That changes how you work with the world: instead of one answer off the top, you get a loop of hypothesis, simulation and check.

For physical reasoning this is especially useful. When a model gets access to even the traces of that procedure — through data from executable worlds — it answers quantitative questions about size, velocity and acceleration better. And the QuantiPhy results show the approach can outperform large closed models.

The core idea is simple: to understand physics, a system needs not just a visible world but an executable model of it. Code turned out to be a convenient format for that model — because you can run it, edit it and check it against reality.