Orca learns one latent world state and reads out text, images and robot actions

When models stop talking and start understanding what is happening in the world

For the past couple of years AI has been learning a grab bag of tricks. Some models continue text well. Others fill in an image. Others try to control a robot. But nearly all of them solve narrow problems: predict the next token, the next frame or the next action.

The authors of Orca propose a different framing. What if a model should be learning neither the “next word” nor the “next pixel,” but the next state of the world? Not guessing the shape of an answer, but grasping internally what changes in a scene when a person does something, an object shifts, a cup falls, a robot misses its grip and tries again.

Ambitious, and in places overly so. But the paper is interesting for its engineering bet rather than its promises: build one general world model, train it on video, events and questions about video, then check whether text, images and robot actions can all be read out of the same internal representation.

In short: the idea is strong, some of the results are genuinely impressive, especially in robotics. But this is an early step, not a finished general-purpose machine for understanding the world.

The core idea behind Orca

Instead of building separate models for text, video and robots, Orca tries to learn a single latent state of the world. That internal representation is meant to answer one simple question: “what is happening right now, and how might it change next?”

The authors bet on two training regimes.

The first is “unconscious” learning. The model watches continuous video and learns to predict what the scene's next state will be. No labels. Just a stream of observations. That is how it is supposed to pick up dense, natural dynamics: objects moving, collisions, occlusions, inertia, everyday physics.

The second is “conscious” learning. Here language is added to the video: event descriptions, instructions, questions and answers. The model learns not just to guess the immediate future but to understand meaningful transitions: the “before” and “after” of an action, the goal, the intent, the causal link.

Orca's overall design: one encoder learns the latent world state, and separate lightweight decoders turn it into text, an image or an action.

The architecture is tidy. There is an encoder and a decoder. The encoder learns the shared internal space. It is then frozen, and lightweight readout modules are trained on top of it for whichever modality is needed: text, image or robot control. That detail matters. It is there to show the gains do not come from separately fine-tuning a large model for every task. They come from the quality of the shared latent state.

Why does that matter? Because, at least in principle, this setup is closer to how we think about the world ourselves. We do not have one brain for text and another for picking up a spoon. We have a single picture of what is going on, and words, expectations and actions come out of it.

How Orca was trained

The authors went big on data. They assembled a “world training” corpus that includes:

Only a tenth of the video data was actually used in this version. That alone says the authors see this as a long program rather than a one-off experiment.

The data Orca uses: continuous video for natural transitions, text-annotated events for meaningful changes, and video QA for semantics and common sense.

Pretraining rests on three tasks.

The first is predicting the next state from the current frame. This is the “unconscious” part.

The second is predicting a state specified by an event or an instruction. The model sees a scene together with a text condition pointing to the previous or the next significant event. This is the “conscious” part.

The third is answering questions about video. That is there to push language, semantics and everyday common sense into the latent state.

Notably, Orca is not trained to reconstruct pixels directly. It predicts the representation of the next state inside the visual encoder's space. The bet is not on a pretty picture but on the semantic dynamics of the scene.

After that the encoder is frozen, and three readouts are trained separately:

It is a good design for testing the hypothesis. If performance on three different tasks improves while the core stays frozen, the internal representation really does carry useful information about the world.

The results: the paper's central claim holds up

The first check is scaling. If the idea is right, training loss should fall as the model and the data grow. It does.

The larger the model and the data, the better the decoders perform: text, image prediction and robot actions all improve.

The second check is far more interesting: does a stronger world model help on downstream tasks? Here too the answer is yes. As pretraining scales, all three readouts improve — text, image and actions.

That is probably the paper's most important result. It shows the authors did not just train one more video model — they got a shared internal representation that transfers across tasks.

The gain shows up in robotics as well, even though the model never saw labeled actions during pretraining. In a field where real robot data is always scarce and expensive, that is a strong signal: some skills may be growable from ordinary video, provided the model can understand state transitions.

Text: Orca is better at change, causality and motion

On the text side, Orca was evaluated on four benchmarks covering video understanding, temporal dynamics, spatial relations and causal questions.

Orca-4B does well: on average it beats models of comparable size and some specialized approaches. The gain is clearest on tasks where recognizing an object is not enough and you have to work out how the scene changes over time.

The authors break performance down into four broad categories:

Orca looks strongest on state transitions and dynamics. That follows: train on video and events, and you should be better at telling what leads to what.

No one here is claiming magic — that it understands everything better than everyone. On the spatial tasks the margin is already thin. But the overall picture is clear: betting on next-state prediction does help with questions that require thinking in time rather than looking at a single frame.

Images: not prettier, but more physical

The image section is the most unusual. The authors are not competing on artistic generation. They want something else: can Orca's latent state help predict how a real scene changes after an action.

For that they built their own benchmark of real-world interactions. The input is a source image plus an instruction. The output should be a picture of the result. What is scored is not beauty but fidelity to the action, preservation of the scene and physical plausibility.

Orca-4B posts the best average score among the models compared. The description and the examples show where the advantage lies: fewer objects “teleporting,” fewer extra items appearing out of thin air, better preservation of the robot's pose, of contact relations and of the scene's logic.

Image prediction on real scenes: Orca does better at preserving scene structure, objects and plausible changes after an action.

It is a subtle but important shift. Image generators are usually excellent at visual plausibility and can be poor at causal plausibility. The picture looks good, but there is no telling how the world got into that state. On these results, Orca holds the causal thread better.

Robots: the strongest part of the paper

The liveliest and most convincing part of the work is the experiments on a real bimanual robot. Five tasks. Two hard generalization settings: new environments and new objects. Fine-tuning on just 200 trajectories per task. That is very little.

And this is where the world-model idea proves useful beyond comprehension benchmarks, in physical action. On average Orca clearly beats Qwen3.5 running the same action module, and comes out comparable to — in places better than — π0.5, a strong specialized robotics model pretrained on large robot datasets.

Orca recovers better from failed grasps: the robot does not freeze, it tries again and makes progress on the task.

The most interesting thing here is not the bare success rate. The authors look at intermediate progress and at the ability to recover from errors. Orca looks stronger precisely as a system that “notices what went wrong” and tries to get back to a working state. If the robot fumbles a spoon, Orca more often makes another sensible attempt instead of stalling.

That is a very good sign. In the real world, failures happen constantly. A robot that simply follows a template breaks at the first deviation. A robot with even a crude model of the world's state can notice the goal was not reached and correct itself.

What the paper gets right

The paper has several genuine strengths.

First, the problem is framed correctly. Moving from “next token” to “next state” is not just marketing. It is a useful frame if what we want is systems that understand the world rather than imitate answers.

Second, one idea tested across three channels at once: text, images, actions. That makes the case far more convincing.

Third, methodological honesty. The authors freeze the core and train only the lightweight readouts. That reduces the risk that the results come down to a larger number of trainable parameters on a particular task.

Fourth, a focus on the real world. Not synthetic data, not pretty generation, but scenes with objects, contact, failure and a second attempt.

The weak spots, and why a general world model is still far off

Strong as the idea is, the paper is honest about its limits.

First, Orca lives mostly in vision and language. The real world also has sound, touch, force, temperature, and the state of your own body. Without those, the world model stays incomplete.

Second, the model learns to predict states not directly but inside the space of an off-the-shelf visual encoder. That is convenient engineering, but it means the “world” partly inherits the limits of someone else's visual space.

Third, the scale is modest so far — 0.8B and 4B. For an idea this ambitious that is clearly not enough. The authors write themselves that a small model ends up trading off between text, image and action.

Fourth, the prediction horizon is short. Orca learns local state transitions well, but this is not yet a model of processes that run for hours or days.

And finally, some of the evaluation, especially on images, is still rough. Building your own benchmark is fine, but it has to be put through its paces by the community.

The takeaway

Orca is not “another LLM with video.” Nor is it one more robotics model. It is an attempt to redefine what is being learned in the first place: instead of words, frames and actions — world states and the transitions between them.

Not everything worked. A general world model is still an enormous distance away. But the paper already delivers a practical conclusion: train a system on the natural dynamics of video, on meaningful events and on language conditions, and you get an internal representation that is useful for several different tasks at once.

Orca's main value is that it returns the AI conversation to a more fundamental question: does the model understand how the world changes. And on that question the paper gives, for the first time, an answer that is engineering rather than philosophy.

Cautious so far. But already very interesting.