How to control a robot using a small AI model

A Robot That Learns to Write Its Own Programs

A small open model is given a task: pick up a cup and place it on a saucer. It looks at an image, writes code to move the robot, gets a report on what happened, and decides what to do next. There’s no prebuilt “pick and place” skill, and no large model guiding it at every step.

That’s how CaPtain works: a pipeline that teaches a 4-billion-parameter model to control a robot through code. On standard manipulation tasks, its success rate rises from 55.4% after training on examples to 73.8% after reinforcement learning. On some tests with altered scenes, it also outperforms several systems built on frontier models.

How CaPtain works: the model writes code and receives reports from the robot, while a critic helps assess individual steps during training.

Programs Instead of Prebuilt Skills

In the “code-as-policy” approach, the model controls the robot by writing short programs on the fly. It receives camera images, calls tools to recognize objects and measure distances, and sets the manipulator’s position. After each action, the model gets a response—for example, how far the robot moved, whether it touched an object, and why it stopped.

These systems typically rely on either closed frontier models or motion-control models trained in advance. The code then selects which prebuilt skill to run, while that skill determines how to approach an object, grasp it, and move it. CaPtain works differently: the model itself decides where to place the gripper and which direction to move.

To make this possible, the authors built CaPtain-Harness, a wrapper with 22 low-level tools. These include object recognition, geometry measurements, and collision-protected movement. There are no ready-made commands like “pick up the cup” or “open the drawer.” Instead, the robot returns a short report after every tool call, helping the model understand what happened.

First, the model learns from examples of how to use the wrapper. Larger models generate those examples by completing tasks in simulation. The smaller model learns not just to produce code, but to follow the full workflow: inspect the scene, choose an object, check the gripper’s position, and assess the result.

Then comes reinforcement learning. The model performs tasks on its own, and training adjusts its behavior based on the outcome. To make this work, the authors had to solve a separate problem: figuring out which of dozens of requests helped the robot complete a task.

How to Measure Each Step’s Contribution

Imagine a long episode: the robot looks at a cup, measures the distance to the saucer, moves the gripper into position, closes it, lifts the cup, and carries it over. If the cup falls at the end, a simple “failure” label isn’t enough. It doesn’t show where things went wrong. And if the task succeeds, it’s hard to tell which of the many requests made the difference.

In simulation, the system has access to information hidden from the model during operation: the exact positions of objects, the gripper’s state, and how much of the task has been completed. CaPtain uses this information to train a separate evaluator, called a critic. The critic analyzes the robot’s physical state and estimates the probability of completing the task.

It then compares the state before and after a request. If a movement increases the estimated chance of success, it gets a positive score. If the model simply looks at the scene and the physical state doesn’t change, that step gets little direct credit. It can still be useful: for example, an observation may help the model choose the right movement next.

The critic sees the simulation’s exact state, but the model does not. During evaluation, CaPtain acts only on images and reports about its actions. This separation lets the researchers train the model with detailed data without turning that data into hints the robot could rely on in real-world operation.

The evaluator also has to avoid crediting the mere passage of time. To prevent that, the authors compare states at the same request index. Otherwise, the critic might conclude that the odds of success changed simply because the episode was closer to ending. When analyzing the training data, the evaluator most often associated success with physical movements, grasping, and releasing objects. Observations alone accounted for only a small share of the total score.

From Simulation to Altered Scenes

The base model, Qwen3.5-4B, completed none of the tasks in the initial evaluation without training. After learning from examples, CaPtain reached a 55.4% success rate on held-out LIBERO tasks. Reinforcement learning raised that to 73.8%.

Its success rate also improved on tests that changed object positions, lighting, camera angles, or task wording. On LIBERO-PRO, the model solved 78.5% of the short tasks and scored 68.6% on the full benchmark. On LIBERO-Plus, its score rose from 48.6% after training on examples to 66.1% after reinforcement learning. The improvement held across all seven dimensions tested by the benchmark.

Reinforcement learning improves success rates on both standard tasks and tests with altered conditions.

Models trained to directly control robot movements often perform well in familiar scenes but struggle when conditions change. The paper gives one example: a model that reaches 97% on standard LIBERO scores just 12.8% on LIBERO-PRO. CaPtain falls short of the top systems on some evaluations, but after reinforcement learning it holds up better when conditions change.

Among other code-based control systems, CaPtain-4B also outperforms several solutions built on frontier models. Its LIBERO-PRO score is higher than those of K1, WAA, and ASPIRE. The authors estimate the model’s compute cost at around three to six cents per episode, excluding separate recognition services. That’s considerably less than the cost of several systems that call frontier models during every attempt.

After reinforcement learning, CaPtain-4B improves across different task types and sources of disruption.

The model was also tested on tasks it hadn’t been trained on. Without additional training, it scored 83.3% in SimplerEnv on tasks involving a different robot and new objects. On CaP-Bench, it scored 75 points; when given more time, it completed 39 of 40 episodes across two tasks. This suggests that its learned skills can transfer to new conditions, though the transfer isn’t equally effective everywhere.

Results are much lower on challenging two-arm tasks. CaPtain completed 22.6% of episodes on RoboTwin 2.0 and just 0.8% on RoboDojo. The authors point to tasks requiring coordinated movement of both arms and the unfamiliar manipulator’s limited reach as major challenges. A model trained mainly on one arm doesn’t automatically learn to coordinate two.

Testing on a Physical Robot

The authors also ran CaPtain on a physical manipulator. Beforehand, they adapted the model in a simplified simulation of the new robot, with different dimensions, a different camera, and a different gripper. They used no real-world data for this adaptation.

In one episode, the robot had to place a yellow cube in a wooden tray. In another, it had to interpret an indirect description: pick up the sky-colored cube and place it in the container that isn’t round. The model chose a light-blue cube and a rectangular tray, then completed the task using the same low-level tools.

Two episodes on a physical robot: the model handles new objects and an instruction using an indirect description.

The test covers only two tasks, but it shows that transferring from simulation to a real robot is possible without a separate, large training dataset collected on the physical device.

Where the System Still Struggles

CaPtain’s limitations stem from both its training and the actions available to it. The model was trained only on single-arm LIBERO tasks. It doesn’t transfer those skills well to two-arm movements or robots with different geometry. Its capabilities are also limited by the wrapper. For example, it can’t handle tasks that require pressing an object against a surface for an extended time: its collision-protected movements aren’t designed to apply that kind of force.

There’s another issue: the model sometimes keeps checking a task after it’s already complete instead of stopping. The authors found that, in later training stages, the scene-based success rate stayed roughly the same while the number of episodes that ended correctly fell. The robot had already done the job, but it kept observing and measuring until it ran out of time. The researchers chose an earlier model version, which had fewer of these cases.

Conclusion

CaPtain makes it possible to train a small open model to control a robot through programs, without delegating manipulation to prebuilt skills or a separate motion-control model. The key idea is to use the simulation’s exact state to identify which actions brought the robot closer to its goal, while keeping that information available only during training.

The system transfers some of its skills to new tasks and can run on a physical robot after adaptation in simulation. But coordinating two arms, handling unfamiliar manipulators, and reliably recognizing when a task is complete remain difficult. Further progress depends on whether this approach can teach models not only to move, but also to know when the job is truly done.