RoboBrain 2.5 gives robots metric depth and a running sense of progress

Robotics has an old sore spot: even strong vision-language models reason about a scene well enough, yet sometimes fail at acting in the physical world. In household terms it sounds simple — "move the mug 10 centimeters to the right," or "pour the water over the flowers from a height of 1–5 cm." For a person those instructions are close to trivial; for a robot they are a minefield. It has to recognize not just where the object sits in the image, but also real depth, distances and collision constraints — and, no less important, keep track of whether the task is moving forward or going nowhere.

The authors of RoboBrain 2.5: Depth in Sight, Time in Mind take on exactly those two bottlenecks. They propose the next version of a foundation model for embodied AI, one that learns to think more precisely in 3D and to feel the pace of an action as it unfolds.

Depth in sight: from 2D pointing to metric 3D

Plenty of VLMs handle 2D grounding well: pointing at an object, marking out a region, and so on. Manipulation asks for a different level — how many centimeters, at what height, how to move the arm without clipping anything. In RoboBrain 2.5 this takes the form of a 3D spatial trace: a sequence of keypoints that amounts to a draft of the manipulator's future trajectory.

The key engineering choice is representing a point as (u, v, d) — image coordinates plus absolute depth. A 3D point follows easily from that if the camera parameters are known, which on robots they usually are. The format spares the model from guessing camera geometry and fits neatly with existing 2D datasets: you can drop depth for the moment, or reduce the task to start and end points.

Training is laid out as a curriculum: first the model learns to locate objects in 3D, then to measure metric quantities such as clearances and distances, and only then to generate a manipulation trace that crosses the scene without collisions. That is an important bridge between understanding an instruction and being able to carry it out.

Time in mind: telling progress from stalling

The second half of the update is dense temporal value estimation. The idea is straightforward: a robot needs continuous feedback while it works, not just a final success/failure verdict. In the real world a robot can get stuck, drift off target, take a step backward or end up in a broken state. A model that recognizes those modes from video can be used as a dependable signal for closed-loop control and as a reward for RL.

The authors label progress not crudely by elapsed time but through normalized hops between states. That keeps the signal stable: it stays within a bounded range and drifts less when video is sliced densely into frames. On top of that, three views of progress are fused — step-to-step, from-start and to-goal. There is also a forward/backward consistency check: when the estimates contradict each other, the model lowers its confidence in its own signal. It reads like a guard against hallucination in unfamiliar situations, which matters a great deal if you plan to build an automatic reward on top of it.

Data and training: betting on scale, but with physical meaning

RoboBrain 2.5 is trained on roughly 12.4 million curated examples, mixing general multimodal data with spatial and temporal tasks. The value-estimation pool stands out in particular: tens of millions of frames to begin with, then a sample drawn from them for training.

An interesting practical detail is the infrastructure. The authors discuss training stability on long sequences separately (there are a lot of visual tokens in play), along with fighting memory fragmentation and even cross-accelerator training on Nvidia and Moore Threads hardware with closely matching convergence. That is a signal the model really is being prepared as a production artifact, one you can carry between platforms.

How it held up under evaluation

On 2D spatial benchmarks RoboBrain 2.5 pulls clearly ahead of its predecessor RoboBrain 2.0 and of several general-purpose models. The real story, though, is 3D metric accuracy and tasks with physical constraints. On VABench-V and ShareRobot-T, where distance error is what counts and lower is better, the model posts the best numbers of anything compared.

On the temporal side the results look especially convincing because of the time-reversal check: play the video backward and a sound model should flip the sign of progress. Many strong models score high on VOC+ but not on VOC- — they are guessing the direction rather than understanding it. RoboBrain 2.5 holds up evenly in both directions on real, synthetic and human datasets, which is what dependable feedback requires.

Why this matters

RoboBrain 2.5 helps robots be robust, metrically accurate and capable of noticing mid-action that something has gone wrong. Pairing 3D traces, as a legible intermediate plan, with dense progress estimation, as a continuous quality signal, looks like a solid base for systems that have to work outside the lab — in an actual home, workshop, factory or store.

💾 Code