NVIDIA's RoboLab says 70 runs can't tell two robot policies apart

NVIDIA has released RoboLab, a simulation platform for evaluating general-purpose robot policies, along with RoboLab-120, an opening benchmark of 120 human-curated tabletop grasp-and-move tasks. The sharpest part of the release is not the software but a statistical argument aimed at everyone else's papers: at an observed success rate of 90% over 70 runs, the 95% Clopper–Pearson confidence interval runs from 80.5% to 95.9% — a width of 15.4 percentage points. NVIDIA states that most published robotics benchmarks do not use enough runs to compare two policies with statistical significance. The platform was built by a team in NVIDIA's research division that includes researchers affiliated with the University of Sydney and the University of Toronto; the paper and source code are on GitHub.

Start with that claim, because it reframes most of the numbers the field currently reports. Both the physics engine and the policy contain randomness. A single success rate over N runs says almost nothing about how much to trust it as a measure of real capability. A robot that finishes 9 out of 10 attempts might be a policy with a true success rate of 90%, or 80%, or 95% that got lucky on a small sample. NVIDIA's proposed remedy is Clopper–Pearson, which builds an exact binomial confidence interval for the success fraction directly from the binomial distribution.

The cost of precision here is brutal and rarely paid. Tightening the interval at 90% success from 10 percentage points to 2 requires roughly 15 times more runs — from 70 to 1,030. At 1,030 runs the margin falls to about ±2 percentage points, an interval of 88.0% to 91.8%. That is the whole argument: the difference between a benchmark result that means something and one that does not is an order of magnitude more compute per policy.

The reason simulation carries this load at all is that testing on real robots is expensive, slow and reproduces badly, so large-scale evaluation happens in simulation. Existing simulated benchmarks carry their own defects. The first is contamination: in most cases the training data and the evaluation environment come from the same visual source. If a model is fine-tuned on simulation data and then tested in that same simulation, a high score only shows that it memorized a particular configuration, and says nothing about whether it generalizes. The gap persists because simulated imagery still does not match real-world visual quality. Real-to-sim pipelines address part of this by reconstructing photorealistic environments from captures of real scenes, including through Gaussian splatting — but preparing a single scene can take more than an hour, which rules the approach out for testing at volume.

The second defect is saturation. Most benchmarks fix their task set and update it rarely. Models reach ceiling performance on a static set, and it becomes hard to tell which one is genuinely more capable. Once every system reports above 90% success on the same benchmark, the numbers stop carrying information.

RoboLab also refuses to commit to a single robot. Building a general-purpose policy means solving a long tail of specialized tasks, and one team rarely holds large data volumes for every robot type — a lab may have thousands of hours for a Franka manipulator and almost nothing for a humanoid, or the reverse. A benchmark built around one embodiment forces everyone into the same constraint regardless of what they are actually building. RoboLab's tasks are tied to neither a robot nor a policy architecture: the same task set runs across different embodiments and different policies, with the user choosing the robot and the approach while RoboLab assembles the same scenes and tasks for the selected system. As the number of robot designs grows, this keeps the focus on whether the robot solves the task rather than which machine was used to collect the data and train.

The task design follows the same logic. A useful benchmark should isolate individual capabilities rather than simply record that a task was completed; by the authors' observation, general manipulation tasks draw on at least three distinct competencies. Each RoboLab task is designed to probe one or more specific abilities and tagged with the competencies it requires, so skill coverage stays visible and balanced, and can be rebalanced as new tasks are added.

Success rate alone also does not show how the robot did the job. A policy may pick up the right object and drop it early, scoring a failure. Another may finish only after jerky, slow or erratic motion and still be logged as a success. RoboLab adds three more evaluation instruments alongside the success rate. An event log automatically records grasping the wrong object, dropping an object, and gripper collisions with the environment. Given the task "put all the plastic bottles in the container," a policy might collect every bottle and deposit it correctly while also adding an orange along the way — formally complete, but the robot grabbed the wrong item before finishing.

A built-in dashboard shows what happened during an episode and lets the user jump straight to the frame where the error occurred. That turns diagnosis from manual post-hoc review into something closer to a debugger for robot behavior: not only whether the policy worked, but where it stopped working and under what conditions.

The robustness tests are where the interesting failures are. Real instructions vary in phrasing, real scenes are cluttered, and real tasks run long. RoboLab lets a task carry several instruction variants — the initial benchmark provides three, vague, standard and specific — selectable at run time. The authors found that vague instructions consistently produce errors: current models remain sensitive to wording. Excessive detail sometimes degrades results too.

Scene complexity gets the same treatment, with distractor objects and visual noise added to test whether a policy still selects the right target. And many real tasks are chains of dependent subtasks, where a small early error produces total failure at the end — to put mugs in a cabinet, the robot has to open the door first. RoboLab lets a developer specify the expected subtask sequence and track how far along it the policy gets. The authors report that most policies struggle with long tasks: none of the systems tested completed more than four complex subtasks in sequence. Which systems those were is not stated.

That four-subtask ceiling is the most consequential number in the release, and it sits several sections down from the top. It is also the one result that a simulation benchmark can deliver cleanly, since chaining failures do not depend on photorealism. The rest of the argument deserves more scrutiny than it will get. NVIDIA is telling the field that credible evaluation requires roughly fifteen times more runs per policy, and NVIDIA sells the hardware and the simulation stack those runs consume. The statistics are correct; the party making the case is not disinterested.

There is also a gap in the diagnosis. NVIDIA identifies the visual fidelity of simulation as a core reason existing benchmarks mislead, notes that the real-to-sim reconstruction methods which close that gap cost over an hour per scene, and then proposes a platform built for scene setup in minutes. How RoboLab itself avoids the sim-to-real visual gap it names is the thing the announcement is quietest about — and it is precisely the property no amount of in-simulation confidence interval can verify.

For the sensitivity question, RoboLab avoids testing one variable at a time, which becomes impractical at scale. It runs evaluations across many scene variants simultaneously and applies sensitivity analysis to show which environment parameters correlate most with success or failure. For episodes run with condition variation \(\theta\) and observed outcome \(x\), such as task success, it uses the posterior distribution \(p(\theta \mid x) \propto p(x \mid \theta)p(\theta)\), estimated via neural posterior estimation (NPE). That turns a hunch like "maybe camera position matters" into a measurable result.

NVIDIA's own conclusion is that robotics benchmarking lags visibly behind other areas of AI research, and that without a common industry platform progress is hard to measure. As policies improve, success rate alone will not show whether a model generalized or merely memorized the test conditions. The field needs benchmarks that extend rather than saturate, metrics that explain failures, and analysis that tells researchers how to improve a policy rather than only how well it scored.

RoboLab's research is already feeding NVIDIA Isaac Lab-Arena, the company's open simulation framework for large-scale policy training and evaluation, with key RoboLab capabilities slated for the product in August 2026. Which means the ruler and the workbench will ship from the same vendor — and the claim that nobody else's robot numbers survive a confidence interval will be tested inside the stack of the company that made it.