NVIDIA hands robot navigation training to a coding agent

NVIDIA has published a workflow that hands most of the labour of training a robot navigation policy to a coding agent. Working with COMPASS, its cross-embodiment mobility framework, a developer names a robot, a scene source and a navigation goal. The agent then checks dependencies, stages assets, runs smoke tests, starts training, investigates failures and compares checkpoints. The human signs off on three things: accepting the scene, running the single-environment test, and promoting a checkpoint to the next stage. The reference robot is Boston Dynamics' quadruped Spot; the reference environments are the built-in COMPASS warehouse, the SAGE-10K generated scene set, and reconstructions built with NVIDIA Omniverse NuRec. The tutorial drives it with Codex; in Claude Code the same workflow is invoked with /compass.

Navigation is the part of robot autonomy that refuses to transfer. Locomotion keeps a machine upright and moving. Navigation has to localise it continuously, interpret a changing scene, choose a route and avoid obstacles well enough to reach a goal safely. Carrying that capability to a new robot or a new scene means new data, simulation assets, robot interfaces, training, diagnostics and evaluation. Repeat the exercise for every robot-and-scene pair and the cost compounds while reproducibility decays.

COMPASS stands for cross-embodiment mobility policy via residual RL and skill synthesis, and its central trick is that it does not learn navigation from scratch. It reuses the navigation behaviour of NVIDIA's pretrained X-Mobility policy, then trains a residual specialist — a reinforcement learning policy that corrects the base action for the chosen robot and environment. Expert demonstrations collected on a single robot type do the heavy lifting. Several specialists can later be distilled into one policy covering multiple embodiments.

That residual structure is what makes an agent workflow plausible in the first place. There is no frontier training run for an agent to supervise here — there is a long chain of setup, conversion, registration and comparison steps, each of which fails in dull, diagnosable ways. The workflow the agent runs is packaged as skills inside the COMPASS repository, and the $compass blocks in the tutorial are prompts for a Codex chat started at the repository root, not shell commands. Codex exposes repository skills through .agents/skills and selects COMPASS via /skills or a mention of $compass; symlinked skill directories are supported, so the working skill can stay where it lives. When a run fails, $compass-doctor performs a read-only health check and reports likely causes without changing the environment without permission.

The one place the workflow draws a hard line is secrets. The agent may clone the repository, build the project, download non-gated assets and verify the software stack. Accepting the terms for the gated Hugging Face repositories nvidia/COMPASS and nvidia/X-Mobility, and supplying the token, is the developer's job outside the chat. The agent must not request, display or log the token. The token should be read-only, confined to the current shell, and never pasted into an agent prompt or committed to version control. Registered simulation assets land in ./assets/usd/ and the pretrained X-Mobility checkpoint in ./assets/x_mobility.ckpt. A 401 or 403 usually means access was never fully granted or the token is underscoped — fix authentication before blaming Isaac Lab.

COMPASS accepts three scene sources: the built-in COMPASS warehouse, a generated scene from SAGE-10K, and a reconstructed environment prepared with Omniverse NuRec. Each has its own registration, traversability map and scene-approval checks before training starts. The fastest path is the registered combined_multi_rack warehouse, where robot, scene and traversability map are already in the system, which makes it the right way to validate an installation before attaching anything new.

SAGE-10K contains 10,000 generated indoor scenes across 50 room types, each with geometry, materials, layout metadata and a preview. It is a scene set, not a policy and not a simulator. Living rooms and warehouses are prepared identically, so there is no reason to download the whole thing — one suitable candidate is enough. SAGE-10K carries two human checkpoints: open the converted USD file in NVIDIA Isaac Sim and inspect geometry, materials, scale and collision meshes; then, after the scene is registered and a traversability map is built, approve a single-environment preview before committing to full training. The traversability map marks free and blocked space, and from it come the valid robot spawn positions and navigation goals.

NuRec is the option for teams that want to fine-tune and evaluate inside a reconstruction of the actual environment the robot will work in. It converts stereo RGB captures into an Isaac Sim-compatible reconstruction with consistent visual geometry, collision meshes and, where needed, additional scene elements. The COMPASS NuRec process registers the rendered scene, verifies the supplied traversability map and origin convention, evaluates clearances for the robot, and runs a single-environment smoke test before training. NVIDIA keeps it an optional path in this material and points to the COMPASS NuRec workflow and the Isaac Sim NuRec guide, including a living-room example, which cover scene preparation, training, evaluation, export and ROS 2 deployment.

Before any training scales up, the scene goes through a single-environment preview. The checks are unglamorous and exactly the ones that catch a broken asset: Isaac Sim starts, the scene loads, Spot spawns in a valid pose, camera images are available, the robot responds to policy commands, and it neither clips objects nor falls nor produces unresolved simulation errors. The agent collects preview logs and visual evidence into a report, flags blocking problems and stops for human confirmation. Residual training begins only once scene, robot, observations and action interface demonstrably work together.

Residual training itself is a long job and the workflow treats it accordingly. Run it in a persistent session or through a managed scheduler, write logs and checkpoints to a configured results directory, and report progress without holding an interactive session open. Before launch, record the command, the repository revision, the scene key, the configuration, the checkpoint interval and the stopping criteria. If the run is interrupted, verify the integrity of the last checkpoint and resume only in a supported mode. The --num_envs parameter follows available GPU memory — one environment for the smoke test — and during training the things worth watching are reward components, progress toward the goal, contacts and falls, episode terminations, throughput and GPU memory. Checkpoints are saved regularly and evaluated under identical conditions, because the final iteration is not automatically the best one. Distributed multi-GPU training is supported for larger runs.

Failures route through the COMPASS diagnostic workflow before anyone touches the environment or the training parameters: authentication problems to Hugging Face access, scene-loading and collision errors to scene preparation, camera and action-interface errors back to the smoke test, out-of-memory to the environment count or multi-GPU mode.

Evaluation compares the base policy against residual candidates under identical conditions. The standard COMPASS report covers goal success rate, fall rate and route traversal time. Anything beyond that — goal progress, contact behaviour, timeout overruns, command stability — has to be labelled as derived analysis or the team's own instrumentation. A checkpoint is promoted only once comparable results pass the project's navigation and safety checks and a human approves packaging.

The striking thing about this workflow is how much of it consists of not letting the agent proceed. Scene acceptance, single-environment preview, checkpoint promotion, dependency changes, reward changes, training-parameter changes — each is gated on a person. The agent's remit is real but narrow: it is a build engineer and a triage engineer, not a decision-maker. That reads like an honest account of where coding agents are actually useful in robotics right now. They are quick at the plumbing that makes simulation runs reproducible, and they have no way of knowing whether a converted USD scene is geometrically wrong in a manner that will quietly poison a policy. NVIDIA is not pretending otherwise, which distinguishes this from most agent demonstrations.

What the material is quiet about is what "good" looks like. It states plainly that scene quality, training duration and checkpoint results depend on embodiment, environment, reward design and available compute, and so the workflow defines no universal success threshold. It also stops at checkpoint evaluation: export to ONNX, JIT or TensorRT, ROS 2 integration and running on physical hardware each require their own robot- and site-specific checks. So the one question a developer brings to this — is my policy good enough to put on the robot — is precisely the part left entirely to them, after the automated part has finished.

The reference integration at least specifies the boundary. In ROS 2, the compass_inference component converts front camera images, a navigation goal or route, and robot velocity computed from odometry into inputs for the exported policy, and publishes forward linear and angular velocity commands on /cmd_vel. Recurrent state and previous action stay internal to the inference implementation rather than crossing the ROS boundary as separate inputs. Export and deployment package the trained policy for inference; they do not hand the developer a base policy and a residual policy as two ROS 2 components to wire together. Before running on a target robot, check coordinate frames, update rate, normalisation, command limits, stop behaviour and the physical controller itself.

If the robot cannot supply compatible and verified odometry and coordinate transforms, NVIDIA cuVSLAM — a CUDA-accelerated visual odometry and simultaneous localisation and mapping library — can provide them at deployment, which matters in GPS-denied or GPS-degraded environments. cuVSLAM odometry can support the COMPASS navigator, but the cuVSLAM map is not an input to the navigation policy. It plays no part in training and needs no agent skill of its own: run it as a standalone ROS 2 component with matching versions, route or remap its odometry into /chassis/odom, publish the required odom-to-base_link transform, and verify calibration, timestamps, topic names and frame conventions. The optional $cuvslam-onboard and $cuvslam-troubleshoot skills exist for setup and diagnosis during development.

For an unregistered robot, $compass-newembodiment walks a developer through robot setup, environment registration, action mapping and a visual smoke test in a single environment. Adding a new embodiment is a distinct engineering job from training a specialist for a robot the system already knows, even though it runs on the same discipline of checks and confirmations.

Once training is done, the agent leaves. The policy and the robot's controller handle navigation on their own; no AI agent is in the loop while the robot is moving. That puts the exercise in its proper place. COMPASS is not making robots smarter — it is making the dozen steps between an asset download and a validated checkpoint cheap enough to repeat. The payoff is measured in how many robot-and-scene pairs a team is willing to attempt at all, and NVIDIA's own recommended order — start with the reference route, then add one component at a time — is an admission that today the number is still close to one.