NVIDIA re-renders old drive data for cameras a car doesn't have

NVIDIA has published a workflow for adapting autonomous-driving perception software to a vehicle that may not exist yet. Omniverse NuRec reconstructs a drive recorded by an existing car into a 3D scene, then renders fresh camera streams from the sensor positions of a target vehicle with different placement, field of view, resolution and calibration. The company says it ran this inside its own automated driving program, where a new camera configuration needed support and the target vehicle's data was not yet available but a library of drives from another car already was. The steps are packaged as skills for coding agents in the public NVIDIA/nurec-skills repository.

The problem being solved is unglamorous and expensive. A perception stack is bound to the car it runs on. Move the same software from an SUV to a sedan or another variant and the sensor placement and calibration change, along with fields of view, overlap zones, body geometry, synchronization and scene coverage. A traffic light lands in a different part of the frame. A curb becomes less distinct. A pedestrian at the edge of the field of view becomes more ambiguous. None of that requires the perception system itself to change for the engineering work to be real.

Collecting and labeling a new real dataset for every vehicle configuration is costly, and early in development it can be impossible: the fleet may not exist, and rare conditions cannot be recorded on demand. Real drive data is still needed to ground and validate the system. What synthetic data buys is the ability to adapt models before a complete target-vehicle dataset exists.

NuRec uses a 3D Gaussian splatting point representation to rebuild a real environment from sensor data and render it back out in simulation. For carline adaptation, two capabilities matter: reconstructing a recorded drive as a scene, and rendering new streams from cameras that differ from the originals. The workflow pushes an existing reconstructed drive through the target sensor rig, cleans up the resulting frames and prepares them for perception training.

The tutorial draws its scenes from the NVIDIA Physical AI NuRec dataset on Hugging Face, which holds more than 1,500 neural-reconstructed drive scenes. Each runs about 20 seconds and was captured from six viewpoints:

a 120-degree wide front camera;

a 30-degree front telephoto camera;

120-degree left and right cameras;

70-degree rear left and right cameras.

Access is gated: accept the license, authenticate with a Hugging Face token, download a single scene.

The target rig in the tutorial is a synthetic sensor configuration from an open example, with camera name, pose, resolution and lens parameters serving only as illustration. The shipped configuration contains one target camera, so the example produces one stream. Each sensor entry holds a colon-separated camera name, image resolution, an F-Theta lens model, intrinsics and the camera's pose relative to the rig. NRE normalizes the name into a logical identifier, turning `camera:front:synthetic:120fov` into `camera_front_synthetic_120fov`. Adding a second camera means a new entry in `rig.sensors` with a unique name, width and height, lens model, intrinsics and a `nominalSensor2Rig_FLU` transform.

NVIDIA recommends pinning the public NuRec container to an immutable digest, and running trajectory export and rendering as two separate operations inside it. The target camera trajectory is exported first, then a sparse single-camera render validates the configuration — the tutorial uses `camera_front_synthetic_120fov` because the target camera faces forward, and the render command loads the exported trajectory and selects the camera with `--camera-id`. Frames are then checked for correct pose, field of view and timestamps. If the sparse render passes, the full sequence is rendered with `--frame-step=1`, repeating `--camera-id` for every validated camera.

The USDZ scene was reconstructed from six source cameras, but the target camera does not have to correspond to any of them. It can differ in position, orientation, resolution, field of view and calibration, provided its pose is expressed in the same rig coordinate system and its lens model is supported by the exporter. Cameras placed far outside the observed trajectory, or aimed at regions the original sensors covered poorly, render worse.

Neural rendering can leave view-dependent artifacts, inconsistent color and tone, and poorly reconstructed dynamic objects. NVIDIA Harmonizer is a publicly available, temporally aware post-processing model that corrects artifacts in NuRec and other neural renders. NVIDIA is explicit about what it does not do: it will not fix bad calibration, will not restore missing scene coverage, and does not substitute for validating the target camera. The `nurec-fixer` skill covers setup, inference, evaluation and optional fine-tuning; getting there means accepting model licenses on Hugging Face, cloning the Harmonizer repository, building a runtime image and downloading the published checkpoints. Camera sequences are processed one at a time.

That disclaimer, read alongside the coverage limit, is the most useful thing in the post, and it defines the method more honestly than the framing does. This is viewpoint interpolation inside a volume that real sensors already observed. You can shift a camera and change its lens; you cannot conjure a viewing angle the original rig never saw, and no amount of post-processing will invent the pixels. "Adapt perception to a car that does not exist" is more precisely "adapt to a car whose cameras sit near where the old car's cameras sat." That is still valuable — most carline variants are exactly that — but it sets the ceiling, and it explains why NVIDIA keeps saying real drive data is still required for grounding and validation.

The more interesting question is one the post never raises: how well does any of this work. NVIDIA states it validated the process on an internal program, and then reports no result from it. There is no accuracy figure anywhere for a perception model trained on NuRec-rendered target-rig data, and nothing comparing such a model against one trained on real data from the target vehicle. The post is meticulous about failure modes, container digests and coordinate conventions, and silent on whether the adapted models drive better. For a workflow whose entire premise is that synthetic frames can stand in for a dataset that does not exist, that is the number that settles the argument.

The bulk of the material, tellingly, is data plumbing rather than rendering. Reconstructing a recorded drive requires synchronized video from one recording rig and matching metadata from the same recording, arranged in a consistent structure: MP4 files from the vehicle's camera recorder, Parquet files carrying rig metadata, with all three Parquet files mandatory for the tutorial's converter and reconstruction script. Calibration must describe every selected camera, ego-motion must span the first to the last exposure, and object trajectories must overlap the same interval in a single world coordinate system; a semantic validator checks those relationships in the resulting NCore sequence. Pose math is to be done in `float64`, with the local `world` frame rebuilt so the first rig pose is the identity. The supplied converter handles a seven-video, three-table schema and produces a sequence in the `separate-sensors` profile — one camera component archive per camera, a shared archive for poses, intrinsics, masks and bounding boxes, with each camera's real resolution and color interpretation preserved. Anything else means copying the open NCore converter template, which is a code skeleton rather than a universal converter, and mapping your own source to camera video and frames, calibration, poses and ego-motion, masks, and object trajectories and bounding boxes. The result can be inspected in the open NCore viewer at `http://localhost:8080`.

The agent packaging is the quiet part of this release. NVIDIA/nurec-skills wraps the stages into skills an AI agent can run: `physical-ai-datasets` for finding and downloading scenes, `nre` for rendering, `nurec-fixer` for post-processing, plus skills for NCore and DiffusionHarmonizer, and `nurec-carline-adaptation`, which adds a compatibility and data-provenance layer without distributing NuRec source or models. Pointed at a cloned repository, a compatible coding agent will validate the USDZ and the cleaned target sensor configuration, print the exact local Docker commands, run the sparse single-camera test, and stop short of the full render if the camera or windshield check fails.

That last behavior is the tell. The judgement about whether a reconstructed dataset is fit to train on — the judgement NVIDIA warns Harmonizer cannot make for you — is being moved from an engineer who knows the rig to an agent working through a checklist NVIDIA wrote. It makes the pipeline reproducible for teams that would never have staffed it, and it pushes the one step that still requires knowing what a correct frame looks like into software.