Nvidia opens its medical physics simulator, with vessel walls still rigid

Nvidia has released Medical Physics Simulation, an open-source GPU-accelerated platform inside Nvidia Isaac for Healthcare for building digital twins of anatomy, simulating how devices interact with the body, and training reinforcement-learning policies on the result. The endoluminal module — long flexible instruments moving inside body cavities — is now generally available; a surgical module is in early access. Everything runs inside Nvidia Isaac Sim and Nvidia Isaac Lab, with simulation and training on the same GPU, and Nvidia is shipping generative world models alongside the classical solvers.

Medical robotics has constraints that self-driving cars and factory arms do not. There is no internet-scale corpus to collect and no way to run unlimited experiments on real patients: every demonstration needs specialized hardware, clinical specialists and access to patients or laboratory environments. Most teams have a few hundred demonstrations. Robust systems need tens of thousands.

Volume alone would not close the gap. Medicine is a long-tail problem — rare anatomy, difficult patient physiology, complications and failure scenarios show up too infrequently to accumulate in real datasets, and those are precisely the cases that decide clinical safety. Imitation learning runs out at the edges of its distribution. Reinforcement learning can push past that edge, exploring millions of interactions and stress-testing strategies, but only against a simulator that is realistic enough to produce useful behaviour and fast enough to train at scale.

The third constraint is calendar time. Medical robots are checked on physical organ models, cadavers, animals and limited clinical studies — necessary stages that run sequentially, cost a great deal and scale badly. A change to a design or an algorithm takes months today, and a full development cycle runs four to seven years.

The platform combines classical physics solvers with simulators built on generative world models, packaged as modules for interactive surgical and interventional simulation, robot learning, synthetic data generation and procedure development. Keeping simulation and training on one GPU removes the constant transfers between CPU and GPU that add latency to a machine-learning loop.

The endoluminal module ships as a standalone package that can be attached independently to different workflows and environments. Its first version demonstrates catheter navigation through the vasculature under fluoroscopy. It is written in Python on Nvidia Warp and Newton Physics, and represents flexible instruments as Cosserat rods — a mathematical model describing bending, twisting and stretching. Extended position-based dynamics, XPBD, was chosen as the main method for computing the nonlinear dynamics of such one-dimensional rods efficiently on GPU.

XPBD normally works through local iterative constraint projections, which for a long instrument can take many iterations to carry motion from the steered proximal end to the distal tip. Here the coupled rod constraints are assembled into a matrix system instead. Each segment contributes six constraint equations — three for stretch and shear, three for bending and twisting — producing a block-tridiagonal XPBD system with 6 × 6 blocks, solved by the Thomas algorithm in time linear in the instrument's length. Independent instruments are processed simultaneously in vectorized GPU environments.

The globally coupled solver carries an input — inserting the catheter, rotating its proximal section — along the whole instrument in a single simulation step, so the distal tip responds immediately even on very long devices.

Contact between catheter and vessel is also computed in parallel on GPU. Signed closest-point queries run directly against the patient-specific triangle mesh, with no separate distance field built in advance; samples that penetrate the wall are returned to the lumen while keeping the ability to slide along the surface. The current version supports rigid one-sided containment inside the vessel. Two-way interaction with deformable vessel walls is planned for the future.

Control is expressed through boundary conditions and the rod's rest state. Insertion moves the clamped proximal end along the introducer axis, hub rotation sets its orientation, and steering the distal end changes the rest curvature of the final segments rather than applying an artificial external force; the bending and twisting constraints propagate the change along the instrument. The module talks to other Newton solvers through a shared API, and wp.from_torch and wp.to_torch keep the control and reinforcement-learning interfaces on GPU with no data copying. That is what lets it plug into Nvidia Isaac Lab, which manages environments and runs while the solver computes dynamics and returns tensors that stay on the device for the observation, reward and control loops.

Nvidia ran experiments with 512 parallel environments and 1500 training iterations. Physics reached roughly 1300 Hz in a single environment and about 60 Hz across all 512; the full simulation-and-rendering loop ran at about 63 frames per second, at 256 × 256 pixels. Nvidia presents this as groundwork for training policies directly from fluoroscopic images and other pixel observations.

The catheter workflow joins two linked patient-specific components. A vascular digital twin package produces the attenuation and geometry data — a mu_volume, a vessel mask or mesh, and a centerline — and the endoluminal package computes catheter mechanics inside that anatomy under XPBD constraints. On each frame, solver state is converted into catheter segments for rendering, and a fluorosim system composites them with an image in the same space derived from CT, with detector-response effects added to make the output more realistic.

The surgical module, in early access, is a GPU-accelerated platform for real-time surgical simulation: soft-tissue deformation, surgeon actions, haptic feedback and rendering in a single pipeline. It is also Python on Warp and Newton, but structured differently from a monolithic physics engine — its functions are organized into explicitly specified sequential simulation systems, and each procedure is configured by the order of operations: deformation, collisions, grasping, cutting, clipping, diathermy, rendering. The GPU compute pipeline can be inspected, reconfigured and extended.

Soft tissue is represented as tetrahedral meshes, deformed by position-based dynamics with distance- and volume-preserving constraints; corrections are accumulated and averaged on GPU, which reduces artifacts caused by constraint ordering and suits a parallel Jacobi solver. Deformation, collision queries, grasping, heat propagation, topology updates and surface reconstruction all run on GPU, CUDA graph capture cuts kernel launch overhead, and CUDA Interop hands data straight to the renderer instead of copying it back through the CPU. The result runs in real time above 30 frames per second, on a single consumer graphics card, with eight physics substeps and eight inner deformation-constraint iterations.

The first version supports a complete cholecystectomy. The scene opens with robotic instruments already inserted into an insufflated abdomen, showing liver, gallbladder and connective fat tissue; the procedure runs through dissection of the hepatocystic triangle, clipping and cutting the cystic duct and artery, and separating the gallbladder from the liver.

The generative side works from the opposite direction. Instead of programming each interaction explicitly, a world model learns from video and other data what real scenes look like and how they change over time, tying visual observations to motion and to control signals including robot actions, then predicting the next video frames at inference. Because the output lives in the visual space that cameras and robot policies actually see, it can reproduce detail and appearance that would be laborious to author by hand in a conventional simulator, and it cuts the manual work of building objects, materials and lighting. The limitation is stated plainly: learned predictions can look realistic without offering the explicit physical guarantees a numerical solver provides.

Nvidia supplies Cosmos world foundation models for this. The Cosmos video models use a flow-based diffusion transformer architecture with a rectified-flow formulation, iteratively turning noise in a compressed video representation into a coherent sample from the learned distribution; an encoder compresses frames from pixel space into a latent representation and decodes generated latents back into video, while multimodal conditioning sets properties of the scene. For medical work, the broad visual model is further trained on domain data covering specific anatomy, devices, procedures and dynamics.

Three Cosmos-H models are shown. Cosmos-H-Surgical-Predict generates plausible future surgical video from an initial image and text. Cosmos-H-Surgical-Transfer produces surgical video guided by signals such as depth, edges, segmentation or blur. Cosmos-H-Surgical-Simulator predicts the next frames from the kinematics of a surgical robot. Together they cover synthetic data generation, controlled domain transfer, action-conditioned robot learning and policy evaluation. They do not replace explicit physics everywhere, but they remove manual scene construction where first-principles modelling is impractical.

Cosmos-H-Dreams pushes this into interactive territory. A teacher model fine-tuned from Cosmos-H-Surgical-Simulator is turned into a causal student model that autoregressively produces future surgical video in a small number of steps in response to robot actions. Running through the FlashDreams inference mechanism, it converts a high-quality offline world model into a real-time environment generating above 30 frames per second, drivable by hand or queryable in closed loop by a trained robot policy on a single workstation GPU.

The performance table is worth reading as a statement of where this is rather than where it is going. 1300 Hz in a single environment becomes 60 Hz across 512, and parallel throughput is the number reinforcement learning actually consumes; 1500 training iterations shows that the loop closes, not that a policy exists. The more telling limit is the one Nvidia states outright — rigid one-sided containment. The vessel does not push back. To me that is the line between navigation and safety: you can learn to steer inside a rigid tube, but what you cannot learn there is what happens when the anatomy yields, and two-way deformable interaction is on the roadmap rather than in the release.

The announcement is also quiet about validation. It opens by naming a four-to-seven-year development cycle as the problem to be solved, and nothing in it claims that cycle has shortened. It describes simulators, not evidence that a policy trained inside one behaves the same way against real anatomy — and that step, not frame rate, is what stands between a fast solver and a device that touches a patient.

Nvidia sets the terms itself: classical simulation gives the physically consistent, deterministic environments that trust and quantitative evaluation rest on, and generative simulation gives diverse, visually realistic environments at a scale classical simulation cannot reach. The cheap experience comes from the models with no physical guarantees; the dependable experience comes from a solver running 512 environments at 60 Hz. Whichever of the two ends up supplying most of the training data will also decide what kind of evidence exists for the robot that comes out of it.