Nvidia teaches robots to copy human movements from video

How GR00T 2 transfers movements

Engineers usually have to operate a robot remotely for hours to teach it a new task. This approach, known as teleoperation, is expensive and slow, and the data must be collected separately for each robot model. In a research report, Nvidia presents GR00T 2 as a foundation model designed to reduce that work.

Unlike earlier approaches, including GR00T 1, the model does not directly translate a camera image and a language instruction into movements. Nvidia calls it an “action-to-future-video” model: GR00T 2 calculates the next movement while also predicting a short video of the near future. For example, the system estimates how the camera image should change as the robot’s hand lifts an object.

Nvidia trained the model on a cluster with more than a thousand GB300 chips. The training data included about 400,000 hours of video showing people performing different tasks, with cameras mounted on their heads.

To show the robot a new task, a person can demonstrate the required movements using sensor-equipped gloves, special hand grippers or a finger exoskeleton. During training, these recordings are combined with general movement data for the target robot. In the model’s video prediction, the human hand is replaced by a robotic one, after which the system generates the corresponding control commands.

In the lab, a robot with five-fingered hands used this method to light a match, cut a banana and toss packages onto a conveyor belt. It had not been trained on these specific tasks through teleoperation in advance.

Assembling complex objects and a version for humanoids

Task success rates in the tests ranged from 10% to 80%. The model also learned to hammer nails after watching a single video of a human hand in motion, using computer simulation. For 50 selected tasks that GR00T 2 could already perform well after a few demonstrations, less than three minutes of standard training raised success rates from 40% to 95%.

For extended assembly tasks, Nvidia combined GR00T 2 with a higher-level language model, such as OpenAI’s GPT-6 Astra. GR00T 2 handles grasp movements, while the language model breaks the task into steps, marks grasp points in color on the camera image and runs auxiliary procedures if a part falls out.

When researchers scattered toy-model parts randomly across a table, the combined system assembled them in at least 97% of trials. Without the corrective language model and additional fine-tuning, the toy car was assembled successfully in only 23 out of 100 attempts. The team worked out precise actions, such as connecting network cables and chargers, through trial and error on a real robot.

For bipedal robots, Nvidia developed a lightweight version, GR00T 2 Turbo, which processes fewer image details. The model runs directly on the humanoid robot’s Jetson Thor onboard computer. In tests, the robot scooped popcorn, cleared a table and wiped up liquid with a towel held under its foot. It managed the last task in just under half of the attempts.

What stands in the way of reliable operation

The researchers say significant obstacles remain before the system can be used reliably in real-world conditions. The model does not yet process tactile sensor data directly, and sometimes misses small errors, such as screws that have not been fully tightened.

The language-model planner also slows things down. While sorting waste, the robot stood idle for about a third of the total time, waiting an average of more than seven seconds for the model to calculate its next step.

Other labs are also looking for ways to teach robots new tasks with less teleoperation. Dataist covered some of these approaches in issue #5 of AI Radar, on Neolabs.

Dataist’s take: GR00T 2 shows how video data can help transfer human movements to robots and reduce the amount of teleoperation needed for training. But success rates varied considerably across tasks, and waiting for the language model took up a significant share of the time. To work in real-world conditions, the system needs to learn to account for touch and detect errors more reliably.