Google's AgentHands syncs XR agent gestures to speech, word by word

Google has built an XR prototype in which the assistant has hands. AgentHands, the work of Google XR research scientist Xun Qian and Ruofei Du, who leads interactive perception and graphics there, takes a language model's spoken answer and attaches synchronized three-dimensional hand gestures to it — pointing at an orchid, tracing its aerial roots, miming the turn-and-press on a 3D printer's control knob. In a within-subjects study with 12 participants, the gesturing agent scored clearly above a speech-only baseline on understanding where objects were, understanding what to do, and noticing warnings. Both conditions spoke identical pre-written text. The hands were the only variable.

The problem AgentHands is aimed at is specific to headsets. Project Astra and Gemini 3.1 Flash Live already let a user talk about physical space in real time, usually by drawing boxes around objects in the camera image. On a flat screen that works. On an immersive platform like Android XR it is a category error: the interface elements float over the world rather than being in it, and the conversation stays two-dimensional while the user is not. The alternative the team went after is to give the agent the same channel people use — in ordinary conversation a person does not only point, they describe shape, act out a procedure, and mark which part matters.

The project builds on Google's earlier Human I/O and Sensible Agent work, and its contribution is to turn verbal instructions into physical demonstrations. Before building anything, the authors ran a formative study with Google experts in XR and human-computer interaction, asking what makes a virtual hand legible in a three-dimensional environment. The answers became a multidimensional taxonomy describing how an agent should use hands to anchor a conversation to the user's physical space. It has six dimensions: leading hand, gesture type, spatial properties, temporal dynamics, interactivity and visual effects.

The pipeline starts with knowing what the user is looking at. A lightweight object registration module combines gaze with scene reconstruction; the user marks an item quickly — an orchid, a laptop — and the system builds a spatial registry of three-dimensional bounding boxes the agent can refer to during the conversation.

On top of that sits a gesture library, sorted into three semantic categories: deictic, iconic and expressive. Deictic gestures address specific objects. Iconic ones depict actions or shapes. Expressive ones carry social signals and mood.

When the user asks a question, a server-side language model produces an answer with gesture events embedded in it. Each event is bound to particular trigger words and carries motion parameters drawn from the taxonomy's dimensions. A local parser running on the headset then synchronizes speech output with hand movement using word-level timestamps, so the hands start moving at the exact fragment of speech they belong to and point at objects in the room while doing it.

That is the whole trick: a standard language model response becomes a multimodal performance, with the agent speaking and executing spatially accurate movements at the same time. Complicated instructions can be shown in the exact places in the physical environment they refer to.

The authors demonstrated three uses. In orchid care, the agent does not stop at telling you to check the roots — it brings its hands to the base of the plant, traces the aerial roots and explains what they do. On a 3D printer it can show the precise sequence, turning the knob and pressing it to open the right menu item and select a file, which is a meaningful gain when the physical interface is dense. In a habit-coaching scenario the agent acts as a wellness coach and can hold the user's hand back, perform an interactive warning gesture and add a visual effect to discourage an unhealthy action.

To measure the effect, the researchers ran a within-subjects study with 12 participants against a speech-only baseline, using the same researcher-scripted text in both conditions. Participants performed two procedural tasks: plant care and a technical operation on the 3D printer. Across the system-perception questionnaire, AgentHands came out well ahead of the voice baseline on object location understanding, action understanding and warning noticeability.

The study is where I would put a flag. Twelve participants, scripted speech and a ten-item questionnaire on a seven-point scale measure whether people felt the gesturing agent was clearer — not whether they were faster, made fewer errors, or actually turned the right knob. For two procedural tasks with obvious objective measures sitting right there, the absence of task-completion data is conspicuous. Hands are also a strong novelty stimulus: a floating blue avatar doing something in a headset will rate well against a disembodied voice almost regardless of whether the gestures were correct. None of this makes the result wrong. It makes it a perception finding, which is a smaller claim than the demos imply.

The engineering is quieter about a harder number. The gesture events come from a server-side language model, unnamed, while word-level synchronization happens locally on the headset — a split that exists precisely because round-tripping to a model is slow. Nothing in the write-up says how long the user waits between asking about the orchid and watching the hands move, and in a conversational medium that latency is the product. Nor is there any figure for how often gaze-based object registration picks the wrong thing, which in a system whose entire value is pointing accurately at the right object is the failure that matters most.

Google frames AgentHands as a step toward systems that not only interpret the world but act in it through movement, with speech-synchronized gestures binding conversation to physical objects, lowering cognitive load on complex tasks and making spatial computing more approachable. For the Android XR ecosystem the team is also looking at gesture personalization, so the agent can account for a user's dominant hand and remember their characteristic spatial habits. The bulk of the research was done by Ziyi Liu during a student researcher stint at Google, with the project spanning several teams; the authors credit David Li, Zhongyi Zhou and David Kim for contributions, and Adarsh Kowdle, Guru Somadder and Shahram Izadi for guidance and review.

An agent that gestures convincingly at one registered orchid is a demo. An agent that gestures convincingly at your kitchen needs a persistent, accurate, continuously updated model of every object in it, and the registration step here still starts with a human looking at a thing and marking it. The hands are solved well ahead of the world they are supposed to point at.