Embodied-R1 points instead of acting and hits 87.5% on real robot tasks

Why robots need a language of pointing

Robots increasingly see the world through a camera and read our written instructions. But that "knowledge" often fails to turn into the right action: the model knows what a cup is, yet not where to put it or how to get around the objects next to it. This distance between vision and action is the seeing-to-doing gap. The Embodied-R1 team proposes a simple and unexpectedly powerful idea: teach the robot to speak in pointing — points and visual traces drawn on the image. A point, or a sequence of them, anchors the reasoning directly in the scene: here is the object, here is the functional part, here is free space, here is a safe path. The representation ends up independent of any particular robot and legible to both a human and a motion planner.

What the model can actually do

The authors reduce pointing to four basic skills that cover the overwhelming majority of everyday manipulation:

1. REG — find an object from a description and point at it.
2. RRG — pick a free region from a relative query, for example "between the cup and the bowl".
3. OFG — indicate the functional part of an object, for example the handle of a knife.
4. VTG — draw an ordered visual trace of points that spells out a readable motion plan, obstacles included.

What matters is that the trace is object-centric rather than tied to a specific arm or its joints. That is why the solution carries over between platforms.

How it works inside

At the core is a compact 3-billion-parameter VLM (built on Qwen2.5-VL) that first reasons about the scene, then returns an answer: a point or a trajectory. It does not generate low-level actions — a standard planner handles that (CuRobo, for instance), fed with the key grasp and placement points, or with a 3D trajectory recovered from the 2D trace. This middle language buys two things: the generalization of large vision models and predictability for the arm.

Where the data came from, and how training worked without magic

To teach the model not just to understand but to point, the team assembled Embodied-Points-200K: nearly 200,000 verifiable tasks for REG, RRG, OFG and VTG. On top of that there is a corpus of spatial questions, plus a small but important layer of commonsense data so the model does not forget the wider world. Training runs in two stages via RFT — reinforcement fine-tuning on verifiable feedback (a PPO variant called GRPO).

• • Stage 1 strengthens spatial reasoning.
• • Stage 2 teaches pointing and trace drawing, with multi-part rewards: answer format, whether the point lands inside the mask, distance to the target, similarity of the trace to the reference, and even binary success in simulation.

What came out in practice

Across 11 benchmarks for spatial tasks and pointing, Embodied-R1 sets the bar among open models at a modest 3B parameters. On REG/RRG/OFG/VTG it consistently beats specialized systems (FSD and RoboPoint, for example), and on VTG it posts the best RMSE/MAE errors and the best score under an LLM judge. In the SIMPLEREnv simulator the model reaches 56.2% success with no fine-tuning, above strong modular and end-to-end alternatives. In the real world it hits 87.5% success on eight tasks with an XArm manipulator, with gains over strong baselines reaching 62%. Robustness to noise stands out on its own: changing the background, the lighting and the camera height barely dents the result — pointing is what makes the robot's behavior hold up.

Why this works

Points and traces are a compact bridge between perception and action.

They

• • remove the translation step from abstract text into scene geometry;
• • read the same way for different robots and different planners;
• • score cleanly: you can check automatically whether the point landed inside the mask or whether the trace matched.

RFT is what makes this work: plain SFT tends to memorize the shape of the answer and falls apart out of distribution, while training on reward teaches the model to make the decision that actually helps the robot.

What it looks like through the model's eyes

Step by step, Embodied-R1 locates the target object, checks the spatial relations ("to the left of", "between", "closer to"), picks the affordance — the handle of a mug, say — and then emits coordinates or an 8-point trace. That train of thought makes the actions transparent and checkable.

It holds up outside familiar scenes

The model transfers confidently to new environments: other simulators, a different robot configuration (even a two-armed one), and even rough hand drawings — where what counts is not texture but the spatial idea itself.

Where this goes next

Pointing has limits: 2D points and traces will not always be enough for fine force control or for handling soft objects; that calls for pairing with a learned policy in 3D. Another step is long, multi-step tasks: a hierarchical plan in which the top level splits the goal into subplans and Embodied-R1 points them out on the image. Finally, more reliable depth handling and RGB-D fusion would make the model more precise in difficult geometry.

Want to try it?

Project: https://embodied-r1.github.io/

Code: https://github.com/pickxiguapi/Embodied-R1

Datasets: https://huggingface.co/IffYuan

Contact: yuanyf@tju.edu.cn

The main idea is simple: when a robot understands by pointing, it acts with more confidence. Embodied-R1 shows that this language — legible to people and machines alike — really does shorten the path from seeing to doing.