Review · 2025-10-01

Pointing at a pixel beats text commands for drone navigation

Navigating from written instructions has been a hard problem for autonomous drones for years. Classic reinforcement learning approaches need large datasets and transfer badly to new domains. The recent wave of vision-language-model solutions promised generality, but usually asked the model to emit its commands as text: turn, fly, ascend. Language turned out to be a clumsy carrier for precise numbers, and it teaches the model nothing about the three-dimensional mechanics of flight. The researchers propose an unusually sensible replacement: stop describing the motion in words and just point at the spot in the frame the drone should move toward.

Language-prompted UAV control with no fine-tuning: (a) the drone replans continuously to keep pace with a moving person; (b) it chains targets together through a hallway; (c) it locates a person on the ground and flies around obstacles. The colored 3D boxes mark successive camera viewpoints and the full flight path over a reconstructed point cloud. Every waypoint comes straight from the vision-language model, with no task-specific training.

Pointed pixels instead of words

SEE-Point-Fly (SPF) is an open framework for aerial vision-and-language navigation. At each step the system feeds the VLM the current camera frame and the instruction. The model returns a small structured plan: the image coordinates of a waypoint and a discrete estimate of how far to travel. In the same pass it draws boxes around obstacles, so the drone is never steered into an obvious hazard. None of this requires teaching the model to fly — the basic visual-spatial understanding it already acquired from general data is enough.

The camera frame and the instruction go into a frozen vision-language model, which returns JSON with a 2D waypoint and obstacle boxes. An Action-to-Control layer turns that into low-level motion commands — yaw, throttle, pitch — for the UAV; the loop repeats until the task is finished.

From a point in the frame to motion in the air

From there it is geometry. The pointed pixel and the step distance are lifted out of 2D into local 3D offsets using a standard camera model. The system converts them into yaw, pitch and throttle commands and hands them to the flight controller. The key ingredient is adaptive step scaling. With open space around it the drone accelerates confidently; close to objects it eases off. That is what makes the flight both fast and safe. The loop is closed: the VLM re-estimates the point a few times per second (roughly 0.3–1 Hz) while low-level control runs faster (around 10 Hz), for a total latency of 1.5–3 seconds. That is enough to follow even a moving target reliably.

The control geometry of VLM-guided flight: the frozen model predicts a 2D point (u,v) and a discrete depth; (a) a nonlinear scale turns it into an adaptive step — larger in open space, smaller near obstacles; (b) the 2D point plus the step, passed through a pinhole model, gives a 3D offset in the UAV frame; (c) the offset becomes yaw, pitch and throttle; the commands are issued over a timed window, and the loop repeats until the instruction is carried out.

What it does in practice

In the DRL simulator SPF reset the bar: 93.9% successful completions against 28.7% for PIVOT and 0.9% for TypeFly. The gap is widest on tasks with obstacles (92% vs 16%), on long routes (92% vs 28%) and in search (92% vs 36%). On real hardware — a DJI Tello EDU — it reached 92.7%, far ahead of strong baselines that either misread the instruction or came apart on physical details. Adaptive stepping paid off in speed as well: across three real-world scenarios completion time fell, in one case from 61 to 28 seconds, with no loss of reliability; where the baselines never finish at all, SPF gets to the goal consistently.

Qualitative comparison of flight paths in simulation: the new method in green, PIVOT in blue, TypeFly in purple. A missing colored path means the baseline failed to produce a flight command at all.

Qualitative comparison of flight paths in the real world: the new method's trajectory next to the other baselines. The takeoff path is marked in green, the task path in magenta.

Completion time by task. The new approach is faster on most tasks, and especially in the harder scenarios. Bars capped at ∞ mark failures of the baseline method.

Why it works

The decisive change is the action language itself. Instead of words and numbers written out as text, the researchers get visually grounded points. An image is the natural medium for a VLM, and the camera parameters make it possible to recover an exact 3D command without elaborate training. Folding obstacle labeling into the same model costs less latency and is more accurate than a separate detector. The system also turned out to be model-agnostic: in simulation the success rate reached 100% on several VLM backends (Gemini 2.0 Flash, Gemini 2.5 Pro, GPT-4.1), held at 93.3% on Claude 3.7 Sonnet and Llama 4 Maverick, and came in at 87% on the compact Gemini 2.0 Flash-Lite.

There are weak spots. The VLM can misread a scene and hallucinate small or distant targets. The step estimate is an indirect quantity, not real depth, and it sometimes produces inaccurate distances. Reaction to fast-moving obstacles is bounded by the VLM's latency. Search behavior is driven by heuristics and prompt style, so an optimal trajectory is not guaranteed. Even with those caveats, the shift from text commands to 2D grounding looks like a robust and practical idea.

💾 Code

Full paper on arXiv

AI paper breakdowns

Every day we read the new AI papers and retell what matters in plain language — no hype, no filler. If you want to see where AI agents are heading before everyone else, subscribe.

New breakdowns every day.

On Telegram