An inverse dynamics model turns YouTube videos into agent training data

AI agents promise to be useful inside real applications: setting up a browser, editing images, driving a media player. But to hit the right button and not get lost in a menu, they need thousands of good demonstrations recorded inside the target software. That data barely exists: the available datasets are narrow and go stale fast, and synthetic traces tend to be too simple and a poor match for real tasks. The authors of Watch & Learn propose something different: take tutorial videos off the internet and convert them into precise step-by-step trajectories of clicks, scrolls and typing.

The find hiding in plain sight: learning from YouTube

The key engineering move is to frame the problem as inverse dynamics: given two consecutive screen frames, recover the action that connected them. That is simpler and more reliable than guessing where to click from a single frame plus a text instruction. The authors train a dedicated vision model that takes a pair of frames and predicts the action type, the coordinates of a click or mouse movement, and — when the action is typing — the text itself. No brittle heuristics, no dependence on HTML.

How Watch & Learn works

Three steps under the hood

1) Teach the model to read an action. First comes a large corpus of state transitions: roughly 500k automated interactions with live websites plus 132k human ones from Mind2Web — more than 630k triples in total (screen before, action, screen after). On these they train an inverse dynamics model: a SigLIP-2 encoder plus a few Transformer layers, with three output heads for action type, coordinates and text. Coordinates are predicted as classes over a grid, which lifts accuracy noticeably.

2) Mine trajectories out of video. Given a task description, a retriever searches YouTube for relevant tutorials. Videos are filtered frame by frame at 1 fps, keeping only clean, high-quality screencasts. The inverse dynamics model is then applied to every adjacent pair of frames, turning a clip into an executable sequence of actions. At inference time the system keeps the best videos for the task at hand; for training it keeps the whole pool that passed the filter.

3) Use the data twice. The trajectories serve both as in-context examples while solving a task and as training data for SFT. In the first case the steps are annotated with short reasoning; in the second they are plain state-action pairs.

What it does in practice

The pipeline produced 53,125 trajectories across 69 applications — from browsers and GIMP to VLC and system utilities. On OSWorld-Verified the gains are consistent.

• In-context for general-purpose models: Gemini 2.5 Flash +3% (19 → 22), o3 +2.5% (21.8 → 24.3), Claude 4 Sonnet +1.6% (43.9 → 45.5).
• The Jedi agent stack: +2.2% (50.6 → 52.8).
• Training open models: Qwen 2.5-VL 7B — +11.1% (1.9 → 13), UI-TARS-7B — +3.8% (27.3 → 31.1).

The quality of the action labels is what makes it work. On the Mind2Web test set the inverse dynamics model reaches 91.6% action accuracy, against 82.7% for TongUI and 72.8% for baseline systems. Clicks and scrolling are especially strong; typing is harder but still holds up. The ablations make the point clearly: raw frames on their own barely help, structured actions deliver a steady gain, and short reasoning adds a little more on top.

Where the gains are largest

The biggest wins land in domains with a deep library of clear tutorials: Chrome, GIMP, VLC. There the examples are informative and line up well with the tasks. The gains are more modest in VS Code and in OS-level tasks, which are heavy on text and code, and in interfaces that need fine-grained actions such as drag-and-drop, which the current action model does not support.

Why it matters

The work shows that video from the internet is a real source of executable skills for AI agents. A dedicated inverse dynamics model removes the labeling bottleneck and yields data that is equally useful for in-context prompting and for training. That is a practical route to agents that operate confidently inside live applications, without paying for manual annotation.

What to improve next

The obvious next steps are widening the action repertoire (drag-and-drop, keyboard shortcuts), learning to cut long clips into tasks more cleanly, and adding reinforcement learning on top of the extracted trajectories. The video corpus itself can keep growing, too: as the Qwen curve shows, once the data passes a critical mass it starts to pay off sharply.