Human experience helps an AI agent handle unfamiliar programs.

AI agent skills can be written in plain English

Ask an agent to book a table, find backup restaurants, and save the best ones to its favorites. It might manage in a quick demo. But in a real interface, the task has several interdependent parts: the agent must remember the booking details, explore alternatives, mark the right restaurants, and only then confirm the reservation. A successful click doesn’t mean the whole task is done.

This article looks at how to turn people’s experience with graphical interfaces into instructions AI agents can use. The authors propose Work2Skill: a person completes a task, the system records their actions and what appeared on screen, and another agent studies the recording to create a reusable skill.

The core idea is simple: preserve the rules that help get the task done. For example, what makes a restaurant a suitable backup, what order to follow, and how to check that changes were saved.

Work2Skill captures human experience, organizes it, and turns it into skills for AI agents.

Why agents need more than action logs

Coordinates show where someone clicked, but not that they clicked the “Calendar” button—or why they chose it. A screen recording may show the outcome but miss an important intermediate step. And a long video is difficult to fit into a model’s context.

Work2Skill addresses these problems with three connected components. WorkRecorder captures the screen, mouse and keyboard actions, and—when available—information about interface elements, such as a button’s name and role. WorkDemo-350 stores selected recordings. An agent then studies the relevant parts and writes a skill.

Human actions are linked to screen frames and available information about interface elements; the resulting skills are then tested on tasks.

The recordings are aligned on a shared timeline. A click can be matched to a screen frame and any available information about its target. If it’s unclear what the click did, the agent can check later frames or watch the relevant section of the video. It decides which materials to inspect and can return to the recording after drafting the skill.

This is different from imposing a rigid sequence such as “read all the action logs, then watch the video, then write the instructions.” Questions may come up as the agent works on the skill. For instance, it might realize the instructions should say to verify that changes were saved, but find no evidence of that in the recording. It can then search for the relevant moment.

The result is a file of step-by-step rules for a particular type of task. It should describe how to choose between options, which steps depend on others, and what visible signs confirm that the task is complete. The agent doesn’t need the original recording when it later performs the task.

Data and evaluation

The authors built WorkDemo-350, a collection of 350 recordings of successful computer use. It includes 64 tasks from MyPCBench and 286 from OSWorld-Verified. The recordings were reviewed, and unsuccessful or incomplete attempts were excluded. On average, each task involved about 32–35 actions and roughly 24 images. The videos add up to about 8.5 hours.

The researchers compared four models: GPT-5.5, Opus-4.8, and two versions of Qwen3.5. Each model received the same task and interface controls; only the instructions changed. They compared performance with no skill, with instructions from two existing methods, and with Work2Skill skills. They measured average scores on a 0–100 scale and the share of tasks completed without errors.

On MyPCBench, Work2Skill raised average scores by 4.67–10.54 points over runs without a skill. The share of tasks completed in full increased by 6.25–25 percentage points. On the selected subset of OSWorld-Verified, scores rose by 3.11–7.50 points, while the full-success rate increased by 1.40–7.69 percentage points.

Work2Skill outperformed the comparison methods on both average score and full-success rate for all four models across both benchmarks. But whether it saved steps depended on the model: the skill helped both Qwen versions and Opus work more efficiently, while GPT-5.5 took more steps on some tasks. The number of tokens needed to create a skill also varied. In other words, a more useful instruction doesn’t necessarily cost less.

Work2Skill skills improve average scores and the share of tasks completed in full across models on MyPCBench.

A skill should work on a changed task

Testing a skill only on the task it was created from says little about whether it can be reused. So the authors modified 64 MyPCBench tasks: they changed objects, dates, numbers, or requirements while keeping the same application. For example, a booking task might use a different destination city and different seat preferences.

The agent received the original skill without additional training or access to the recording. Compared with runs without a skill, average scores rose by 3.93–8.64 points, and the share of tasks completed in full increased by 6.25–9.38 percentage points. Agents also took fewer steps on average.

This suggests that an instruction can convey a rule, not just specific values from a recording. It’s more useful for an agent to know how to choose a suitable option than to memorize the one a person picked.

The authors also tested whether skills transfer between models: one model created a skill, and another used it to complete the task. Skills improved performance for both models compared with runs without instructions. However, the weaker model still lagged well behind the stronger one. A well-written procedure doesn’t erase differences in the models’ capabilities.

When intermediate steps matter

The restaurant-booking example shows why a skill needs to spell out dependencies and checks. The task was to book a table for ten at Cotogna, find three backup restaurants, and add the two best to favorites. The main reservation was to be confirmed only after the backup options had been handled.

Two earlier methods helped complete the main booking but missed the additional requirements: they didn’t check all the restaurants, save the favorites, or verify that the reservation was still there after refreshing the page. Each scored 55 out of 100. Work2Skill met all eight evaluation criteria.

On the booking task, the two comparison methods scored 55 out of 100, while Work2Skill met all eight requirements.

The skill lays out the sequence explicitly: keep the date and party size for searching alternatives, don’t include the main restaurant among the backups, save the right restaurants to favorites, and only then confirm the reservation. At the end, check that both the reservation and the favorites are still there after refreshing the page.

Instructions like these help the agent keep the whole task in view. It can complete the main steps and still forget a secondary requirement, especially when switching between screens. A skill reminds it not just what to click, but what else must be done before the task is complete.

What remains to be tested

Additional experiments suggest that interface-element information, images, and action logs are generally useful. Removing them lowered performance. A prompt that prescribed an order for reviewing the materials also performed worse than Work2Skill’s more flexible approach, while using more tokens to create skills.

Still, the dataset contains only selected successful recordings. It shows how to extract rules from tasks completed successfully, but doesn’t fully answer how reliably the system learns from mistakes, incomplete recordings, or ambiguous actions. Interface information also depends on the application: names and roles for elements aren’t always available. And checking that a skill file is well-structured doesn’t guarantee its instructions are correct.

The practical takeaway is that human experience can be turned into verifiable procedures that spell out the order of operations, rules for choosing between options, and signs of successful completion. Work2Skill shows that these skills can help different models and transfer to modified tasks in the same application. The next question is how well the approach will handle unfamiliar software—and recordings in which the person makes mistakes.