182 applications helped AI agents solve unfamiliar tasks.

AI Agents Don’t Need More Models. They Need More Software

An AI agent can open a spreadsheet, find the right cell, change its value, and save the file. That looks convincing in a short demo. But add a few dependent actions, a different interface, and state that must be tracked—and the results become unreliable.

The reason is simple: computer-use agents need large collections of examples. They must encounter different applications, interfaces, and multi-step tasks. Existing environments are poorly suited to this. Real applications are expensive to integrate and maintain. Browser-based environments are easier to scale, but they are often limited to similar pages and shallow workflows.

The paper on SoftwareGym proposes a practical way forward: automatically generate small desktop applications, test them through their graphical interfaces, and use the resulting interactions to train agents.

The authors generated 182 applications across 12 categories. On average, agents successfully completed 91.5% of the tasks. A model trained on this data raised its Pass@3 score on the external OSWorld benchmark from 41.39% to 56.97%, even though the training set contained none of OSWorld’s applications or tasks.

Examples of SoftwareGym applications: image, video, spreadsheet, and code editors with multi-step workflows.

Why Existing Applications Aren’t Enough

An agent can be trained on interactions with existing programs: a browser, image editor, email client, or spreadsheet. This approach provides realistic states and genuine workflows. But it has two limitations.

First, the pool of applications is limited. Adding a new one means installing, configuring, and isolating it, then creating tasks and a verification system. Second, an agent may memorize the exact locations of buttons and particular action sequences. It learns to use one editor, but gets lost in a different interface.

Browser-based environments solve part of the problem. They are easier to launch and reset. But they all share the same foundation: a browser, web pages, and markup elements. As a result, their interfaces quickly start to look alike, and complex workflows are difficult to reproduce.

SoftwareGym aims to provide three things at once:

🟠 interface diversity — different navigation schemes, panels, menus, and interaction methods;

🟠 real functionality — the application must actually save files, change its state, and respond to user actions;

🟠 scalability — new environments should be created without manually developing every application.

This matters for generalization. An agent should learn not to memorize instructions like “click the button in the top-right corner,” but to understand the task: find the right object, change it, verify the result, and leave the application in the required state.

What Is SoftwareGym?

The pipeline consists of three connected stages: design, implementation, and verification.

The first step is to specify the type of application—for example, an image editor, music player, email client, or code editor. The system then expands the brief description into a detailed specification.

The specification defines:

🟣 application objects — files, emails, spreadsheets, projects, or media tracks;

🟣 operations — creating, opening, searching, editing, sorting, exporting, and saving;

🟣 complete workflows — sequences of actions with dependencies between steps;

🟣 checkpoints — what should change in the application’s state and what the user should see.

SoftwareGym then creates several interface variants for the same functionality. They must differ in more than just colors and fonts. The system changes the navigation, panel layout, screen structure, and placement of key actions.

To prevent the variants from becoming ten copies of the same application, each interface is represented as a component tree containing windows, regions, panels, and controls. The system compares these trees. Variants that are too similar are discarded and regenerated.

In the second step, another agent turns the specification into a runnable application built with Avalonia. It implements not only the visual appearance, but also the connections between commands, application state, data persistence, and user feedback.

In the third step, the application is tested through mouse and keyboard input. Inspecting the source code alone cannot show whether an agent will be able to open the right window, find an element, perform an action, and see the correct result.

The SoftwareGym pipeline: application design, implementation, GUI-based verification, and correction of detected errors.

If verification uncovers an error, the report is sent back to the implementation stage. The application is fixed, relaunched, and put through the same test suite again. Up to three versions are produced in total.

The results are clear:

🟣 the first version of the applications completed 54.9% of GUI tasks on average;

🟣 after the first round of fixes, the figure rose to 81.1%;

🟣 after the second, it reached 91.5%;

🟣 each application took about 237 minutes, 1,718 model calls, and roughly $70.76 in resources on average.

That is a significant cost, but it covers the creation and testing of an entire application. For generating environments at scale, it is still cheaper than manually developing and integrating a large number of real programs.

How Tasks Are Generated from Applications

SoftwareGym does not ask a model to invent tasks at random. First, the system compiles a list of functions confirmed by successful GUI tests. Unverified or broken capabilities are excluded from the task set.

This prevents a common training-data failure: a function is listed in the description but is not actually available in the working application.

Tasks are then divided by difficulty:

🟠 simple tasks have one main objective and few dependencies;

🟠 standard tasks combine several related operations;

🟠 complex tasks require multi-step state changes, with each step preparing the way for the next.

There are two initial-state modes. In the first, the user provides an existing file. In the second, the application opens in an empty state. Tasks also include distracting objects and elements that must not be changed. The agent therefore has to account for the application’s current state rather than mechanically repeat a familiar sequence.

For example, a task might ask the agent to replace the value in a cell, save the spreadsheet, and leave the search panel open. Success is determined not by whether the agent clicked the right buttons, but by the final state: the value has changed, the file has been saved, and the panel remains visible.

Generating tasks from verified functions: the system connects application capabilities, difficulty level, initial state, and expected outcome.

For each task, SoftwareGym stores the initial state and a programmatic verifier. The environment is reset before every attempt. Once the task is complete, the system checks the outcome: Was the file created? Did the required objects change? Were protected elements left untouched?

This makes the data suitable for both training and evaluation. It captures more than a sequence of clicks: it links the user’s intent to the actions taken and the resulting state.

What the Training Showed

The authors collected agent trajectories in the generated applications and fine-tuned the multimodal Qwen3.5-9B model. Each trajectory was evaluated separately on two dimensions:

🟣 outcome — whether the user’s request was completed;

🟣 process — whether the agent acted consistently, safely, and without unnecessary repetition.

Only trajectories with sufficiently high scores on both criteria were included.

On OSWorld, the model trained exclusively on SoftwareGym data raised its Pass@3 score from 41.39% to 56.97%. Pass@3 means that a task counts as solved if at least one of three independent attempts succeeds.

The improvement appeared across all eight OSWorld domains tested. The transfer itself is significant: the model had not seen OSWorld’s applications during training, yet became better at working with them.

On the SoftwareGym-Bench benchmark, the model reached 40%, compared with 25.8% for the original Qwen3.5-9B. The benchmark contains 120 tasks across 12 categories.

Leading closed models remain stronger, however. GPT-5.4, for example, scored 74.2%, while GPT-5.5 scored 73.3% on SoftwareGym-Bench. SoftwareGym therefore does not turn a small model into a universal agent. It demonstrates something else: carefully verified synthetic environments can teach transferable skills even to a compact model.

One additional experiment is also revealing. Training on a mixture of SoftwareGym and OSWorld data raised the SoftwareGym-Bench score to 47.5%, while preserving an OSWorld score of 58.61%. In other words, synthetic and real applications complement each other.

What Actually Works

The authors separately tested the contribution of the pipeline’s main components.

When strict control over interface structure is removed, diversity drops sharply. The average layout-difference score falls from 83.6 to 4.9 on a scale of 0 to 100. Independent generations often converge on similar screens.

Fixing applications based only on the source code barely improves quality: the score goes from 50.4% to 48.8%. When the developer receives reports based on actual interface interactions, the score rises to 83.1%.

That is why SoftwareGym is built around a closed loop:

🟠 the application is created from a specification;

🟠 it is tested through mouse and keyboard actions;

🟠 errors are tied to specific steps and checkpoints;

🟠 the implementation is corrected;

🟠 the tests are run again.

Interface testing serves as a source of behavioral observations about the program. Without it, a model may consider a function implemented simply because the necessary handler exists in the code.

Limitations

SoftwareGym has several notable limitations.

First, the generated programs are still synthetic. They may look and behave realistically enough for training, but they do not fully capture the chaos of real applications: unexpected dialogs, crashes, long histories of changes, and incompatible file formats.

Second, overall reliability is measured on tasks that the system itself can formalize and verify. This is useful for reproducibility, but it may not cover every type of error that occurs in real-world work.

Third, the cost of tens of dollars and several hours of computation per application is reasonable for a research dataset. But expenses will still grow sharply as the number of categories and functions expands.

Finally, Pass@3 depends on the number of attempts. One successful run out of three is useful for estimating the probability of success, but it does not mean the agent can reliably complete the task on its first try.

Conclusion

SoftwareGym moves computer-use agent training from a world of scarce, manually built environments toward automatically assembled applications. The system creates an interface, implements its functions, tests them through real interactions, and uses only verified capabilities to generate tasks.

Synthetic applications can teach transferable interaction with graphical interfaces, rather than merely memorizing a particular program. The improvement on OSWorld demonstrates this: the model had never encountered the benchmark’s original applications during training.

Future AI agents need environments with three properties:

🟣 diverse interfaces and workflows;

🟣 verifiable final states;

🟣 feedback from real action execution.

SoftwareGym shows how to combine all three in a single scalable pipeline. Real applications are still essential. But they can now become part of a broader system in which new environments are generated automatically, tested before release, and used to help agents transfer skills across different programs.