How an AI agent improves interfaces on its own

An Interface That Learns Alongside the AI Agent

Imagine asking an agent to match emails against bank transactions. It opens your inbox, clicks through messages, scrolls around, and copies details by eye. The information is already there, but the agent can’t retrieve it with a single request.

These workarounds are a common problem for computer agents. They have to operate through predefined buttons, screenshots, and tools. When the interface is awkward, a model may reason well and still spend dozens of steps on simple tasks.

The article’s authors propose changing how agents access a computer. Their system, Agent S4, studies agents’ attempts, spots recurring difficulties, and creates new software interfaces. Instead of clicking through emails, for example, an agent could use an interface that returns a list of messages and their contents as text.

Agent S4 studies the agent’s attempts and changes the set of interfaces available to it.

Interfaces Become Part of the Learning Process

People usually design the ways an agent can interact with a computer. They decide what actions are available, what information the agent receives in response, and how it should use its tools. The model itself may change, but the set of interfaces often stays the same.

That leaves the agent to work around the tools’ limitations. To edit a spreadsheet, it might click cells and check the results in screenshots, even though it could simply pass a range of data to a program. To find a function in a file, it might dump the entire file into a terminal window.

Agent S4 proposes letting agents improve their own interfaces. The results are saved in a repository containing the tools’ code and guidance on when and how to use them. The model and its wrapper stay the same. What changes is what the agent can see and do.

The system has two agents. One performs tasks on the computer. The other, a meta-agent, reviews records of those attempts and updates the repository. It can add an interface, modify an existing one, or combine ideas from two different versions.

The meta-agent doesn’t get the correct answers to training tasks or learn whether the agent completed each one successfully. It sees the process: actions, screenshots, tool outputs, and errors. It uses those records to look for recurring problems. For example, if the agent repeatedly edits spreadsheet cells one at a time, the meta-agent might create a function for bulk updates.

New versions are tested on separate evaluation tasks. The meta-agent cannot see their prompts, results, or explanations of errors. These tasks are used only to choose which versions the system should develop further. The authors designed this setup to reduce the risk that a new interface will help only on the tasks it was built for.

How new versions are selected: the meta-agent updates repositories, while evaluation tasks help choose which versions to develop further.

What Changes in Practice

Agent S4 doesn’t just accumulate individual tricks. It changes the actions available to the agent and the responses its tools return. Instead of buttons and screenshots, the agent can get page contents, data from an application, or confirmation that an operation succeeded.

In one example, the agent had to collect expense information from an email account and a banking system. With the standard tools, it opened emails one at a time, worked through download warnings, and scrolled through a list of transactions. The version with an improved interface used a browser to retrieve the email text and attachment links, downloaded the documents, and extracted their text. It then used the same approach to retrieve the bank transactions.

A fixed set of tools forces the agent to rely on clicks and screenshots, while new interfaces provide direct access to data.

The changes also extended to specialized software. For the ScienceBoard benchmark, the system added tools for a geographic information system, the molecular modeling program ChimeraX, and the mathematics application KAlgebra. For example, the agent could run a command on an open map and get statistics as text, rather than searching for a console and checking the result on screen.

The repositories didn’t evolve only through a series of sequential edits. The system kept multiple versions and sometimes returned to an earlier branch or combined ideas from two branches. That meant it could preserve earlier solutions that might prove useful later.

Results on Two Benchmarks

On OSWorld V2, which includes 108 multi-step tasks using real computer applications, the same model, Qwen3.8-27B, achieved different results depending on the interface. With a fixed set of tools, it fully completed 5.56% of tasks. With a repository created by Agent S4—without giving the meta-agent access to the test virtual machine—the figure rose to 21.3%.

The average cost per task was $1.10. For comparison, the results cited by the authors show GPT-5.5 fully completing 13% of tasks at a cost of $25.50, and Claude Opus 4.8 completing 20.6% at a cost of $72.40. These figures reflect specific benchmark configurations and runs. They show how the systems compared in this experiment, not a universal ranking of models.

Share of tasks fully completed and cost per task on OSWorld V2.

The system was also tested on 169 tasks across six scientific applications in ScienceBoard. With the original repository, the same agent completed 55.03% of tasks; with the improved repository, it completed 58.58%. That’s a modest gain, but it suggests the system can find useful interfaces for software beyond the usual suite of office applications.

The authors also compared different ways of developing the repository. Access to a test virtual machine helped the meta-agent open applications and check new code. In one run, it increased the share of partially completed tasks on OSWorld V2, but the highest share of fully completed tasks came in a run without the machine. In other words, being able to test an interface doesn’t guarantee that it will perform better on new tasks.

Where the Evidence Ends

The results measure the combined effect of several repository changes. The researchers can’t always isolate the contribution of a particular tool from the other edits. Examples in which a new interface cuts dozens of actions down to a handful of calls help explain how performance might improve, but they are individual cases, not rigorous comparisons across every task.

The evaluation setup matters, too. Evaluation tasks influence which repository is selected, even though the meta-agent never sees them directly. This separation reduces the risk of overfitting, but doesn’t eliminate it. Also, the cost estimates include model calls made while solving tasks, but not the costs of creating and testing the repositories.

Interfaces can give agents broad access to data and applications. That can speed up their work, but it also makes access controls and the consequences of actions important. In the experiments described, the wrapper handled permissions, code execution, and restrictions. That layer of control remains essential: an agent that can access email or edit documents programmatically needs clear limits on what it can do.

Conclusion

Agent S4 shows that progress in computer agents depends on more than the model itself. Sometimes, it’s more useful to give an agent a way to retrieve the data it needs directly than to make it better at clicking a mouse.

The system learns to create these access methods from records of its work, saves them in a repository, and tests new versions on separate tasks. This improved results on both benchmarks, particularly OSWorld V2. The next question is how reliably the approach transfers to other applications—and how safely agents can expand their own toolkits.