A million action steps from Chinese apps put UItron ahead of UI-Tars

Could AI agents ever work a computer the way people do — see the screen, understand it, click, launch apps and carry out long chains of tasks? That is no longer science fiction. A new generation of models, UItron among them, promises to reset what automation on desktop and mobile can look like.

The trouble with today's AI agents is less a shortage of powerful models than a shortage of lived experience. Teaching a machine to operate an interface takes more than text instructions: it takes an enormous store of real mouse clicks, swipes and decisions made in situations nobody scripted. The earlier approach was to convert the whole interface into code or HTML and parse it tag by tag — which threw away the "liveness" of the picture and the details a person actually sees.

The arrival of vision-language models (VLMs) — where the model reads not only text but the entire screen image — was the breakthrough. The machine now sees what we see, and it holds up on hard tasks: picking the right button out of a hundred, telling the live element from the dead one, coping even when the app has been redesigned. But further progress, the UItron authors argue, needed three things: a large and varied supply of operational traces (action trajectories), a workable environment for simulating them in real time, and a big trained base to start from.

What UItron did

Most of the work went into gathering and processing data. It turns out that even if you take hundreds of open datasets across different systems, the formats disagree, the examples are scattered over many platforms, and the scenarios run thin. So the UItron team:

• collected over a million action steps from the top 100 Chinese mobile apps — the ones where AI had been failing;
• built its own infrastructure for collection and testing on phones and PCs, so the agent could be trained on real interaction rather than scripts alone;
• worked out how to unify heterogeneous data and widen the task set — the model now learns from text, from images, and from traces of sequential commands taken from real users.

Collecting data is not enough on its own. The agent also has to plan a chain of actions when the user asks it to, say, send a file in a messenger — which is not one click but a cascade of clicks, searches, scrolls and typed messages. UItron works out which route will do the job, reads the variables of the interface and tries different strategies, allowing for the fact that the reward for getting it right — the file actually sent — often arrives only at the end of a long path.

UItron trains in three phases

1. First, perception: the agent learns to recognize interface elements — text, buttons, icons — down to very small details.
2. Then planning: it builds a model of what to do next and learns to respond correctly, step by step, to the state of the interface around it.
3. Finally, a phase of reinforcement learning (RL): the agent is given a reward system, and the more tasks it completes, the better it gets at drawing lessons even from the attempts that fail.

On Chinese apps, notably, UItron had no peer: its accuracy and its ability to see a task through — even in interfaces it had never met — left other systems behind, closed commercial ones included. The jump comes from two things: the sheer volume of data labeled specifically for China, and a carefully designed evaluation and training setup.

The other significant result is that the agent can be trained fully online: new tasks are handed to it as it runs, and the system checks immediately how well it read the interface, built a plan and reached the goal. That interactivity is a step past earlier setups.

In testing, UItron beat many well-known rivals at perceiving elements (where accuracy and speed of decision matter), at grounding actions to the right objects on screen, and at planning long action sequences — on desktop as well as mobile. Against the strongest open models, the gain in task success rate runs to 10–20%, and in Chinese super-apps it is higher still.

What matters most here is that UItron is fully open to researchers. It can be developed further, wired into other projects, extended to new languages, new classes of application and new scenarios. It is a base for the multi-agent systems to come, where AI does not simply execute a user's command but works out its own route to the result in the digital world.

What's next? The plans call for a tighter coupling between interface-perception analysis and the precision of actions, for multi-agent systems whose members learn from one another, and for a move beyond the screen into interaction with physical devices. What is in front of us is a genuine prototype of the digital assistant of the future.

💾 Code