ColorAgent hits 77.2% on AndroidWorld by splitting the work across agents

We are used to systems where you press a button and the machine silently carries out the command. Mobile changes that. In anything complicated — ordering food, configuring an app — you need an intermediary that can understand what the user actually wants, hold the context of the conversation, ask clarifying questions and act consistently even when the interface shifts underneath it. The team behind ColorAgent built an OS agent that takes requests in natural language, finds its way around Android's changeable GUI, and is meant to make the interaction feel friendly and intuitive.

• The model is trained in two stages, which improves the accuracy of its screen perception, its reasoning and its actions.
• The system is built on a multi-agent architecture — separate modules, each with a specific job: knowledge retrieval, task allocation, and reflection on its own actions.

The result is an agent that does not just tap the right pixels but adapts to the user. It remembers habits, tracks preferences and, when it helps, anticipates what is needed and asks a clarifying question.

How the model was trained

First it was taught to do everything step by step, like a small child. Instead of trying to solve the whole task at once, it works out what to do on each screen and why. The developers deliberately allowed several correct actions for the same goal — open an app from its icon, or through a command. They also wrote clear rules for rewarding it for good answers and correct actions. That helps the model understand what is being asked of it and keeps it from proposing nonsense it cannot actually carry out.

Then comes the interesting part: it starts teaching itself. It comes up with its own questions, then tries to answer them under different conditions, as in a game. The answers are checked and only the best are kept — was the task solved, were the steps sane, did the reasoning hold together. Those good examples go back into training. And round it goes. The model assembles its own training data, so nobody has to spend weeks on manual annotation. Step by step it gets smarter.

Why one general-purpose agent is not enough

The analysis turned up three problems with these agents: they stall on unfamiliar interfaces, they lose track of what they have already done on long tasks, and they fall apart after the first mistake. Something has to be done about it.

The multi-agent system: who handles what

Here is what they came up with: the task is broken into parts where that is needed, and the system holds on to what matters between them. If the agent has looked at a receipt, the next subtask already knows the numbers and the categories.

The system can also pull in information from several places: instructions, the web, past experience. That helps when something unfamiliar appears on screen and it has to work out what to do.

And the key part — the system keeps checking itself at three levels: each step, the last few steps, and the task as a whole. If something goes wrong, it corrects course immediately.

What the benchmarks show

On AndroidWorld the agent completes 77.2% of tasks successfully, and on AndroidLab 50.7%. That is state of the art among open systems and a number of commercial ones. Where the gains come from is worth a look:

• training the model itself (step-level RL + self-evolution) already lifts the numbers noticeably: for one of the base models the gain is +29.3 pp on AndroidWorld and +14.5 pp on AndroidLab;
• adding the multi-agent architecture carries it the rest of the way, to 77.2% and 50.7% respectively.

The authors note separately that the larger model overfits more easily: it earns a higher reward during training but generalizes worse. The balance between capacity and robustness remains a challenge.

Closer to the user: memory and initiative

ColorAgent wants to be a friend rather than a plain robot. If it remembers what you did before, it takes that into account — both what you asked for directly and your habits. And if it does not remember, no problem: it will ask what you want before doing anything. That matters most when you say something vague, like order me a burger. Better to check the size and the sauce than to get it wrong outright.

The user-facing benchmarks show it: 58.66% on MobileIAR and 68.98% on VeriOS-Bench, a clear margin over the baseline. Behind those numbers is a careful pairing of system design (multiple agents, memory, knowledge retrieval) with disciplined training (step-level RL, scenario filtering).

The authors say outright that better evaluation is needed: more apps, harder situations, edge cases, and metrics that go beyond the share of tasks completed. Safety and control matter as well — working out when to hand off to a human, how to set permission limits, how to operate inside a sandbox. Beyond that, interaction between agents is worth developing, along with architectures that avoid these weak spots and let that interaction scale.

ColorAgent's point is that the future of operating systems lies not in one all-powerful intelligence but in the coordinated work of many narrow specialists, which:

• Learn and improve on their own, gradually, without hand-labeled data.
• Share experience and knowledge with each other.
• Can correct their own mistakes and adjust to what the user needs.

A structure like that lets the system run reliably, understand context and stay usable for a person even in an environment as changeable as Android.

💾 Code