Semi-online RL lifts a 7B GUI agent to 34% on AndroidWorld

Automating the interfaces on a screen is a long-standing wish: open an app, find the right button, run through a series of steps and see the task through. Today that job falls to agents built on large language models, which can look at screenshots, reason and act. But once the scenario runs to many steps, progress tends to run into the question of how we train these systems in the first place.

Why agents get stuck on multi-step tasks

Traditionally there are two routes. Offline reinforcement learning takes ready-made expert trajectories: stable, cheap, but the model learns from someone else's history and never sees its own mistakes. The result is individual steps that are over-optimized and a weak memory of the goal. Online learning looks more attractive: the agent interacts with the environment itself, gets feedback at the episode level, learns to take context into account. But in GUI navigation the rewards are sparse and delayed, the infrastructure is expensive, and covering a wide range of apps and sites is a headache of its own.

The authors of UI-S1 propose a middle route — Semi-online RL. The idea is simple: imitate online dynamics without ever entering a real environment. You stay offline, but you make the model live through its own history, and then carefully splice that history back onto the reference trajectory so the useful tail of the episode isn't lost.

How it works in practice

The semi-online rollout: for every expert trajectory the agent generates its own actions and thoughts at each step, conditioned on the history it has built up so far. If the predicted action matches the expert's, the rollout continues. If it doesn't, the run would break off without help, and the information about how the episode ends would be lost.

This is where the patch module comes in. On a mismatch it substitutes the expert action and, depending on the setting, a synthetic thought — for instance no reasoning at all (the cheapest mode), reasoning from an outside helper, or reasoning from the current policy itself. The history stays the model's own, and the trajectory runs to the end. The model learns on a live context without losing the useful signal from the reference.

On top of that, the authors optimize the policy against two signals at once: a step-level one (correctness of format, action type and the action itself) and an episode-level one (whether the whole task succeeded). Rewards propagate forward with a discount, to push the model toward seeing the goal rather than only the nearest button. The resulting objective looks like PPO with clipping and a soft KL penalty, plus a check on advantage diversity so training doesn't collapse into a narrow policy.

What to measure: SOP instead of guesswork

One of the hidden problems in GUI research is evaluation. The popular offline metrics often fail to predict how an agent will behave live. The authors introduce Semi-Online Performance (SOP) — a cheap proxy that accounts for the model's own history and scores progress and success across the steps of a task. In their experiments SOP correlates far better with real online performance than the usual offline metrics do.

What the benchmarks show

UI-S1-7B reaches the level of the best open 7B systems on dynamic benchmarks. On AndroidWorld the gain over a strong baseline runs up to +12 percentage points; on AITW-Gen, up to +23.8. The final 34% on AndroidWorld is comparable to far heavier systems and close to closed-source ones. Single-step skills don't suffer for it: AndroidControl-High and GUI Odyssey both improve. Unlike classic offline RL, which in places even slips backward, semi-online training consistently lifts exactly the multi-step behavior.

What matters is how you patch

The ablations show that the thought-free patch delivers nearly the best quality at minimal cost — and that is what goes into the final configuration. The higher the threshold on allowed patches, the more late steps the model sees and the better it uses the data. The optimal discount for distant goals turned out to be a middling one (γ≈0.5): ignoring the future makes results worse, while too long a horizon adds noise.

Case study: memory, context and precision

On real tasks of the "pull the data out of the gallery and create a file in notes" variety, the semi-online-trained model holds on to the plan through the end of the episode, doesn't mix up fields and doesn't lose context when moving between apps. The baseline and pure offline RL go off track or stop early more often — precisely because the history and the final goal have drifted apart.

Why this matters and what comes next

The semi-online paradigm joins the simplicity of offline with the goal-directedness of online. What you get is training that scales with data, doesn't require expensive infrastructure cycles, and still brings out exactly the agent's multi-step abilities. Plus a practical metric, SOP, that lets you rank models quickly before committing to real runs.

It would be interesting to extend thought patching to the policy itself without shifting distributions; to learn to choose more intelligently where to patch and where to let the agent make its own mistakes; to add a lightweight world model that approximates transitions between screens without leaning on an expert. All of that could make training cheaper still and improve transfer to new apps.

💾 Code