Simulating 30,000 APIs as databases lets a 4B agent match a 30B one

Most useful agents are missing one thing: robust, accurate function calling. Not a well-phrased answer — the right tool calls, with the right arguments, in the right order. The trouble is that data with scenarios like that barely exists, and hand-written scenarios are brittle and scale badly. The authors of AgentScaler suggest looking wider: expand the agent's world and both task diversity and rich experience follow. In their framing every function call is a read or a write against a shared state store, and tools from different domains are unified by a common data schema. The rest is engineering: build many different worlds, teach the agent to live and act in each one, then consolidate the shared skills.

How the environment pipeline works

It starts with a large pool of tools: more than 30,000 APIs from public and internal sources. From these the authors build a compatibility graph over arguments — an edge means one tool's output fits logically into another tool's input. Clustering that graph carves out domains, the semantic groups of tools. Inside each domain, a large language model refines the connections.

The environment is then materialized as code: parameter descriptions are turned into a database schema, and the tools become operations over that database. The same single idea — tools as reads and writes — turns into thousands of reproducible, heterogeneous, checkable worlds.

How tasks are born and correctness is checked

To keep the agent from learning on artificial dialogue, the authors simulate the full user–agent–environment loop. The database state is initialized, logically consistent tool sequences are drawn from the domain graph, arguments are generated, everything is executed, and the evolution of the state is recorded. Verification is strict: database integrity, a match on the final state, and an exact match on the call sequence wherever nothing is written. One curious detail: if a tool errored out but the goal was still reached, the trajectory is not thrown away — that makes the model more robust to the noise of the real world.

What the model is taught, and how

The model learns to predict tool calls and final answers. Tokens from the user prompt and from tool responses stay in the context but are not penalized in the loss — that keeps the focus on actions. Training runs in two stages. First, general tool-use skills across many domains. Then specialization: fine-tuning on verticals with their own parameters and goals. For some task sets the second phase adds multi-turn agentic fine-tuning, which helps where long-horizon strategy matters.

What the experiments showed

Evaluation ran on τ-bench and τ²-Bench (retail, airline, telecom) and on ACEBench-en. Closed systems are still strong, but AgentScaler, at much smaller model sizes, comfortably catches and passes open counterparts of up to a trillion parameters. The 4B model reaches the level of a 30B, and the 30B variant comes close to closed systems in places and sets SOTA on those benchmarks. The contribution of two-stage training shows up on every ACEBench subset: a consistent gain on Normal, Agent and the overall metric.

Endurance and chain length

On τ²-Bench the pass^k curve shows the familiar picture: the more attempts allowed, the lower the accuracy — and still AgentScaler-30B stays ahead of the base Qwen from the same family at every k. That is a good sign for planning stability.

The authors look separately at the length of tool-call chains: accuracy falls as the number of calls grows. This is a general property of current LLMs, and it is visible here too. Long plans remain a problem.

Bottom line

Existing approaches either deploy real APIs at a high cost or simulate everything with an LLM and risk hallucination. AgentScaler sits in between: simulated tools run as code over state, and correctness is judged by how the database changed. That buys both scale and rigor, and it removes a large share of the manual work.

No reinforcement learning yet, although the environment is ready for it. Experiments stop at the family up to 30B parameters; larger models were not tested. The main weak spots are long tool-call chains and stability across repeated attempts. The authors point toward RL, multimodality and a tighter link to production.

The core idea is simple: widen the world and the agent gets smarter. Unifying tools as operations over shared state, building domains automatically and training in two phases turn scarce agentic data into rich experience. The results on τ-bench, τ²-Bench and ACEBench back the claim that scaling the environment works for compact models too — which matters for low latency and for running models on device.

💾 Code