AgentScaler turns 30,000 tools into verifiable simulated environments

For the past year everyone has been arguing about how to make agents use tools with confidence: book a ticket, check a delivery status, pull a balance, assemble an answer out of several APIs. The bottleneck is always the same — there aren't enough realistic, varied trajectories in which an agent calls functions in sequence, sees the responses and changes the state of the world. A new paper takes an unexpected route: scale up not only the data but the environments the agent learns to act in. And it does all of it in simulation, with every step checked.

Why tool calling stalls

Function calling looks easy until you try to scale it. Real APIs are unstable and expensive, and LLM-driven simulations tend to hallucinate. Models end up weak on long scenarios, picking the wrong tool and the wrong parameters, and — most of all — they rarely see enough variety of situations. The researchers put the emphasis on expanding the environments and collect experience not from ready-made logs but from fully reproducible dialogues between the agent and the world.

How they build a world for the agent

The idea is simple: any tool call is a read from or a write to a database defined for its domain. Which means tools can be expressed as code that changes the state of that database correctly.

The rest is engineering:

• First they assemble a large pool of tens of thousands of tools (more than 30,000), clean up the specifications and normalize inputs and outputs.
• They build a compatibility graph: which tools can follow one another by parameter type. Clustering that graph yields hundreds of domains — from retail to air travel to telecoms.
• For each domain they materialize a database schema and wrap the tools in executable read/write code.
• Tasks are generated as a walk over the dependency graph, with real execution and tracking of how the database evolves. It ends with strict verification: the final database state and the accuracy of the call sequence.

What it looks like in practice

The agent talks to a simulated user and calls tools.

If it slips somewhere but recovers and reaches the correct database state, that experience is not thrown away — it makes the model more robust.

Before training, the data goes through a funnel of filters: dialogue correctness, a match on the final database state, and an exact match on the function sequence wherever that matters.

How the agent is trained

Training runs in two phases. First the model learns the basics: how to fill in parameters, how to alternate between turns with the user and calls to tools, how to read responses. Then comes domain tuning on verticals: choosing the right tools in the right order, parameterizing them correctly, giving the user coherent answers.

Crucially, only the tokens of tool calls and of the assistant's final answers are optimized; user turns and tool responses stay in the context but never enter the loss. That points the model at exactly what makes it an agent rather than a talker.

What the benchmarks show

The AgentScaler family is built on Qwen3 — three models: 4B, 8B and 30B-A3B. On τ-bench, τ2-bench and ACEBench they post a striking gain over open models of the same size. For reference: AgentScaler-30B-A3B scores 70/54 on τ-bench, 70.2/60/55.3 on τ2-bench and 76.7/82.7/60/75.7 on ACEBench-en (Normal/Special/Agent/Overall). Even the 4B model holds its own against 30B base systems — a good sign for compact LLMs.

The pass^k behavior on τ2-bench is interesting on its own: AgentScaler-30B-A3B stays above the original Qwen3-30B-A3B at every k, though the overall decline as k grows underlines the old problem of unstable tool selection.

What's wrong and what's still ahead

The most visible weakness is long tool chains: as the number of steps grows, accuracy falls off noticeably. That's no surprise — long-range dependencies, unstable feedback and the need to plan are still hard for most LLMs. The authors state the long-horizon problem honestly and see the way forward through RL inside the same simulated environment.

Most approaches lean on synthetic dialogues, hand-written scenarios or loose execution. The novelty here is in three things: scalable construction of domains out of a tool graph, reducing tools to code with verifiable state changes, and filtering data by whether the world actually changed. Together that yields cleaner and more varied training experience, which is what the metrics reflect, including the OOD tests on ACEBench-zh.

The researchers plan to put RL on top of the finished simulation, extend the modalities and test transfer to larger models. But something else already matters more: a carefully built environment and a focus on correct actions give compact models a real shot at acting as agents. That's useful in production and in edge applications alike, where milliseconds and watts are what count.

💾 Code