Agents with 18,000 MCP tools clear just 1 of 7 hard Azure tasks

Researchers at Microsoft have built a new benchmark for agents that solve tasks not through a browser but by calling tools directly over MCP. They assembled more than 18,000 tools from Azure, GitLab, RocketChat, Plane and ownCloud, paired every task with the correct set of tools needed to finish it, and tested six models. The result cuts both ways: hand an agent the right tools up front and it performs noticeably better and cheaper than browser agents. Make it find tools on the fly and today's models handle the basics but collapse on complex enterprise scenarios that require combining thousands of functions and planning several steps ahead.

Why it matters

With MCP servers, AI agents can open tasks in Jira, merge branches in GitLab or work with Azure without ever touching a browser. That is faster, cheaper and more transparent — and more dangerous: agents have to find their way through thousands of near-identical tool descriptions, where a single wrong pick can take down an entire infrastructure.

How the environment works

The authors took TheAgentCompany, a simulated software company, and wired real MCP servers for five platforms into it. The key move: they converted the full REST APIs into MCP. Azure alone ended up with 16,800 tools, GitLab with 1,085, RocketChat with 520. Every task comes with verification scripts and a short list of the "correct" tools without which it cannot be closed. That is how they measure how far an agent can climb.

Azure gets its own track. They wrote 10 easy operations for it — attaching a tag to a resource, say — and 7 hard scenarios along the lines of "restore a downed application." Those require editing code, service settings, secrets and access policy all at once. This is where models fail most often: many interlinked services, a strict order of operations, and dependencies to untangle.

The agent that finds its own tools

You cannot hand a model 18,000 tool descriptions at once. So the authors built MCPAgent: inside it, tool search runs on embeddings — given a query it returns the top-k matches — and calling the chosen tool is a separate step. The familiar actions are still there: write Python, edit files. Search sits directly inside the reasoning loop: the agent sets a goal, finds a tool, runs it, looks at what came back, and reworks the query if it has to.

What happened in practice

• Hand the agent exactly the tools it needs and it beats its browser-based counterparts by 13.79% on quality while saving $2.29 per task.
• When tools are picked on the fly, the gain is far smaller: +5.39% and under $2.06 saved. For GPT-5, the difference between a random tool set and the correct one is only 2.13%. Smaller models give up part of the upside, and GPT-5-mini comes out worse than browser agents in several scenarios.
• On the easy Azure tasks, GPT-5, Sonnet-4 and Opus-4.1 handle 9 out of 10. On the hard ones their fate is grim: the leaders clear 1 out of 7, and some cannot finish a single task. Agents most often get stuck looping on IAM, never verify that their changes actually landed, and never check the other paths.

A few numbers on how the runs behaved

On enterprise tasks the models typically select 15–26 tools and make 7–12 calls, up to a third of which can fail. On the hard Azure tasks — GPT-5, for instance — search queries get longer and carry more information, but the share of failed calls reaches 25%. That points to two problems: no robust retrieval over ambiguous descriptions, and no systematic planning that checks intermediate state.

What this means for us

The study makes clear that direct tool access is dangerous. In one run the agent went as far as deleting a virtual machine. Extra human checks are required. The core bottleneck is navigation at scale across thousands of tools, where descriptions are imperfect and a task has several valid ways to be solved.

MCP is a convenient interface for working with tools. But for real enterprise work, agents are missing two things: more accurate tool search across a huge space of options, and a dependable plan of action with checks along the way. The next step is smarter retrievers, tracking dependencies between tools, memory of partial state, and verification of the "does the system actually work" kind rather than "did the call come back empty." That is when AI agents get close to reliable operation inside real companies.

💾 Code