Even GPT-5 solves fewer than 60% of live multi-tool agent tasks

MCP-based agents can already do a lot: search the web, work with files, draw charts, run calculations, call external APIs. But a demo on a single task is one thing, and sustained work in a realistic, shifting environment is another — one where service responses differ from run to run and several dozen tools are on offer at once. Most existing benchmarks miss this: they are short, synthetic, usually free of noise, and almost always built around a fixed correct answer that goes stale. The authors of LiveMCP-101 set out to close that gap.

What LiveMCP-101 is

LiveMCP-101 is a set of 101 real queries that force an agent to coordinate several MCP tools. The domains are web search, file handling, math and data analysis. There are three difficulty tiers: 30 easy, 30 medium and 41 hard. On average a task takes about five or six meaningful steps, sometimes ten or more.

To keep scoring robust against live responses from external services, the authors gave up on freezing raw API output. Instead, every task comes with a reference execution plan. The plan was refined against a reference agent's trajectory until it reliably produced the right result. It is not a script for its own sake but a sequence of meaningful actions and parameters that can be compared against what the model under test actually does.

How the live evaluation works

Each task runs in two threads. The first is the reference: the agent follows the plan strictly and produces the reference output. The second is the test: the model gets only the query text and a large pool of MCP tools (15 servers and 76–125 tools). Scoring uses several metrics: the share of fully solved tasks (TSR), average result score (ARS), trajectory quality (ATS), plus average tokens and number of tool calls. The judge is an LLM whose scores were checked for agreement with human raters: agreement is high, especially on final answers.

What the experiments showed

The main finding is simple and important: even today's strongest LLMs succeed on fewer than 60% of tasks. The leader is GPT-5, with a TSR of 58.42% averaged across the benchmark and 39.02% on the hard tier. Next come o3 and GPT-5-mini, then Anthropic's flagship models. The middle tier trails noticeably, and open models trail further still: the best of them land around 20–23% TSR, and some score in the low single digits.

Trajectory quality almost always tracks answer quality: the cleaner and more sensible the path to a solution, the better the outcome. Verbosity does not rescue anything — past a certain point, spending more tokens buys top models very little. Open models often burn more tokens and call tools more often, and it does not convert into reliable answers. Extended reasoning chains help noticeably, through better planning and better recovery from mistakes.

Where agents trip up

The failure analysis identifies seven typical breakdowns, which fall into three classes:

• Planning and orchestration: ignoring requirements or stopping early; trying to solve the task in its head without the tools it needs; long unproductive reasoning; picking the wrong tool.
• Parameters: syntactic errors (wrong format or type) and semantic errors (the form is right, the meaning is not: wrong identifier, forgotten constraints).
• Post-processing: the tool returned a correct response and the agent parsed it wrong.

Semantic parameter errors dominate — 16–25% of all failures for strong models, over 40% for small ones. Overconfidence is common too, with the agent skipping the tools and describing the answer in words.

What the settings change

The authors examined two levers. First, the iteration limit: raising it to roughly 25 rounds steadily improves success, after which the effect fades. Second, the width of the tool pool: the more servers, the more often weak and mid-tier models lose accuracy. Top models hold up better because they filter out noise and plan more economically.

Why it matters

LiveMCP-101 moves agent evaluation closer to reality: not a "perfect" offline answer, but work against a changing world, in a large tool space, checked against a reference plan. This format captures not only correctness but also the quality of the path, which is critical for practical use. The results point to planning, reliable tool selection, getting parameters right and assembling the final answer carefully as the biggest sources of headroom. Token efficiency is a topic of its own: it needs a sensible balance between thinking, checking and stopping early. For the open stack, the obvious moves are fine-tuning on MCP schemas and noisy tasks, plus methods for verifying intermediate steps more strictly.