Routing across eight LLMs beats GPT-5-medium by 7% at the same cost

Why you'd want smart routing

Anyone who has wired a large language model (LLM) into a real product has run into the same choice: more accurate but expensive, or cheaper but worse. GPT-5, the authors note, is already moving toward a fix through test-time routing: easy queries go to a faster, cheaper model, hard ones to the powerful one. The Avengers-Pro team pushes further — it pools several models from different families into one system and teaches it to hold a steady balance between accuracy and cost, without depending on any single large model.

The core idea

Avengers-Pro is a lightweight test-time router sitting on top of a pool of LLMs. It encodes the query into a vector with an embedding model, matches that vector against several nearby clusters, and picks the one model with the best balance of performance and efficiency. The router is driven by a single parameter α: the higher the α, the more willingly it hands tasks to expensive, more accurate models; the lower it goes, the more often it falls back on cheap ones.

How it works under the hood

• Embeddings. The query is turned into a vector (here, Qwen3-embedding-8B, 4096 dimensions).
• Clustering. On the training split (70% of the dataset), the authors run k-means with k=60 and get clusters of similar tasks.
• Model profiles. For every model, average accuracy and token cost are computed per cluster.
• Scoring. Inside a cluster, each candidate gets a score x = α·p + (1−α)·(1−q), where p is normalized accuracy and q is normalized cost.
• Inference. At runtime the query retrieves its top-p nearest clusters (p=4), sums the scores across them and picks the model with the highest total.

So: no heavy multi-agent system, no elaborate orchestrators, no prompt chains. One pass over a simple rule, one model per query, and a single slider α to tune it.

What was tested

The lineup is strong and varied: GPT-5-chat and GPT-5-medium, Claude-4.1-opus and Claude-4-sonnet, Gemini-2.5-pro and Gemini-2.5-flash, plus two Qwen3-235B variants (including a "thinking" one). The benchmarks are six different sets totaling 2,603 tasks, from GPQA-Diamond and ARC-AGI to LiveCodeBench and τ2-bench. Cost was computed at OpenRouter prices, counting input and output separately.

What came out of it

• At comparable cost, Avengers-Pro beats the best single baseline, GPT-5-medium, by +7% in average accuracy: 66.66% against 62.25%, for roughly the same total spend ($47.13 against $47.96).
• Pin accuracy at GPT-5-medium's level and the system cuts the bill by 27%: 62.66% for $35.05.
• For the "cheap but decent" case: roughly 90% of GPT-5-medium's accuracy at 63% less cost; against Gemini-2.5-pro, savings reach 81% at comparable accuracy.

And the main point: in the accuracy-cost plane, no single model beats the router on both axes at once.

Balance on a single slider

α behaves predictably: as it rises, the share of heavy models grows, and so do both the scores and the price. The authors point to two elbows — around 0.4 (still cheap, but with a noticeable quality gain) and around 0.6 (past which accuracy plateaus while cost climbs faster).

Who actually carries the load

At low α the system routes mostly to Qwen3 and Qwen3-thinking — cheap models that handle part of the workload perfectly well. As α climbs, GPT-5-medium's share grows, and Gemini-2.5-pro and Claude-4.1-opus get pulled in for the most demanding queries. It reads like a sensible multi-model strategy, learned automatically.

Why it matters

• Predictable budgets. You can fix a target cost or accuracy up front and get the optimal operating point without micromanaging anything by hand.
• Easy to adopt. The router needs no complicated logic and drops on top of an existing stack, whatever the LLM provider.
• Transparency. The decision is interpretable: the clusters, the model profiles and the effect of α are all visible.

Closing thoughts

Avengers-Pro makes the case that sensible routing across a zoo of LLMs can pay off more than the hunt for one perfect universal model. A single parameter controls the tradeoff, and the models themselves stay interchangeable. The result is better quality at the same price, or real savings without a collapse in the scores. For teams that watch token spend and SLAs at the same time, that is a very practical recipe.

💾 Code