How AI Chooses the Right Model for Each Task

One AI, Many Specialists

Imagine asking the same model to handle everything: answer questions, write code, generate video, and control a robot. It’s convenient, but expensive—and not always effective. One model may be better at coding, another at understanding images, and a third at controlling a robot in a complex scene.

The authors of Route Anything propose building general-purpose AI systems from specialized models and tools. For each task, the system selects the right model or tool to handle it. This process is called routing.

The key question is how to choose: should a request go to the strongest model, or to a cheaper option that can do almost as well? The researchers explored this question across text, images, video, speech, coding, and robotics.

Diagram showing how the system selects a candidate for each request and balances performance against cost.

How to Measure the Benefits of Routing

The authors collected data from 25 benchmarks and 452 model, tool, and robot-control policy configurations. They tested every candidate on the same inputs, for a total of 374,000 evaluated runs.

That matters. If one model is tested on easy tasks and another on hard ones, the results say little about which candidate to choose. Here, the researchers could compare how each candidate handled the very same request.

They compared three approaches:

🟠 One fixed candidate for every request in a benchmark.

🟠 A router selects a candidate based on available information, such as the task type or the request itself.

🟠 An oracle chooses the best candidate for each input after the fact. This is an unattainable upper bound in real-world use, since making the choice would require knowing the result in advance.

On average, the oracle outperformed the best fixed candidate by 5.4 points on a 100-point scale. The gap was especially large for video and robot control—about 19 and 18 points, respectively. In other words, the models really do complement one another: no single candidate is best at every task.

But most of the potential gain comes from differences within a task type. Knowing that a request is, say, about image generation barely changes the choice. To get close to the oracle, a router needs to distinguish between individual requests and the conditions under which they’re handled.

Routers Still Fall Short of the Oracle

To test routing in practice, the authors trained routers on task data and candidate performance results. The main router predicted each candidate’s expected result, then chose the one with the highest predicted score. A separate router first identified the task type, then selected a suitable model from a lookup table.

Across 16 benchmarks, the router scored 79.98 points, compared with 79.68 for the best fixed candidate. The margin was small, but it showed that a trained router can outperform a fixed model. The gain was 2.1 points for video generation and 0.8 points for coding and work-agent tasks.

That’s still well short of the oracle, which scored 84.80 in the same comparison. The router captured only a small share of the available headroom. For image generation, the researchers tested 124 routing configurations, but none convincingly outperformed the fixed model.

Comparison of fixed selection, a trained router, and the oracle across different task groups.

A separate robot experiment showed why a router may benefit from seeing the input images. The system received an instruction, the robot’s state, and three camera frames. In clean scenes, it tended to choose one control policy; in scenes with random changes, it tended to choose another.

Across 2,000 test scenes, the router succeeded 71.6% of the time, compared with 70.5% for the best single candidate. It also cut policy runtime by 24%. The difference in success rate was small and statistically uncertain, but the combination of comparable performance and lower runtime looked promising.

In clean and altered scenes, the router chooses different robot-control policies.

Predict the Result, Not the Winner

Another finding concerns what a router should be trained to predict. It can be asked to name the winner—the candidate with the best result on a given example. But that throws away information: even if several models succeed, the label still names only one.

The difference was especially clear in the robotics tests. A model that predicted the success rate of each policy separately achieved a 71.6% success rate. A classifier trained to guess the winner managed only 42.8%.

The problem lay in how the winner was selected. When several policies completed a task, the faster one was often preferred. As a result, the classifier learned to select for speed rather than likelihood of success. The authors argue that routers should instead predict each candidate’s expected result and then compare those predictions.

Results also depend on what the router is compared against. If a fixed model is excluded from the candidate set because a few of its evaluations are missing, the baseline score drops. That can make the router look better even when the apparent gain comes from the scoring method. Candidates should therefore be compared on the same inputs, with the same rules for handling missing results.

Lower Costs Without a Noticeable Drop in Quality

The biggest practical benefit was not better performance, but lower cost. The researchers compared routed systems with a strong model or another fixed candidate used as the baseline. On average, efficiency—quality per dollar or per second of runtime—increased 8.82-fold.

The gain was 25.5-fold for coding tasks and 7.1-fold for text question answering. In these cases, the router used cheaper models more often, calling on the baseline only when it was really needed.

But the details matter. On one coding benchmark, cheap models performed almost as well as the baseline, so the router could avoid using it altogether. For text questions, weaker models lost too much quality. There, the system needed a check-and-escalate setup: if the first model failed, the request went to a stronger one.

The authors also tested whether routing could keep quality loss within a guaranteed limit. They first tuned the router on training data, then evaluated it on a separate sample. In their experiments, this approach stayed within the promised quality-loss bound in all 200 tests across two task sets, while randomly selected routes often failed the check.

Routing cuts costs compared with always choosing a strong model, while maintaining similar quality.

More elaborate setups can also prove uneconomical. For example, checking code with a free compiler helped filter out bad solutions. But a language-model judge cost so much at typical prices that it wiped out the savings. The value of a check depends not on how sensible it sounds, but on how many errors it prevents—and what it costs.

Takeaway

Routing is most useful when candidates perform similarly but differ significantly in cost. It can already reduce expenses without a noticeable drop in quality, and in some cases improve results slightly by choosing different candidates for different inputs.

Choosing the ideal candidate for every request remains a distant goal. Trained routers capture only a small share of the potential gains, especially when the differences between tasks are subtle. Practical systems should follow three rules: compare candidates on the same inputs, predict each candidate’s result rather than simply guessing the winner, and check separately that savings don’t push quality below an acceptable threshold.

A general-purpose AI system can be made up of many specialized parts. But its usefulness depends not just on which models are available. It also depends on knowing when to call each one.