Why replacing an AI model takes weeks

Companies use gateways and routers to switch AI models with a single line of code. But in applications already used by customers and employees, making that switch often takes weeks or months of engineering work: the application has to be reworked, or the company has to return to its previous model. In an October VentureBeat survey of employees at companies with at least 100 workers, 47% said their company had changed the model in a production system in the past 12 months without replacing the application itself. Among those companies, 70% spent several weeks or more on the transition, and 36% spent more than a month. Just 13% said switching was almost as easy as changing a setting.

Why Switching an AI Model in a Production App Takes Weeks

The transitions had different outcomes: 55% of respondents at companies that had changed models said the new one performed at least as well as the old one. The other 45% said performance fell and the application had to be reworked, or the company returned to its original model. That share was not statistically significantly different from September’s.

Routing makes it easier to choose a model

Companies can route requests to cheaper models using a router, a program that selects a model for each request. This is one of the functions of an AI gateway, which sits between a company’s applications and model providers.

On Monday, Anthropic released Claude Fable 5.2 at $8 per million input tokens and $40 per million output tokens, 20% less than the model it replaced. VentureBeat executive editor Karl Franzén reported the release. The VentureBeat survey had closed before the model was released. It also did not ask why companies changed models or whether the new one was cheaper.

Adobe’s marketing agent first uses a cheaper model to select the right context, then sends the request to a larger model. Adobe senior vice president of engineering Anjul Bhambri described the approach on October 28 at VentureBeat’s Agentic Infrastructure Stack Summit in San Francisco.

Startups also offer routing. Tetrate’s solution includes Auto Mode, which selects a model for each request. Some of the company’s customers no longer care which specific model they get, Tetrate co-founder and CTO Varun Talwar said at the summit. But, he added, the system has not yet learned to reliably select the right model for every task. Customers start with a single command and enable Auto Mode by default only if answer quality holds up while costs fall.

Even when model selection is automated, companies need to check how their application works with the new option. That is the kind of switch within an existing system that the VentureBeat survey asked about.

Sudhanshu Bist, vice president of platform infrastructure engineering at Upwork, the online freelance marketplace, made a similar point at the summit. For an open-ended coding task, he would let a smaller model finish work started by a larger one. But in a production system, he said, he would most likely keep the existing model. The choice depends on the task: Upwork assigns repetitive work with clear rules, such as matching freelancers to job openings, to small fine-tuned models, and open-ended tasks to larger ones.

A production application may depend on prompts, tools and tests configured for a specific model. Replacing it can mean reworking those components. The survey did not ask which parts of their applications companies changed or whether they used AI gateways.

A new model may behave differently in the same system

Some companies fine-tune models on their own examples to better adapt them to their field. But the results of that tuning do not automatically carry over to the next model.

PayPal has changed the model inside an application many times, the company’s vice president and global head of AI products, Ren Zhang, said at a VentureBeat event on October 8. For a shopping agent that helps customers explore products and make purchases, PayPal moved to a faster, smaller model and fine-tuned it. This improved accuracy, but Zhang said the improvement was not enough. The work took place in internal research and testing, before the agent launched for customers, the company said.

PayPal later decided to rely less on fine-tuning: it requires a team of specialists, and the process has to be repeated with every new model. The company now combines fine-tuned and off-the-shelf models and accounts for how computing resources are allocated to balance accuracy, speed, cost and control. A PayPal spokesperson relayed Zhang’s comments on Tuesday.

Box CTO Ben Kus explained why a new model can make an application work worse: engineers tune software to the original model, and the next one may behave differently. In an interview for VentureBeat’s Beyond the Pilot podcast on October 30, he noted that executives often expect a switch to a new model to bring improvements, but that does not always happen.

According to Kus, the software wrapper around a model can have the greatest effect on results. It is the code that gives the model instructions and determines which tools it can access. The choice between leading models matters less now: they are all already good enough. That is why Box tests models with different wrappers.

A wrapper can also affect the cost of completing a task. In the Runta FrontierHarness Eval benchmark, Moonshot AI’s Kimi K3 solved 63.3% of tasks in both Claude Code and the DeepSeek Harness configuration. But the average cost per solved task was $18.34 in Claude Code, compared with $3.28 in DeepSeek Harness — more than a fivefold difference at the same success rate.

In the FrontierHarness Eval test, Kimi K3 ran in nine wrappers and 12 configurations on the same 30 coding tasks. Runta, the company that provided the testing environment, announced the results in early September. Thirty tasks are not enough for a reliable ranking, so the results should be treated as approximate. Sam Witvin, co-host of a VentureBeat podcast, makes the same point in his video analysis of the benchmark.

When testing a new model, companies should use the same wrapper that will run in the application. If the result is unsatisfactory, it makes sense to try another wrapper before giving up on the model: the wrapper may be the reason it failed.

At the October 28 summit, Mika Hill-Smith, co-founder and CEO of model evaluation company Artificial Analysis, said that in the company’s research, wrappers had less effect on the results of the largest models than expected. Smaller models, he said, are less flexible. In tests of coding agents, results for these models can vary by 20, 30 or 40 percentage points depending on whether they were trained on the style of a particular wrapper.

Switching to a smaller model can significantly reduce costs, but it requires additional engineering work. Hill-Smith explained that companies put more effort into using these models. A wrapper designed for a large model may therefore be a poor fit for a smaller one.

The price per token does not show the cost of a good result

A cheaper model can cost more if it makes mistakes more often. At the summit, Bist gave a hypothetical example: a small model costs 10 cents per run and completes a task successfully 20% of the time. On average, it will take five attempts, putting the cost of a successful result at about 50 cents. A larger model costs 30 cents per run but succeeds 90% of the time — about 33 cents per successful result. It costs three times as much per run, but less per completed task. The calculation assumes the company retries failed runs until it succeeds. Bist clarified these figures in an email on Wednesday.

Upwork’s model gateway does not account for all the costs of a successful result. It tracks what teams spend on model calls, but does not capture the cost of GPUs for training and fine-tuning, employees’ time spent correcting an agent’s output, or the compute required to call tools, Bist explained.

According to Tetrate’s Talwar, some companies are starting to measure the cost of a successful result rather than just the price of a token. WEKA chief AI officer Val Berkovich said at the summit that the industry is still tied to per-token pricing, even though it should be measuring the cost of an outcome. WEKA was one of the event’s sponsors.

For some tasks, savings are not the main priority. T-Mobile uses leading models for customer support agents and is willing to pay more for them, the company’s chief information officer, Jeff Simon, said at the summit. In customer support, the company’s goal is to give customers fast and accurate answers, not primarily to cut costs.

Uber can switch an agent’s model in a few days

If an open model proves more cost-effective for AI code review, Uber could switch to it in a few days, the company’s distinguished engineer Uday Kiran Medisetty said at the summit. Code review is one of Uber’s centrally managed agents, which the company develops and runs itself.

For code review, Uber built a benchmark that lets it test a model on code changes from previous years. The company later developed tests for a background coding agent, too, using real coding sessions. Medisetty said this task turned out to be much harder.

Benchmark results help Uber choose a model for automatically cleaning up outdated feature flags in code. Testing and model selection have enabled the company to cut the cost of each such operation by 90%, Medisetty said. Every managed agent at Uber has room for cost reductions.

The company also tracks what share of AI requests it can move from one model to another. Uber’s managed agents can be switched between models, but sessions engineers run manually in tools such as Claude Code are much harder to move: doing so requires changing how people are used to working.

How companies prepare to switch models

PayPal stores business knowledge separately from the model. The company’s internal data analysis assistant uses a knowledge layer with company definitions, approved queries and step-by-step instructions for analysis, Zhang said. According to PayPal, this layer increased the accuracy of answers to internal business questions from 43% to 95% with the same model. It was built separately so company knowledge would not depend on a particular model. If that knowledge is tied to one model, it has to be configured again with every transition, Zhang said.

Databricks customers turn answers from a leading model into an evaluation set. The company’s director of product management, Brian Clark, said they first build a simple version using a leading model and a text index — an approach Clark described as “a smart agent, but not a smart back-end component.” They save the leading model’s answers for comparison. Then they switch to a cheaper open model, add vector search and compare its answers with the saved set. Clark described the approach at the summit.

Box evaluates each change against three measures: accuracy, cost and speed. It tests them on a set of queries based on tasks customers give the company’s agents. According to Kus, these three results are what matter most, and gathering evaluation data is usually the hardest part.

T-Mobile tests changes on real customer calls and checks every session after launch. When engineers change customer support agent settings, the old and new versions run at the same time, and the company compares them on real calls. The company expands use of the new version only if it performs better. In addition, a small model checks every text and voice session and flags answers that may be wrong. Preliminary tests do not cover every possible question, so T-Mobile looks for rare cases that would not turn up in a lab, Simon said.

At JPMorgan Chase, new models must receive internal approval before employees can access them. This process affects how quickly the bank can switch: as it expands its use of Anthropic’s Claude, it tests every new Claude model against its own rules. Anthropic said this on Monday.

What to check before switching

The need to spend time replacing a model is a reason to prepare carefully, not to give up on the switch. William Allen, who leads product for Snowflake’s CoCo, the company’s coding agent, and CoWork, its assistant for office workers, told VentureBeat on November 3 that assumptions about AI that seemed true three or four months ago may already be out of date. He also warned against refusing to change a system simply because a lot of work has already gone into it.

The experience of companies in these discussions points to the following preparation steps:

VentureBeat plans to ask the same questions about model switching in its November survey and find out whether 70% of companies that changed models still report spending several weeks or more on the work.

VentureBeat Intelligence conducted its VB Pulse survey on agent orchestration in October 2026. Respondents answered about companies with at least 100 employees; 28% were consultants answering about the client they spent the most time on. Only participants whose companies had changed models answered the question about engineering effort. The reported transition outcomes also refer to this group, so their estimates are less precise.

For comparison with October, the September data included only respondents who would have passed October’s eligibility screening. In September, 49% of respondents at companies that had changed models said the transition took several weeks or more, and 12% said it took more than a month. VentureBeat Intelligence considers both October increases indicative rather than conclusive because the respondent mix changed. In October, 35% of respondents said they use AI systems or track them from a business perspective. That was considerably higher than in September, and these respondents were more likely than others to select “more than a month.” Excluding them, the share of transitions lasting several weeks or more would be 66%, and the share lasting more than a month would be 29%. Most participants in the October survey were recruited through a research panel, so VentureBeat cannot determine how much of the change between September and October is explained by the respondent mix.

Dataist’s take: switching a model with one line of code is possible only at the routing level. A reliable transition depends on tests, the software wrapper, the cost of a successful result and the company’s internal rules. The examples also show that a separate knowledge layer and testing on real queries help companies prepare, but do not remove the need to reassess every model-application combination.