AI projects are held back more by costs than computing shortages

AI projects are more often held back by cost than by computing shortages

The VB Pulse survey on AI infrastructure and computing included 156 people who work on AI at organizations with at least 100 employees or consult for such organizations.

Rising data center costs are also worrying investors. Fortune reported that on November 5, tech stocks fell sharply amid concerns that the results of the midterm elections could slow construction of AI data centers. But the VentureBeat survey was conducted in October and asked about project delays over the previous year, not expectations for future computing capacity.

How to assess the cost of AI systems

Even when AI budgets are growing, companies need to understand exactly what they are paying for. A prototype may use a model that proves too expensive to scale. And repeated requests from an AI agent can drive up data retrieval costs.

Open-source code and open interfaces make it possible to switch models and providers without rebuilding every integration. Commercial providers can help operate these systems. But the options should be judged by how they affect the cost and quality of the work completed.

How Uber cut feature-flag cleanup costs by 90%

At VentureBeat’s October Agentic Infrastructure Stack Summit, Uber vice president and distinguished engineer Uday Kiran Medisetti described how the company reduced the cost of cleaning up feature flags. Uber tests managed AI agents on real tasks and tracks quality metrics, including rollbacks and incidents. After comparing models and selecting the right one, the cost of each such cleanup fell by 90%.

Uber’s published account also says costs can be assessed by the result—for example, a completed review or cleanup. Medisetti gave a figure for a specific Uber task. He recommends that other companies test their model choices on real work and consider the quality of the result alongside the cost.

Few measure financial returns

Of the 91 respondents whose organizations fully or partially track returns on AI infrastructure investments, 34% said the finance team had reviewed and approved the return calculation. That is about 20% of all 156 respondents.

Industry groups are trying to make costs and results comparable. Jay R. Strommen, executive director of the Tokenomics Foundation—a Linux Foundation group developing open standards for measuring AI costs and returns—described the need for unified accounting that connects technology work to business value. He said the work resembles maintaining a ledger and differs from projects that connect AI agents to tools and models.

Strommen also leads the FinOps Foundation. It already publishes FOCUS, a widely used format for billing and resource-usage data. Its full name is FinOps Open Cost and Usage Specification. The next step is to connect costs and usage with telemetry, outcomes and value classification. That would help show what a workflow delivered for the money spent.

Open models do not guarantee an easy switch between providers

In the October survey, 56% of respondents said their organizations planned to adopt a new, additional or replacement AI computing platform in the next 12 months. Choices made during experimentation may need to be revisited as deployments expand.

Open weights are one factor in that choice. 17% of respondents said their organizations use open-weight models in production; another 33% use them only internally or in experiments. But publishing model weights does not by itself mean the model’s license qualifies as open source. The survey results also do not show that such a model will be cheaper for any specific task.

Self-managed open-source infrastructure is less common. Among respondents who named one of the listed platforms, 7% said their organization uses such a stack in production. Just 2% of all respondents named it as their primary AI computing platform. These answers about platforms do not show which open-source components are included in managed services and packaged software products.

Open interfaces, open-source implementations and published weights offer different kinds of flexibility. Buyers should check the license, compatibility and support separately rather than treating openness as a single, general property.

One way to switch between models is an AI gateway that routes requests and applies rules. Tetrate co-founder and CTO Varun Talwar described such software as a layer that works with external model APIs, models on internal servers, tools and security controls. Tetrate develops Agent Router, previously known as Envoy AI Gateway. In September, it joined the Agentic AI Foundation. This layer handles traffic and addresses a different problem from Kubernetes workload management and inference projects such as vLLM and llm-d.

Talwar said traffic handling remains open, while Tetrate sells management tools for multi-tenancy, administration and auditing. Buyers need to find out where that boundary lies, how difficult it would be to switch providers, and whether automatic request routing can save money without reducing quality. Talwar recommends assessing workload telemetry, costs and quality—including the cost of repeated requests and human review—before enabling automatic model selection by default.

Agentic AI Foundation executive director Mazin Gilbert described competing companies developing protocols, gateways, skills and isolated runtime environments. AI agents from different teams may use different identities and have different access rights. The protocol used to interact with a tool does not determine who is allowed to use it.

Gilbert cautioned against treating every new foundation project as an established standard. When assessing projects, he looks at the diversity of participants, not the number of downloads. Buyers should consider independent developers, working integrations and project maturity alongside vendor demos.

Costs go beyond model inference

Choosing a model is only one part of an AI system’s economics. AI agents may repeatedly request corporate data, retain context and generate monitoring logs. Each of these actions consumes resources, even if the model call itself is inexpensive.

OpenSearch Software Foundation executive director Bianca Lewis noted a risk: a company may choose a data platform and later discover that its price, capabilities or operating requirements make an expensive system redesign necessary. At September’s OpenSearchCon in San Jose, discussions covered how to gather context from different systems and prevent AI agents from accessing data that their users are not allowed to see.

Amazon OpenSearch Service CEO Tia White of AWS spoke about the ecosystem of developers and service providers around OpenSearch. Intel demonstrated hardware optimizations contributed to the core open-source project, rather than released only as part of a closed product. That is why platform and data teams evaluating open platforms need to consider not just search latency but also repeated requests, storage, monitoring, access control and flexibility for future changes.

OpenShift 5 adds AI to cluster management

Infrastructure costs also include the work of operating a platform. Red Hat’s OpenShift 5 announcement includes AI-assisted tools for managing Kubernetes clusters. At a preview briefing, product director Ashesh Badani discussed analysis ahead of cluster upgrades, troubleshooting recommendations and creating dashboards with natural-language commands.

Badani also described MCP integration and support for external orchestration and AI-agent workflows. This will expand the ways to manage the platform alongside its console and existing automation. Buyers need to find out whether the integration will reduce effort in their particular environment.

Red Hat says confidential computing support on GPUs is already available for supported bare-metal systems, while an AI-agent sandbox remains in technology preview. The company also set out a three-year cycle for major releases to make the platform lifecycle more predictable.

When an AI agent requests a change, the platform must establish its identity, limit the actions it can take and give an operator the ability to intervene. Before expanding access beyond a pilot, teams should measure upgrade preparation time, manual work and the number of policy exceptions. Even a more autonomous workflow should keep records of actions, errors and recovery costs.

Red Hat distinguished engineer and vice president of the Office of the CTO Steven Watt described an internal pilot combining open-model inference on the company’s own servers, a gateway and developer budgets. The team had to maintain the inference service and persuade engineers to switch to an alternative when it suited the task. In its published account, Red Hat says token costs fell by 40–70% compared with the previous prices of commercial models, depending on caching and request routing.

But the savings added operational work. Buyers need to determine who will maintain the service, how policies will be enforced and whether users are ready to adopt the new system. Those costs should be counted alongside hardware and inference expenses.

How to test savings from open infrastructure

The October survey does not prove that open source improves returns on investment; that has to be established by the economics of each deployment.

Platform teams should start with one AI-powered workflow. Together with its owner, they should define what counts as success and how much human review is acceptable. The data team should identify authoritative data sources and the access rules that apply. Platform engineers should add tools to measure costs, latency and rework, while the finance team agrees on a baseline and checks the results.

Then change one component at a time: test a different model, serving configuration or data-retrieval method while keeping the same quality requirements. The comparison should account for integration, support and human review. Cheaper requests will not save money if the results require more manual checking or rework.

At KubeCon + CloudNativeCon North America, taking place this week in Salt Lake City, the AI Inference + Agentic track focuses on model serving and AI operations. In conversations with providers and project maintainers, teams should ask which interfaces are stable, which improvements are making their way into core open-source projects and where commercial support begins. They should then compare the answers with the economics of a specific workflow. A new platform should justify its selection by helping the company do useful work more reliably or at lower cost.

Dataist’s take: The survey points to a gap between expectations of growing budgets and companies’ ability to justify spending through financial returns. Open models and platforms offer options for switching providers, but they do not guarantee savings on their own. Those savings need to be demonstrated on specific tasks, accounting for quality, maintenance and human labor.