Nvidia’s AI factory pitch meets a gap in infrastructure returns

The missing measure is economic value

At GTC Berlin on October 21, Nvidia CEO Jensen Huang said companies need to produce AI, not just consume it, to participate in the AI economy. The “AI factory” idea, however, now extends beyond computing capacity.

Infrastructure choices increasingly involve integration and security as well as compute. For agents handling longer tasks, teams also have to decide how they access company knowledge, retain task state, use different models and recover from errors.

The survey points to a more basic gap: companies are planning platform changes without a financially validated measure of what their current infrastructure returns. Across all 102 respondents, 63% planned to adopt a new, additional or replacement platform within a year. Just 10% said their AI infrastructure return metric had been reviewed and approved by finance. Among those planning a platform change, 91% lacked such a measure. Among those not planning one, the figure was 89%; the difference was not statistically significant.

Respondents whose primary platform was in production rated overall satisfaction at 4.0 out of 5.0, ease of implementation at 3.9 and value for money at 3.6. Those scores can describe a platform that works day to day. They do not establish whether its output is worth its cost.

For agent applications, a more useful measure may be the cost of a successfully completed task. That calculation should include inference, information retrieval, storage, retries and human review, then compare the total with the value delivered against an agreed baseline. Success also needs a quality bar: a support ticket closed incorrectly is not a successful task, and the cost of fixing a code change belongs in the cost of completing it.

Architecture changes the cost of a task

Respondents cited integration with existing cloud and data stacks (48%), performance (42%) and GPU availability (35%) as factors in infrastructure selection. The difference between integration and performance was not statistically significant.

The practical question is where a workflow loses time, repeats work or spends resources without improving the result. A more capable model may help, but so might better information retrieval or context, a different model-serving configuration, or a clearer division of work between models and tools.

That makes “memory” an architectural question with three distinct parts:

These are not interchangeable. Company knowledge needs to be current, relevant and permissioned. Task memory should preserve useful progress without turning temporary output into official organizational knowledge. Inference-memory techniques need to be tested on the workloads they are meant to serve.

Anthropic’s context-engineering guidance describes structured notes outside the context window as one way to preserve information during long-running work. Automatic prefix caching in vLLM operates at a different layer: it reuses computation when requests begin with the same prompt. The vLLM documentation says this reduces prompt-processing work, not the time needed to generate new tokens.

In the September survey, 41% of respondents said they used or were evaluating Nvidia for inference memory, and 39% said the same of Dell; the difference was not statistically significant. Open-source key-value cache tools and model-level efficiency techniques each drew 21%. Another 16% said they were not working on inference-memory constraints or did not know about them.

What companies are not measuring

The survey suggests infrastructure plans are advancing faster than financial validation. My read is that GPU utilization alone cannot resolve that tension: even a well-used system might be expensive per successful task, while low utilization does not by itself show that replacing the platform will improve the economics.

The same problem applies to model selection. Small models may suit bounded tasks, while harder stages need a different model or a person. The test is whether a model placed in the right part of a workflow meets the quality requirement at lower total cost—not whether its token price is lower. The September report named token price as a platform-selection factor for 23% of respondents, up from 11% in August; the difference between the two monthly samples was statistically significant. Total task cost still depends on token use, retries, tools and human intervention.

Responses also show a divide between leadership and individual contributors. Total cost of ownership was cited as a selection factor by 52% of vice presidents and directors, compared with 4% of individual contributors, a statistically significant difference under the report’s test for small independent groups. Individual contributors most often chose integration, at 58%.

A shared measure could connect those priorities: cost per successful task. It gives finance a way to assess value and gives technical teams a way to test whether changes to retrieval, model choice, caching or task-state management justify their expense.

Before buying new infrastructure, teams can start with a specific agent workflow:

I think the consequential question is not whether companies should build AI factories, but what they will count as output from one. More capable models widen the range of tasks agents can attempt; without a quality threshold and an agreed cost measure, a new platform can add capacity without establishing that the work is worth doing.

VentureBeat will hold the Agentic Infrastructure Stack Summit on October 28 in San Francisco, with planned discussion of inference economics, including ways to use less memory and fewer tokens.

The figures come from VentureBeat Intelligence’s September VB Pulse report on AI infrastructure and computing. The survey included 102 respondents from organizations with at least 100 employees, including consultants answering for their primary client. Results are self-reported by a self-selected group. GPU utilization data covers 68 respondents running AI in production who provided a specific figure: 55 reported average utilization of 50% or less, and 13 reported higher utilization. The role comparison is based on 23 vice presidents and directors and 26 individual contributors. The August and September token-price figures come from different respondent groups. The survey’s inference-memory question covered both use and evaluation of technologies. The discussion of institutional memory, agent task memory and small models is the analyst’s interpretation, not a direct survey finding.