Val Bercovici, WEKA’s director of AI, says enterprises should evaluate multi-agent systems by the cost of completed work and store agents’ working context in fast memory outside GPU memory. At the VB Agentic Infrastructure Stack Summit, he named two sources of excess spending: paying by token count and recalculating the KV cache when agents hand tasks to one another. If the context does not fit in GPU memory, the next agent has to process tokens that have already been accounted for, and companies buy additional accelerators to fit more data in fast memory.
Why additional GPUs do not always improve AI agents
Measure the cost of the result
More agents do not necessarily mean better quality. According to independent evaluator Vals AI, agent teams on the Vibe Code Bench benchmark use significantly more tokens for a small gain in speed. Their work costs 1.8–5.1 times more than that of single agents, with barely any improvement in quality. At the start of the year, customers were looking to increase token consumption. Now, Bercovici says, finance teams are limiting it and comparing the cost of each run with the value of the work produced.
Bercovici notes that the industry still bills for token consumption, not results. He named Vals AI and Epoch AI as independent evaluators: they assess the quality of agents’ work on legal, financial and medical tasks, and calculate the cost of a completed task or session. For these evaluations to be trustworthy, evaluators must remain independent of model developers and agent platforms. Buyers should pay attention to whether that independence holds.
Bercovici expects systems to process more tokens at a lower unit cost in the future, reducing the cost of results. But calculating total expenses is not straightforward. At a separate panel, Sudhanshu Bisht, Upwork’s vice president of platform infrastructure engineering, said the company’s AI gateway tracks model spending and allocates it across teams and services. But it does not account for model training and fine-tuning, the human work involved in correcting outputs, or the compute resources used by tool calls. Those expenses are recorded in different systems.
How agents use memory
In real-world production systems, work is usually distributed among several agents. They may coordinate with one another or receive tasks from a single agent that launches subtasks. When work is handed off, new agents need the context accumulated so far.
Two types of memory are used for this. Short-term memory includes the KV cache — the attention keys and values that a model computes for each token. The cache lets the model continue generating without processing the entire sequence again. Long-term memory stores context from previous sessions that an agent may need later — for example, when a user or another agent returns.
Bercovici believes that it is now important not only to save information, but also to make it quickly accessible. Agents’ continuous operation makes this harder. During a recent wave of hacks dubbed “Black October,” attackers exploited a twenty-year-old PHP vulnerability. Researchers found that the attacking traffic appeared to have been generated by automated agents working around the clock. Defenders need agents for continuous background tasks in response. Both types of memory therefore need to be available at all times: the cluster has no downtime in which it can rebuild the context.
Why companies buy excess GPUs
Bercovici named high-speed GPU memory and the onboard SRAM in Cerebras accelerators as some of the most expensive resources. Scientific computing has long required a balance between floating-point performance and memory capacity, but that balance is now disrupted: long context and swarms of agents consume memory quickly. As a result, companies pay for computing capacity they do not use.
In Bercovici’s view, one reason is the availability of capital. Chinese labs operate under hardware constraints, so they build more efficient models and inference systems. US companies, especially in Silicon Valley, have the resources to treat buying more hardware as the default solution. Bercovici considers Nvidia CEO Jensen Huang’s claim that the more you buy, the more you save misleading: additional resources lead to additional consumption.
DeepSeek’s prices have made this approach especially costly. Competitors seeking to offer a comparable cost per token have to eliminate repeated context processing. That requires storing computed context somewhere cheaper than high-speed memory, then loading it as needed.
Where to store the KV cache
Developers often use the word “memory” to mean different things. In agent systems, it usually refers to logical storage — such as a database, file system, knowledge graph or vector store — from which context is added to the prompt. At the hardware level, memory is the state a model creates while processing a prompt, including the KV cache.
There are three tiers for storing this state:
If a system supports this, saved context can be loaded from shared storage instead of being computed again. The GPU can then process new tokens. To illustrate the scale of these needs, Bercovici cited DeepSeek’s 100 pebibyte cluster.
How to give GPUs back their idle cycles
Software changes can also reduce waste. Megakernels combine operations in the inference pipeline and cut the overhead of handing work between them. Cerebras, SambaNova, d-Matrix and Groq have built specialized inference chips that deliver thousands of tokens per second to a single user. Bercovici says megakernels could help Nvidia and AMD GPUs approach that speed if operators choose faster responses for individual users over serving more users at once.
Modern systems have to coordinate millions of cores across parallel racks, compared with thousands in processors in the past. Bercovici notes that managing millions of operations every few microseconds is difficult, leaving room to improve performance and efficiency.
Dataist’s take: Additional GPUs do not solve the problem of repeated context processing or reveal the full cost of agent work. Comparing systems requires the cost of completed tasks and expenses for memory, compute and corrections. But, as the Upwork example shows, that data may still sit in different systems.
Daily AI news
Every day we pick what actually matters in AI and explain it plainly — no hype, no filler. Subscribe if you want to follow where the industry is going.
Only what matters — every day
Follow on X