Choosing an AI model is an infrastructure decision

Different workloads call for different deployment approaches

Paying for inference by the token is convenient when flexible consumption matters. Dedicated capacity can be a better fit for workloads with steady demand or strict requirements for resource availability, latency, throughput and cost predictability.

These approaches have different priorities:

Managed Compute in Microsoft Foundry is a managed deployment option that leaves customers in control of decisions affecting inference performance and cost: the choice of model, accelerator, capacity and operating mode. Microsoft takes on a substantial part of maintaining the runtime environment. As requirements change, companies can combine managed compute in Foundry with other deployment approaches.

How dedicated capacity changes the cost calculation

Moving open-model inference into production changes both the economics and the choice of technology. With dedicated capacity, resources are planned around expected workload requirements rather than relying only on flexible pay-as-you-go pricing. This helps companies assess available capacity, latency and throughput, while budget owners can see how infrastructure consumption, utilization, performance and spending are connected.

As AI use grows, companies can compare workloads by utilization, performance, predictability and cost—not just by the stated price per token or infrastructure price. Total cost of ownership includes the cost per GPU-hour, GPU utilization, throughput and the amount of work completed. Workload characteristics and expected demand therefore matter: they determine the actual cost per token or inference request.

Several metrics are important to the calculation:

Less operational work, more time for development

Running open models on dedicated infrastructure can require configuring and optimizing the software environment, maintaining software, addressing vulnerabilities and installing security updates. A managed runtime environment reduces much of that work. Teams retain control over decisions that affect workload performance and cost, while spending less time managing the underlying infrastructure.

Models have different infrastructure requirements

Different models need different infrastructure resources, which affects performance, GPU requirements and workload cost. Video memory is used both by the model itself and by the number of users it serves.

AMD Instinct MI300X is an accelerator with a large memory capacity, available through Foundry managed compute. Each GPU has 192 GB of high-bandwidth HBM3 memory, and an eight-GPU configuration has about 1.5 TB. That capacity makes it possible to run Z.ai GLM 5.3 Flash, a model with 320 billion parameters, on just four MI300X accelerators in Foundry and use a 512K context window. For the same model, using fewer GPUs means less dedicated capacity and lower cost. The software environment is built on ROCm, AMD’s open-source software stack for GPUs. It supports running open models and the tools used to operate them on Instinct accelerators in Foundry.

The accelerator choice affects how well a workload’s infrastructure, performance and cost requirements align.

The AT&T example

AT&T’s OTel2.0 initiative, focused on the telecommunications industry, used managed compute in Foundry to process large volumes of industry data and apply different open models at different stages of the work.

According to Mark Austin, AT&T’s vice president of data science and AI, Managed Compute in Foundry gave teams access to GPU capacity at the scale they needed and let them focus on developing OTel2.0 rather than managing infrastructure.

AT&T did not move all its AI workloads to one model or a single infrastructure configuration. The company used several open models and GPU architectures—530 accelerators in total, about 430 of them AMD Instinct MI300X. With managed compute, it processed about 1 trillion tokens on those accelerators and created synthetic training data. AT&T said it saved tens of millions of dollars by using open models to generate data instead of frontier models.

The example shows that different parts of a workload may need different models and GPU architectures. CIOs have to find a combination of model, infrastructure and capacity that delivers the required performance at an appropriate cost.

Managing an AI infrastructure portfolio

As enterprise AI moves into production, companies need to choose a combination of model, deployment method and infrastructure for each workload that delivers the right balance of performance, predictability, utilization and cost.

For open-model inference on dedicated capacity, managed compute in Foundry combines access to that capacity with customer control over workload settings. AMD Instinct MI300X offers a high-memory infrastructure option for demanding models and tasks, including optimized configurations with Z.ai GLM 5.3 Flash. This approach gives CIOs a basis for controlling infrastructure costs, planning capacity, and assessing total cost of ownership and return on investment as AI use grows.

Dataist’s take: the Foundry and AT&T examples show why enterprise AI infrastructure needs to be planned around specific workloads, not just the price of a model or token. The economic outcome also depends on GPU utilization and the amount of work completed, while AT&T’s reported savings apply to its own case.