Microsoft spent $37.5 billion on infrastructure in the quarter ending December 2025, with roughly two-thirds going mainly to GPUs and CPUs. At this scale, even modest waste is expensive. NVIDIA found that idle workloads consumed about 5.5% of GPU capacity across its research clusters. By combining GPU telemetry with job data and automatically clearing stalled or abandoned workloads, it reduced that waste to about 1%, potentially saving millions.
Scale down the arithmetic and the pattern holds: a team running 500 GPUs at 5% waste is still burning six figures a year on hardware that's doing nothing, and unlike Meta, they don’t have the balance sheet to shrug it off.
Training clusters and inference clusters waste capacity on opposite schedules. A training run saturates its GPUs for days, then sits mostly idle until the next one starts. Inference does the reverse — it tracks user traffic, so it peaks during the day and drops off at night — but the hardware underneath is provisioned for the peak and billed around the clock either way. Enterprises are on track for $401 billion in new AI infrastructure spend in 2026 globally, and Kubernetes-managed production clusters are running at just 5% average GPU utilization, that is one-twentieth of the capacity being paid for. Peak demand, latency targets, and scheduling constraints all justify holding some slack. What's missing is the visibility to tell justified slack from waste at the level of an individual workload.
This article is about closing that gap: the orchestration and visibility work behind scalable AI infrastructure, and the TCO decisions that determine what it costs to run.
Without GPU-aware orchestration, a scheduler can see that a GPU has been requested, but not whether the workload holding it is doing anything useful. Occupied capacity reads as occupied capacity regardless of what's actually running on it, so new work queues up behind jobs that may be idle, stalled, or finished in all but name. Cost-efficient orchestration combines job state, GPU telemetry, scheduling policy, and cleanup rules, a layer allocation-based scheduling doesn't have on its own.
This shows up concretely in how Kubernetes schedules: once a pod requests and receives a GPU, Kubernetes has no built-in mechanism to reclaim that capacity mid-job, even if the GPU sits idle waiting on data or a stalled process for the job's entire runtime. There's no partial credit and no automatic recovery: the allocation holds until the pod releases it or is killed. Without that visibility, GPU capacity governance moves outside the scheduler entirely: quota decisions, priority exceptions, access approvals, and manual cleanup become tickets instead of automated policy
The real waste comes from workloads that remain alive after useful GPU work stops: idle notebooks, stalled pods, abandoned interactive sessions, or services holding more GPU capacity than they need. Without telemetry and cleanup rules, that capacity stays allocated while new work waits.
Getting past allocation-only scheduling starts with the NVIDIA GPU software stack. GPU Operator automates the components Kubernetes needs to provision and observe GPUs: drivers, the NVIDIA device plugin, container toolkit, GPU Feature Discovery, DCGM-based monitoring, and related components. That does not make native scheduling utilization-aware by itself, but it gives the cluster the device inventory and telemetry needed for GPU-aware policy, sharing, and cleanup automation.
Pipeline tooling choices are expensive to reverse: overbuilding means months maintaining orchestration infrastructure instead of shipping models; underbuilding means losing visibility into what's running on the cluster and why. The decision tree below covers the four questions that matter: audit or compliance requirements, whether Kubernetes is already running, how often the model updates, and how many pipelines need coordinating.
Auditability is the constraint that decides the left branch: a manually coordinated pipeline can't reliably produce the lineage and approval trail regulated workflows require. On the right, pipeline count decides it. Several concurrent pipelines justify Airflow's orchestration overhead. One or two don't, no matter how often the model updates.
In practice, it's easier to justify the overbuilt path than the lightweight one: approving Kubeflow or Airflow "to be safe" is a simpler conversation than telling stakeholders a small team with infrequent updates doesn't need either yet. The case study below shows what the lightweight path looks like in production: no wasted overhead, and it still holds up under real load.
A healthcare technology provider needed semantic search across standardized medical terminologies, such as SNOMED CT, RxNorm, LOINC, with sub-second latency and consistent behavior across environments. The challenge was pipeline reliability and ownership: embeddings were being versioned manually, transferred via Google Drive, and deployed by hand, and the ML team had no visibility into full-system behavior, requiring cross-team coordination for every integration issue. Every update was a coordination exercise with room for error at each step.
The solution was a modular containerized stack: 11 Docker containers total, with the ML component isolated in two dedicated containers: a Flask API and a Qdrant vector database. GPT-4 via Azure OpenAI normalized clinical input through an endpoint, while embeddings and vector search handled retrieval. No local model training or fine-tuning was required . Embeddings were generated offline, packaged as versioned archives, and loaded into Qdrant at deploy time through a Jenkins pipeline. Environment configuration lived in .env files; Graylog handled centralized logging and monitoring across DEV and PROD.
Domain experts created a 200-concept benchmark with validated mappings, and each release was checked against 100+ curated test prompts. That reduced reliance on expert review for every release without removing clinical oversight before production promotion.
The result: semantic search across collections of up to several million medical terms, under one second latency maintained even after a forced migration from RAM-based to disk-based vector indexing when embedding volumes outgrew available memory. Reproducible deployments, version traceability across environments, and operational ownership of the semantic-search pipeline gave the ML team the ability to control, debug, and iterate on the service without cross-team coordination for every deployment or integration issue. For a healthcare workflow, that is the lightweight path done properly: reproducible releases, traceable artifacts, centralized logs, and clinical validation without the overhead of Kubeflow or a managed platform.
A GPU can be correctly scheduled and still sit half-idle: the scheduler did its job, and the waste happens anyway.
NVIDIA's own cluster team found this out the hard way. By joining DCGM telemetry with Slurm job metadata, they identified five recurring sources of GPU waste across their HPC fleet — a more granular version of the same pattern: CPU-only jobs running on GPU nodes, misconfigured jobs over-provisioning GPUs, active-looking jobs that were actually stalled, container downloads or data fetching holding resources during setup, and unattended interactive sessions. The cluster could look busy while specific jobs were tying up GPUs without using them productively. Once they built tooling to catch each pattern automatically (an idle job reaper, a job linter, and automated defunct-job cleanup) GPU waste fell from roughly 5.5% to about 1%, representing millions in recovered capacity across a fleet where even small inefficiencies compound fast.
Under-provisioning wastes capacity the opposite way: instead of nobody releasing it, nobody's assigned it fast enough. Every step in a manual request: the approval, the configuration, the assignment, is a point where a GPU sits unclaimed while a team waits. Our multi-tenant virtual datacenter project is one example of what that automation looks like in practice — covered in the case study below.
The orchestration problem in a GPU cluster and the problem of running a multi-tenant virtual data center share the same question: how do you allocate shared resources across isolated tenants without making every new request a manual project? A virtual data center platform Sciforce built for a hardware and infrastructure provider started from a familiar place: a small DevOps team, multiple enterprise tenants each needing fully isolated environments, and a provisioning process that required dedicated effort every single time. Security and efficiency were being solved separately, which made both more expensive than they needed to be.
The architecture that replaced it has two layers. A management cluster handles overall control, while per-tenant clusters spin up automatically via Cluster API and KubeVirt. Each tenant gets its own compute, storage, and networking, with dedicated subnets, routing, and DNS zones. Firewall and network policies block cross-tenant traffic.
Where GPU access is required, tenants receive GPU resources within predefined quotas, managed through the same Kubernetes-based provisioning workflow as CPU, memory, storage, and networking. When a workload stops, compute and network resources release automatically. Persistent storage is retained or deleted based on policy rather than wiped by default.
The results were operational as much as financial: costs fell by more than 60%, and tenant deployment time dropped from several hours to under 15 minutes. More importantly, provisioning stopped being a bespoke DevOps task for every new tenant. A two-person platform team could operate the environment because tenant creation, resource allocation, network setup, and cleanup were handled through automation.
Part of the cost reduction came from replacing commercial platforms (VMware and OpenShift were both evaluated and rejected on cost grounds) with open-source KubeVirt and in-house automation. AI infrastructure comparisons often over-index on hardware and cloud hourly rates while underweighting platform licensing; this project treated licensing as one of the primary levers. The broader saving came from paying the automation cost once at the architecture level, rather than absorbing the coordination cost on every new tenant onboarding. That trade-off sits at the center of GPU provisioning as well: build the automation once, and manual onboarding stops being a recurring expense.
Queue-based scheduling gets the right workload onto the right hardware. But a small inference job claiming a full A100 still holds 100% of it: native scheduling has no utilization-aware mechanism to share that capacity by default. Three approaches address that, and they make genuinely different tradeoffs.
MIG's strongest-isolation guarantee comes at a cost: it's also the least flexible of the three: a fixed profile can't be resized without downtime, so getting the initial sizing right matters more than it does with vGPU or time-slicing. Implementing any of these solves the allocation problem, but doesn't automatically tell you whether the resulting slice is being used well.
The default tool for checking GPU activity is nvidia-smi. According to NVIDIA's own documentation, GPU utilization is measured as the percentage of time a kernel was scheduled and running. It can show that kernels were active during the sample window, but not whether those kernels were compute-efficient, memory-bound, poorly parallelized, or part of a workload that's correctly sized. More importantly, nvidia-smi's utilization metrics are explicitly unsupported on MIG-enabled GPUs: once MIG is active, the default tool loses its utilization signal specifically, even though it still reports process and memory information.
The lower-level GPU signals needed to judge whether partitions are sized and loaded correctly are available through NVIDIA's profiling metrics: compute activity, memory activity, SM occupancy, and bandwidth use.
A peer-reviewed study of MIG partitioning on A100 found that a BERT inference workload only reached about 50% GPU utilization on the largest partition available, while smaller partitions were too slow to meet the workload's latency requirement. MIG's partition sizes are fixed, so there's often no size that fits the workload exactly. For a single workload, accepting that waste is usually the right call — rebuilding the workload to fit a fixed partition takes more engineering time than the wasted capacity is worth. That changes at fleet scale. The same 50% gap repeated across hundreds of workloads adds up to real cost, and that's when restructuring, or switching to a different partitioning approach, starts to pay off. DCGM is the practical way to tell which side of that line you're on.
In 2026, cloud compute for startups and scaling AI teams is no longer the obvious default. H100 one-year rental contract pricing has climbed nearly 40% since October 2025, from $1.70 to around $2.35 per hour, according to SemiAnalysis, whose price index is built from monthly surveys of more than 100 market participants and validated against transaction data. On-demand GPU rental capacity has been effectively unavailable across the market. At sustained utilization, it's worth comparing renting capacity against buying hardware outright. Separately, infrastructure you already own can run in colocation, with someone else handling the power and cooling.
The standard comparison of hardware purchase price against hourly cloud rate misses most of what determines the actual cost. Three questions worth asking before making the call:
1) What are you paying for? Both sides of the comparison have costs that don't appear on the rate card. On-premise adds power, cooling, networking, maintenance, and ops headcount on top of hardware. Cloud adds storage, licensing for optimized inference engines, and managed service tiers that turn a predictable hourly rate into an unpredictable monthly bill. What matters is total ownership cost against total consumption cost over three years: roughly one hardware refresh cycle, long enough for both sides' hidden costs to show up.
2) What are you not counting? Egress, or the fee cloud providers charge when data leaves their network, is easy to miss entirely. Moving data out of AWS costs $0.09 per gigabyte at common first-tier rates. If your training data lives in cloud storage but your GPU cluster is on-premise, or vice versa, you pay that fee on every data movement between environments. At a petabyte of training data, that's $90,000 before a single GPU runs, and every iteration on that data pays the fee again. It's a cost that falls in the gap between the two columns in a standard comparison
3) What are you assuming that might be wrong? Most TCO models assume utilization levels that production workloads rarely hit. Deloitte's 2026 infrastructure research puts the tipping point at 60–70%: once cloud spend reaches that share of what equivalent hardware would cost, on-premise is worth a serious look. Cloud still fits variable training workloads best; on-premise fits steady production inference.
Our virtual datacenter project is a direct example: the licensing swap alone was a meaningful share of a 60% infrastructure cost reduction, without changing the build-vs-rent decision. Check what your current hypervisor, orchestration, and monitoring stack costs in licensing before assuming hardware procurement is the primary lever.
For prior-generation GPU capacity, the market is less constrained than it was at the peak of the 2023 shortage, but availability still depends on provider, region, contract length, and volume. The pressure has shifted to the newest systems. At GTC 2026, Nvidia’s CEO said it now sees at least $1 trillion in revenue from 2025 through 2027, driven by Blackwell and Rubin demand. For teams planning around B200-class capacity, the practical problem looks familiar: waitlists, limited reserved availability, and premiums on whatever short-term capacity exists.
Two common procurement levers help manage that constraint, and they solve different problems. EC2 Capacity Blocks for ML reserve actual GPU capacity for a defined future window, up to six months, so the hardware is available when the workload needs to run: priced separately from on-demand rates rather than at a fixed discount. That is a different guarantee than a regional Reserved Instance or Savings Plan provides: those can lower the rate, but they do not reserve physical capacity. The catch is that reserved capacity still has to be used well: an unfilled reservation window just relocates the underutilization problem from the scheduling layer to the procurement layer.
The second lever is Spot: cheaper, interruptible spare capacity. It suits work that can absorb an interruption: distributed training checkpointing its progress along the way, or batch inference queued to retry on failure. It is a poor fit for primary production serving without failover, where an interruption becomes a customer-facing outage rather than a delayed job.
Capacity Blocks feel safer because the guarantee is explicit. But a workload already built with checkpointing for fault tolerance may not need one — it's already Spot-viable. The workloads worth reserving are the ones where an interruption would actually cost you: a customer-facing outage, not just a delayed batch job. The decision between them comes down to one question: which workloads need guaranteed availability and which can tolerate interruption? Answering that requires visibility into actual GPU activity — allocation records alone won't tell you.
Single-provider GPU infrastructure has a predictable failure mode: quota limits, regional outages, and spot evictions don't arrive separately. When capacity gets tight, they hit together. Spreading workloads across providers reduces that exposure, though only if the operating layer underneath is consistent, which is the harder half of the problem. The case study below shows one team's version of it, built around a shared schema instead of duplicated per-provider configuration.
The hybrid decision comes down to where each workload belongs. Deloitte's 2026 research offers a starting framework, shown below. Real placement decisions also weigh data gravity, compliance, contracts, and team capability, not workload pattern alone
None of this works without a consistent way to manage infrastructure across environments. Kubernetes and Terraform standardize infrastructure definitions and deployment patterns; MLflow and OCI-compliant container formats do the same for model portability. These tools don't close every cross-environment gap on their own: IAM, secrets management, and observability still tend to diverge across providers unless someone unifies them deliberately. Left alone, hybrid infrastructure becomes multiple stacks drifting out of sync with each other. The case study below shows how a shared schema and drift correction close that gap.
DevOps platform company needed to run infrastructure consistently across Azure, GCP, AWS, and on-premises for clients operating in mixed environments. The core problem was fragmentation: different APIs, authentication flows, and IAM systems across providers, with no single way to define or enforce state across all of them. Every client onboarding meant hours of manual setup and provider-specific configuration.
The solution was a unified provisioning layer on Kubernetes, using custom resource definitions to manage cloud resources as Kubernetes-managed objects across supported providers. Infrastructure is described in a single schema and translated into provider-native resources automatically. A GitOps controller enforces the repository as a source of truth: if someone manually changes a cluster in Azure or GCP, the system detects the drift and restores the approved state. Azure AD and Google IAM were mapped to one centralized access model, with AWS IAM support built on the same framework. SIEM and CAPM (Cloud Application Performance Monitoring) integration added near real-time visibility across security events and infrastructure performance.
Running the same configuration across Azure and GCP repeatedly surfaced hidden differences in APIs, quotas, and default behaviors — problems that only appear when you run the same schema against both providers at once. Addressing them took significant cross-cloud testing before the abstraction layer held consistently across environments.
The result: onboarding time dropped 80%, from 2–3 hours to under 30 minutes, because the same schema now handles most requests without provider-specific workarounds — over 90% of them, per the case data. Misconfiguration errors fell 70% once schema validation moved into CI/CD, and the team now saves 15–20 engineering hours per project. Multiplied across a full client base, that's the difference between onboarding scaling with headcount and onboarding scaling with clients.
Preventing vendor lock-in means solving two separate problems: infrastructure and the ML platform layer on top of it. Kubernetes and infrastructure-as-code tools such as Terraform or OpenTofu reduce infrastructure lock-in at the workload and provisioning layers. The same core workload definition can often run across EKS, GKE, or AKS, though each provider's own networking, IAM, storage, and GPU setup still needs separate handling. Terraform can target multiple providers from a shared definition, but rarely without a translation layer in between: a shared schema that gets translated into provider-native resources, not a one-size-fits-all config.
ML platform lock-in is harder to see and more expensive to escape. Training pipelines built on SageMaker's APIs, models registered in Vertex AI's model store, experiment tracking tied to Azure ML — the control plane and metadata semantics don't transfer cleanly when the provider changes, even when individual artifacts and model weights can be exported.
Gartner predicts that by 2028, 30% of total global enterprise GenAI spend will be on open GenAI models tuned for domain-specific use cases. MLflow can reduce that coupling at the registry and tracking layer, giving models a provider-neutral version history and packaging convention, but serving elsewhere still requires compatible runtime, dependencies, and deployment code. OCI-compliant container formats standardize packaging and distribution for model artifacts the same way, without making runtime behavior itself portable.
Inference is where the preceding infrastructure decisions either pay off or compound into waste. It runs continuously, and its cost scales with every request the system handles in production. Sequencing matters here more than any individual tactic.
1) 0–90 days: measure before you optimize
One metric makes the cost side of every subsequent decision visible: cost-per-inference. The formula GPU hourly rate divided by tokens processed per hour, split by input and output where pricing or internal allocation differs, scaled to per-million-token units. Cost-per-inference doesn't replace p95/p99 latency, throughput, or error rate. Those metrics still gate whether a cheaper setup is actually viable. But without a cost baseline, none of the tradeoffs between them are visible either.
2) 3–9 months: make architectural decisions with data
With utilization numbers in hand, queue-based scheduling, MIG partitioning, and the cloud-versus-on-premise call all stop being intuition and start being decisions backed by data. Skip the measurement step and move straight to architectural changes, and you're optimizing against assumptions instead of evidence.
3) 9–18 months: apply the efficiency levers
Inference costs for GPT-3.5-level model quality dropped from $20 to $0.07 per million tokens between late 2022 and late 2024 — a 280-fold drop in under two years. Three mechanisms explain why teams can keep pushing inference cost down inside their own stack. Quantization often cuts memory requirements by 2–4x, with cost and accuracy effects depending on the model, task, and quantization method. Speculative decoding runs a small, fast draft model alongside the main model: the draft proposes several tokens at once, and the main model verifies them in parallel, increasing throughput while preserving the target model's output distribution when implemented with correct rejection sampling. Hardware right-sizing means matching the GPU partition profile to what the workload actually needs — a small inference job on an oversized partition wastes capacity the same way a full GPU allocation does, just at a smaller scale. Quantization strategy and carbon-aware scheduling are covered in more depth in Sustainable AI: Model Optimization and Carbon-Aware Computing.
Orchestration, cloud strategy, and inference optimization look like three separate problems. They're one problem at three different layers — and fixing any layer in isolation tends to reproduce the original cost spike in a new form.
A practical test: can you identify, right now, which workload drove last month's GPU bill and whether it used the hardware efficiently? If that took more than a few minutes to answer, we can find out for you. Talk to us about a utilization audit.
