Average GPU utilization inside enterprise Kubernetes clusters is 5%. That is not a typo. CAST AI analyzed roughly 23,000 clusters running on AWS, Google Cloud, and Azure for its 2026 State of Kubernetes Optimization Report and found that companies provision about 20 times more GPU capacity than they actually use. CPU utilization sat at 8% and memory at 20% in the same dataset, so GPUs, the most expensive silicon in the building, are also the most wasted.
That gap is where the money goes. A single reserved 8-way H100 node runs tens of thousands of dollars a month, and most organizations run dozens to hundreds of them. When 95% of that capacity idles, the loss is not a rounding error on the cloud bill. It is often the single largest correctable line item in an AI budget. This is a financial governance problem wearing an engineering costume, and closing it requires Finance and IT leaders to manage GPU spend on Kubernetes differently from the way they manage everything else in the cluster.
Why standard Kubernetes cost tactics break on AI workloads
Traditional Kubernetes cost optimization assumes horizontally scalable, predictable microservices. Add more pods, spread the load, bin-pack the nodes, done. AI workloads violate almost every one of those assumptions.
GPUs do not share the way CPU and memory do. A training job that needs 8 GPUs with high-bandwidth interconnect cannot be scattered across nodes without a severe performance penalty. Inference traffic arrives in bursts that horizontal pod autoscaling was never tuned to absorb. And a data scientist who requests “one GPU” for a notebook holds that hardware whether the tensor cores are pinned at 95% or the person went to lunch three hours ago.
The FinOps Foundation framework still applies (Inform, Optimize, Operate), but each phase needs an AI-specific lens:
- Inform: Standard cost allocation tags miss the metrics that matter. You need GPU memory utilization, tensor core activity, and multi-instance GPU (MIG) partition efficiency, not just pod resource requests. A pod that requests a GPU and uses 4% of it looks identical to a fully loaded one in request-based tooling.
- Optimize: Right-sizing recommendations from general-purpose tools are calibrated for CPU. A suggestion to downsize a node because average CPU is 20% ignores that the attached GPU may be carrying a latency-sensitive model.
- Operate: Chargeback based on resource requests understates real AI cost. A team that grabs a single GPU node for “experimentation” can quietly generate several thousand dollars a month that showback reports never surface.
Across production AI platforms I have reviewed, the cost distribution on a Kubernetes ML stack tends to look like this:
| Cost Category | Share of Total Spend | Typical Waste Factor |
|---|---|---|
| GPU compute (training) | 45-55% | 60-70% idle time |
| GPU compute (inference) | 20-30% | 40-50% over-provisioned |
| Persistent storage (models, datasets) | 10-15% | 30-40% orphaned volumes |
| Data transfer (egress) | 5-10% | Variable, often unmeasured |
| CPU and memory (preprocessing, orchestration) | 5-10% | 20-30% over-provisioned |
GPU compute dominates, and it is where the waste concentrates. That is the lever.
The 5% utilization problem, and what finally moves it
GPU utilization is the single biggest cost lever in AI, and it is also the least visible metric in most cost tooling. The economics are blunt. An NVIDIA H100 SXM instance rents for roughly $2.50 per hour on-demand from efficient providers, and 8-way nodes on the hyperscalers run far higher. At 5% average utilization, you are paying full freight for compute that sits dark 19 hours out of every 20.
Three structural issues drive the waste inside Kubernetes:
Reservation-based scheduling. Kubernetes places GPU workloads on resource requests, not actual use. Interactive notebook workloads routinely show utilization under 10% for most of their allocated hours, because a human is thinking, reading, or away, while the GPU stays reserved and billed.
GPU sharing that never ships to production. NVIDIA MIG and time-slicing have been available for years, yet adoption stays low. Teams cite performance isolation worries and setup complexity, so they default to one workload per GPU even when four could coexist.
Training pipeline stalls. A distributed training job can hold 95% GPU utilization during forward and backward passes, then collapse below 10% during data loading, checkpointing, and inter-node communication. Monitoring that samples once a minute misses the stalls entirely and reports a rosy average.
The tactics that reliably move utilization:
- MIG partitioning for inference. Splitting an H100 or A100 into isolated instances lets multiple model replicas run on one physical GPU. Teams that do this commonly cut inference endpoint cost 40-60%. The caveat: MIG needs current CUDA and driver versions, and some legacy model containers need updating first.
- Time-slicing for development clusters. NVIDIA’s device plugin lets 4 to 8 workloads share a GPU. Expect 30-50% savings on dev clusters, at the cost of 2-4x higher latency per workload, which is fine for experimentation.
- Idle timeout enforcement. Automatically hibernate or terminate GPU pods after a set idle window. A 30 to 60 minute threshold on development environments typically recovers 25-35% of dev GPU spend on its own, and it is the cheapest policy to implement.
- Right-size the GPU, not just the count. Plenty of inference runs on H100 or A100 80GB hardware when a smaller GPU would serve the model comfortably. Audit actual GPU memory in use, not the request, and downsizing candidates worth 40-70% per workload usually surface fast.
The 2025-2026 shift: Dynamic Resource Allocation and open-source scheduling
The biggest structural change since this problem was first framed is that Kubernetes itself got better at GPU sharing. Two developments matter for anyone budgeting AI infrastructure.
Dynamic Resource Allocation (DRA) reached general availability in Kubernetes 1.34 in September 2025 and is now enabled by default. DRA replaces the rigid device-plugin model with an API where a workload describes what it needs (say, two GPUs on the same node with NVLink and at least 40GB of VRAM), and the scheduler allocates accordingly. It supports GPU sharing across pods through time-slicing and MIG rather than the old static one-pod-to-one-GPU mapping. Kubernetes maintainers and vendors report utilization moving from the 45-60% range typical of device plugins toward 70-85% with DRA-based packing and sharing. Red Hat brought DRA to GA in OpenShift 4.21 in March 2026, and the managed Kubernetes services on all three hyperscalers now support it. If your clusters are still on the device-plugin model, upgrading is now a direct cost lever, not just a maintenance chore.
NVIDIA open-sourced the Run:ai scheduling engine as the KAI Scheduler in April 2025, under Apache 2.0, following its acquisition of Run:ai in late 2024. KAI is now a CNCF Sandbox project and brings gang scheduling, GPU sharing, hierarchical quota fairness, DRA support, and topology-aware placement to any cluster without the enterprise license that used to gate those features. For teams that could not justify commercial GPU scheduling software, the calculus changed: the core scheduling capability that used to cost real money is now free to adopt, and the paid Run:ai platform layers management and support on top. Evaluate KAI before you assume you need a commercial scheduler.
Together these two shifts mean the “GPU sharing is too hard for production” excuse is weaker in 2026 than it was even a year ago. The tooling caught up. The gap now is adoption and governance, not capability.
Spot and preemptible GPUs: the discount most teams still skip
Cloud providers sell GPU spot capacity at 50-80% off on-demand, and AI platform teams still leave most of it on the table. The fear is legitimate. A preempted training job that loses hours of progress is a real productivity hit. But checkpointing and Kubernetes-native spot handling make that risk manageable for a large share of AI work.
Current on-demand and spot economics on efficient providers, for orientation (rates move constantly, so confirm live pricing before committing):
| GPU | Representative On-Demand ($/GPU/hr) | Typical Spot Discount |
|---|---|---|
| H100 SXM | ~$2.50 | 50-70% |
| H200 | ~$3.70 | 50-70% |
| B200 (Blackwell) | ~$6.00 | 55-70% |
One pricing note worth flagging for budget planning: the hyperscalers are no longer the cheapest option for any GPU model. AWS, Azure, and Google list H100 and H200 capacity well above the neocloud and specialist providers, even after AWS cut its H100, H200, and A100 instance prices by up to 45% in 2025. If your GPU workloads run exclusively on a hyperscaler, that premium is part of what your 5% utilization is buying.
A workable framework for moving GPU work onto spot:
Tier 1, migrate now (low risk): hyperparameter tuning, data preprocessing pipelines, model evaluation runs, and general experimentation. These are parallel and failure-tolerant by nature.
Tier 2, spot with checkpointing (medium risk): training jobs that checkpoint under 30-minute intervals, fine-tuning jobs, and batch inference that is not time-sensitive.
Tier 3, on-demand only (high risk): real-time inference with SLAs, training jobs in their final convergence phase, and any job where a preemption costs more than the spot savings.
Implementation leans on node pools configured for spot or preemptible capacity plus pod disruption budgets that allow graceful checkpoint-and-terminate. Karpenter on AWS and GKE’s node auto-provisioning both handle spot natively, though each has quirks: Karpenter’s consolidation can be too aggressive for long-running training, and managed autopilot modes restrict the custom node configs some ML frameworks expect. Test the interruption behavior before you trust it in production.
Kubernetes cost tools for AI workloads, as of mid-2026
The tool landscape reshuffled since this guide first ran, and ownership changed hands. IBM acquired Kubecost in September 2024 and folded it into its FinOps suite alongside Cloudability and Turbonomic. Run:ai became NVIDIA and its scheduler went open source as KAI. Here is the honest state of play:
| Tool | GPU Cost Attribution | AI-Specific Optimization | Spot Integration | The Catch |
|---|---|---|---|---|
| IBM Kubecost / OpenCost | Basic (requests-based) | Limited | Good | GPU utilization needs DCGM integration; recommendations not tuned for training jobs |
| CAST AI | Strong (utilization-aware) | Moderate to strong | Excellent | Automation can interrupt training without careful policy tuning |
| Vantage | Good (multi-cloud) | Limited | Moderate | Visibility-first; light on in-cluster optimization actions |
| CloudZero | Good (unit economics) | Moderate | Moderate | Best at cost-per-unit allocation; GPU optimization less mature |
| NVIDIA KAI Scheduler | Scheduling-level, not billing | Strong (sharing, gang, quota) | Good | A scheduler, not a cost dashboard; pair it with a cost tool |
The honest assessment has not changed: no single product delivers comprehensive AI cost management out of the box. The teams getting best-in-class results combine a cost platform (IBM Kubecost or CAST AI) with GPU-level telemetry (NVIDIA DCGM exporter feeding Prometheus) and a workload-aware scheduler (KAI, or native Kubeflow and MLflow capabilities). If you want the broader tradeoffs across cloud cost platforms beyond Kubernetes, our FinOps tools comparison covers the full field.
A chargeback model that actually changes behavior
Namespace-based chargeback keyed to resource requests creates exactly the wrong incentives for AI teams. If you charge per GPU-hour regardless of utilization, a team pays the same at 5% and at 95%, so there is no reason to optimize. Worse, teams learn to request maximum capacity “just in case,” which inflates the baseline for everyone.
A chargeback framework built for AI looks different:
- Split training and inference pools. Allocate training cost to project and experiment budgets with clear cost-per-model-trained. Allocate inference cost by request volume and latency tier, which ties spend to business value.
- Bill on utilization, not reservation. Charge a base rate for reserved capacity plus a premium for GPU-hours actually consumed above baseline. This rewards teams that consolidate and penalizes idle reservations.
- Offer tiered GPU pricing. Make a development tier (shared GPUs, spot capacity, higher latency tolerance) available at a 40-60% discount to the production tier. Publish the differential and let teams self-select.
- Set experimentation budgets. Give research a fixed monthly GPU budget separate from production. When it runs out, teams wait for the next period or request more with justification.
- Report unit economics. A team spending $50,000 a month serving 100 million inferences ($0.0005 each) is more efficient than one spending $10,000 serving one million ($0.01 each). Aggregate spend hides that; cost-per-inference and cost-per-training-run reveal it.
Organizations that move to utilization-adjusted chargeback typically see a 20-35% GPU cost reduction within two quarters, and the savings come mostly from behavior change rather than any technical trick. The prerequisites are executive sponsorship from both Finance and Engineering, pricing published before it goes live, a 60 to 90 day shadow-billing period where teams see costs without financial consequence, and a clear exception path. Skip the shadow period and you get disputes and slow adoption instead.
A 90-day plan for Finance and IT leaders
A structured sequence beats scattered optimization. Here is a program that works.
Days 1-30, build visibility. Deploy the NVIDIA DCGM exporter across all GPU nodes. Stand up IBM Kubecost, CAST AI, or an equivalent with a GPU-aware cost model. Establish baselines for GPU utilization, cost-per-team, and cost-per-workload-type. Identify your top five cost-generating workloads and your five lowest-utilization resources. Record current spot usage as a percentage of GPU hours.
Days 31-60, capture quick wins. Enforce idle timeouts on development GPU pods at a 30 to 60 minute threshold. Move hyperparameter tuning and preprocessing to spot. Right-size at least three over-provisioned workloads from the assessment. Turn on MIG or time-slicing for development clusters, and if you are on the device-plugin model, evaluate the DRA upgrade path. Start a weekly cost review with the platform team.
Days 61-90, make it structural. Roll out utilization-adjusted chargeback. Automate spot handling for checkpointed training. Set GPU budget allocations by team and project. Build cost-per-inference and cost-per-training-run dashboards. Document the policies and train teams on cost-aware development.
Teams that complete a program like this generally cut GPU spend 30-45% inside the first 90 days, with more available in the following quarter as behavior and automation mature. The common failure modes are worth naming: stalling in the visibility phase over GPU metric integration, capturing quick wins but failing to sustain them without governance, and resistance from data science teams who read cost controls as productivity barriers. Dedicated engineering support for tooling, a real FinOps operating rhythm, and policies built with the AI teams rather than imposed on them address all three. If broader cluster and cloud waste is the concern, our guide to finding and eliminating cloud waste covers the non-GPU categories, and the rightsizing AI infrastructure playbook goes deeper on the sizing decisions.
The macro trend gives this urgency. Flexera’s 2026 State of the Cloud Report found estimated cloud waste rose to 29%, its first increase in five years, and named AI workloads as the driver. GPU-heavy Kubernetes is where that waste concentrates. The teams that fix it now are the ones whose AI programs survive the first serious budget review.
Frequently asked questions
What is a realistic GPU utilization target on Kubernetes?
It depends on the workload. Production inference should target 60-75% average utilization, since pushing higher risks latency spikes during traffic bursts. Active training phases should target 80-90%, accepting that data loading and checkpointing create periodic dips. Development and experimentation environments often top out at 25-40% even after optimization, which is acceptable given their exploratory nature. Mature organizations reach 55-65% aggregate utilization across all workload types, against the 5% average CAST AI measured across the broad population. The fastest win is eliminating zero-utilization hours before you optimize partially loaded capacity, because most environments carry a surprising number of GPU-hours doing effectively nothing.
Do reserved instances and savings plans make sense for GPUs?
They do, but with more caution than for standard compute. AWS reports roughly 35-40% savings on one-year commitments and 55-60% on three-year, yet GPU hardware turns over every 12 to 18 months, which makes three-year commitments risky when a better price-performance instance may launch mid-term. A balanced split: commit reserved capacity to 40-50% of your stable baseline (production inference and always-on services), cover another 20-30% of predictable training with flexible savings plans, and keep 20-30% on spot or on-demand for variable and experimental work. Favor commitments that flex across instance families over ones locked to a specific instance type. Our deeper comparison of reserved instances versus savings plans walks through the tradeoffs by provider.
Should Finance or IT own AI infrastructure FinOps?
Neither should own it alone. Finance should own budget allocation, chargeback policy design, and executive reporting on unit economics. Platform Engineering should own tooling, tagging standards, and technical optimization. The ML platform team should own workload-level decisions and capacity planning for training. A dedicated FinOps practitioner reporting jointly to Finance and Engineering acts as the coordination layer. Put FinOps entirely under Finance and you get good visibility but weak technical follow-through; put it entirely under IT and you optimize well but lose the line back to business outcomes. The FinOps Foundation’s federated model, central standards with embedded practitioners, is the pattern that holds up.
What metrics prove the program is working?
Track financial and operational measures over time. On the financial side: total GPU spend trend, cost per training run by model type, cost per 1,000 inferences by model, and GPU cost as a share of the AI program budget. On the operational side: aggregate GPU utilization (target 55% or better for mature teams), spot as a percentage of GPU hours (target 40-60% for training), idle GPU hours as a share of total (target under 15%), and time-to-terminate for idle resources (target under 60 minutes). Watch out for vanity metrics like raw “dollars saved” with no baseline: a 50% cut from a bloated starting point is less impressive than 15% off an already-lean environment.
How do we allocate cost across teams sharing a GPU node pool?
Three approaches work, each with a tradeoff. Request-based allocation charges teams for the GPU resources their pods request; it is simple through Kubecost but rewards over-requesting. Utilization-weighted allocation measures actual GPU use per pod through DCGM and splits node cost proportionally; it rewards efficiency but needs richer monitoring and invites disputes when measurement has gaps. Time-slice allocation charges by GPU-seconds actually consumed, the most accurate option, but it requires scheduling infrastructure such as the KAI Scheduler that tracks consumption below the pod level. Most teams start request-based and migrate to utilization-weighted as the practice matures. Whichever you pick, set an explicit policy for shared overhead (cluster management, monitoring, networking), usually a 10-15% surcharge on direct GPU cost or a proportional split. Getting these allocation rules right is what turns shared pools from a source of orphaned, unclaimed spend into an accountable, optimizable resource.