GPU cost optimization on AWS in 2026: a working playbook

TL;DR

• GPU instances are the single most expensive line item in most AI infrastructure budgets — and the most commonly mis-provisioned, with typical utilization sitting at 30-45%.
• The fix isn’t “use less GPU” — it’s matching instance type to workload shape, mixing Spot/Capacity Blocks/On-Demand deliberately, and building visibility so idle capacity gets caught in days, not months.
• Teams that apply this playbook typically cut GPU spend 40-60% without slowing down model development or degrading inference latency.
• Use our GPU cost calculator to model your own numbers against the scenarios below.

The problem: GPUs are expensive, and “just provision more” is the default

A single p5.48xlarge (8x H100) instance runs north of $90K/month on-demand. At that price, the difference between 35% and 70% utilization isn’t a rounding error — it’s the difference between a sustainable AI infrastructure budget and a board-level conversation about whether the AI initiative is worth it.

The trap is that GPU procurement decisions get made under deadline pressure — “we need to ship this model by Friday” — and once a team has working capacity, nobody revisits the provisioning decision. Six months later, that team is running the same instance type for inference that they provisioned for training, at 3am on a Tuesday when nobody is sending requests.

We’ve audited AI infrastructure budgets ranging from $15K/month to $400K/month. The waste pattern is nearly identical at every scale: 40-60% of GPU spend goes to capacity that’s either the wrong shape for the workload, or simply idle.

The framework: four levers, applied in order

1. Match instance type to workload shape — not the other way around

Training and inference have fundamentally different resource profiles, and provisioning them the same way is the single most common mistake we see.

• Training (large batch, long-running, fault-tolerant): Spot-friendly. Checkpoint frequently (every 15-30 minutes for large runs) and design the job to resume from the last checkpoint on interruption. p4d/p5 Spot pricing runs 50-70% below on-demand, and a training job that takes 18 hours instead of 14 because of one interruption is a non-event if your checkpointing works.
• Inference (low-latency, bursty, user-facing): On-demand or Reserved for the steady baseline; Spot only for overflow capacity behind a queue that can absorb a 2-minute interruption notice. For 7B-13B models, g5.xlarge/g6.xlarge (A10G/L4) covers most production traffic at a third of the cost of A100-class instances — and most teams over-provision to A100 “just in case.”
• Fine-tuning / experimentation: This is where Spot plus aggressive auto-shutdown earns its keep. Idle notebook instances and abandoned fine-tuning jobs are a quiet but persistent leak — we routinely find GPU instances that have been “temporarily” running for six weeks.

2. Build a deliberate Spot / Capacity Block / On-Demand mix

AWS gives you three distinct purchasing models for GPU capacity, and most teams use exactly one of them.

• Spot for fault-tolerant, interruptible workloads — 50-70% discount, with a 2-minute interruption notice you need to design around.
• Capacity Blocks for ML for predictable, time-boxed training runs — you reserve a block of GPU capacity for a defined window (1-14 days typically) at a discount versus on-demand, with the guarantee that spot doesn’t give you. Underused by teams that don’t know it exists.
• On-Demand / Reserved for the inference baseline that has to be there at 3pm on a Tuesday no matter what.

The math: a workload that’s 70% predictable training (Capacity Blocks), 20% experimentation (Spot), and 10% production inference (Reserved baseline + on-demand burst) costs roughly 45% less than the same workload run entirely on-demand — without any change to the actual compute performed.

3. Instrument utilization before you provision more

The question “do we need more GPUs?” should never be answered by “the team is asking for them.” It should be answered by a dashboard showing actual GPU utilization (nvidia-smi-derived metrics via DCGM exporter, scraped into Prometheus) against current capacity, segmented by team and workload.

In our experience, the first time a team sees this dashboard, they find at least one of: a forgotten notebook instance, a job that’s been queued (not running) for days while holding reserved capacity, or an inference deployment sized for a launch-day spike that never repeated.

4. Right-size before you reserve

Reserved Instances and Savings Plans lock in a discount — and lock in whatever instance type you choose, for one to three years. Reserving capacity before you’ve gone through steps 1-3 means committing to your current waste at a discount. Get utilization visibility first, rightsize the workload, then reserve the steady-state baseline.

Real-world example

An AI-native startup came to us spending $187K/month on GPU infrastructure — almost entirely on-demand p4d instances, provisioned during a six-month model development sprint and never revisited after launch. DCGM-based monitoring showed average utilization at 38%, with three instances effectively idle (under 5% utilization) for the prior five weeks.

We restructured the fleet: training moved to Capacity Blocks plus Spot with checkpoint-resume (cut training compute cost by ~55%), inference moved from p4d to a right-sized g5 fleet behind an autoscaling group keyed to request queue depth (cut inference compute cost by ~60%), and the idle instances were terminated. New monthly spend: $94K — a 50% reduction, with model iteration speed unchanged and inference p99 latency actually improving slightly because the new fleet was better matched to the traffic pattern.

Trade-offs and what we’d avoid

• Don’t move latency-sensitive inference to Spot wholesale. A 2-minute interruption notice is fine for a batch job; it’s a customer-facing incident for a chat interface. Keep a Reserved or On-Demand baseline for anything users are actively waiting on.
• Checkpoint-resume has to actually work before you bet a training run on Spot. Test the failure path deliberately — kill the instance mid-run — before you rely on it for a multi-day job.
• Capacity Blocks require forecasting discipline. They’re a great deal if you can predict your training windows two to eight weeks out. If your roadmap changes weekly, the flexibility of Spot may be worth more than the Capacity Block discount.

What to do next

1. Get a utilization baseline — DCGM exporter + Prometheus, two weeks of data, segmented by team. You can’t optimize what you can’t see.
2. Run your numbers through our GPU cost calculator to model the Spot/Capacity Block/On-Demand mix that fits your workload shape.
3. Talk to us about a structured GPU cost audit — our AI Infrastructure and Cloud Cost Optimization teams typically find 40-60% in recoverable spend within the first two weeks of an engagement, before any architectural changes are made.

If GPU spend is the line item nobody on your team wants to own, book a 30-minute call — we’ll walk through what we’d expect to find in your environment, free, before you commit to anything.

Related reading: GPU node pools on Kubernetes — five sharp edges to know for the scheduling and quota issues that compound the cost problem.