How Much Does LLM Infrastructure Actually Cost in 2026?

TL;DR

• API pricing in 2026: frontier models run $2.50-$15 per million tokens in/out; open-weight hosted APIs (Llama, DeepSeek-class) run $0.07-$0.90 per million tokens.
• Self-hosted inference can get as low as $0.10-$0.18 per million tokens at high GPU utilization with batching — but that number assumes you’re already running it well.
• The real crossover point where self-hosting beats an API is roughly 2-5 million tokens a day on reserved capacity, not the much larger numbers some vendors imply.
• The cost almost every comparison leaves out: 20-30% of a senior engineer’s time, ongoing, to run self-hosted inference well. That’s $3,000-$6,000/month most sticker-price comparisons never mention.

The question behind the question

“How much does LLM infrastructure cost” almost always really means “should we call an API or run our own GPUs” — and the honest answer depends on volume, not on which option has the lower headline number. Both numbers below are real. Which one applies to you depends on how much you’re actually generating.

What the API costs

Frontier model APIs (the GPT and Claude class) run $2.50 to $15 per million tokens, combined input and output, depending on model tier. Open-weight models hosted by an API provider (Llama-class, DeepSeek-class) run meaningfully cheaper — $0.07 to $0.90 per million tokens — since you’re paying for hosting and margin, not a frontier lab’s training cost.

The detail that catches most cost estimates off guard: output tokens cost 2-5x more than input tokens, across virtually every provider. A workload that generates long responses (summaries, code, reports) costs far more per request than one that mostly reads long context and returns a short answer, even at identical total token counts.

What self-hosting actually costs

GPU instance pricing, as of now: an 8x A100 instance (AWS P4d.24xlarge) runs about $32.77/hour on-demand — roughly $4.10 per GPU-hour. An 8x H100 instance (P5.48xlarge) runs about $98.32/hour on-demand — roughly $12.29 per GPU-hour, though AWS Capacity Blocks (prepaid, reserved) bring that down to $3.93-$5.19 per GPU-hour for teams willing to commit. H100 costs roughly 3x an A100 per hour, and delivers roughly 3x the throughput for most transformer workloads — close to a wash on raw compute-per-dollar, with H100 pulling ahead specifically on larger models and memory-bound workloads.

Under strong conditions — high GPU utilization, proper batching with something like vLLM, a model that fits efficiently on the hardware — self-hosted inference cost can get down to $0.10-$0.18 per million output tokens. That number is real, and it’s also the best case, not the typical case. Most teams don’t hit 85% GPU utilization on day one; that’s an outcome of tuning, not a starting point.

The actual break-even point

Depending on GPU choice and usage pattern, self-hosting becomes cheaper somewhere between 2 million and 22 million tokens a day, with the more conservative, realistic estimate — reserved capacity over a 12-month commitment, not idealized batch conditions — sitting around 2-5 million tokens a day. Below that volume, a hosted API is almost always the better financial call once engineering time is counted. Above it, self-hosting starts winning on raw compute cost — but “winning on compute cost” and “actually cheaper for your team” aren’t automatically the same thing, which is the next section.

The cost almost every comparison forgets

Self-hosted inference isn’t a one-time setup. It’s an ongoing operational commitment: monitoring GPU utilization so you’re not silently paying for idle capacity, tuning batch sizes as traffic patterns shift, handling model version upgrades without a regression in output quality, and debugging the failure mode that’s specific to LLM infrastructure — a model that responds fast and looks healthy while quietly producing degraded output, which doesn’t show up in standard monitoring at all.

Realistically, that’s 20-30% of a senior engineer’s time, on an ongoing basis — roughly $3,000-$6,000/month in fully-loaded staffing cost. Add that to the GPU bill before comparing it to an API’s per-token price. A raw compute-cost comparison that leaves this out isn’t wrong about the GPU math; it’s just answering a narrower question than the one most teams are actually asking.

Trade-offs and what we’d avoid

• Don’t compare API pricing to self-hosted GPU pricing without adding engineering time to the self-hosted side. It’s the single most common way this comparison gets skewed toward “self-hosting is obviously cheaper.”
• Don’t assume you’re at 85% GPU utilization on day one. That’s a tuned outcome, not a default — budget your early self-hosted cost estimate closer to the higher end of the range until you’ve actually measured it.
• Don’t ignore the input/output token asymmetry when estimating API cost. A workload-specific estimate using your actual input/output ratio will be far more accurate than a blended average.
• Don’t over-commit to reserved GPU capacity before you have real usage data. The same principle that applies to cloud cost optimization generally applies here — commit to your measured steady state, not a forecast.

What to do next

Related reading: GPU Cost Optimization on AWS, Running vLLM on EKS, and DevOps for AI Startups — the deeper technical setup behind the numbers above.