How Much Does LLM Infrastructure Actually Cost in 2026?

TL;DR

• Frontier model APIs (October 2026 official prices): about $1-$10 per million input tokens and $5-$50 per million output tokens. Mid-priced models like Claude Sonnet 5.5 and GPT-6.1 Sol are $2 in / $10 out.
• Open-weight models from an API provider cost much less: popular models on Together AI run $0.15-$1.32 per million input and $0.60-$3.96 per million output.
• One H100 GPU on AWS costs about $5,000 a month on-demand (about $2,200 with a 3-year commitment).
• Self-hosting starts to pay off at roughly 10-20 million tokens a day against a top frontier model, much more against a mid-priced one, and maybe never against a cheap open-weight API.
• The cost most comparisons forget: by our estimate, 20-30% of a senior engineer’s time, every month. That’s roughly $4,000-$6,000 a month on top of the GPU bill.

Updated October 27, 2026 with current official API prices and AWS’s lower GPU prices. Earlier versions of this post used older prices.

The question behind the question

“How much does LLM infrastructure cost” almost always means “should we call an API or run our own GPUs?” The honest answer depends on two things: how many tokens you use every day, and which API you would compare against. The numbers below show why.

What the API costs

Prices below are from the providers’ own pricing pages, checked October 2026.

Type	Example models	Input (per 1M tokens)	Output (per 1M tokens)
Top frontier	Claude Fable 5.1, GPT-6 Astra	$10	$50
Mid frontier	Claude Opus 5.5	$4	$20
Mid frontier	Claude Sonnet 5.5, GPT-6.1 Sol	$2	$10
Small / mini	GPT-5 mini	$0.25	$2
Open-weight (via Together AI)	DeepSeek V4 Pro	$1.32	$3.96
Open-weight (via Together AI)	DeepSeek V4.1 Flash	$0.30	$1.20
Open-weight (via Together AI)	gpt-oss-120B	$0.15	$0.60

Two things stand out. First, output tokens cost about 5x more than input tokens on most frontier models. A feature that writes long answers (summaries, code, reports) costs far more than one that reads a long document and gives a short reply, even with the same total tokens. Second, the gap between model tiers is huge: the top frontier model costs about 80x more per output token than gpt-oss-120B. Picking the right model for each task usually matters more than any infrastructure choice.

What self-hosting costs

GPU prices on AWS dropped a lot in June 2025 (up to 44% for H100 instances, up to 33% for A100). Current on-demand prices in US East:

• 8x H100 (p5.48xlarge): about $55 an hour, which is about $6.88 per GPU-hour, or about $5,000 a month for one GPU running all the time. With a 3-year Savings Plan it’s about $2.97 per GPU-hour, roughly $2,200 a month.
• 8x A100 (p4d.24xlarge): about $22 an hour, or about $2.74 per GPU-hour.

How many tokens you get for that money depends on the model, the GPU, and above all how busy you keep it. A GPU costs the same per hour whether it’s serving requests or sitting idle. Batching with a model server like vLLM helps a lot, but high utilization is something you get after tuning, not on day one.

Where self-hosting starts to pay off

Here’s simple math for one H100 at on-demand price (~$5,000 a month), assuming a typical mix of 80% input and 20% output tokens. The question is: how many tokens a day would you need to send to an API before the API costs more than the GPU?

Compared against	Blended API price (80/20 mix)	Break-even, GPU only	Break-even, GPU + engineer time
Top frontier ($10 / $50)	~$18 per 1M tokens	~9M tokens a day	~18M tokens a day
Mid frontier ($2 / $10)	~$3.60 per 1M tokens	~46M tokens a day	~90M tokens a day
Open-weight API ($0.15 / $0.60)	~$0.24 per 1M tokens	~700M tokens a day	~1.4B tokens a day

Three things to keep in mind:

• This assumes one GPU can actually serve that volume at the speed your users need. At the higher numbers, you’ll need more GPUs, which moves the break-even point up again.
• It assumes your self-hosted model is good enough to replace the API model. An open-weight model you run yourself is usually compared against a frontier API, but it may not match its quality for your use case. Test this first.
• Against cheap open-weight APIs, self-hosting rarely wins on cost. Those providers run GPUs at very high utilization across many customers. It’s hard for a single team to beat that. If you self-host, it’s usually for data control or a custom model, not to save money.

The cost almost every comparison forgets

Self-hosted inference isn’t a one-time setup. Someone has to watch GPU usage so you’re not paying for idle capacity, tune batching as traffic changes, upgrade models without the output getting worse, and catch the problem that’s specific to LLMs: a model that responds fast and looks healthy while quietly giving worse answers. Standard monitoring doesn’t catch that.

By our estimate, that’s 20-30% of a senior engineer’s time, every month. A senior MLOps engineer in the US earns around $187,000 a year (median). Add roughly 25% for taxes and benefits and 20-30% of that person comes to roughly $4,000-$6,000 a month. Add it to the GPU bill before comparing against an API.

Trade-offs and what we’d avoid

• Don’t compare API prices to GPU prices without adding engineering time. It’s the most common way this comparison gets tilted toward “self-hosting is obviously cheaper.”
• Don’t compare against the wrong API. Self-hosting looks great next to a top frontier model and much worse next to a cheap open-weight API. Compare against the cheapest API that meets your quality bar.
• Don’t assume high GPU utilization on day one. Budget your early estimate at lower utilization until you’ve measured it.
• Don’t use old prices. GPU and token prices both changed a lot in the last year. Check the providers’ own pricing pages before you decide.
• Don’t commit to reserved GPU capacity before you have real usage data. Commit to what you’ve measured, not to a forecast.

What to do next

Related reading: GPU Cost Optimization on AWS, Running vLLM on EKS, and DevOps for AI Startups.