Alan West

Posted on Oct 27 Originally published at blog.authon.dev

Self-Hosted LLM vs API Cost: Below $10.5K a Month, an H200 Node Can't Beat the API

#ai #llm #devops #selfhosted

IBM's CEO told investors last week that open-weight models now run at "anywhere from 25 percent to 10 percent of the token cost of the frontier models" and are "90 percent as effective." According to The Register's write-up of the Q3 call, Arvind Krishna also thinks a useful on-prem AI setup needs just two or three racks.

I wanted to know if self-hosted LLM vs API cost actually works out like that, so I took public GPU prices, power rates, published throughput benchmarks and today's API price sheets and did the arithmetic. Short version: an 8x H200 node costs about $10.5K a month before anyone touches it. Against Claude Opus 5.5 it pays for itself at 6% utilization. Against DeepSeek's own off-peak API it needs 54%. If your API bill is under about $10.5K a month, the API beats that node no matter what.

Krishna's own benchmark says 71%, not 90%

On the call he described a broad benchmark where the top closed model scored 70 and open models scored 20 or 30 in 2024, then "70 to 50" by late 2025. 50 out of 70 is 71%.

Maybe the gap has closed further since then, but the only figures he actually gave don't reach 90%. That matters because everything below assumes the open model can do your job, and if it can't, the cost per token is irrelevant.

Checking 10-25% against Claude, GPT and DeepSeek API prices

Current frontier list prices, next to the best-known hosted open-weight model:

Model	Input $/1M	Output $/1M	Source
Claude Opus 5.5	$4	$20	Anthropic
Claude Sonnet 5.5	$2	$10	Anthropic
gpt-6.1-sol	$2	$10	OpenAI
DeepSeek V4 Pro (peak)	$1.32	$3.96	DeepSeek
DeepSeek V4 Pro (off-peak)	$0.66	$1.98	DeepSeek
DeepSeek V4.1 Flash (peak)	$0.30	$1.20	DeepSeek

V4 Pro output at peak is 19.8% of Opus 5.5 and 39.6% of Sonnet 5.5. So "10-25%" holds if you compare against the most expensive tier and ignore the mid tier. That's fair enough for an earnings call, but most teams aren't sending everything to Opus.

That's API to API, though. IBM sells hardware and Red Hat, so the claim it cares about is on-prem.

What an 8x H200 node costs per month

The inputs:

Hardware: an 8x H200 HGX server goes for about $340,000 new in volume (Rack Report via AMCompute, October 2026). Spread over 36 months, that's $9,444/month.
Power: that page puts the server at 10.2 kW. 10.2 kW × 720 h = 7,344 kWh. At the US commercial average of 14.47¢/kWh (EIA, August 2026), that's about $1,063/month before cooling overhead.

That comes to $10,507/month, without colo, networking, spares or people.

As a sanity check, DeepSeek published its own fleet cost in February 2025: $87,072/day across an average of 226.75 eight-GPU H800 nodes, assuming $2/GPU-hour (DeepSeek open-infra-index). That's about $384 per node per day, or $11,520 per month, close enough to my number.

How many tokens that node actually produces

This is where people argue most, because published throughput numbers vary by about 8x depending on who tuned the stack.

Untuned, one node: Verda ran DeepSeek-V3 FP8 on 8x H200 with SGLang v0.4.1, 1024 tokens in and 1024 out, and got 2,864 output tok/s at 8 requests/s (Verda). They say themselves it had no H200-specific tuning.
Tuned, multi-node: LMSYS ran DeepSeek with prefill/decode disaggregation and large-scale expert parallelism on 12 nodes of 8x H100, and reported 22.3k output tok/s per node with 2000-token inputs. They put the cost at $0.20 per 1M output tokens (LMSYS).
Production: DeepSeek's own fleet averaged about 14.8k output tok/s per H800 node while decoding.

Over a 30-day month (2,592,000 seconds), the Verda figure works out to 7.42B output tokens, plus the same number of input tokens. The LMSYS figure works out to 57.8B.

Break-even utilization vs each API

I priced one fully loaded month of the untuned Verda workload (7.42B in + 7.42B out) at each API's rates, then divided the node's $10,507 by that. The result is the utilization the node needs just to break even:

Compared against	API cost of the same tokens	Break-even utilization
Opus 5.5	$178,164	5.9%
Sonnet 5.5 / gpt-6.1-sol	$89,082	11.8%
DeepSeek V4 Pro, peak	$39,196	26.8%
DeepSeek V4 Pro, off-peak	$19,598	53.6%

On the untuned setup at 100% load, self-hosting costs $1.41 per 1M output tokens, with the matching input tokens included. At the LMSYS throughput it's $0.18, which lines up with their $0.20 claim. Compared with Opus 5.5's $20, that's 7% and 0.9%, which is well below Krishna's range.

So at enterprise volume he's being conservative. If the open model really does the job and you can keep the hardware busy, on-prem beats the frontier API easily.

The off-peak row is the one to stare at. DeepSeek's peak window is only 01:00-04:00 and 06:00-10:00 UTC on weekdays, so most of the week is billed at the lower rate. And these are cache-miss input prices. Cache hits on V4 Pro are $0.022-0.044 per 1M, so an agent that resends the same long context is cheaper again on the API side.

One caveat on the table: Verda benchmarked V3, not V4 Pro. I'm using it as a stand-in for a large open MoE on one node. Your model and prompt lengths will move these numbers.

Where self-hosting falls apart for small teams

Look at the fixed cost again: $10,507/month before anyone touches it. If your whole API bill is under about $10.5K/month, a node can't win, whatever its utilization. And DeepSeek's own API at off-peak rates needs you above 53.6% load before your hardware pays off.

Then there are the people. Krishna said it himself: "Where are my kernel engineers? Where are the people who worry about the underlying network?" The Register notes he "didn't have answers" beyond IBM building software to help. I'm not putting a salary in the table because it varies too much, but every dollar of engineering time you spend on the node adds a dollar to the break-even bill. If babysitting the node costs about as much as the node itself, the break-even roughly doubles to around $21K/month.

The HN thread mostly agrees. jacquesm, who runs an 8-GPU rig that cost about $25K, says it "is not cheaper than using a frontier lab (or at least, I don't think it is)." Forgeties79 says: "I can't 'invest' $25k into an LLM. The vast majority of people can't." The people defending local models mostly bring up privacy, no token caps and no refusals. Hardly anyone argues on price.

A single RTX 5090 dev box: a much lower bar

One RTX 5090 is a different calculation. Hardware Corner lists the card at $3,699 (average US market value) and 575 W TDP, and measured these llama.cpp generation speeds at 4K context:

Qwen3 32B Q4_K: 61.4 tok/s (43.8 at 32K)
Qwen3 30B A3B Q4_K: 226.1 tok/s (76.8 at 128K)
gpt-oss 20B Q4_K: 298.2 tok/s

Over 36 months the card costs $103/month. Run it flat out 8 hours a day, 22 days a month, and power adds about $15 at the same EIA rate. So the break-even is roughly $117/month of API spend you'd actually stop paying, assuming you already have the rest of the PC.

That's easy to hit with agentic coding, which burns huge amounts of input tokens. $117 buys 58M input tokens on Sonnet 5.5. It's also why the cheap tiers are hard to beat. At full speed, 176 hours a month on Qwen3 32B is about 38.9M output tokens. At Haiku 5.5's $0.50/1M output (prompts up to 100K tokens), that's worth $19. The dev box only saves money if it replaces mid- or top-tier calls, and one 5090 owner on HN (dekhn) says the local models "generate an enormous amount of noise" next to Claude or ChatGPT.

What to do before you buy GPUs

Pull last month's bill and split it by model tier. Only spending on Sonnet- and Opus-class models is worth trying to replace.
Rent a node by the hour (H100s run about $2.65/GPU-hour on-demand at Spheron) and benchmark with your real prompt shape:

vllm bench serve --model <your-model> \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 1024 \
  --request-rate 8 --num-prompts 2400

Run your own eval set against the open model. Don't trust benchmark leaderboards, and don't trust "90%."
Break-even utilization = monthly node cost ÷ (monthly tokens at full load × your API price). If you can't keep the node above that number during your real traffic hours, keep paying for the API.

My take: IBM's math is right for companies that already run data centers and have steady, boring workloads like the email summaries and contract checks Krishna mentioned. For a 10-person team paying $3K a month, it's a sales pitch for hardware.

So where did it tip for you? At what monthly API bill did self-hosting start making sense, if it ever did? And if you run your own node, what utilization do you actually see outside office hours?

DEV Community