DEV Community: Rey Kingers

Token Economics: Why Your LLM Bill Is 3 What the Pricing Page Promised

Rey Kingers — Mon, 13 Jul 2026 03:01:02 +0000

Every LLM provider publishes a pricing table.$2.50 per million input tokens. $10 per million output tokens.` Clean. Transparent. Easy to spreadsheet.

So you run the napkin math: 10,000 requests/day × 2,000 input tokens × $2.50/M = $18.25/day on GPT-4o. Annualized: $6,660. The CFO approves it.

Three months later the bill is $54/day — $19,710/year — and nobody can explain the gap.

It's not a billing error. It's five structural leaks between the pricing page and your credit card.

The Five Leaks, at a Glance

Leak	What It Is	How Much It Costs You
Workload ratio	Output tokens cost 3–4× more than input	2.9× spread across use cases
Tokenizer variance	Same text = different token counts per provider	5–15% (EN), 15–30% (multilingual)
Prompt caching	Anthropic gives 90% off, OpenAI 50% — nobody configures it	24% of total bill
Batch processing	50% off for async workloads	15–30% blended
Retry overhead	Failed requests consume tokens twice	1–3% + architectural waste

These aren't additive. They're stackable. Combined, the difference between naive pricing and optimized reality is 40–65%.

Leak 1: Workload Ratio — Your Use Case Is the Multiplier

Output tokens cost 3–5× more than input tokens. The ratio between them is determined by your workload — and it's the single largest cost variable.

Same model (GPT-4o). Same request count (10,000/day). Different workloads:

Workload	Input	Output	Annual Cost	vs Chat
💬 Chat	20M	8M	$47,450	1×
🔍 RAG / Q&A	60M	8M	$83,950	1.8×
📝 Summarization	80M	10M	$109,500	2.3×
💻 Code Generation	15M	30M	$123,188	2.6×
🌐 Translation	30M	30M	$136,875	2.9×

2.9× spread — same model, same request count. Before comparing providers. Before factoring any other leak.

Fix: Measure your actual input-to-output token ratio in production. Most teams guess 1:1. Almost no real workload is 1:1.

Leak 2: Tokenizer Variance — You're Comparing Different Units

Every provider's tokenizer is different. The same text produces different token counts on each:

Provider	Tokenizer	Relative Efficiency
OpenAI	`cl100k_base` (tiktoken)	Baseline
Anthropic	Proprietary BPE	5–10% fewer tokens (EN)
Google	SentencePiece	5–10% more tokens
DeepSeek	BPE (optimized for Chinese+English)	5–15% more tokens (EN-only)

Why this matters: comparing per-token prices without benchmarking your actual text = comparing different units. Provider A at $2.00/M with a 10% hungrier tokenizer = Provider B at $2.20/M. The cheaper sticker price may be more expensive after tokenization.

Fix: Run your actual production text through 2–3 candidate tokenizers before committing. At 1M+ requests/day, a 10% efficiency gap is thousands/month.

Leak 3: Prompt Caching — The 90% Discount Nobody Turns On

Anthropic introduced prompt caching in August 2024. OpenAI followed with automatic caching. Google launched context caching in early 2025. The discounts are the largest cost lever in LLM APIs — and most teams never configure it.

Provider	Standard Input	Cached Input	Discount
Anthropic Claude Opus 4	$15.00/M	$1.50/M	90%
Anthropic Claude Sonnet 4	$3.00/M	$0.30/M	90%
OpenAI GPT-4o	$2.50/M	$1.25/M	50%
Google Gemini 2.5 Pro	$1.25/M	$0.3125/M	75%

What's actually cacheable in your app:

Token Category	Typical Size	Cacheability
System prompt	500–2,000 tokens	100%
Few-shot examples	500–3,000 tokens	100%
RAG context	2,000–8,000 tokens	20–40%
Conversation history	1,000–10,000 tokens	0%

Real example: a customer support chatbot with 1,500-token system prompt, 1,000-token few-shot examples, 3,000-token RAG context per query. Total input: 5,500 tokens. Cacheable: 2,500 tokens (45%).

Scenario	Annual Cost (Claude Sonnet 4)
Naive (no caching)	$104,025
With caching configured	$79,388
Saved by one config change	$24,638 (24%)

Fix: Identify your cacheable prefix tokens. Structure API calls so they appear at the beginning of every prompt. Anthropic requires explicit cache point marking; OpenAI and Google handle it automatically.

Leak 4: Batch Processing — Half Price, No Catch

OpenAI and Anthropic offer batch endpoints at 50% off standard pricing. The tradeoff: up to 24-hour completion SLA instead of real-time response.

For offline workloads — evaluation runs, dataset labeling, embedding generation, nightly summarization, synthetic data generation — there is literally zero downside. The 50% discount is free money.

Stacked with prompt caching:

plaintext Cached input + batch = 5% of sticker price (90% off × 50% off) Uncached input + batch = 50% of sticker price Output + batch = 50% of sticker price

Moving 60% of the support chatbot's traffic to batch: $55,572/year vs $104,025 naive = 47% saved.

Fix: Segment traffic into realtime and async. Route async to batch endpoints. The infrastructure change is an API endpoint swap — no model changes, no prompt changes.

Leak 5: Rate Limit Retries — Paying Twice

When your app hits API rate limits, the client retries — and the failed tokens are charged. At 2% retry rate, 10,000 requests/day: $365/year in wasted input tokens. Small, but the architectural cost is larger: teams over-provision multiple providers to avoid limits.

Fix: Exponential backoff with jitter. Monitor retry rate (if >1%, you need higher limits or a queuing layer). Route async traffic to batch endpoints (separate, higher limits).

The 2026 Provider Landscape

Tier	Models	Output Price	Best For
Premium	Claude Opus 4	$75/M	Non-negotiable quality + caching
Standard	GPT-4o, Claude Sonnet 4, Gemini 2.5 Pro, Mistral Large 2	$5–15/M	General purpose
Budget	GPT-4o-mini, Claude Haiku, Gemini Flash, Llama 4 Scout (Groq)	$0.50–1.25/M	Classification, extraction, filtering
Disruptor	DeepSeek-V3, DeepSeek-R1	$1.10–2.19/M	Flagship capability at budget prices

The caching twist: Anthropic's 90% cache discount makes Claude Opus 4's effective cached input ($1.50/M) cheaper than GPT-4o's standard input ($2.50/M). At high cache hit rates, the premium tier beats the standard tier on price.

Self-Hosted vs API: The Breakeven Math

Scale	GPU Cost	Breakeven vs DeepSeek	Breakeven vs GPT-4o-mini
8B model	1× H100 = $1,800/mo	Wins at 35% utilization	Wins at 50% utilization
70B model	3× H100 = $5,400/mo	Wins at 40% utilization	Wins at 3% utilization

The utilization reality: most teams overestimate their GPU utilization. Self-hosted GPUs idle during nights, weekends, holidays. The API charges zero for idle time. Bursty traffic → API wins. Steady high throughput → self-hosting wins.

The hidden cost: self-hosting a 70B model across 3 GPUs requires understanding tensor parallelism, quantization (AWQ/GPTQ/FP8), continuous batching (vLLM/TGI), and GPU node management. Budget 0.25–0.5 FTE for production self-hosting.

Five Questions That Determine Your Bill

What's your actual input-to-output ratio? Measure it. Don't guess 1:1.
What % of input tokens are cacheable? If >20%, Anthropic's 90% cache discount may beat GPT-4o despite the higher sticker.
What % of traffic tolerates 24-hour latency? Batch = 50% off. Moving 30% of traffic to batch cuts blended cost by 15%.
Is traffic steady or bursty? Steady → self-host. Bursty → API. Be honest about utilization.
Need multi-provider for reliability? A three-tier routing strategy (budget/standard/flagship) cuts blended per-token cost by 60–80% vs routing everything to the flagship.

Interactive calculator: jslet.com/llm-api-pricing-calculator — compare 12 models across 6 providers with caching, batch, and workload presets. All client-side, no signup.
`

DNS Propagation Time: How Long Until Your DNS Change Goes Live?

Rey Kingers — Mon, 13 Jul 2026 02:51:28 +0000

You change the A record forapi.example.comfrom203.0.113.10to203.0.113.20`. Your browser shows the new IP. The deployment dashboard says green.

Then the Slack messages start rolling in:

"API is down from Singapore."

"Connection refused from Frankfurt."

"Works fine here in Virginia."

This is DNS propagation. It is not a single number. It's a probability distribution — shaped by your TTL, the cache policies of 8 major resolver populations, and the geographic topology of the DNS hierarchy itself.

The user on Google DNS sees the change in 60 seconds. The user on Deutsche Telekom might wait 24 hours. Both readings are correct from their vantage point.

The Hidden Architecture Nobody Explains

DNS is not a push protocol. When you update a record at your authoritative nameserver, it doesn't notify anyone. It waits. Resolvers come to it when their cached copy expires. The time between "change made" and "every resolver has the new answer" is your propagation window — and it's determined entirely by cache expiration, not network physics.

The Four-Step Lifecycle of a DNS Change

Step	What Happens	Timer
1	You update the zone file and increment SOA serial	t=0
2	A resolver cold-queries your authoritative NS, gets the new record, starts its TTL countdown	t + RTT
3	While TTL ticks, all users behind that resolver get the cached answer	TTL window
4	TTL expires, resolver re-queries, gets the new record	t + RTT + TTL

The key insight: each resolver's countdown started when it last cached the record — not when you made the change. If a resolver cached the old record 30 seconds before your update with a 3600s TTL, it will serve the old value for 59 minutes and 30 seconds after your change.

The 8 Resolver Populations

Tier 1: Strict TTL Honor (Fastest)

Google Public DNS — 8.8.8.8

Walks the DNS hierarchy from root to authoritative for every cold query. No upstream forwarding. No minimum TTL override.
Propagation: TTL + 0–60s. At 300s TTL: 5–6 minutes.
Market share: ~10% of global DNS. Default on many Android OEM builds.

Cloudflare 1.1.1.1 — 1.1.1.1

330+ edge locations. Cache is per-edge, not global — Mumbai and London may see the record expire at slightly different times.
Propagation: TTL + 0–90s.
WARP catch: Cloudflare WARP users layer an additional cache at the egress point. Adds 30–60s delay vs native 1.1.1.1.

Tier 2: Mostly Honoring (Slight Lag)

Quad9 — 9.9.9.9

Honors TTL but enforces a 30s minimum. Every query passes through IBM X-Force + 18 threat-intelligence feeds; filtering adds 5–50ms to cold queries but doesn't affect cached answers.
⚠️ DNSSEC landmine: Quad9 validates DNSSEC by default. Broken DNSSEC → SERVFAIL → cached for the negative TTL. Frequently misdiagnosed as "propagation failure."

OpenDNS / Cisco Umbrella — 208.67.222.222

Honors TTL with 1–3 min processing overhead from content filtering. Enterprise Umbrella users may have admin-configured cache overrides.
NXDOMAIN handling: free tier replaces NXDOMAIN with a search page. Irrelevant to propagation but confusing during testing.

Tier 3: ISP Resolvers — The Wild West

Comcast / Xfinity — 75.75.75.75

Minimum TTL: 300s for A/AAAA. Regional cache clusters — East Coast ≠ West Coast.
Propagation: max(your TTL, 300s) + 0–15 min.
~30 million subscribers. Largest single ISP resolver population in the US.

BT / EE (UK)

Minimum TTL: 900s for A/AAAA. Independent regional clusters.
Propagation: max(your TTL, 900s) + 0–10 min.

Deutsche Telekom — the reason "48 hours" exists

Minimum TTL: 3,600s (1 hour) for A/AAAA. The most aggressive override among major ISPs.
Three-tier hierarchical cache: edge → regional → central. Each tier caches independently. All three must expire sequentially before the change propagates.
Propagation: 1–24 hours for A records. NS changes: 48+ hours.
45 million subscribers. This single ISP is why "DNS takes 48 hours" persists as industry lore — for their users, it genuinely can.

General ISP defaults — the catch-all model

Most Tier 2/3 ISPs run BIND or Unbound with default configs. BIND default min-TTL: 300s. Unbound: no minimum, but 86400s maximum. Many ISPs customize upward.
Safe assumption: ISP resolvers enforce 300–900s min for A/AAAA, 3600s for MX/NS, 600–3600s for negative caching.

The Pre-Warming Playbook

The difference between a 5-minute cutover and a 48-hour outage for half your users:

Day	Action	Why
T–1 day	Lower TTL to 60–300s. Wait ≥ old TTL duration.	Expire the old long cache everywhere before the change.
T=0	Make the DNS change.	Every resolver now has a short stale window, not a long one.
T+1 hour	Verify propagation across all 4 public resolvers.	Confirm the change is live.
T+1 day	Restore original TTL.	Operational TTL back to normal; short TTL already expired.

The logic: you're not fighting stale caches — you expired them before the battle. A 60s TTL means 60s propagation on Google DNS and Cloudflare. 300s on Comcast (their minimum). 3600s on Deutsche Telekom (their minimum). Pre-warming eliminates the old-cache variable; it cannot override ISP minimum TTL policies.

When pre-warming fails: NS record changes. Parent zone NS records have registry-level TTLs (.com = 2 days, set by Verisign). No amount of pre-warming your own zone accelerates registry TTLs. Nameserver migrations are inherently 24–48 hour affairs.

Quick Propagation Check

`bash

Test each major resolver

dig +short A example.com @8.8.8.8 # Google
dig +short A example.com @1.1.1.1 # Cloudflare
dig +short A example.com @9.9.9.9 # Quad9
dig +short A example.com @208.67.222.222 # OpenDNS

Google DNS-over-HTTPS (machine-readable)

curl -s "https://dns.google/resolve?name=example.com&type=A" | jq '.Answer[] | {name, data}'
`

Golden rule: if Google DNS returns the new IP but your ISP doesn't — it's your ISP's cache policy, not your DNS configuration. Don't touch the zone file. Wait.

Try the interactive propagation modeler at jslet.com/dns-propagation — plug in your TTL, record type, and see per-resolver estimates across all 8 populations. 100% client-side.
`

LLM Inference Latency: Why Your 7B Model Gets 15 tok/s on a T4 but 3,500 tok/s on an H100

Rey Kingers — Mon, 13 Jul 2026 02:16:01 +0000

`NVIDIA's spec sheet says the H100 delivers 989 TFLOPS of FP16 compute. The A100: 312. The T4: 65. Simple arithmetic says the H100 is 15× faster.

So a 7-billion-parameter LLM should be 15× faster on an H100, right?

It's 150× faster.

The T4 struggles at ~15 tok/s. The H100 cruises at ~2,200 tok/s — and with continuous batching, north of 3,500.

The 15× TFLOPS gap doesn't explain the 150× throughput gap. The missing variable is the one thing NVIDIA's marketing pages bury on line three of the spec table:

Memory bandwidth.

The T4 has 300 GB/s. The H100 has 3,350 GB/s. That's an 11× gap in raw bandwidth, and closer to 100–150× in effective throughput once cache size and clock speed are factored in.

This post traces the arithmetic from first principles: why memory bandwidth is the bottleneck, how quantization turns it into a lever, and what tok/s you should actually expect across 12 real GPU × model combinations.

1. Memory Bandwidth: The Bottleneck Nobody Talks About

Autoregressive LLM inference has two phases:

Phase	What Happens	Bottleneck
Prefill	Process the input prompt (all tokens in parallel)	Compute
Decode	Generate output tokens (one at a time)	Memory bandwidth

The decode phase dominates total latency for any response longer than a few tokens. And in decode, the GPU spends 98%+ of its time doing nothing — waiting for the next chunk of model weights to arrive from VRAM.

The Arithmetic, Step by Step

Each generated token requires reading every single parameter from VRAM. For a 7B model at FP16:

`plaintext
Weights: 7,000,000,000 params × 2 bytes = 14 GB per token

H100: 14 GB ÷ 3,350 GB/s = 4.18 ms → 239 tok/s (theoretical)
T4: 14 GB ÷ 300 GB/s = 46.7 ms → 21 tok/s (theoretical)
`

Now consider the compute cost: ~14 TFLOPs per token. The H100's 989 TFLOPS could execute that 70 times in 4.18 ms. The compute units finish their work and sit idle, waiting for the next 14 GB of weights to trickle in.

You cannot compute faster than you can read.

Every meaningful LLM optimization — quantization, KV cache compression, FlashAttention, speculative decoding — is fundamentally about reducing bytes moved per token.

The Full GPU Landscape

GPU	Bandwidth	7B FP16	7B INT4	70B INT4	VRAM
NVIDIA B200	8,000 GB/s	571	2,286	229	192 GB
NVIDIA H200	4,800 GB/s	343	1,371	137	141 GB
NVIDIA H100	3,350 GB/s	239	957	96	80 GB
NVIDIA A100-80GB	2,039 GB/s	146	583	58	80 GB
NVIDIA RTX 4090	1,008 GB/s	72	288	28.8	24 GB
NVIDIA L40S	864 GB/s	62	247	—	48 GB
NVIDIA T4	300 GB/s	21	86	—	16 GB

Theoretical tok/s at 100% bandwidth utilization. Real-world: multiply by 0.6–0.85. "—" = doesn't fit in VRAM.

2. Quantization: 4× Faster Without Changing GPUs

If your bottleneck is bytes-per-token, and you can't change the GPU's memory bandwidth, there's exactly one lever left: reduce bytes per parameter.

Precision	Bytes/Param	7B Model	70B Model	Speedup	Quality
FP16	2.0	14 GB	140 GB	1×	Reference
INT8	1.0	7 GB	70 GB	~1.8×	Negligible
INT4 (GPTQ/AWQ)	0.5	3.5 GB	35 GB	~3.5×	1–3% perplexity
INT3	0.375	2.6 GB	26 GB	~4.5×	3–8% loss

The rule of thumb: the larger the model, the harder it is to break with quantization. A 405B model at INT2 often outperforms a 70B model at FP16 on knowledge tasks — despite using fewer bytes per token.

What quantization doesn't improve: time-to-first-token (prefill latency). Prefill is compute-bound. Quantization can even make it slower due to dequantization overhead. The gains apply almost entirely to the decode phase.

3. Real Benchmarks

Theoretical ceilings are clean math. Reality includes framework overhead, attention kernel efficiency, and KV cache management. These numbers are from vLLM 0.6.x, continuous batching, bare-metal H100 instances — mid-2026.

Model	Precision	GPU(s)	Batch 1	Batch 8	Batch 32
Llama 4 Scout (8B)	FP16	1× H100	185	1,200	3,200
Llama 4 Scout (8B)	INT4	1× H100	620	3,800	8,500
Llama 4 Scout (8B)	INT4	1× RTX 4090	95	310	—
Mistral Small 3 (7B)	FP16	1× H100	195	1,250	3,500
Llama 4 Maverick (70B)	FP16	2× H100	35	190	480
Llama 4 Maverick (70B)	INT4	1× H100	78	440	1,050
Mistral Large 2 (123B)	FP16	4× H100	15	70	140
DeepSeek-V3 (671B MoE)	INT8	8× H100	18	85	180
Qwen 2.5 (72B)	INT4	1× H100	72	410	960
Phi-4 (14B)	INT4	1× RTX 4090	78	—	—
Gemma 3 (27B)	INT4	1× RTX 4090	55	—	—

"TP=N" = Tensor Parallelism across N GPUs. "—" = VRAM insufficient at listed context length. Output token decode only.

The headline: INT4 turns a 70B model from "needs 2 GPUs and is still slow" into "runs on 1 GPU and is faster than FP16 on 2."

4. Batch Size: The Latency–Throughput Tradeoff

Batching fills the idle time between memory reads. While one request waits for weights, another's compute can run. But per-request latency climbs with batch size.

Workload	Batch	Latency Target
Real-time chat / copilot	1–4	<200ms TTFT
Code completion	1–2	<20ms/tok
Support chatbot	4–16	<1s TTFT
Summarization (async)	16–32	Throughput priority
Dataset labeling	32–128	Throughput only

Continuous batching (vLLM, TensorRT-LLM, SGLang) is the critical innovation. Traditional static batching waits for all requests to finish. Continuous batching evicts completed requests and admits new ones at every decode step — eliminating the "one slow request holds up 31 others" problem.

If you deploy without continuous batching, you're leaving 50–70% of your GPU throughput on the table.

5. Multi-GPU: The Scaling Efficiency Problem

When one GPU isn't enough, you split the model with tensor parallelism. But all-reduce communication between GPUs eats into your gains:

Configuration	Batch 32 tok/s	Per-GPU Efficiency	NCCL Tax
2× A100 (TP=2)	380	92%	~8%
4× A100 (TP=4)	610	80%	~20%
8× A100 (TP=8)	840	63%	~37%

At TP=8, nearly 40% of interconnect bandwidth goes to synchronization.

This is why expert-parallel inference for MoE models is revolutionary: each expert lives on a dedicated GPU, and only the active experts participate in each forward pass. DeepSeek-V3 has 671B total parameters but only 37B active — so it's faster than a 70B dense model despite having 10× more total weights.

The One-Sentence Takeaway

LLM inference throughput = GPU memory bandwidth ÷ bytes per token. Everything else is overhead management.

Try the interactive calculator at jslet.com/llm-inference-latency — plug in your model, GPU, and quantization level. All 100% client-side, no signup.
`