Every LLM provider publishes a pricing table.$2.50 per million input tokens. $10 per million output tokens.` Clean. Transparent. Easy to spreadsheet.
So you run the napkin math: 10,000 requests/day × 2,000 input tokens × $2.50/M = $18.25/day on GPT-4o. Annualized: $6,660. The CFO approves it.
Three months later the bill is $54/day — $19,710/year — and nobody can explain the gap.
It's not a billing error. It's five structural leaks between the pricing page and your credit card.
The Five Leaks, at a Glance
| Leak | What It Is | How Much It Costs You |
|---|---|---|
| Workload ratio | Output tokens cost 3–4× more than input | 2.9× spread across use cases |
| Tokenizer variance | Same text = different token counts per provider | 5–15% (EN), 15–30% (multilingual) |
| Prompt caching | Anthropic gives 90% off, OpenAI 50% — nobody configures it | 24% of total bill |
| Batch processing | 50% off for async workloads | 15–30% blended |
| Retry overhead | Failed requests consume tokens twice | 1–3% + architectural waste |
These aren't additive. They're stackable. Combined, the difference between naive pricing and optimized reality is 40–65%.
Leak 1: Workload Ratio — Your Use Case Is the Multiplier
Output tokens cost 3–5× more than input tokens. The ratio between them is determined by your workload — and it's the single largest cost variable.
Same model (GPT-4o). Same request count (10,000/day). Different workloads:
| Workload | Input | Output | Annual Cost | vs Chat |
|---|---|---|---|---|
| 💬 Chat | 20M | 8M | $47,450 | 1× |
| 🔍 RAG / Q&A | 60M | 8M | $83,950 | 1.8× |
| 📝 Summarization | 80M | 10M | $109,500 | 2.3× |
| 💻 Code Generation | 15M | 30M | $123,188 | 2.6× |
| 🌐 Translation | 30M | 30M | $136,875 | 2.9× |
2.9× spread — same model, same request count. Before comparing providers. Before factoring any other leak.
Fix: Measure your actual input-to-output token ratio in production. Most teams guess 1:1. Almost no real workload is 1:1.
Leak 2: Tokenizer Variance — You're Comparing Different Units
Every provider's tokenizer is different. The same text produces different token counts on each:
| Provider | Tokenizer | Relative Efficiency |
|---|---|---|
| OpenAI |
cl100k_base (tiktoken) |
Baseline |
| Anthropic | Proprietary BPE | 5–10% fewer tokens (EN) |
| SentencePiece | 5–10% more tokens | |
| DeepSeek | BPE (optimized for Chinese+English) | 5–15% more tokens (EN-only) |
Why this matters: comparing per-token prices without benchmarking your actual text = comparing different units. Provider A at $2.00/M with a 10% hungrier tokenizer = Provider B at $2.20/M. The cheaper sticker price may be more expensive after tokenization.
Fix: Run your actual production text through 2–3 candidate tokenizers before committing. At 1M+ requests/day, a 10% efficiency gap is thousands/month.
Leak 3: Prompt Caching — The 90% Discount Nobody Turns On
Anthropic introduced prompt caching in August 2024. OpenAI followed with automatic caching. Google launched context caching in early 2025. The discounts are the largest cost lever in LLM APIs — and most teams never configure it.
| Provider | Standard Input | Cached Input | Discount |
|---|---|---|---|
| Anthropic Claude Opus 4 | $15.00/M | $1.50/M | 90% |
| Anthropic Claude Sonnet 4 | $3.00/M | $0.30/M | 90% |
| OpenAI GPT-4o | $2.50/M | $1.25/M | 50% |
| Google Gemini 2.5 Pro | $1.25/M | $0.3125/M | 75% |
What's actually cacheable in your app:
| Token Category | Typical Size | Cacheability |
|---|---|---|
| System prompt | 500–2,000 tokens | 100% |
| Few-shot examples | 500–3,000 tokens | 100% |
| RAG context | 2,000–8,000 tokens | 20–40% |
| Conversation history | 1,000–10,000 tokens | 0% |
Real example: a customer support chatbot with 1,500-token system prompt, 1,000-token few-shot examples, 3,000-token RAG context per query. Total input: 5,500 tokens. Cacheable: 2,500 tokens (45%).
| Scenario | Annual Cost (Claude Sonnet 4) |
|---|---|
| Naive (no caching) | $104,025 |
| With caching configured | $79,388 |
| Saved by one config change | $24,638 (24%) |
Fix: Identify your cacheable prefix tokens. Structure API calls so they appear at the beginning of every prompt. Anthropic requires explicit cache point marking; OpenAI and Google handle it automatically.
Leak 4: Batch Processing — Half Price, No Catch
OpenAI and Anthropic offer batch endpoints at 50% off standard pricing. The tradeoff: up to 24-hour completion SLA instead of real-time response.
For offline workloads — evaluation runs, dataset labeling, embedding generation, nightly summarization, synthetic data generation — there is literally zero downside. The 50% discount is free money.
Stacked with prompt caching:
plaintext
Cached input + batch = 5% of sticker price (90% off × 50% off)
Uncached input + batch = 50% of sticker price
Output + batch = 50% of sticker price
Moving 60% of the support chatbot's traffic to batch: $55,572/year vs $104,025 naive = 47% saved.
Fix: Segment traffic into realtime and async. Route async to batch endpoints. The infrastructure change is an API endpoint swap — no model changes, no prompt changes.
Leak 5: Rate Limit Retries — Paying Twice
When your app hits API rate limits, the client retries — and the failed tokens are charged. At 2% retry rate, 10,000 requests/day: $365/year in wasted input tokens. Small, but the architectural cost is larger: teams over-provision multiple providers to avoid limits.
Fix: Exponential backoff with jitter. Monitor retry rate (if >1%, you need higher limits or a queuing layer). Route async traffic to batch endpoints (separate, higher limits).
The 2026 Provider Landscape
| Tier | Models | Output Price | Best For |
|---|---|---|---|
| Premium | Claude Opus 4 | $75/M | Non-negotiable quality + caching |
| Standard | GPT-4o, Claude Sonnet 4, Gemini 2.5 Pro, Mistral Large 2 | $5–15/M | General purpose |
| Budget | GPT-4o-mini, Claude Haiku, Gemini Flash, Llama 4 Scout (Groq) | $0.50–1.25/M | Classification, extraction, filtering |
| Disruptor | DeepSeek-V3, DeepSeek-R1 | $1.10–2.19/M | Flagship capability at budget prices |
The caching twist: Anthropic's 90% cache discount makes Claude Opus 4's effective cached input ($1.50/M) cheaper than GPT-4o's standard input ($2.50/M). At high cache hit rates, the premium tier beats the standard tier on price.
Self-Hosted vs API: The Breakeven Math
| Scale | GPU Cost | Breakeven vs DeepSeek | Breakeven vs GPT-4o-mini |
|---|---|---|---|
| 8B model | 1× H100 = $1,800/mo | Wins at 35% utilization | Wins at 50% utilization |
| 70B model | 3× H100 = $5,400/mo | Wins at 40% utilization | Wins at 3% utilization |
The utilization reality: most teams overestimate their GPU utilization. Self-hosted GPUs idle during nights, weekends, holidays. The API charges zero for idle time. Bursty traffic → API wins. Steady high throughput → self-hosting wins.
The hidden cost: self-hosting a 70B model across 3 GPUs requires understanding tensor parallelism, quantization (AWQ/GPTQ/FP8), continuous batching (vLLM/TGI), and GPU node management. Budget 0.25–0.5 FTE for production self-hosting.
Five Questions That Determine Your Bill
- What's your actual input-to-output ratio? Measure it. Don't guess 1:1.
- What % of input tokens are cacheable? If >20%, Anthropic's 90% cache discount may beat GPT-4o despite the higher sticker.
- What % of traffic tolerates 24-hour latency? Batch = 50% off. Moving 30% of traffic to batch cuts blended cost by 15%.
- Is traffic steady or bursty? Steady → self-host. Bursty → API. Be honest about utilization.
- Need multi-provider for reliability? A three-tier routing strategy (budget/standard/flagship) cuts blended per-token cost by 60–80% vs routing everything to the flagship.
Interactive calculator: jslet.com/llm-api-pricing-calculator — compare 12 models across 6 providers with caching, batch, and workload presets. All client-side, no signup.
`
Top comments (0)