Last month I did something I should have done a year earlier: I exported my token usage across every AI service I run, put the numbers in a spreadsheet, and multiplied. My agent infrastructure, the cron jobs that draft articles, the summarizers, the API calls stitched into side projects. The total was not catastrophic, but it was growing every month while I could not point to a single feature that got better.
Here is the strange part. While my bill climbs, the underlying price of intelligence is collapsing. Stanford's 2025 AI Index measured it precisely: querying a model at GPT-3.5-level performance on MMLU cost $20 per million tokens in November 2022. By October 2024, Gemini-1.5-Flash-8B hit the same quality bar for $0.07 per million tokens. That is a 280-fold drop in roughly 18 months. Epoch AI tracks the broader trend: inference prices for a fixed benchmark level fall a median of 50x per year, and since January 2024 the median has accelerated to around 200x per year.
So why do real-world bills go up? Because price per token is not the same thing as cost per task. The frontier labs cut prices, but teams respond by running bigger models, longer prompts, and always-on agents. Spending is exploding even as unit costs crater: enterprise LLM API spend doubled from $3.5 billion to $8.4 billion in just six months according to Menlo Ventures, and The Information reported OpenAI's 2025 inference bill alone near $8.4 billion.
Yesterday a piece by Philip Kiely at Baseten, "The efficient frontier of LLM inference," hit the front page of Hacker News, and it names the mental model I was missing. This article is my attempt to turn that model into decisions you can actually make, whether you self-host models or just pay an API bill like I do.
Full disclosure up front: I have never operated a large GPU inference cluster. I run API-based services and small local experiments on a MacBook Pro. The cluster-side numbers below come from published engineering sources, cited inline. The decision framework is what I have actually applied to my own stack.
The one idea worth stealing from portfolio theory
The efficient frontier is a concept borrowed from investing. For a fixed budget, there is a curve of optimal tradeoffs between two things you want. Everything below the curve is waste. Everything on the curve is a tradeoff: you can have more of one only by giving up some of the other.
For LLM inference, the two axes are latency (how fast a user sees tokens) and throughput (how many tokens per second your system serves in total). Baseten's framing splits every optimization technique into two categories, and this is the distinction that changed how I look at my bill:
- Tradeoff techniques move you along the frontier. You trade latency for throughput, or quality for speed. Nothing gets universally better; you just pick your position.
- Frontier-pushing techniques move the curve itself. Quantization, speculative decoding, better kernels. These create genuinely more efficiency, which you can spend on lower latency, higher throughput, or both.
The expensive mistake is applying tradeoff techniques while believing you are optimizing. Rebalancing batch sizes rearranges waste; it does not eliminate it.
Why one GPU cannot win at both phases
Every LLM request has two phases with opposite hardware personalities, and this mismatch is the root cause of the frontier existing. Google Cloud's inference engineering blog breaks it down clearly:
- Prefill is compute-bound. The GPU processes your entire prompt at once to build the key-value cache. All those matrix multiplications run in parallel, so tensor cores stay busy. Longer prompts mean more compute, and the GPU handles it efficiently.
- Decode is memory-bandwidth-bound. Generating each new token requires streaming the full model weights and the growing KV cache from high-bandwidth memory into the cores. One token at a time, no parallelism to hide the latency.
A single deployment tuned for one phase leaves the other phase starved. That is why "just add more GPUs" feels necessary when the real problem is that one rigid system is serving two incompatible workloads.
The techniques, ranked by how quickly they cut a real bill
1. Route intelligently before anything else. Not every request needs a frontier model. A lightweight classifier at the gateway can send hard reasoning to a large model and simple formatting, classification, or summarization to a small quantized model that costs orders of magnitude less per token. Google's GKE Inference Gateway case study is the proof this is not theory: intelligent L7 routing alone, with no hardware or model changes, cut time-to-first-token by 35%, improved P95 tail latency by 52% for bursty chat workloads, and doubled the prefix cache hit rate from 35% to 70%. Routing was the single highest-leverage move in their entire writeup, and it is the one I applied first to my own stack: my article-drafting cron jobs now run classification and linting steps on a small local model and reserve the expensive API calls for the actual writing.
2. Cache aggressively at the prefix level. If your prompts share a long, stable prefix (system prompts, tool definitions, retrieved documents), that prefix should be computed once and reused. A cache hit rate moving from 35% to 70%, as in the GKE case above, is essentially halving your prefill bill. On the API side, this maps directly to provider prompt-caching features: structure every prompt so the reusable part comes first and the per-request part comes last. I was interleaving static instructions with per-request data for months. Reordering them was a five-minute fix with a real effect on the invoice.
3. Quantize, but measure quality, not vibes. Running weights, activations, or the KV cache at lower precision improves both latency and throughput, which makes it one of the rare techniques that pushes the frontier out rather than just trading along it. Baseten notes the gains are especially large with modern microscaling formats like MXFP4 and NVFP4, where big serving improvements often cost little to no quality. The trap is that the quality-versus-efficiency frontier here is jagged: some precision drops are nearly free, others silently degrade exactly the tasks you care about. I ran a quantization bakeoff on my own local models earlier this year and the variance between formats was bigger than I expected. The rule that survived: benchmark on your own workload before trusting any general claim.
4. Use speculative decoding for predictable outputs. The idea is elegant: a small draft model guesses the next several tokens, and the big model validates them in one pass. Accepted guesses skip expensive forward passes entirely. Baseten points out that modern methods like EAGLE-3 make this work especially well on code generation, where output sequences are predictable, and that the technique now delivers throughput gains in addition to its traditional latency win. The cost: the draft model competes with the main loop for resources, so maximum batch sizes shrink. Great for interactive coding, less attractive for massive batch jobs.
5. Disaggregate prefill and decode when volume justifies it. At high volume, running prefill and decode on separate, separately-tuned worker pools lets you match the pool ratio to your actual traffic. Google's analysis is blunt about what this buys: mostly higher throughput at the same or slightly better latency, not magic. This is the heaviest lever and the one I have never pulled, because it only makes sense if you operate your own serving infrastructure. If you are on APIs, your provider has already made this decision for you, which is exactly why comparing providers on price per token alone understates the real differences.
6. Pick your batch size and parallelism deliberately. These are the pure tradeoff knobs. Bigger batches mean better throughput and worse per-user latency. More tensor parallelism can cut latency for large models while reducing maximum throughput. There is no configuration that wins both, which is the whole point of the frontier. The only wrong move is not knowing which axis your product actually needs.
The decision checklist I now use
Before touching any configuration, answer these in order:
- What does the user actually feel? Interactive chat needs low latency. Overnight summarization needs throughput. If your workload is batch, stop paying latency prices for nothing.
- Can routing remove the request from the expensive model entirely? Cheapest optimization, applies even on pure API stacks.
- Is your prompt structured for cache reuse? Static prefix first, variable content last. Check hit rates, not assumptions.
- Are you on the newest quantized format your hardware and quality bar allow? Revisit this every few months; the frontier moves.
- Is your output predictable enough for speculative decoding? Code and structured formats, yes. Creative prose, usually not.
- Only then: do you need more hardware? In that order, because every step before this one shrinks the hardware bill you were about to approve.
One honest caveat: Gartner projects that inference on a trillion-parameter model will cost more than 90 percent less in 2030 than in 2025, and the trend data backs the direction. But do not wait for prices to save you. The teams getting crushed are the ones whose token volume grows faster than the price falls.
What I would do differently if I started today
I treated model choice as the only cost lever for most of a year: swap providers, hunt for cheaper per-token rates, repeat. That is optimizing one point on the curve while ignoring the curve. The order that actually works is routing first, caching second, and only then negotiating over the residual. My own stack got meaningfully cheaper from prompt restructuring alone, which cost nothing and took an evening.
I write about Java, Spring Boot, and AI engineering every week, including the ongoing experiments from my own agent infrastructure. Subscribe, it's free, and you will get the next bakeoff before I optimize it into a footnote.
Have you audited your own inference costs recently? Did you find waste in routing, caching, or somewhere I have not thought of? Tell me in the comments, I am still tuning this.
Top comments (0)