Phone prices are climbing this year largely because memory is scarce - AI data centers are buying DRAM faster than fabs can make it. That same squeeze shows up in your cloud bill, and it changes which knob you turn first when inference gets expensive.
Memory Is the Binding Constraint, Not Compute
When teams hit capacity limits serving a language model, the reflex is to ask for more GPUs. But on most serving setups the shortage isn't raw compute - it's the memory those chips carry. Model weights take a fixed slice. Everything left over goes to the KV cache: the running store of intermediate values for every token the model has already seen in a conversation. That cache grows linearly with context length and with the number of requests you serve at once.
So the real decision isn't "buy more GPUs or not." It's: do we cap context length and retrieve the relevant parts, or do we pay for the memory to keep stuffing everything in the prompt? Long-context models made the second option easy to reach for. Chipflation is making it expensive to keep reaching.
Running the Numbers Before You Ask for Budget
The formula is simple enough to put in a planning doc:
kv_bytes_per_token = 2 * layers * kv_heads * head_dim * bytes_per_value
# 7B-class model, GQA: 2 * 32 * 8 * 128 * 2 (fp16) = 131,072 bytes
# = 128 KB per token, per request
# 8k context -> 1.0 GB per request
# 32k context -> 4.0 GB per request
On an 80 GB accelerator with roughly 15 GB going to weights, you have ~65 GB for cache. At 32k context that's about 16 concurrent requests. At 8k, about 65. Same hardware, four times the throughput - purely from a context decision, no new chips required.
That's the trade-off stated honestly: you're spending retrieval quality to buy concurrency. The way to tell whether it's worth it is not vibes. Build a held-out set of 100 - 200 real user questions, answer each one twice - once with the full long context, once with an 8k window plus retrieval - and score them. If the quality gap is inside your tolerance, cap the window. If your users routinely need reasoning across a whole document, it isn't, and the extra memory is a legitimate line item.
There's a middle lever too: quantizing the KV cache to 8-bit halves the per-token cost with a smaller quality hit than truncation. Worth testing before you commit to either extreme.
Key Takeaways
- Memory, not compute, usually sets your inference ceiling - and memory prices are rising for reasons outside your control.
- Context length and batch size drive KV cache size linearly; a 4x context cut is roughly a 4x concurrency gain on the same hardware.
- Decide with a scored comparison on real questions, not intuition - long context is a cost you should be able to justify.
If you ran that comparison on your own traffic, how much of your current context window is actually load-bearing?
Sources referenced: The Verge
Top comments (0)