The model loaded. The first prompt worked. Then longer prompts or multiple users showed up, and suddenly the same setup stopped feeling stable. A lot of the time, that is KV cache.
What KV cache changes
- more context means more memory tied up during generation
- more concurrent requests make the problem worse
- a setup that fits one short prompt can fail on real workloads
- people blame the model when the cache is the thing quietly growing
The common mistake
People test with one short input and assume the model fits.
Then product prompts get longer, users stack up, or batching gets turned on. The model did not change. The memory footprint did.
When KV cache becomes the real problem
- Short prompt, single user: Everything looks easy
- Longer prompt: Latency rises and memory margin shrinks
- Longer prompt + concurrency: This is where people suddenly think they need a bigger GPU
What we would do before upgrading
Measure the real prompt length. Measure concurrent requests. Then decide whether the better answer is quantization, shorter context, or a bigger card.
The expensive mistake is skipping that step and upgrading blind.
Top comments (0)