Every token in a session, from either side, leaves a permanent entry in the key/value cache. What it costs is exactly derivable. What is not obvious is that under the most common local runtimes the memory does not grow as the chat goes on — it was all taken at load time, and the ceiling arrives as silent truncation instead.
What one token costs
Each layer stores one key vector and one value vector per key/value head per token:
bytes per token = 2 * L * H_kv * D * b
Llama 3.1 8B 2 * 32 * 8 * 128 * 2 = 131,072 = 128 KiB
Yi-34B 2 * 60 * 8 * 128 * 2 = 245,760 = 240 KiB
Llama 3.1 70B 2 * 80 * 8 * 128 * 2 = 327,680 = 320 KiB
Llama 2 13B 2 * 40 * 40 * 128 * 2 = 819,200 = 800 KiB
The cost depends on depth and on the number of key/value heads, and on nothing else — not on the parameter count, not on the quantization of the weights, not on the feed-forward width. This is why a 13B without grouped-query attention costs 2.5x more per token than a 70B with it, and it is the single most counter-intuitive number in local sizing. The mechanism itself is covered under the KV cache.
How a session accumulates
Every token counts once and stays: the system prompt, every user message, every assistant reply, every tool result. There is no per-message reset, because the model is re-reading the whole sequence on every step.
Llama 3.1 8B at 128 KiB/token, fp16 cache:
system prompt, 400 tokens 0.05 GiB
after 10 turns of ~300 tokens each (3.4k) 0.42
after 40 turns (12.4k) 1.51
after pasting a 20,000-token file (32.4k) 3.96
at the full 131,072-token window 16.00 GiB
Llama 2 13B at 800 KiB/token, same session:
after 40 turns (12.4k) 9.46 GiB
The interesting row is the file paste. Forty turns of ordinary conversation cost less than a single dropped document, because the growth is linear in tokens and a document is thousands of them at once. If your assistant gets slow and heavy at an unpredictable moment, look at what was pasted rather than at how long you have been talking.
Two things people expect to reduce the total do not. Deleting a message from the interface does not free anything unless the client rebuilds the request without it, in which case the cache is invalidated from that point and re-computed rather than shrunk. Summarising earlier turns genuinely does help, but only for the next request: it produces a shorter sequence to send, and the saving appears when the runtime processes that shorter sequence, not retroactively. The cache holds a prefix of a specific token sequence, so anything that changes a token in the middle discards everything after it.
It is allocated up front
Here is the correction. llama.cpp and Ollama allocate the key/value cache for the whole configured context when the model loads, not as tokens arrive. The loader prints its size, and it does not change for the life of the process:
llama_kv_cache_init: CUDA0 KV buffer size = 4096.00 MiB
for -c 32768 on an 8-layer-group model at 128 KiB/token:
32768 * 131072 = 4.295e9 bytes = 4.00 GiB — taken at load, not at turn 40
Which changes three things about how you reason:
- An empty chat costs the same as a full one. The memory is committed by
-c, so the question “how much will this grow to” has already been answered before the first message. - Leaving
-cunset is the dangerous case. If the runtime takes the model’s advertised window as the default and that window is 128k, it will try to allocate 16 GiB of cache for an 8B, and the failure looks like the model being too large when it is not. - A long chat does not cause an out-of-memory error. If it loaded, the cache fits. Something else is wrong if it dies mid-session — most often a second process, or a compute buffer that grew with a longer prompt.
Not every runtime works this way. Servers built for concurrency allocate the cache in pages and hand them out as sequences need them, precisely so that many short conversations do not each reserve a full window. On those, memory really does grow with use, and the failure mode is eviction or a rejected request rather than truncation.
What happens at the ceiling
When the session reaches the configured context, the runtime does one of three things, and which one decides whether you notice:
- It refuses. The server returns an error saying the requested tokens exceed the context size. This is the honest behaviour and the easiest to handle — see the local context-length-exceeded error.
- It shifts. The oldest tokens are discarded and the remaining cache entries are re-indexed so generation continues. The conversation keeps working and the model quietly loses its earliest instructions, including, frequently, the system prompt.
- The client truncates. Some front-ends drop old messages before sending, which is the same loss one layer up, and they rarely say so.
All three present as “the model stopped following the system prompt after a while”, which is why that complaint is usually a memory-budget symptom rather than a model quality one.
Context shift deserves particular suspicion because it is the one that keeps working. A refusal is a bug report; a shift is a model that continues to answer, fluently, having quietly forgotten its instructions and the first half of the document it was asked about. There is nothing in the output that says so. If a long session starts producing answers that contradict a constraint you set at the top, check whether the total token count has passed your -c value before you conclude anything about the model. Setting the context explicitly and choosing refusal over shift, where the runtime lets you, converts a silent quality failure into a visible one — which is almost always the better trade.
When the conversation outweighs the model
Set the cache term equal to the weights term and solve for context. Weights at Q4_K_M, using the 4.8944 bits per weight published in llama.cpp’s quantize README, against an fp16 cache:
C = weights_bytes / kv_bytes_per_token
Llama 3.1 8B 4.58 GiB / 128 KiB = 37,482 tokens
Yi-34B 19.59 GiB / 240 KiB = 85,545 tokens
Llama 3.1 70B 40.20 GiB / 320 KiB = 131,687 tokens
Llama 2 13B 7.42 GiB / 800 KiB = 9,720 tokens
Above those lines, more than half of what your card is holding is conversation. It also tells you which lever to pull: below the crossover, a smaller quant frees more memory; above it, halving the cache with --cache-type-k q8_0 --cache-type-v q8_0 frees more, and costs less in quality because the cache stores activations rather than learned structure. The budgets for specific cards are worked through on 8 GiB, 12 GiB and 24 GiB.
A local assistant with a 16,000-token window is fine until somebody pastes a contract. The two honest options are to summarise and lose detail, or to send that one request somewhere with room for it — which means one call site handling a local runtime and a hosted model with different limits and different truncation behaviour. A gateway is a reasonable place to put the length check and the fallback, so the overflow case does not become application code in three places.
Top comments (0)