It’s been 3 weeks since the first article of this series.
A lot has changed for me.
In the first article there was a lot of excitement, because I found evidence, data and something new.
The whole thing seemed a lucky coincidence.
Over time I had the chance to revise my way to identify that situation - not only that.
How do you prevent - or at least recognize - when a model will degrade for a given context?
My answer: there's no rule, no magic formula - it has to be grounded in data. You cannot predict it. **It’s a sum of different
signals you read from your hardware.**
You start to see shared memory going up.
The ram usage growing also. Generation takes a heavy hit - at least 50%. These numbers are measured across real runs, over and over with different models and contexts.
In calibr, I’ve defined how you can tackle the issue - the strategy differs from the one provided here, since the one below is the logic I’ve used to narrow down the cliff token range.
First you run a baseline with llama.cpp.
You get the numbers:
- Check for VRAM shared memory:
- shared memory usage = shared memory peak - shared memory baseline. where peak is the highest value you save during the model run and, baseline is VRAM usage before starting the run.
- Check RAM delta:
- ram delta usage = ram peak usage - ram baseline. this is important because it shows as the shared memory grows, it’s going into the ram.
- Check eval tok/sec
- A baseline run shows what the model's eval speed can be - say, 17 tok/sec.
- A long run - deep prefill, or filling the KV cache directly - shows how it evolves over time: the same model might drop to 2 tok/sec (real numbers again: Gemma 4 12B, at a deep prefill).
These points show you why a shallow check, might be a false-positive result.
To be clear: test only the baseline and you get happy results with high evals. Dig deeper with prefill and KV fill, and you get the real picture.
There's no way to prevent this. There are workarounds to squeeze more into memory, but they usually cost precision or eval speed.
The blanket is a fixed size - you can't stretch it. Pull it up to your shoulders and your feet stick out.
You can quantize the KV cache from f16 (llama.cpp's default) down to q8_0: it halves the cache footprint with near-lossless quality - PPL within run-to-run noise, mean precision above 99.8% against the bf16 baseline; the only measurable dip is in the worst 0.1% of token positions source.
This might fit more into VRAM, but it doesn't solve the issue.
To identify the cliff precisely, you can approach it this way:
- use the numbers from VRAM shared memory and RAM delta.
- setup at least 3 runs, with different context - for instance 16k, 32k, 65k. Two points give you a slope; a third is what tells you the slope is real and not just noise from that one run. The delta between consecutive runs tells you how much VRAM roughly 1,000 tokens of context allocate.
From here, you extrapolate: take your last two clean points, keep that slope, and project forward until you hit your card's total VRAM.
model total load in vram + ( * / 1024 ) = vram total usage theoretical
Now you can see your hypothesis getting shaped and run some more numbers:
- baseline vram usage
- system vram reserved
- theorical total vram
- avalable vram = theorical total vram - baseline vram usage - system vram reserved
- model total load in vram + ( * / 1024 ) = vram total usage theorical
Does it seems overkill?
Think about the architectures of LLMs.
We have now hybrid models, which don’t scale with a simple formula memory usage, so you cannot make up a different formula for each different model.
Why not just use a formula to compute it?
Because a formula doesn't account for:
- buffer allocation
- real available vram
- real reserved vram
- shared memory vram
- all things that only show up when you actually run the model on your PC.
Different architecture implies different attention strategies and layers, which don’t grow in the same way of a full attention or supposedly predictable architecture.
Top comments (0)