The model card said 262,144 tokens, and nothing else. No bytes per token, no VRAM
figure, nothing about the cache growing linearly while the weights stay put. I did
the arithmetic before touching a config, and the arithmetic ended the request for
500K on paper, before any GPU saw a change.
What the card gave me
The official Qwen3.8-27B config and the GGUF build I installed agree on 262,144
native tokens. My runtime was serving 65,536 with q8_0 K/V at the time. The ask
on the table was 500K per instance, on one RTX 3090, and neither the vendor page
nor my own notes answer whether that fits.
The math nobody publishes
Everything below comes from fields the model itself carries: 16 full-attention
layers, 4 KV heads, head dimension 256, Q8_0 blocks costing 34 bytes per 32
values. That works out to 16.21 GiB of KV cache for 500,000 tokens. The weights
are where cards price things, 18.19 GiB for the Q5_K_M file, so the pair wants
34.41 GiB before overhead, against 24 GB of card.
I never asked llama.cpp to try it, the number stayed on paper, and the gap runs over
10 GiB wide with no rounding closing it.
The native window misses too. Same weights, same quant, 262,144 tokens costs an
estimated 26.69 GiB, still more than one 3090. The advertised limit was always bigger
than the hardware advertising it, and neither document says so.
Where the runtime put it
--no-kv-offload is documented in llama.cpp as leaving the KV cache on host RAM,
which is the only way past 24 GB without a second card. Two staged entries set
engine and capability contexts to 262,144, changed nothing else, both passed
llama-swap --validate. Staging passing tells you the YAML shape is right, that's
all it tells you.
Activation ran under a lifecycle lock with a rollback backup, then the measured
side took over. /props and /slots report n_ctx=262144 on both lanes, one lane
showing four slots at that number. VRAM sat at 19,746 MiB, resident set came back
11,161,320 KiB on one process, 11,254,012 KiB on the other. The cold lane answered
HTTP 200 after 30.69 seconds, the warm one generated 121 completion tokens. Both
were short prompts.
The estimate said 8.5 GiB of KV per loaded instance, the RSS figures landed near
11 GiB, same order of magnitude. Nothing published that number, I built it myself.
The second data point
A smaller quant moves the trade. I pulled a Q4_K_M build, 16,551,316,384 bytes,
SHA-256 verified against Hugging Face before install, staged with q4_0 K/V pinned to
VRAM and host RAM out of the picture. Whether that allocation fits was untested at
staging time, so I applied it and let the card answer: 21,664 MiB and 21,668 MiB used,
decode at 42.26 tok/s and 43.72 tok/s from a short 27-token prompt, 51 output tokens,
thinking disabled on those probes. Two requests in one morning, way too thin to call
a benchmark.
How I picked that build matters too. A "fastest uncensored model" search sorted by
downloads returns the popular abliterated repo first at 3,157,401 downloads against
1,149,119 for the runner-up. Downloads count hands reaching for a file, they say
nothing about speed on any hardware, so that ranking answered a question I hadn't
asked, and the answer I wanted needed an install.
Rules I'm carrying out
Treat an advertised context window as a claim about attention semantics and price the
memory yourself by multiplying layer count times kv heads times head dim times bytes per
element times tokens, then compare that against the card before any config changes.
Label which numbers you measured and which you computed. The 34.41 GiB was arithmetic
on paper, the 19,746 MiB came back from nvidia-smi after activation, and a validation
pass that says valid proves only that the YAML has shape. Those three kinds of fact get
treated as one fact by most writeups, mine included, until you split them.
What I did not test
The full 262K prompt never ran. Long-context quality on either build is untested, the
8.5 GiB KV estimate was never compared byte-for-byte against a real allocation, and the
Q4 speed numbers are two samples deep. What held up under measurement: the memory math,
the config landing at 262,144 on both lanes, and the VRAM plus RSS figures after
activation.
Does your engine document the cost of its context window anywhere, a bytes-per-token
line or a KV table? Mine documented nothing until I wrote down the multiplication myself,
and if some vendor page publishes that figure and I have not found it, put the link in the
comments, because half of what this piece argues would change overnight.
Top comments (1)
The measured-vs-computed split is the right discipline, and I'd add a bucket the VRAM reading quietly straddles: reserved vs resident. llama.cpp pins the KV allocation for n_ctx x slots at load, so the 19,746 MiB from nvidia-smi is almost entirely reservation - at a 27-token prompt the actual KV occupancy is kilobytes. That reading proves the allocation fits the card, not that a full 262K window does; the two only converge once a prompt actually fills the slots, which is the run you didn't make.
Which is also where --no-kv-offload hides its bill. Moving KV to host RAM gets you past 24 GB on the reservation, but the cost of that choice is paid per token of real attention, as PCIe traffic on every decode step - and a 27-token prompt never touches it. So the fit question (does it allocate?) and the speed question (does it stay fast at depth?) have different answers, and short-prompt probes can only see the first. The offload penalty lives entirely in the long-occupancy regime your "what I did not test" section is bounded by.
If you do run the full window, logging decode tok/s at something like 1K / 16K / 128K occupancy would separate the allocation cost (one-time, at load) from the attention cost (per-token, growing) - the second is the one offload actually taxes, and it's invisible until the KV is deep enough to spill across the bus.