Every stack I've seen sells the context window as a model spec, and every fix for oversize is RAG, and what I measured on my own stack this month says the ceiling you actually feel lives in a config line. Mine was 131072. The model held retrieval exact at 509,084 tokens in one request.
The lie I found first
Pushing the local stack on my 3090 Windows box to a 524,288-token window, the agent kept reporting 131k, and the root cause was my Hermes configs advertising context_length 131072 while the engine underneath ran 262144. Engine, proxy, agent each declare their own number, the smallest one caps the rest, and nobody prints a warning when they disagree, so the gap between 131072 and 262144 is just quietly gone.
That afternoon found the same shape behind the classic "shows 131k" complaint, a stale middle-layer number half the size of the engine under it.
The config that made it real
--max-context 524288
--kv q4_0
--kv-resident 32768
--rope-scaling yarn
--rope-scale 2
--yarn-orig-ctx 262144
llama-swap advertised 524288, the Hermes configs got 524288 too, all three layers finally agreed on one number, restart through nssm brought the engine back in about 40 seconds, and /health printed max_context 524288 before I sent anything.
The run
509,084 prompt tokens in a single request, under the 524,288 cap, and a 1.76M-token test got correctly 400-rejected, so the cap was enforced at the engine. Prefill burned 381 seconds, 1,336 tokens per second. Two facts sat buried in that prompt, one near position 50, one near position 394,540, both retrieved exact, and the needle past 394k sits well past the trained 262k window, so yarn quality held across the scaled range on my model.
The honest cost is the prefill, 381 seconds before the first token, 6.4 minutes. Follow-ups on the same document hit checkpoint reuse and came back near-instant, the checkpoint-hit resend probe measured 0.12s, so you pay the prefill toll once per document and every follow-up rides the checkpoint.
The RAM night
Half a million tokens needed RAM, so I ran a RAM night, four 16GB G.Skill sticks in all four slots, 32GB to 64GB, 63.9 visible. DOCP at 3600 failed POST. Four dual-rank DIMMs under MC load took the box down to 3400, which POSTed and Windows verified every stick at 3400. The trap that cost an hour: BIOS "Auto" after a failed POST silently retrains to 2133, so when speeds look dead, check DOCP is still enabled before chasing anything else.
KV math made VRAM a non-issue at this scale, 262k tokens at q4_0 is 1.69GB, it fits resident and the streaming flags are pointless there.
Calibration I measured
3.11 characters per token on the synthetic prose, 32.2 tokens per ledger line, both measured on the actual document, so multiply the lines in your file and "will this fit" becomes arithmetic before you send.
The probe that ranked it
The TTFT probe I ran before and after, 10k tokens on a fresh prompt with thinking on: 7.62s on the 131k window with 32GB, 5.10s on the 262k window with 64GB. Doubling RAM took 7.62s down to 5.10s, and the bigger window still answered first-token faster than the smaller one did the day before.
What this says about RAG
The claim I'm backing with this: retrieval quality at half a million tokens held on my engine with rope scaling set right, and the failure that bit me was a stale config line half the size of the engine under it. RAG exists because people hit that config ceiling and read it as a model ceiling. The two numbers are cheap to check, /health prints max_context and your agent prints its advertised context_length, and if they disagree the small one wins in silence.
So the question for anyone running a local stack: what does your agent advertise, and what does your engine actually load, because this bug hides in the gap between the two, and mine was 131072 against 262144.
Top comments (0)