Gemma 4 ships with a context window up to 256K tokens. Every time a model lands with a headline context number, the same thing happens: people paste their entire corpus into the prompt, watch latency and memory explode, get mediocre answers, and conclude long context "doesn't work."
The window is a ceiling, not a target. Here is the mechanical reason why, and what I do instead.
Where the cost lives: attention vs. KV cache
Two different costs scale with sequence length, and people conflate them constantly.
Attention compute is the O(n²) one everybody quotes. For a sequence of length n, full self-attention compares every token to every other token. Double the input, quadruple the attention work. This dominates prefill — the pass where the model ingests your prompt.
KV cache memory is O(n) and it is the one that actually kills you in production. Every token you process leaves behind key and value tensors that must stay resident for the rest of generation. Rough shape:
kv_bytes ≈ 2 (K and V)
× n_layers
× n_kv_heads × head_dim
× seq_len
× bytes_per_element
Note what is not in there: batch size, which multiplies the whole thing. Eight concurrent 200K-token requests means eight of those allocations live simultaneously. This is why a model that runs fine at 8K falls over at 128K with three users — you did not run out of weights memory, you ran out of cache.
Gemma 4's architecture is a direct response to this. Both workstation-class models interleave local sliding-window attention with periodic full-global attention layers, with the final layer always global. The consequence is worth internalizing: only the global layers pay full O(n²) and hold a full-length KV cache. The sliding-window layers attend within a bounded window, so their cache is capped at window size regardless of how long your input is.
So the cost curve is not the naive one. If one layer in every k is global, your per-layer cache length is:
def kv_len_per_layer(layer_idx, seq_len, window, global_every):
if (layer_idx + 1) % global_every == 0:
return seq_len # unbounded, grows with input
return min(seq_len, window) # capped, flat after the window
This is genuinely good engineering and it is why 256K is feasible at all. It is not a license to use 256K.
Why long context degrades before it OOMs
Even with the memory solved, quality does not stay flat as you fill the window. Two well-documented effects:
Positional degradation. Retrieval accuracy is not uniform across the window. Information at the very start and very end of a long prompt is recovered reliably; material buried in the middle is recovered noticeably worse. This is the "lost in the middle" behavior observed across essentially every long-context model family — a property of how attention distributes mass, not a bug in any one model.
Distractor dilution. Adding irrelevant-but-plausible text alongside the correct passage measurably degrades answers. If your true evidence is 2K tokens and you pad it to 180K with semantically adjacent noise, you gave the model more places to be wrong.
Then there is latency. Prefill on a very long prompt is a large one-time compute cost that lands entirely before the first token appears, and users experience it as the app hanging. Decode speed after prefill is barely affected — a genuinely confusing profile if you have not seen it. It feels fast once it starts. It just takes forever to start.
What I actually do
Retrieve first, then use the big window as slack. The point of 256K is not to skip retrieval — it is that your retriever no longer has to be surgical. Under a 4K budget you needed the top-3 chunks and precision mattered enormously. Now you can pull top-40, include full surrounding sections instead of severed 512-token fragments, and stop building elaborate re-rankers. Long context buys you slack in retrieval precision, not permission to skip retrieval.
Chunk on structure, not character count. Fixed-size chunking splits mid-argument and mid-table. Split on headings, sections, and function boundaries, and carry the heading path into each chunk so a retrieved fragment knows where it came from.
Put instructions last. Given positional degradation, the layout that has consistently worked for me:
[system / role]
[long retrieved context] <- the bulk, in the middle
[the actual question]
[explicit output instructions] <- last, right before generation
Instructions at the top of a 100K-token prompt are competing with 100K tokens of recency. Put them where the model is looking.
Cache the stable prefix. If many requests share a large fixed preamble — a schema, a policy doc, a codebase index — order the prompt so that block is a byte-identical prefix and reuse its KV cache. This is often a bigger latency win than any model swap. It also means: never put a timestamp or request ID at the top of your prompt. One varying token at position 0 invalidates the entire prefix.
Budget explicitly and measure at your real length.
BUDGET = 32_000 # not 256_000
def build_prompt(question, chunks, budget=BUDGET):
reserve = 2_000 # instructions + room to generate
used, kept = 0, []
for c in chunks: # ranked, best first
if used + c.tokens > budget - reserve: break
kept.append(c); used += c.tokens
return render(question, kept)
Then evaluate at the length you actually serve. A benchmark at 4K tells you nothing about behavior at 120K, and needle-in-a-haystack scores tell you nothing about multi-hop reasoning across scattered evidence. For the per-size architecture specs you need to do this math on your own hardware, the Gemma 4 model hub collects the family breakdown in one place.
My working default: start at 8–32K, raise the ceiling only when evals demonstrate the extra context earns its latency. Treat any prompt that fills more than half the window as a design smell — it usually means retrieval is doing nothing and you are paying for the model to do the filtering you should have done.
Top comments (1)
The line about long context buying slack in retrieval precision rather than permission to skip retrieval is the part most teams miss. One thing I would add from chunking work: when you widen top-k, the distractors are rarely random noise - they are near-duplicates of the right answer. Boilerplate headers, an older version of the same policy, the same table repeated across documents. Those hurt more than unrelated text because the model has no way to tell which copy is authoritative. Deduplicating near-identical chunks and keeping the newest version bought me more accuracy than a reranker did.