Your 256K Context Window Is a Ceiling, Not a Budget

ethanlin — Sat, 25 Jul 2026 08:07:59 +0000

Gemma 4 ships with a context window up to 256K tokens. Every time a model lands with a headline context number, the same thing happens: people paste their entire corpus into the prompt, watch latency and memory explode, get mediocre answers, and conclude long context "doesn't work."

The window is a ceiling, not a target. Here is the mechanical reason why, and what I do instead.

Where the cost lives: attention vs. KV cache

Two different costs scale with sequence length, and people conflate them constantly.

Attention compute is the O(n²) one everybody quotes. For a sequence of length n, full self-attention compares every token to every other token. Double the input, quadruple the attention work. This dominates prefill — the pass where the model ingests your prompt.

KV cache memory is O(n) and it is the one that actually kills you in production. Every token you process leaves behind key and value tensors that must stay resident for the rest of generation. Rough shape:

kv_bytes ≈ 2 (K and V)
         × n_layers
         × n_kv_heads × head_dim
         × seq_len
         × bytes_per_element

Note what is not in there: batch size, which multiplies the whole thing. Eight concurrent 200K-token requests means eight of those allocations live simultaneously. This is why a model that runs fine at 8K falls over at 128K with three users — you did not run out of weights memory, you ran out of cache.

Gemma 4's architecture is a direct response to this. Both workstation-class models interleave local sliding-window attention with periodic full-global attention layers, with the final layer always global. The consequence is worth internalizing: only the global layers pay full O(n²) and hold a full-length KV cache. The sliding-window layers attend within a bounded window, so their cache is capped at window size regardless of how long your input is.

So the cost curve is not the naive one. If one layer in every k is global, your per-layer cache length is:

def kv_len_per_layer(layer_idx, seq_len, window, global_every):
    if (layer_idx + 1) % global_every == 0:
        return seq_len              # unbounded, grows with input
    return min(seq_len, window)     # capped, flat after the window

This is genuinely good engineering and it is why 256K is feasible at all. It is not a license to use 256K.

Why long context degrades before it OOMs

Even with the memory solved, quality does not stay flat as you fill the window. Two well-documented effects:

Positional degradation. Retrieval accuracy is not uniform across the window. Information at the very start and very end of a long prompt is recovered reliably; material buried in the middle is recovered noticeably worse. This is the "lost in the middle" behavior observed across essentially every long-context model family — a property of how attention distributes mass, not a bug in any one model.

Distractor dilution. Adding irrelevant-but-plausible text alongside the correct passage measurably degrades answers. If your true evidence is 2K tokens and you pad it to 180K with semantically adjacent noise, you gave the model more places to be wrong.

Then there is latency. Prefill on a very long prompt is a large one-time compute cost that lands entirely before the first token appears, and users experience it as the app hanging. Decode speed after prefill is barely affected — a genuinely confusing profile if you have not seen it. It feels fast once it starts. It just takes forever to start.

What I actually do

Retrieve first, then use the big window as slack. The point of 256K is not to skip retrieval — it is that your retriever no longer has to be surgical. Under a 4K budget you needed the top-3 chunks and precision mattered enormously. Now you can pull top-40, include full surrounding sections instead of severed 512-token fragments, and stop building elaborate re-rankers. Long context buys you slack in retrieval precision, not permission to skip retrieval.

Chunk on structure, not character count. Fixed-size chunking splits mid-argument and mid-table. Split on headings, sections, and function boundaries, and carry the heading path into each chunk so a retrieved fragment knows where it came from.

Put instructions last. Given positional degradation, the layout that has consistently worked for me:

[system / role]
[long retrieved context]      <- the bulk, in the middle
[the actual question]
[explicit output instructions]  <- last, right before generation

Instructions at the top of a 100K-token prompt are competing with 100K tokens of recency. Put them where the model is looking.

Cache the stable prefix. If many requests share a large fixed preamble — a schema, a policy doc, a codebase index — order the prompt so that block is a byte-identical prefix and reuse its KV cache. This is often a bigger latency win than any model swap. It also means: never put a timestamp or request ID at the top of your prompt. One varying token at position 0 invalidates the entire prefix.

Budget explicitly and measure at your real length.

BUDGET = 32_000   # not 256_000

def build_prompt(question, chunks, budget=BUDGET):
    reserve = 2_000            # instructions + room to generate
    used, kept = 0, []
    for c in chunks:           # ranked, best first
        if used + c.tokens > budget - reserve: break
        kept.append(c); used += c.tokens
    return render(question, kept)

Then evaluate at the length you actually serve. A benchmark at 4K tells you nothing about behavior at 120K, and needle-in-a-haystack scores tell you nothing about multi-hop reasoning across scattered evidence. For the per-size architecture specs you need to do this math on your own hardware, the Gemma 4 model hub collects the family breakdown in one place.

My working default: start at 8–32K, raise the ceiling only when evals demonstrate the extra context earns its latency. Treat any prompt that fills more than half the window as a design smell — it usually means retrieval is doing nothing and you are paying for the model to do the filtering you should have done.

Picking a Gemma 4 Quantization: VRAM Math That Actually Matters

ethanlin — Fri, 24 Jul 2026 19:00:12 +0000

Every "run this model locally" guide tells you to grab a Q4 GGUF and move on. That advice is fine right up until you try a long-context run and your machine starts swapping.

The weights are the part everyone budgets for

Quantization maths is straightforward. A model's weight footprint is roughly params x bits / 8:

Quant	Bits/param	12B model	Quality note
Q8_0	~8.5	~12.8 GB	Near-lossless, rarely worth it
Q6_K	~6.6	~9.9 GB	Very close to Q8
Q4_K_M	~4.8	~7.2 GB	The usual sweet spot
Q3_K_M	~3.9	~5.9 GB	Noticeable degradation

Below Q4 the loss stops being subtle. Instruction-following degrades before raw perplexity does, which is why benchmark numbers can look fine while the model quietly stops respecting your system prompt.

The KV cache is the part that bites

Here is what the guides skip. The KV cache scales with context length, and it is not quantized by default:

kv_bytes ~= 2 (K and V) x layers x kv_heads x head_dim x seq_len x dtype_bytes

The practical consequence: a model that loads in 7 GB can need well over twice that at long context. Grouped-query attention helps a lot — kv_heads is much smaller than attention heads — but the term still grows linearly with sequence length while your weights stay fixed.

Two knobs matter more than picking a fancier quant:

--ctx-size: do not allocate 128K if your prompts are 8K. You are reserving memory you will never touch.
KV cache quantization (q8_0 for K/V): roughly halves cache memory for a quality hit most workloads never notice. Underused.

A decision order that works

Start at Q4_K_M
Set context to what you actually use, not the model maximum
If you are still tight, quantize the KV cache before dropping to Q3
Only move up to Q6/Q8 if you have headroom left over

That ordering matters: dropping to Q3 to buy context is the most common mistake, and it trades a permanent quality loss for memory you could have gotten from the cache instead.

Per-quantization benchmarks and deployment notes for the Gemma 4 family are collected at gemma-4.net — every number self-run, raw data published.

Caveat

These are rules of thumb, not guarantees. Backends differ in how they allocate, and Apple Silicon unified memory behaves differently from discrete VRAM. Measure on your own hardware before trusting any table, including mine.

DEV Community: ethanlin