A chat server runs clean for three hours. Then, at the exact turn where the rolling KV cache first exceeds its 4096-token window and evicts the oldest block, the model stops producing English. Not degraded output — broken output. Repeated tokens, random punctuation, perplexity in the thousands. Nothing else changed: same weights, same sampler, same prompt template.
The cause is attention sinks. Your eviction policy threw away the first few tokens of the sequence, and those tokens were not carrying information — they were carrying the softmax normalizer that every other head depends on.
TL;DR
- Attention sinks are the first few tokens of a sequence (usually token 0, often the BOS token) that absorb a large share of attention probability mass in most heads and layers, despite being semantically empty.
- They exist because softmax must sum to 1. A head that wants to attend to nothing still has to put its mass somewhere, so training pushes it onto a token every query can see under causal masking — position 0.
- Evict those tokens from a sliding-window KV cache and the mass gets redistributed onto real tokens, shifting every value vector. Perplexity explodes by orders of magnitude within a few tokens.
- The fix is StreamingLLM's: keep the first 4 tokens pinned forever, roll the window over everything else, and assign RoPE positions by cache slot, not by original token index.
- If you use a hosted API (Claude Opus 4.x, GPT-5.x), you never touch a KV cache — but the same logic bites you when you hand-roll conversation truncation and drop the prefix your prompt cache and your model both depend on.
What is an attention sink in a transformer?
An attention sink is a token position that consistently receives disproportionate attention weight across heads and layers while contributing almost nothing semantically. Instrument any decoder-only model — Llama, Mistral, Qwen — and print the attention distribution for a mid-depth layer. Past layer 2 or so, a large fraction of heads put most of their probability on position 0, regardless of the query. In many heads the first token takes more mass than the entire rest of the context.
This is not a bug in the checkpoint. It is a structural consequence of the architecture, and it shows up in every model trained with causal masking and standard softmax attention.
The companion finding is massive activations: the residual-stream hidden state at those sink positions contains a handful of feature dimensions with magnitudes hundreds to thousands of times larger than typical. Those dimensions are nearly constant across inputs. The model is using the sink token as a fixed bias vector — a place to park attention and a place to store a learned constant.
Why does softmax force the model to create attention sinks?
Because softmax has no "none of the above" option.
attn_weights = softmax(QK^T / sqrt(d) + causal_mask) # rows sum to exactly 1
out = attn_weights @ V
Consider a head that has already found what it needs — say, an induction head with no matching prefix in this context. The correct output is "add nothing to the residual stream." There is no way to express that. The row must sum to 1, so something gets multiplied by its value vector and written into the residual.
The trick the model learns: designate a token whose value vector is near-zero (or whose contribution is a constant the rest of the network compensates for), give it a large key-query alignment, and dump the surplus mass there. Attending to it is a no-op.
Which token? It has to be visible to every query. Under a causal mask, exactly one position satisfies that for every sequence length: position 0. Positions 1–3 usually get recruited too, since early tokens are visible to nearly everything and carry little content.
This is also why the "softmax off by one" proposal (adding a +1 to the denominator so rows can sum to less than 1) exists, and why some recent open-weight models — OpenAI's gpt-oss family among them — ship a learned per-head sink logit appended to the attention logits. That gives every head a real null option, so it does not need to hijack a token to get one. Models trained that way are far more tolerant of window eviction, because the sink is a parameter rather than a cache entry.
What actually breaks when you evict token 0?
The moment position 0 leaves the KV cache, every head that was dumping 40–80% of its mass there has to renormalize over the remaining keys. That surplus does not vanish — softmax redistributes it proportionally onto the surviving tokens.
So a head that was effectively writing nothing now writes a large weighted average of whatever happens to be in the window. Every value vector in that layer shifts. The perturbation compounds through the remaining layers, and the residual stream leaves the distribution the LM head was trained on.
The failure is abrupt, not gradual. StreamingLLM (Xiao et al., 2023) demonstrated the clean version of this: dense sliding-window attention that evicts the oldest tokens shows stable perplexity right up to the eviction threshold, then jumps by orders of magnitude within a handful of tokens. Re-adding just four initial tokens restores stable perplexity over millions of tokens of streaming input.
Four is the number to remember. Not one — BOS alone is usually not enough, because positions 1–3 typically carry sink duty too.
How do you fix sliding-window KV eviction?
Pin the sinks, roll everything else. The subtlety is positional encoding: with RoPE, cached keys are already rotated by their original absolute position. If you evict a middle block and leave the survivors rotated at their original indices, you have punched a hole in the position sequence — the model sees positions [0,1,2,3, 5000,5001,...] with a 5000-slot gap it never saw in training.
RoPE is relative, so the correct fix is to encode by position within the cache, not position within the stream. Two ways to do it:
import torch
class SinkWindowCache:
"""Keep n_sink initial tokens forever; roll a window over the rest.
Stores UNROTATED keys so RoPE can be applied by cache slot at read time.
"""
def __init__(self, n_sink: int = 4, window: int = 1020):
self.n_sink, self.window = n_sink, window
self.k, self.v = None, None # [B, H, T, D], keys pre-rotation
def update(self, k_new, v_new):
self.k = k_new if self.k is None else torch.cat([self.k, k_new], dim=2)
self.v = v_new if self.v is None else torch.cat([self.v, v_new], dim=2)
budget = self.n_sink + self.window
if self.k.shape[2] > budget:
s = self.n_sink
self.k = torch.cat([self.k[:, :, :s], self.k[:, :, -self.window:]], dim=2)
self.v = torch.cat([self.v[:, :, :s], self.v[:, :, -self.window:]], dim=2)
# positions are cache slots, NOT original token indices
pos = torch.arange(self.k.shape[2], device=self.k.device)
return apply_rope(self.k, pos), self.v
The alternative, used by llama.cpp's context shift, is to keep keys rotated and apply a delta rotation to the survivors when you discard a block. Because RoPE rotation composes, multiplying cached keys by the rotation matrix for -Δ is exactly equivalent to having encoded them at the shifted position. That is what llama_kv_cache_seq_add does, and it is why --keep N exists on the CLI:
# keep BOS + first 4 tokens pinned, shift the rest down on overflow
./llama-server -m model.gguf -c 8192 --keep 4
What you must not do is the naive version — slice the rotated key tensor and carry on. It is two lines of code, it runs, it produces tokens, and it is wrong in a way that only shows up after the window fills.
Also note the ordering constraint: sinks must stay at the front of the cache. Some hand-rolled implementations concatenate [window, sinks] because it is easier to index. Under causal attention the relative offsets are then inverted, and you get a different flavor of the same corruption.
Why don't production inference servers hit this?
Because they refuse to evict. vLLM under memory pressure preempts a sequence and recomputes its prefix rather than dropping keys from a full-attention model. Sliding-window attention is only applied when the model config declares it — Mistral 7B's 4096-token window, Gemma's alternating local/global layers — because those models were trained with that mask and learned their sinks inside the window.
That distinction is the whole thing: window eviction is safe if and only if the model was trained under the same mask. Applying a sliding window at inference to a model trained with full attention is a train/test mismatch on the attention denominator.
Learned-eviction schemes (H2O, SnapKV, and their descendants) all rediscover this empirically. Their "heavy hitter" sets always include the first tokens, because those tokens have the highest accumulated attention score by a wide margin. If you implement a KV compression policy and don't hard-pin the first few slots, your policy will spend its budget rediscovering them anyway — or fail catastrophically the one time it doesn't.
Do attention sinks matter if I only call Claude or GPT-5?
Directly, no — you never manage a KV cache through the Anthropic or OpenAI API, and the serving stack handles this correctly. Indirectly, yes, in two places.
First, conversation truncation. When your agent loop trims history to fit a context budget, the safe move is the same shape as the sink fix: pin the prefix (system prompt, tool definitions, the first few turns), and drop from the middle or the tail. Sliding a window that eats the head of the conversation is the semantic analogue of evicting token 0, and it also invalidates every prompt-cache prefix downstream of the cut.
Second, quantization. Massive activations live at sink positions. Per-tensor activation quantization computes a scale over a tensor containing outliers hundreds of times the median, which crushes everything else to a handful of representable levels. Any calibration set that under-samples sink positions produces scales that break the model at exactly those tokens. Per-channel or per-token scales exist largely because of this.
How do you detect attention sink loss in production?
Log two things. First, at generation time, the fraction of attention mass on the first four cache slots for a mid-depth layer — run with output_attentions=True on a canary request and check it is not near zero. Second, rolling mean token logprob. Sink loss shows up as a step function in the logprob trace within 5–20 tokens of the eviction event, not a slow drift. If your degradation curve looks like a cliff, correlate its timestamp with the moment your cache crossed its budget.
The direct answer
Sliding-window KV eviction breaks your LLM because the first few tokens of a sequence are attention sinks: softmax rows must sum to 1, so heads that want to output nothing dump their surplus probability mass onto the one position every causal query can see. Evicting those tokens forces that mass onto real content, perturbing every value vector in every layer at once, and perplexity jumps by orders of magnitude the instant the window rolls past the start. Fix it by pinning the first four tokens permanently, rolling the window over the rest, and assigning RoPE positions by cache slot (or applying a delta rotation to survivors) so the model never sees a gap in its position sequence. If the model was trained with a sliding window or ships learned sink logits, this is already handled — otherwise, do not evict.
Top comments (0)