Attention Sinks: Why Evicting Token 0 Wrecks Sliding-Window KV

You ship a KV cache eviction policy: keep the last 4k tokens, drop the rest. Perplexity on your 3k-token eval set is unchanged. You roll it out, and long sessions start producing fluent nonsense — grammatical, on-topic-ish, completely detached from the conversation. The degradation begins exactly when the sliding window passes the start of the sequence.

You didn't hit a context-length limit. You evicted the model's attention sinks, and the softmax had nowhere to put the mass it is mathematically obligated to spend.

TL;DR

Softmax attention weights must sum to 1, so a head with nothing relevant to read still has to distribute full attention mass somewhere. Trained transformers dump that mass on the first few tokens, whose value vectors have near-zero norm — a learned no-op.
Drop those tokens from the KV cache and the mass gets redistributed onto real tokens with real value vectors, injecting spurious content into the residual stream. Perplexity blows up within a few hundred tokens.
Keeping just 4 initial tokens plus a rolling window restores stable streaming quality (StreamingLLM, Xiao et al. 2023). The fix is cheap; the failure is catastrophic.
Position IDs must be assigned by cache index, not original text index. Get this wrong and RoPE sees a hole where the evicted tokens were.
Sink tokens carry massive activations, so they also break per-tensor KV quantization, H2O-style importance eviction, and any agent loop that trims the top of its own transcript.

What are attention sinks in a transformer?

An attention sink is a token position that soaks up a large fraction of attention probability across many heads and layers, regardless of what the query is asking about. In practice it's the first token of the sequence — BOS if you use one, or whatever token happens to be at index 0 — plus the two or three positions after it.

This is not a bug in training. It's the only way a softmax head can express "nothing here is relevant."

Consider one head, one query. Scores s_i = q·k_i / sqrt(d), weights a = softmax(s), output o = Σ a_i v_i. There is no a_i = 0 for all i state. The head must spend its full unit of probability mass. If the honest answer for this query is "skip this operation," the head needs a target whose value vector is approximately zero — a token it can attend to hard while contributing nothing to the residual stream.

Trained models converge on the same solution: make v_0 ≈ 0, make k_0 easy to score highly, and route all idle mass there. Measure it and you'll see heads in mid-to-late layers putting the majority of their attention on the first handful of positions on most queries.

Why does dropping token 0 break sliding-window attention?

Because the remaining weights get rescaled by 1 / (1 - a_sink), and everything left in the window has a non-trivial value vector.

Say a head puts 0.80 of its mass on the sink and spreads 0.20 over 4,000 real tokens. Evict the sink and renormalize: every real token's weight is multiplied by 5x. The head that was supposed to be a no-op now writes a 5x-amplified average of whatever happens to sit in the window into the residual stream.

That output feeds the next layer's queries and keys. The corruption compounds depth-wise, then step-wise as the poisoned states get cached. The output stays locally fluent — the language modeling head still produces well-formed tokens — while the actual retrieval and instruction-following signal drowns.

The tell is that quality collapses at a position, not at a length. A 40k-token session with a 4k window fails; a 4k-token session with a 4k window is fine. Same model, same window, same prompt style.

Why do sinks land on the first tokens specifically?

Causal masking. Position 0 is the only position visible to every subsequent query in every layer. If the model needs a globally reachable dump site, that's the only candidate that always exists.

Two consequences follow:

Sinks are positional, not semantic. Cut the first sentence off your prompt and the new first token becomes the sink. The model doesn't care that it's "The" instead of <|begin_of_text|> — but the KV entry it learned to lean on is gone, and a fresh prefill has to rebuild the role from a token that wasn't trained for it. Fresh prefill mostly recovers; mid-stream eviction does not, because the surviving cache entries were computed against the original sink.
Sinks carry massive activations. The hidden states at sink positions have norms orders of magnitude above the rest of the sequence. This is the same phenomenon that forces outlier handling in activation quantization, and it means a sink token is the single most expensive thing in your cache to quantize naively.

How do you evict KV cache without losing the sink?

Reserve the first N entries permanently, roll the rest, and re-derive positions from cache slot index. N = 4 is the standard choice.

class SinkKVCache:
    """Rolling KV cache that pins the first `n_sink` tokens.

    Positions are re-derived from cache slot index, not the original
    text index. RoPE must see a contiguous 0..len-1 run, otherwise the
    evicted span shows up as a gap in relative distance.
    """

    def __init__(self, n_sink=4, window=4096):
        self.n_sink = n_sink
        self.window = window
        self.k = None   # [B, H_kv, S, D]
        self.v = None

    def append(self, k_new, v_new):
        self.k = k_new if self.k is None else torch.cat([self.k, k_new], dim=2)
        self.v = v_new if self.v is None else torch.cat([self.v, v_new], dim=2)

        cap = self.n_sink + self.window
        s = self.k.shape[2]
        if s > cap:
            keep_tail = cap - self.n_sink
            self.k = torch.cat([self.k[:, :, :self.n_sink], self.k[:, :, -keep_tail:]], dim=2)
            self.v = torch.cat([self.v[:, :, :self.n_sink], self.v[:, :, -keep_tail:]], dim=2)
        return self.k, self.v

    def query_position(self):
        # The next query's position = current cache length, NOT the
        # number of tokens actually generated so far.
        return self.k.shape[2]

The query_position detail is the part people get wrong. If you evicted 30k tokens and keep using absolute text positions, RoPE rotates the incoming query as if it were 34k positions away from the sink while the surviving neighbors sit at 30k+. Every relative distance in the window is now wrong, and you're extrapolating far past the trained range for no reason.

Two other requirements that bite in real serving code:

Cache keys with rotation already applied are not relocatable. If you store post-RoPE keys (most implementations do), you cannot re-index them after eviction without un-rotating. Either store pre-rotation keys for the tail, or accept that only contiguous suffix eviction is safe. Pinning a prefix plus a contiguous suffix satisfies this; arbitrary token-level eviction does not.
Paged/block allocators need a pinned block. In a block-based KV allocator, the sink lives in block 0. Make it non-evictable explicitly, or your LRU policy will reclaim it precisely because nothing "recently wrote" to it.

How do I check whether my model has attention sinks?

Read the attention weights directly. One forward pass with output_attentions=True is enough to see the structure.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "meta-llama/Llama-3.1-8B"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(
    name, torch_dtype=torch.bfloat16, device_map="auto",
    attn_implementation="eager",  # required to get attention probs back
)

ids = tok("The mitochondrion is the powerhouse of the cell. " * 40,
          return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model(**ids, output_attentions=True)

for layer_idx in (0, 8, 16, 24, 31):
    attn = out.attentions[layer_idx][0]       # [H, S, S]
    last_q = attn[:, -1, :]                   # final query row per head
    sink_mass = last_q[:, :4].sum(-1)         # mass on first 4 positions
    print(f"L{layer_idx:2d}  median sink mass {sink_mass.median():.3f}  "
          f"max {sink_mass.max():.3f}  heads>0.5: {(sink_mass > 0.5).sum().item()}")

Layer 0 is usually clean. From roughly the second layer onward you'll see a large share of heads with most of their final-row mass parked on the first four positions, on a prompt with zero informational reason to look there. That's your sink budget. Also print the value-vector norms at those positions — they'll be conspicuously small relative to the sequence median, which is the "no-op" half of the mechanism.

Where else do attention sinks bite?

Importance-based eviction. H2O-style "heavy hitter" policies score tokens by accumulated attention and usually retain the sink for free, since it's the heaviest hitter by construction. Policies that score by recency, semantic salience, or embedding similarity will happily throw it away. If you built a custom eviction heuristic, check that the sink survives it.
KV cache quantization. Sink keys and values sit far outside the distribution of the rest of the cache. Per-tensor scales get dragged toward the outlier and everything else loses precision. Keep the pinned sink entries in fp16/bf16 — it's 4 tokens, the memory cost rounds to zero.
Trained-in sliding windows are different. Models trained with a rolling buffer from the start (Mistral-style sliding window attention, or interleaved local/global layer stacks) learned to operate without a permanently visible token 0 in their local layers. Retrofitting a window onto a model trained with full attention is where this fails. Don't assume the architecture rescues you; check the training config.
Learned sink logits. Several recent open-weight models, including OpenAI's gpt-oss family, add a learned per-head scalar to the softmax denominator — an explicit "attend to nothing" slot that costs no KV entry. If your model has it, the head has a real no-op and the positional-sink pressure drops. Make sure your inference stack actually implements that extra denominator term; silently ignoring it renormalizes every attention distribution in the model.
Hosted APIs are not exposed to this. With Claude Opus 4.x or GPT-5.x you don't own the KV cache, and every request re-prefills whatever prefix you send, so a token 0 always exists. The analogous cost when you trim the top of an agent transcript is prefix-cache invalidation and lost instructions, not sink loss. Different problem, different fix — don't apply this article's reasoning to your API context-window trimmer.

So why does evicting token 0 wreck sliding-window attention?

Because softmax attention weights are forced to sum to 1, and trained transformers solve the "nothing here is relevant" case by dumping that mandatory mass onto the first few tokens, whose value vectors are near zero. Those positions are attention sinks: high attention, no content. Evict them mid-stream and the same mass is redistributed onto real tokens with real value vectors — scaled up by 1/(1 - a_sink), often a 5x amplification — which writes garbage into the residual stream and compounds across layers and decode steps. Pin the first 4 KV entries, roll the rest, assign positions by cache slot index rather than original text index, and keep those pinned entries out of your quantizer. The fix costs 4 tokens of cache; skipping it costs the whole session.