Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.
Give a Transformer a long stream of text and inspect its attention patterns. You may find that some of the earliest tokens receive a surprisingly large fraction of attention, even when they have little semantic relevance to the token being generated.
Why?
The answer turns out to be connected to something deceptively mundane: softmax has to put probability somewhere.
This odd behavior became operationally important in 2023, when researchers at MIT, Meta AI, CMU, and NVIDIA discovered that preserving a handful of these seemingly useless tokens could make a sliding-window LLM remain stable for millions of tokens.
That work, Efficient Streaming Language Models with Attention Sinks, became known as StreamingLLM. The paper reported stable language modeling out to 4 million tokens and up to a 22.2x speedup over a recomputation-based baseline.
For developers building inference systems, attention sinks are a beautiful example of an LLM implementation detail becoming an algorithmic primitive.
Let's work from intuition all the way down to the math and the GPU economics.
1. First: what exactly is an attention sink?
Recall the basic attention equation from the original Transformer paper:
Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V
For one query token, imagine that we have five previous tokens:
Token: A B C D E
score: 2.1 1.7 -0.4 0.2 1.1
Softmax converts these arbitrary scores into probabilities:
attention:
A = 0.40
B = 0.27
C = 0.03
D = 0.07
E = 0.23
The probabilities must add up to exactly 1.
Now imagine a query for which none of the previous tokens is particularly useful.
You might intuitively expect:
A = 0
B = 0
C = 0
D = 0
E = 0
But softmax cannot do this.
It has to distribute the probability mass somewhere.
A token that repeatedly becomes the recipient of this otherwise-unwanted attention is called an attention sink.
The important observation from Xiao et al. was that the first few tokens frequently become these sinks. They can receive substantial attention even when their semantic content is irrelevant to the current prediction.
Think of it as a drain in a plumbing system.
The drain doesn't mean anything about the water. It simply gives the water somewhere to go.
2. Why would the first token become the drain?
This is where autoregressive training creates an interesting asymmetry.
Consider:
"The cat sat on the mat because it was tired."
When predicting later tokens, the first token is visible to almost every subsequent position.
The first token therefore has a peculiar property:
token 1 -> visible to token 2
token 1 -> visible to token 3
token 1 -> visible to token 4
...
token 1 -> visible to token N
The last token doesn't have this property.
During causal language-model training, the first token is consequently available as a potential destination for attention across essentially the entire sequence.
Over many training examples, the model can develop a useful convention:
"I don't have a particularly useful thing to attend to?
Fine. Put some attention on this globally available token."
This is an emergent computational convention.
And there is an important subtlety here:
The attention sink is not necessarily carrying useful information through its Value vector.
The interesting quantity is often its attention score.
Suppose:
attention scores -> [large, small, small, small, small]
The first token absorbs probability mass.
Its Value can then contribute relatively little useful semantic information.
Recent empirical work has investigated this interpretation and found evidence that attention sinks behave partly like extra key biases: they can store attention score mass without necessarily contributing corresponding semantic information through the Value computation.
That is a much stranger phenomenon than "the model remembers the first word."
It is closer to:
The model has learned a place to park unused attention.
3. The Transformer history makes this especially interesting
To understand why this matters, it helps to remember where the machinery came from.
In 2017, Ashish Vaswani and colleagues at Google introduced the Transformer in Attention Is All You Need. The central idea was radical for its time: dispense with recurrence and convolutions and build sequence modeling primarily around attention.
The Transformer's attention mechanism gave us an extremely powerful abstraction:
query -> "What am I looking for?"
key -> "What information do I represent?"
value -> "What information should I provide?"
The query-key interaction determines how much attention a token receives.
Then the weighted Values are aggregated.
The catch is the normalization:
softmax(scores)
Softmax isn't an optional implementation detail. It fundamentally couples all the attention probabilities.
Suppose:
scores = [10, 0, 0, 0]
The first token dominates.
But suppose:
scores = [0, 0, 0, 0]
You don't get:
[0, 0, 0, 0]
You get approximately:
[0.25, 0.25, 0.25, 0.25]
The model must allocate attention somewhere.
That seemingly innocuous property becomes extremely important when we start deleting tokens from the KV cache.
4. The real engineering problem: KV cache
During autoregressive generation, recomputing attention over the entire conversation for every generated token would be absurdly expensive.
So inference engines cache the Keys and Values.
For a simplified Transformer:
new token
|
v
compute Q, K, V
|
+----> K/V added to cache
|
v
attention over cached K/V
|
v
next token
The cache grows with every token.
A useful back-of-the-envelope calculation is:
KV bytes/token
= 2 * layers * KV_heads * head_dim * bytes_per_element
The 2 is because we store both K and V.
For example, consider a hypothetical model with:
layers = 32
KV heads = 32
head_dim = 128
precision = FP16 = 2 bytes
Then:
KV bytes/token
= 2 * 32 * 32 * 128 * 2
= 524,288 bytes
≈ 0.5 MB/token
So:
4,000 tokens ≈ 2 GB
32,000 tokens ≈ 16 GB
100,000 tokens ≈ 50 GB
These numbers are per sequence for this particular architecture.
This is why modern inference systems care enormously about:
- MQA/GQA
- KV-cache quantization
- paged KV caches
- prefix caching
- speculative decoding
- sliding-window attention
- context compression
The KV cache is frequently a memory-capacity problem before it becomes a FLOP problem.
And this is exactly where attention sinks become useful.
5. The obvious solution: just keep the last N tokens
Suppose you want a chatbot that can run indefinitely.
You don't actually want:
KV cache:
[token 1 ... token 10,000,000]
You might instead say:
KV cache:
[last 4,096 tokens]
This is sliding-window attention.
Operationally, it is beautiful.
Memory becomes approximately constant:
O(context_length)
|
v
O(window_size)
If the window is 4,096 tokens, the cache never grows beyond roughly 4,096 tokens.
But there is a problem.
Suppose the model was trained with:
[token 1 ... token 4096]
and you start evicting old tokens.
Eventually:
[token 1] -> evicted
[token 2] -> evicted
...
[token 4096] -> still present
The model suddenly sees a different attention environment.
And empirically, the model can collapse.
The surprising discovery from Xiao et al. was that keeping just a few initial tokens alongside the sliding window largely restores the model's behavior.
So instead of:
[recent 4096 tokens]
you maintain:
[first 4 tokens] + [recent 4092 tokens]
The first four tokens are effectively anchors.
That is StreamingLLM.
6. Why four useless tokens can save millions of tokens
This is the part that initially sounds almost ridiculous.
Imagine processing:
1
2
3
4
5
...
4,000,000
You maintain:
S = [1, 2, 3, 4]
W = [3,995,908 ... 4,000,000]
You discard almost everything.
Yet the model remains stable because the attention computation still has access to its familiar sinks.
The memory requirement is therefore approximately:
KV_memory ~= (sink_tokens + window_tokens) * KV_bytes/token
With:
sink_tokens = 4
window = 4092
you have roughly the same memory footprint as a 4K context window regardless of whether the stream has processed:
10K tokens
100K tokens
1M tokens
4M tokens
The model is not storing 4 million tokens.
It is processing a stream while retaining a tiny attention state plus the recent working set.
The result reported by Xiao et al. was striking: Llama-2, MPT, Falcon, and Pythia models could maintain stable language modeling behavior out to 4 million tokens, while StreamingLLM achieved up to a 22.2x speedup over a baseline requiring recomputation.
There is a critical caveat.
This does not mean the model has 4-million-token memory.
StreamingLLM stabilizes next-token prediction.
If you tell the model something important at token 100 and ask about it at token 3,000,000, the attention-sink mechanism does not magically preserve that information.
It solves a different problem:
How do I keep the inference dynamics stable
while aggressively evicting old KV entries?
That distinction matters enormously in production.
7. The deeper lesson: attention isn't always about information retrieval
Developers often develop an intuitive model of attention:
high attention
=
important information
That interpretation is frequently useful.
It is also incomplete.
An attention distribution can have at least two roles:
1. Information retrieval
"Find the token containing the information I need."
2. Computational normalization / routing
"I need the attention distribution to have somewhere
to put probability mass."
Attention sinks expose the second role.
This also explains why simply inspecting attention maps can be misleading.
Suppose you see:
Token #1: 70% attention
Token #200: 2%
Token #201: 1%
...
You shouldn't immediately conclude:
"The model considers token #1 extremely important."
It might.
Or token #1 might simply be serving as a learned computational sink.
That distinction becomes particularly relevant for systems engineers because it changes how we think about cache eviction.
A naive cache policy says:
Keep the tokens that are semantically important.
But the model may need:
Keep the tokens that are computationally important.
Those are different optimization problems.
And the latter can sometimes be surprisingly cheap.
8. What this means for real LLM infrastructure
Attention sinks are interesting because they turn a research curiosity into an infrastructure primitive.
Imagine a production voice assistant.
A user talks continuously:
minute 1
minute 2
minute 3
...
minute 60
...
minute 600
Keeping every KV entry indefinitely is expensive.
Suppose your architecture costs roughly:
0.25 MB/token
Then a million-token conversation would require roughly:
250 GB
of KV state for one sequence.
That is obviously incompatible with ordinary GPU inference.
A bounded cache changes the economics:
unbounded context
-> memory grows with conversation
sliding window
-> memory stays bounded
sliding window + sinks
-> bounded memory + stable attention behavior
And the operational consequences are substantial:
GPU memory
A smaller KV cache allows more concurrent sequences.
If one request needs 8 GB of KV memory and your GPU has 80 GB available for KV state, you might support roughly:
80 / 8 = 10
such sequences.
Cut that requirement to 2 GB and the same memory budget theoretically supports:
80 / 2 = 40
before accounting for model weights, activations, fragmentation, runtime overhead, and batching.
That is a 4x concurrency difference.
Latency
Recomputing discarded history can become increasingly expensive.
StreamingLLM's reported 22.2x speedup against a recomputation-heavy baseline gives a sense of how important this can become for genuinely long streams.
Hardware utilization
The economics of LLM serving are dominated by expensive accelerators.
If an algorithm lets you:
use less HBM
+
serve more sequences
+
avoid recomputation
then a seemingly tiny architectural observation can translate into meaningful infrastructure savings.
This is one of the recurring patterns in LLM research:
small mathematical quirk
|
v
inference algorithm
|
v
memory behavior
|
v
GPU utilization
|
v
cost per generated token
Attention sinks are a particularly clean example.
9. A useful mental model for developers
If you're implementing or debugging an LLM inference system, I'd keep three layers in your head.
Layer 1: Semantic attention
"What previous information do I need?"
This is the intuitive interpretation.
Layer 2: Attention mechanics
"How does softmax distribute probability mass?"
This is where attention sinks appear.
Layer 3: Systems behavior
"What K/V states must physically remain in GPU memory?"
This is where StreamingLLM becomes useful.
The crucial connection is:
softmax behavior
|
v
attention sinks
|
v
KV-cache eviction policy
|
v
memory footprint
|
v
serving economics
A phenomenon that initially looks like a weird visualization artifact becomes an answer to a very practical systems question:
Which old tokens can I safely throw away?
The surprising answer is:
Almost all of them.
Except perhaps the few that the model has learned to use
as computational sinks.
That is a much richer way to think about LLM inference than simply treating the context window as a giant text buffer.
Conclusion: the token that does almost nothing
Attention sinks are a good reminder that neural networks develop internal computational conventions that don't necessarily line up with our semantic intuitions.
The first token can become important precisely because it is not important in the usual sense.
It becomes a place where the model can park attention.
And once researchers recognized that behavior, they could exploit it to build an inference system with a constant-size KV cache that remained stable over streams millions of tokens long.
For LLM engineers, that suggests a broader principle:
When you find a strange behavior in an LLM, don't immediately treat it as noise. It may be an undocumented interface between the model's mathematics and the hardware running it.
That interface is where a lot of the interesting engineering is happening.
Have you encountered another LLM behavior that initially looked like a quirk but turned out to have major inference or systems implications?
*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
HexmosTech
/
git-lrc
Free, Micro AI Code Reviews That Run on Git Commit
| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी |
git-lrc
Free, Micro AI Code Reviews That Run on Commit
GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.
git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.
In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…
Top comments (0)