Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
At 1 million tokens, the difficult part of an LLM is no longer simply having a large memory.
It is deciding what part of that memory is worth reading.
DeepSeek's recent architecture work can be understood as a sequence of answers to that problem:
- MLA compresses the representation stored for each token.
- DSA learns which parts of that compressed history matter.
- CSA combines compression and learned sparse selection.
- HCA compresses history even more aggressively and reads it densely.
- mHC solves a different problem: keeping information flowing stably through many Transformer layers.
These are easy to confuse because they all appear in the same family of models, but they operate at different levels.
The useful mental model is:
DeepSeek-V4
┌──────────────────────────────────────────┐
│ Residual stream │
│ │ │
│ mHC │
│ │ │
│ ┌─────────────┴─────────────┐ │
│ │ │ │
│ CSA HCA │
│ │ │ │
│ compress + select compress only│
│ │ │ │
│ DSA dense attn │
│ │ │ │
│ └─────────── attention ────┘ │
└──────────────────────────────────────────┘
The interesting part is that DeepSeek did not discover one magic attention mechanism.
It built a hierarchy of information access.
1. The problem DeepSeek was actually trying to solve
The original Transformer attention mechanism has a simple rule:
For every query, compare it with every previous token.
For a sequence of length L, that gives roughly:
attention work ~ O(L^2)
At 4K tokens, this is manageable.
At 128K, it becomes expensive.
At 1M tokens, blindly comparing everything with everything becomes a systems problem.
This matters even more for reasoning models and agents.
A normal chatbot might generate 500 tokens after reading a 5K-token prompt.
An agent can repeatedly:
read repository
→ call tool
→ inspect result
→ reason
→ call another tool
→ inspect another result
→ continue
The context keeps accumulating.
The model therefore needs something closer to a working memory hierarchy.
DeepSeek approached this problem incrementally.
In December 2025, the DeepSeek-V3.2 paper introduced DeepSeek Sparse Attention (DSA). The interesting detail is that V3.2 did not redesign the entire model. It started from the previous checkpoint and introduced DSA through continued training.
Then, on December 31, 2025, Zhenda Xie and colleagues published the mHC work, attacking a different issue: information propagation through increasingly sophisticated residual connections.
A few months later, DeepSeek-V4 combined these ideas with two new attention forms, CSA and HCA, and pushed the context length to one million tokens.
That sequence is useful because it reveals what each mechanism is actually responsible for.
2. DSA: the model learns where to look
Start with ordinary attention.
For one query q_t, we calculate something like:
score(t, s) = q_t · k_s
for every preceding token s.
Then we use those scores to form the weighted sum of the values.
DSA inserts a cheap indexer before the expensive attention operation.
Conceptually:
query
│
▼
lightweight indexer
│
├── score token 17
├── score token 18
├── score token 19
├── ...
└── score token 100000
│
▼
Top-k
│
▼
expensive attention
Instead of doing expensive attention over the entire history, the model first asks:
Which locations appear relevant to this query?
Then it performs the real attention operation only on those locations.
The DSA paper expresses this as:
I(t,s) = sum_j w(t,j) * ReLU(q(t,j) · k(s,j))
followed by:
S_t = TopK(I(t,:))
and the normal attention operation is performed only on S_t.
The important distinction is that DSA does not merely make attention smaller.
It makes the selection content-dependent.
For one query:
"Where did we define the database schema?"
the indexer might favor a handful of earlier locations containing schema definitions.
For another query:
"What did the user say about authentication?"
a different subset becomes relevant.
This makes DSA closer to a learned retrieval mechanism than to a fixed sparse pattern.
A concrete training trick
There is a nice engineering story in the V3.2 paper.
DeepSeek did not immediately turn on sparse attention.
First, it trained the lightweight indexer while keeping normal dense attention active.
The dense model supplied a target attention distribution, and the indexer was trained using KL divergence:
L_indexer = KL(
dense_attention_distribution
||
softmax(indexer_scores)
)
They used only 1,000 warm-up steps, covering about 2.1 billion tokens.
Only after the indexer had learned something approximating the dense model's attention pattern did they turn on actual top-k selection.
Then they trained the model for another 15,000 steps over about 943.7 billion tokens, selecting 2,048 KV tokens per query.
That is a useful lesson for developers:
sparsity can be trained as a routing problem before it becomes a computational constraint.
3. CSA: compress first, then ask DSA where to look
Now we can understand CSA.
CSA stands for Compressed Sparse Attention.
It essentially says:
DSA is useful, but do not even make the indexer search over every original token.
Suppose:
L = 1,000,000 tokens
m = 4
CSA compresses the sequence dimension by approximately 4x.
So instead of:
1,000,000 KV entries
we get roughly:
250,000 compressed KV entries
The compression itself is learned.
Very roughly, for a group of tokens:
C_comp = sum_j S_j * C_j
where the S_j weights are normalized and learned from the hidden states.
Now DSA operates on this compressed sequence.
For DeepSeek-V4-Pro:
m = 4
top-k = 1024
So the rough information flow is:
1,000,000 original tokens
│
▼
compression / 4
│
▼
250,000 compressed entries
│
▼
DSA TopK
│
▼
1,024 entries
│
▼
core attention
That is the central relationship between DSA and CSA.
DSA is the selection mechanism.
CSA changes the objects being selected.
DSA alone:
tokens → score → TopK → attention
CSA:
tokens → compress → score compressed entries → TopK → attention
This produces an important back-of-the-envelope calculation.
With a million-token context:
dense attention:
1,000,000 KV candidates
CSA core attention:
1,024 KV candidates
Ignoring all other costs:
1,000,000 / 1,024 ≈ 977
So the expensive core attention sees almost three orders of magnitude fewer candidates.
But this does not mean the entire attention layer becomes 977x cheaper.
The lightning indexer still has to inspect the compressed history:
~250,000 compressed candidates
and the compression itself has a cost.
The actual architecture is therefore a trade:
cheap broad search
+
expensive narrow attention
That pattern is familiar from information retrieval systems.
You do not run the most expensive ranking model against ten million documents.
You first retrieve a candidate set.
4. HCA: what happens when even 4:1 compression is unnecessary?
CSA gives us:
moderate compression
+
sparse selection
But DeepSeek asks another question:
What if we compress history so aggressively that we no longer need sparse selection?
That is HCA: Heavily Compressed Attention.
For HCA, DeepSeek uses:
m' = 128
So:
1,000,000 tokens / 128
≈ 7,812 compressed entries
Now dense attention over 7,812 entries is much more manageable.
There is no DSA-style TopK step.
The architecture is simply:
1,000,000 tokens
│
▼
compress / 128
│
▼
~7,812 compressed entries
│
▼
dense attention
This reveals something important about CSA versus HCA.
They are not competing mechanisms.
They represent two different resolutions of memory.
Think of a map.
When navigating a city, you might need:
nearby streets → detailed
nearby neighborhoods → moderately detailed
distant cities → coarse
CSA is the middle layer.
HCA is the far-away layer.
DeepSeek-V4 interleaves the two.
For V4-Pro, the paper specifies:
CSA compression: 4:1
HCA compression: 128:1
CSA top-k: 1024
sliding window: 128 tokens
There is also a local sliding-window branch containing recent uncompressed KV entries.
That last detail matters.
Compression is excellent for long-range information, but language has very strong local dependencies.
The token immediately before the current token matters.
A variable name five tokens ago matters.
The beginning of the current sentence matters.
So DeepSeek effectively gives the model:
recent context:
individual tokens
medium-range context:
compressed blocks + learned TopK
long-range context:
heavily compressed blocks
This is much closer to a multi-resolution memory system than to conventional attention.
5. Where does mHC fit? It is solving a different problem.
This is where the terminology becomes confusing.
mHC has almost nothing to do with deciding which token to attend to.
It changes the residual connection between Transformer layers.
The conventional Transformer update looks roughly like:
x_(l+1) = x_l + F_l(x_l)
The identity path is extremely important.
It gives information and gradients a clean route through hundreds of layers.
Hyper-Connections generalize this idea by maintaining multiple residual streams.
Instead of one stream:
x
we might have:
x1
x2
x3
x4
and allow the layer to mix them.
DeepSeek's mHC uses a formulation of the form:
X_(l+1) = B_l X_l + C_l F_l(A_l X_l)
Here:
X_l = multiple residual streams
A_l = mixes them into the layer input
F_l = the actual Transformer block
B_l = residual-stream mixing
C_l = writes the layer output back
The problem is that unrestricted matrices B_l can make signal propagation unstable when many layers are stacked.
mHC constrains B_l to be doubly stochastic.
That means:
B >= 0
each row sums to 1
each column sums to 1
The set of these matrices is the Birkhoff polytope.
Why is this useful?
Imagine each residual stream as carrying some quantity of information.
A doubly stochastic transformation behaves like a conservative redistribution:
stream 1 ─────┐
stream 2 ──┐ │
stream 3 ──┼──┼──> redistributed streams
stream 4 ──┘ │
│
It can mix information between streams, but it cannot arbitrarily amplify the whole residual transformation.
DeepSeek's stated mathematical property is that:
||B||_2 <= 1
so the residual mapping is non-expansive.
This is why mHC belongs to the stability side of the architecture.
CSA/HCA answer:
What should this layer read?
mHC answers:
How should information survive and mix as it passes through the stack?
That distinction is worth remembering.
6. The connection between them is architectural, not mathematical
The cleanest way to understand DeepSeek-V4 is to separate the system into three axes.
Axis 1: What information is available?
Controlled by KV compression.
MLA → CSA/HCA
The model stores a much smaller representation of history.
Axis 2: What information gets expensive attention?
Controlled by sparse routing.
DSA → CSA
The model uses a cheap indexer to identify a small subset.
Axis 3: How does information propagate between layers?
Controlled by mHC.
mHC → residual stream
The model gets a richer residual topology while constraining the residual mixing for stability.
Put differently:
INFORMATION ACCESS
CSA
/ \
compression DSA
\ /
attention
INFORMATION PROPAGATION
mHC
│
residual streams
│
Transformer blocks
The important insight is that mHC is orthogonal to DSA.
You could imagine a model with DSA and ordinary residual connections.
You could imagine mHC combined with dense attention.
DeepSeek combines them because the engineering constraints are interconnected, but the mechanisms attack different bottlenecks.
This also explains why the V4 paper describes them separately in its architecture section.
7. The economics of this design: memory becomes an architectural resource
The million-token context claim is ultimately an economics claim as much as a modeling claim.
At one million tokens, KV cache is expensive.
Suppose a conventional attention system stores a large KV tensor for every layer and every token.
Multiply:
tokens
× layers
× KV dimensions
× bytes per element
× concurrent requests
and the memory requirement grows rapidly.
Then there is the bandwidth required to read those KV entries during decoding.
This is why the V4 paper reports a particularly useful comparison.
At a 1M-token context:
V4-Pro:
~27% of the V3.2 single-token inference FLOPs
~10% of the V3.2 KV-cache size
And V4-Flash goes further:
~10% of V3.2 FLOPs
~7% of V3.2 KV cache
The model therefore spends a large architectural budget deciding what not to compute.
There is an equally important operational story.
DeepSeek did not merely invent the equations and hope GPU kernels would follow.
They changed the serving system around the architecture:
compressed KV layout
+ sparse-attention kernels
+ sliding-window state
+ on-disk KV caching
+ contextual parallelism
+ fused mHC kernels
For example, mHC increases activation memory and pipeline communication relative to ordinary residual connections. DeepSeek reports using fused kernels, selective recomputation, and pipeline overlap to constrain the wall-time overhead to about 6.7% of the overlapped pipeline stage.
That is a useful lesson for anyone building LLM infrastructure:
an architectural optimization is only real when the hardware sees it.
A paper can turn:
O(L^2)
into something much smaller on paper and still lose in practice to:
kernel launch overhead
memory movement
unfavorable gathers
communication
poor GPU occupancy
DeepSeek's V4 work is interesting precisely because the algorithm, training procedure, kernels, cache format, and distributed runtime were designed together.
A developer's mental model
You do not need to remember every equation.
Remember this pipeline:
1M-token history
│
▼
compress the KV history
│
┌─────────────────┴────────────────┐
│ │
CSA HCA
│ │
4:1 compression 128:1 compression
│ │
DSA TopK selection dense attention
│ │
└──────────────┬───────────────────┘
│
local 128-token window
│
▼
attention output
│
▼
mHC
│
▼
next Transformer layer
So the one-line summary is:
DSA = learned retrieval
CSA = compression + learned retrieval
HCA = extreme compression + dense retrieval
mHC = stable multi-stream residual transport
And there is a broader architectural idea underneath all four:
A million-token context does not require treating a million tokens as equally expensive pieces of memory.
DeepSeek's approach is to represent history at different resolutions and spend computation where the query actually needs detail.
That is a very general idea.
It applies beyond attention to databases, retrieval systems, caches, agent memory, and even compiler architectures: store broadly, index cheaply, compute precisely where needed.
The interesting question for the next generation of LLMs is therefore less "How do we make context windows bigger?" and more:
How many different resolutions of memory should a model have, and how should it learn when to move between them?
What do you think is the more important direction from here: better learned retrieval like DSA, more aggressive hierarchical compression like HCA, or changing the Transformer's information-flow topology like mHC?
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
⭐ Star it on GitHub:
HexmosTech
/
LiveReview
Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.
The exact math, not a black box
Visualize blast radius at a glance
Every factor that feeds the score
Here's the goal:
- A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
- A 300-line UI change in one file, fully covered by…
Click below to try LiveReview with your codebase:
Top comments (0)