This is the second article in my Attention Mechanism Evolution series. The first covered the horizontal (sequence-dimension) attention optimizations — from GPT-2's full attention to Kimi K3's KDA hybrid architecture. That was one axis. This article tackles the vertical axis: can deeper layers selectively attend to shallower layers, instead of just blindly summing via residual connections?
Depth-Attention, proposed by Shanghai Jiao Tong University's LUMIA Lab (arXiv: 2606.05014, accepted at ICML 2026), answers with an elegant design: zero new parameters, zero additional KV cache, under 0.01% extra FLOPs — and a consistent +2.3 point accuracy boost on downstream tasks.
1. The Problem: Transformers Are "Blind" Between Layers
Standard self-attention excels at freely selecting information along the sequence dimension — each token can attend to any position in the sequence.
But switch to another dimension — between layers — and Transformers become startlingly inefficient:
Layer ℓ-1 output → (residual connection: simple summation) → Layer ℓ input
Each residual block just adds the previous layer's output on top of its own — no weighting, no selectivity, no attention. The deeper layers face a hidden state that's just a compressed dump of all prior layer outputs.
Think of it like this: you write a report and hand each page to a colleague, but they only see the last page. They can't tell whether "page 3's analysis is your best work — read more of it" or "page 15 isn't that important — just skim it."
This isn't a new problem. Recent work has tried to solve it — DenseFormer (learns fixed inter-layer weights for weighted averaging), Hyper-Connections/mHC (multiple parallel residual streams replacing single residual), and Attention Residuals (using softmax attention to adaptively select shallow-layer outputs per token — this is what Kimi K3 uses).
But these approaches share a problem
They all operate on hidden states — the model's full intermediate representations, outside the self-attention module itself. This means they need to retain or access these hidden states in addition to the KV cache during inference.
Meanwhile, the trend in large models (GQA, MLA) is to aggressively compress the KV cache — DeepSeek uses MLA to compress from 128×d down to 4×d. Adding extra persistent states outside the KV cache? That's swimming against the current.
Depth-Attention's insight is exactly this contradiction: can we leverage the Q, K, V already inside the attention module, in the same place, without any external state, to achieve cross-layer selective mixing?
2. How Depth-Attention Works: Rotating "Sequence Attention" by 90 Degrees
The answer is remarkably simple.
Standard self-attention operates on the sequence dimension — "the current token attends to all tokens in the sequence."
Depth-Attention rotates this operation 90 degrees — performing the exact same thing on the depth dimension:
For layer ℓ, token t:
q_ℓ^t · k_j^t (current layer query × each shallower layer's key, at the same token position)
→ softmax → depth attention weights α
→ weighted mixture ṽ (depth-mixed value)
The final step is key — everything in self-attention stays unchanged (no Q modification, no K modification, no mask change), only the value is replaced with the depth-mixed version:
O_ℓ = CausalAttn(Q_ℓ, K_ℓ, Ṽ_ℓ) # Ṽ_ℓ replaces the original V_ℓ
Three elegant properties:
1. Zero new parameters. Depth-Attention fully reuses the Q and K projection matrices already in standard self-attention — the same Q serves both sequence-direction and depth-direction attention. No new parameter matrices at all.
2. Zero additional KV cache. The depth-mixed Ṽ has exactly the same shape as the original V (T×d). During inference, it replaces rather than appends — Ṽ goes into the original V cache slot. Subsequent tokens reading that slot automatically get the depth-mixed version — so the persistent state at inference is identical to a vanilla decoder's KV cache. Not a single byte extra.
3. Recursive information propagation — the paper's most ingenious design. Let me write out the formula clearly:
ṽ_ℓ^t = α_ℓ,ℓ · v_ℓ^t (current layer's own value)
+ Σ_{j<ℓ} α_ℓ,j · ṽ_j^t (previous layers' already-mixed values)
Notice the second term: the mixture uses not the raw v_j, but the already depth-mixed ṽ_j. This means information propagates cascadingly — ṽ_4 contains all mixed info from ṽ_0 through ṽ_3. When ṽ_16 reads from ṽ_4, it gets not just layer 4's raw value, but depth-mixed information already processed by layer 4.
If it mixed raw v_j, layer 10 could only directly access layer 2's value — with 8 layers of residual stream attenuation in between, the signal is highly degraded. But mixing ṽ_j means: layer 6 reads layer 4 → layer 4's ṽ already contains layer 0's info → layer 10 reads layer 6 → it also indirectly gains contributions from layer 0 and layer 2.
A recursive structure achieves efficient propagation. This allows shallow representations to penetrate through many intermediate layers and continue influencing deep layers.
3. Counter-Intuitive Ablation: More Layers Isn't Always Better
If every layer attended to all shallower layers, the compute cost would be O(TL²) — non-trivial in deep networks, especially with pipeline parallelism's cross-device communication overhead.
Depth-Attention addresses this with strided sampling: each layer only attends a sparse subset — itself + {0, s, 2s, ...} among the shallower layers.
With s=4, layer 20 attends {20, 16, 12, 8, 4, 0}, skipping the 14 layers in between.
Complexity drops to O(TL/s); with s large enough, it approaches O(T), completely negligible.
But the most counter-intuitive result is the ablation study: the paper tested different strides. The result was not "more is better." s = L/2 (half the total layers) performed best — better than denser s=L/4 and sparser s=L.
Why? The paper doesn't give a definitive explanation, but I think this phenomenon hints at an important principle: there may be an optimal "reception interval" for information propagation across depth — too dense creates redundancy (adjacent layers' values are highly correlated), too sparse loses critical information. s=L/2 happens to provide enough "diversity windows" without falling into the trap where "layer 20's info is highly redundant with neighboring layers 16 and 14."
This echoes the same design philosophy we saw in part one with Kimi K3's KDA hybrid architecture: it's not about stacking more — it's about choosing more wisely where to apply it.
4. GQA Compatibility: Naturally, Freely More Efficient
Modern large models widely use Grouped-Query Attention (GQA), where g query heads share one set of KV heads.
Depth-Attention handles GQA extremely naturally: average the g queries within each group, running depth attention at KV head resolution — not query head resolution.
This brings two additional benefits:
- KV head dimension is already several times smaller than hidden size (typically 4x) — running depth attention in this space further shrinks compute and memory
- No head-dimension expansion or alignment needed — just average g queries, zero hyperparameter changes
In other words: Depth-Attention is not only overhead-free in GQA scenarios, it's actually more efficient than in non-GQA scenarios — because depth attention's operating space is naturally compressed by GQA.
5. Experimental Results: What's the Actual Gain?
The paper ran experiments on Qwen3-style decoder architectures at 1.5B and 3B scales. All models were trained from scratch on 32B tokens from the Pile, with identical data and hyperparameters — a fair comparison.
Baselines included: Vanilla Transformer, mHC (manifold hyper-connections), Attention Residuals (Kimi K3's approach), and DenseFormer.
Key results
At 1.5B scale, zero-shot:
- Vanilla Transformer average downstream accuracy: 51.26
- Depth-Attention: 53.56 (+2.3 points)
- Attention Residuals and mHC fall between Vanilla and Depth-Attention
At 3B scale, the gap widens — Depth-Attention reduces perplexity to 6.66 (Vanilla: 7.10), and average accuracy improves from 53.08 to 55.27 (+2.19 points), comprehensively beating all baselines.
Under 5-shot settings, the pattern holds — Depth-Attention is best at both scales.
Notably: Attention Residuals, as the strongest baseline, does significantly beat Vanilla — the paper acknowledges this clearly — but Depth-Attention surpasses it on every metric, and does so without adding any inference state.
Efficiency: theory vs. measurements
Theoretical FLOPs: The paper's Appendix B provides detailed derivations of extra FLOPs for each method. Counting per decoder layer:
- Depth-Attention extra compute: ≈ 2Td/s FLOPs (s = stride, d = head dim)
- DenseFormer extra compute: ≈ T·d_model·L (weighted summation across all layers)
- Attention Residuals extra compute: ≈ T·d_model·L (similar magnitude)
Plugging in typical values (T=4096, d_model=2048, L=32, s=16), Depth-Attention's extra FLOPs are less than 0.01% of self-attention FLOPs.
Training wall-clock: On a 3B model, Depth-Attention adds only ~1% per-step training time — significantly less than DenseFormer and Attention Residuals.
Inference throughput: At 128K token prefill, Depth-Attention throughput is essentially identical to Vanilla (<0.5% difference). Hidden-state methods, by contrast, need explicit extra state management in long-context scenarios, with noticeably higher memory usage.
Scaling experiments
The paper tested four scales from 360M to 3B. Depth-Attention maintains its advantage at all scales, with gains showing no sign of saturation. This suggests larger models may also benefit — though this hasn't been verified at 70B+ scale (a limitation the paper acknowledges).
6. Weight Visualization: What Did the Model Learn?
The paper visualizes the trained Depth-Attention weight distributions, with several interesting findings:
1. Early layers focus their depth attention on deeper information sources, not shallower ones.
Concretely: Layers 1-8 (early layers) have self-attention weights dominating in depth attention (α_ℓ,ℓ close to 1) — they don't look much at shallower layers, because there simply aren't enough shallower layers.
But layers 9-32 (the second half) start dispersing their depth attention weights: deep layers still retain large self-attention weights (preserving their own features) but radiate uniformly toward middle layers.
2. Shallower isn't necessarily more important.
Intuition might suggest "shallow features are more fundamental, deep layers should attend more to them." But from the weight plots, deep layers (e.g., layer 30) don't assign higher attention weights to very shallow layers (layer 0, 4) than to middle layers (layer 12, 16).
This hints at an interesting learned strategy: the model learns to extract information at different granularities from different depth levels, rather than simply treating shallow layers as an "information repository."
3. Some attention heads are more cross-layer-dependent than others.
Different heads show different degrees of dependence on depth information. Some heads have self-attention weights close to 1 at almost all layers (barely looking at other layers), while others have dispersed weights across the entire depth dimension (actively leveraging cross-layer information). The model thus gains flexibility — use "self-attention-heavy" heads when local fine-grained processing is needed, and "cross-layer-heavy" heads when contextual fusion is needed.
7. Looped Transformer Experiment: Parameter Sharing Works Too
This is one of the paper's most profound experiments. A Looped Transformer has only a few physical layers (e.g., 4 layers) that execute in a loop (e.g., 8 cycles = 32 total effective layers), with all cycles sharing the same parameters.
Depth-Attention's gains persist in this setting — each loop cycle acts as a new "logical layer" that can attend to the values from its own previous cycles.
This result rules out a possible explanation: "Depth-Attention works because different layers have different parameters, so their value semantics differ, making them worth retrieving." In the parameter-sharing scenario, values from different layers are still selectively passed through depth attention — proving that the mechanism's benefit doesn't depend on "different parameters per layer," but rather that selective cross-depth information transfer is inherently valuable.
Taking it deeper: this result hints at a potential relationship between Depth-Attention and recurrence/SSMs. If parameter sharing works, what Depth-Attention is doing is somewhat like a soft state-space model — selectively remembering and retrieving depth-history information in the value dimension, rather than propagating through fixed residuals.
8. Depth-Attention vs. Kimi K3's Attention Residuals: A Design Philosophy Divergence
This is the most fascinating comparison — they solve the same problem but choose different positions.
| Attention Residuals | Depth-Attention | |
|---|---|---|
| Operation point | On residual stream (outside module) | Inside attention (value position) |
| Target | Hidden states | V cache slots |
| Extra inference state | Required (retain shallow hidden) | None |
| Extra parameters | Yes | Zero |
| Information granularity | Full hidden state (rich) | Value state (compressed) |
| Performance (paper comparison) | Better than vanilla | Better (at 1.5B/3B) |
Both share the same conceptual origin — enabling deeper layers to selectively attend to shallower ones. The difference is in the tradeoff:
- Attention Residuals operates at the most information-rich position (hidden state), but at a cost
- Depth-Attention operates at the cheapest position (V cache), at zero additional cost
This divergence doesn't have a final answer yet: who wins at 70B+ scale? Nobody knows. But one thing is clear — Depth-Attention found a "free lunch" path. It proved that you don't need to add anything outside the KV cache to get all the benefits of cross-layer selective information transfer.
This isn't a free lunch — it's the lunch you were already eating, you just didn't realize you could eat it better.
9. What This Means
Architecture evolution: one horizontal, one vertical
The first article covered sequence-direction (horizontal) attention evolution from full attention to KDA — that was "how to remember more information with less state."
This article completes the inter-layer (vertical) direction — "how to let deep layers selectively use shallow representations without paying extra."
Put together:
← Sequence direction (horizontal) →
Full Attn → KV Cache → Linear Attn → DeltaNet → KDA
(compute → memory → compression → precision → hybrid recipe)
↑
Inter-layer direction (vertical)
Residual → DenseFormer → Attention Residuals → Depth-Attention
(sum → weighted → attention → zero-cost attention)
↓
For model users:
- Next time you see a model claiming "cheap long context," don't just look at parameter count and context window — ask "what's your inter-layer communication scheme? How does information flow in the depth direction?"
- Kimi K3 uses Attention Residuals; the next version might switch to Depth-Attention or a variant — keep an eye on this thread
For model trainers:
- Depth-Attention's zero-parameter nature means a lossless drop-in replacement: no weight changes, no cache changes, no pipeline changes — just modify a few lines of attention code and reliably gain ~2 points
- The paper's code is open-source on GitHub, with a pre-trained 3B checkpoint on HuggingFace for verification
For technical investors:
- The design philosophy divergence (hidden state vs. value state path) could influence next-generation model architecture choices
- Inter-layer communication is becoming a core optimization dimension alongside "attention compression"
10. Limitations (Frankly)
Limited validation scale. The 1.5B and 3B experiments are convincing (32B tokens trained from scratch, 8 downstream tasks), but whether results replicate at 70B+ scale remains unknown. The paper acknowledges this.
Mechanism explanation for optimal stride is missing. s=L/2 working best is an empirical finding — why this value, and whether better non-uniform sampling strategies exist, leave significant analysis space.
Interaction with other optimizations under-explored. Effects when combined with MoE (e.g., K3's 896 experts), compatibility with MLA, differential behavior in prefill vs. decode phases — all await follow-up work.
Is the "dual attention" structure of depth + sequence optimal? This is an architectural question — perhaps a future unified primitive will fuse both directional attentions into one operation. Depth-Attention is an important step, but not necessarily the last.
References
- Depth-Attention: Cross-Layer Value Mixing for Language Models — Boyi Zeng et al., Shanghai Jiao Tong University LUMIA Lab, ICML 2026
- Code & Models — GitHub + HuggingFace (3B checkpoint available)
- Attention Residuals — Kimi Team, 2026 (cross-layer mechanism used in Kimi K3)
- DenseFormer — Pagliardini et al., NeurIPS 2024
- Hyper-Connections / mHC — Zhu et al., 2025
- GQA — Ainslie et al., EMNLP 2023
This is the second article in the "Attention Mechanism Evolution" series. Part one covered the horizontal (sequence-direction) evolution — from GPT-2's full attention to Kimi K3's KDA hybrid architecture, with cross-validation across 5 papers, a complete evolution table, and code-level analysis. Part three preview: from sparse attention to MoE — why K3's 896 experts only activate 1.8%.
Top comments (0)