DEV Community: AICDragon

From Sparse Attention to MoE: Why K3's 896 Experts Only Activate 1.8% Per Token

AICDragon — Tue, 11 Aug 2026 03:16:11 +0000

From Sparse Attention to MoE: Why K3's 896 Experts Only Activate 1.8% Per Token

This is the third installment in a series on the evolution of attention mechanisms.

Part 1 covered the sequence dimension: from GPT-2's full attention to KV Cache, linear attention, and KDA — seven years of progress.
Part 2 covered the inter-layer dimension: from simple residual summation to Attention Residuals and Depth-Attention — selective cross-layer communication at near-zero cost.

This article explores the third dimension: orthogonal computation — when Transformers face the demand of "make the model bigger without making inference slower," the answer is "most parameters stay idle." This is the story from sparse attention to MoE to Stable LatentMoE. Let's begin with one number: K3 has 896 experts, but at inference time each token only activates 16 — that's 1.8%.

1. The Origin of the Problem: The Curse of All-or-Nothing

Standard Transformers operate on an implicit assumption: every token at every layer must pass through all parameters.

Layer 0: Token -> (Attention + FFN) -> hidden state
Layer 1: Token -> (Attention + FFN) -> ...
...
Layer 92: Token -> (Attention + FFN) -> predict next token

Every layer's FFN processes every token. "Answer a primary-school math problem" and "write operating system kernel code" engage entirely different cognitive circuits — yet the Transformer is forced to use the same parameters for both.

This creates two fatal consequences:

Parameter count is constrained. More parameters means more operations per inference — compute cost scales linearly. Want a smarter model? Adding parameters = adding cost.
Wasted compute. The simplest token and the hardest token consume exactly the same FLOPs.

MoE's solution is brutally effective: prepare multiple sets of "cognitive circuits" (experts), and have each token call only a small subset.

Sounds like common sense. But turning that common sense into an industrially stable, inference-efficient model took several years.

2. Three Cards in MoE's Evolution

Card 1: Switch Transformer (2021, Google) — Moving MoE from Lab to Factory

MoE itself is an old idea (Jacobs et al., 1991), but before the LLM era it was stuck in the lab due to training instability.

Switch Transformer (Fedus et al., JMLR 2022) did two crucial things:

Solved large-scale training stability. Used capacity factor to cap tokens per expert, and auxiliary loss to balance routing load.
Trained a 1.6T-parameter model on C4, activating far fewer parameters at inference than the total count.

But Switch used Top-1 routing — each token goes to exactly one expert. The advantage is extreme sparsity (minimal activation rate). The problem is obvious: if that one expert fails (overloaded or dropped), the token produces no output — there's no backup.

Card 2: Mixtral 8x7B (2023, Mistral) — Top-2 Becomes the Industry Standard

Mistral's Mixtral 8x7B, released in late 2023, used Top-2 routing — each token activates 2 out of 8 experts.

This marked a key design philosophy shift. Top-1 is "all eggs in one basket." Top-2 begins to accept "spend a bit more compute for significantly better stability and performance." The 8-expert configuration is also clean enough to run on consumer-grade GPUs.

Mixtral, at 46.7B total parameters (12.9B active), matched or approached comparable dense models on multiple benchmarks while using substantially less compute. This proved that MoE isn't just for gigantic models — it pays off at medium scale too.

Card 3: DeepSeek-MoE (2024, DeepSeek) — Fine-Grained Experts + Shared Experts

DeepSeek-MoE (ACL 2024) introduced two of the most important architectural innovations in MoE history:

Fine-grained expert segmentation: Split N experts into mN smaller experts, activating mK per token. This enables more flexible combinations — instead of a few large experts handling broad knowledge domains, many small experts perform precise assembly.
Shared experts: Isolate K_s "shared experts" from the total expert pool, always activated. These handle foundational knowledge every token needs (grammar structure, common phrasing, basic reasoning frameworks), while routed experts focus on domain-differentiated knowledge.

This design solves MoE's biggest contradiction: "knowledge redundancy."

In early MoE, every expert could learn the same basics (e.g., "the" can be followed by a noun) because routing selects experts based on token features, and all experts see these high-frequency patterns. Isn't that just wasting parameters?

The shared expert idea: "Extract that shared part, let everyone use it, stop routed experts from redundantly learning fundamentals."

DeepSeek-MoE 16B achieved LLaMA2 7B performance at roughly 40% of the compute. DeepSeek-MoE 145B matched DeepSeek 67B using only 28.5% of the compute. This was the first paper to clearly demonstrate that the "fine-grained + shared expert" dual-track architecture has overwhelming advantages in compute efficiency.

This is also the direct predecessor of K3's MoE architecture.

3. K3's Stable LatentMoE: Pushing DeepSeek-MoE's Three Paths to the Limit

K3's technical report describes the Stable LatentMoE design in detail. Building on DeepSeek-MoE, it pushes three paths:

Path 1: Expert Count Jumps from 64 to 896

DeepSeek-MoE already made experts smaller and more numerous, but K3 pushes this to another order of magnitude.

DeepSeek-MoE 16B:   ~64 routing experts + shared experts
K3:                  896 routing experts + 2 shared experts

The key isn't "896 is a lot" — it's what 896 implies about how small each expert is.

Let's compute. K3 has 896 routing experts per layer, hidden dim = 7168, latent dim (the compressed dimension before entering routing experts) = 3584, each expert's intermediate dim = 3072.

One expert = two weight matrices (3584->3072 + 3072->3584) = roughly 22M parameters.

K3 has 2.78T parameters total. 896 experts x 93 layers = roughly 83,328 experts. At 22M each, the expert portion is... far below 2.78T. Because experts are just FFN replacements — the total parameter count includes embeddings, all layers' attention weights, shared experts, and other larger components.

But the core logic holds: when experts become small enough, routing ceases to be semantic assignment ("bring in the Python expert") and becomes feature assembly from a pool of fragments.

This leads to K3's most crucial design philosophy, and the most insightful analysis from the lilting channel —

The Router Is Not a "Classifier" — It's a "Compute Allocator"

The standard MoE understanding: "the router classifies each token to a domain expert." This understanding is wrong — at least for K3 with 896 experts.

K3's router is a single line:

scores = Sigmoid(W_r x x)   # W_r: 896 x 7168, x: hidden state
# Take Top-16, weighted combination by normalized scores

A linear projection -> Sigmoid -> Top-16. No complex operations whatsoever.

But this works not because the router is "smart," but because x itself has already been thoroughly processed by preceding layers' Attention and FFN. Just as BERT hidden states are good enough for a simple linear classifier to perform NLU tasks — K3's hidden state, after passing through dozens of layers before reaching the router, is already a highly structured representation space. A single linear projection is sufficient for the final "routing decision."

From another angle: with 896 experts, expecting a simple linear classifier to correctly assign each token to "exactly the right 16 experts" is clearly over-idealized. The router's real job isn't "find the best 16" — it's "under load-balancing constraints, give most tokens a reasonable set of 16."

This is why K3 invested massive engineering effort in load balancing —

Quantile Balancing: The Load-Balancing Challenge of 896 Experts

MoE routing training has a classic dilemma:

Don't intervene in routing -> tokens flood a few experts -> GPU load imbalance -> some experts starve (insufficient training data)
Force balance -> same input routed to different experts across training steps -> experts learn highly similar things -> pseudo-balance

K3's solution is Quantile Balancing: at each training step, compute quantile thresholds from the router score distribution — experts above this threshold receive a target number of tokens. Then update per-expert selection bias based on deviation from the target.

Crucially: this bias is only used during training. Fixed at inference.

This means K3 has already gained complete experience during training on "how to achieve load balance among 896 choices," and at inference it simply uses a fixed bias to prevent catastrophic load skew.

Path 2: Latent Compression + 2 Shared Experts

Another key design in Stable LatentMoE: before entering routing experts, compress the 7168-dimensional hidden state into a 3584-dimensional latent space.

x (7168) -> W_down -> z (3584) -> 16 routing experts -> u (3584) -> W_up -> RMSNorm -> output

Why? Two reasons:

Reduces per-expert parameter count and compute (input dimension halved)
Reduces the information dimension the router needs to handle

But compression loses information. So K3 retains 2 always-activated shared experts — they operate on the full 7168-dimensional hidden space, without latent compression.

Final output:

y = E1_shared(x) + E2_shared(x) + W_up(RMSNorm(u))

The 2 shared experts handle "foundational features everyone needs." The 16 routed experts handle "differentiated features specific to each token." Clean division of labor.

Path 3: 1.8% Activation Rate — It's Not a Bug, It's a Feature

Back to that number: 896 experts, only 16 activated. 1.8%.

Why not more? Say, 32 (3.6%)?

K3's technical report doesn't directly provide this ablation, but we can infer from the architecture:

Latent space capacity bottleneck. Latent dim = 3584. 16 experts already approach this dimension's information saturation. Adding more experts -> additional FFN capacity can't be effectively transmitted through the latent bottleneck -> diminishing marginal returns.
Cross-GPU communication cost. 896 experts are distributed across multiple GPUs. Each additional activated expert means more all-to-all communication across GPUs. K3's official docs explicitly recommend 64+ accelerators for deployment — when you're already doing all-to-all on a cluster that large, each extra expert adds real latency.
DeepSeek-MoE already validated: fine-grained + fewer activations > coarse-grained + more activations. Rather than activating 40 experts each learning a little, let 16 small experts each learn their part well.

1.8% isn't a design target — it's the optimal convergence point shaped jointly by the latent bottleneck, communication costs, and expert granularity.

4. The Evolution at a Glance: From Dense to 896-Expert

Year    Model                  Routing Strategy    Experts/Layer    Active    Shared Experts

2021    Switch Transformer     Top-1               64-2048          1         None
2023    Mixtral 8x7B           Top-2               8                2         None
2024    DeepSeek-MoE 16B       Fine-grained Top-K  ~64              ~6        K_s
2024    DeepSeek-MoE 145B      Fine-grained Top-K  ~192             ~18       K_s
2026    Kimi K3                Quantile Top-16     896              16        2

Clear trends:

Expert count is increasing, but active count stays in a stable range (1-16) — not because 16 is a magic number, but because the latent bottleneck and communication costs impose an invisible ceiling
Shared experts are a watershed — nobody did them before DeepSeek; K3 directly inherited the approach
Load balancing evolved from "aux loss assist" to "quantile direct control" — control precision went from "penalize imbalance" to "direct allocation"
Each expert is getting smaller — from a few large experts to hundreds of small ones, routing's meaning shifts from "domain classification" to "feature assembly"

5. K3 MoE's Real-World Performance

Data extracted from the official blog and technical report:

Total params: 2.78T, active params: 104.2B (roughly 3.7%)
Inference efficiency: Despite the routing overhead of 896 experts, K3's team claims 90%+ cache hit rate (in coding scenarios) through the Mooncake decoupled inference architecture, serving million-token contexts at $0.30/M tokens (cache-hit pricing)
Training efficiency: Introduced fully balanced expert-parallel training with static shapes and no host-synchronization critical path, making 896-expert scale training feasible

More importantly, the synergy between MoE and other innovations: KDA (from Part 1) + Attention Residuals (mentioned in Part 1) + Stable LatentMoE (the protagonist of this article) -> Kimi K3 achieves roughly 2.5x overall scaling efficiency improvement over K2.

This isn't one component's magic — it's three orthogonal optimization dimensions stacked together: cheaper long contexts (KDA), smarter inter-layer communication (AttnRes), and compute concentrated on the most needed parameters (LatentMoE).

6. What's Next: MoE on the Road Ahead

K3 already does conditional computation at the FFN layer — each token uses a different FFN expert combination. This is just the first step:

Current K3 is still a 93-layer fixed-depth Transformer — every token, regardless of difficulty, must traverse all 93 layers. Possible evolutionary directions:

Dynamic depth: Simple tokens exit early, difficult tokens go deeper (ADEPT, 2026)
MoE-ified Attention: Not all tokens need full attention — SALSA uses a router to decide between attention and recurrent (2026)
Layer-level MoE: Different layers may have different expert architectures — input-adjacent layers use shared-type, output-adjacent layers use routing-type
From FFN routing to full-module routing: Not just choosing experts, but choosing "should this layer use attention, SSM, or skip" — turning the model into a token-level dynamic computation graph

If MoE's essence is "most parameters stay idle," these directions' essence is "most computation stays idle." The engineering implementations are entirely different, but the philosophy shares the same origin.

7. What This Means for You

Three "Don'ts":

Don't understand MoE through "domain classification." Thinking in terms of "math expert" and "code expert" misses the essence. Experts are FFN fragments — what the router allocates isn't semantic roles but compute resources. This understanding directly determines whether you can design efficient routing strategies.
Don't equate "more experts" with "better." K3 uses 896 not because it "needs 896," but because under the constraints of fine granularity + latent compression + load balancing, 896 is the optimal tradeoff for training and inference efficiency. Blindly adding experts -> load balancing collapse -> pseudo-balance -> wasted parameters.
Don't treat MoE as an isolated module. K3's 2.5x scaling efficiency improvement comes from the synergy of KDA + AttnRes + LatentMoE. The relationship between MoE and attention mechanisms isn't "this replaces that" — it's "simultaneous optimization across multiple dimensions": horizontal (sequence attention compression), vertical (inter-layer information flow), and orthogonal (selective parameter activation).

Three Numbers to Remember:

1.8% — K3's FFN activation rate per token. The current sparsity-efficiency ceiling for the strongest open-source model.
28.5% — The compute fraction DeepSeek-MoE 145B needs to match DeepSeek 67B Dense. The compute-efficiency baseline for fine-grained MoE.
90% — K3 API's cache hit rate in coding scenarios. Without this level of cache optimization, a 2.78T model could never achieve $0.30/M token pricing.

References

Kimi K3: Open Frontier Intelligence — Kimi Team, July 2026, technical report + open weights
Kimi K3 Tech Blog — Moonshot AI official blog
K3's router picks 16 of 896 experts: an allocator, not a classifier — lilting channel, deep architecture analysis
DeepSeekMoE: Towards Ultimate Expert Specialization — Damai Dai et al., ACL 2024
Switch Transformers: Scaling to Trillion Parameter Models — Fedus et al., JMLR 2022
Mixtral of Experts — Jiang et al., Mistral AI, 2024
The Illusion of Specialization — ACL 2026, analysis of actual MoE expert behavior

This is Part 3 of the "Evolution of Attention Mechanisms" series and the first installment on the "orthogonal dimension." Together, the three parts form a complete coordinate system: horizontal (sequence attention) -> vertical (inter-layer communication) -> orthogonal (selective parameter activation). Coming next: if MoE decouples "parameter count" from "inference cost," what's the next dimension to be decoupled?

Depth-Attention: Opening an Attention Channel Between Transformer Layers

AICDragon — Mon, 10 Aug 2026 12:50:49 +0000

This is the second article in my Attention Mechanism Evolution series. The first covered the horizontal (sequence-dimension) attention optimizations — from GPT-2's full attention to Kimi K3's KDA hybrid architecture. That was one axis. This article tackles the vertical axis: can deeper layers selectively attend to shallower layers, instead of just blindly summing via residual connections?

Depth-Attention, proposed by Shanghai Jiao Tong University's LUMIA Lab (arXiv: 2606.05014, accepted at ICML 2026), answers with an elegant design: zero new parameters, zero additional KV cache, under 0.01% extra FLOPs — and a consistent +2.3 point accuracy boost on downstream tasks.

1. The Problem: Transformers Are "Blind" Between Layers

Standard self-attention excels at freely selecting information along the sequence dimension — each token can attend to any position in the sequence.

But switch to another dimension — between layers — and Transformers become startlingly inefficient:

Layer ℓ-1 output → (residual connection: simple summation) → Layer ℓ input

Each residual block just adds the previous layer's output on top of its own — no weighting, no selectivity, no attention. The deeper layers face a hidden state that's just a compressed dump of all prior layer outputs.

Think of it like this: you write a report and hand each page to a colleague, but they only see the last page. They can't tell whether "page 3's analysis is your best work — read more of it" or "page 15 isn't that important — just skim it."

This isn't a new problem. Recent work has tried to solve it — DenseFormer (learns fixed inter-layer weights for weighted averaging), Hyper-Connections/mHC (multiple parallel residual streams replacing single residual), and Attention Residuals (using softmax attention to adaptively select shallow-layer outputs per token — this is what Kimi K3 uses).

But these approaches share a problem

They all operate on hidden states — the model's full intermediate representations, outside the self-attention module itself. This means they need to retain or access these hidden states in addition to the KV cache during inference.

Meanwhile, the trend in large models (GQA, MLA) is to aggressively compress the KV cache — DeepSeek uses MLA to compress from 128×d down to 4×d. Adding extra persistent states outside the KV cache? That's swimming against the current.

Depth-Attention's insight is exactly this contradiction: can we leverage the Q, K, V already inside the attention module, in the same place, without any external state, to achieve cross-layer selective mixing?

2. How Depth-Attention Works: Rotating "Sequence Attention" by 90 Degrees

The answer is remarkably simple.

Standard self-attention operates on the sequence dimension — "the current token attends to all tokens in the sequence."

Depth-Attention rotates this operation 90 degrees — performing the exact same thing on the depth dimension:

For layer ℓ, token t:
q_ℓ^t · k_j^t  (current layer query × each shallower layer's key, at the same token position)
  → softmax → depth attention weights α
  → weighted mixture ṽ (depth-mixed value)

The final step is key — everything in self-attention stays unchanged (no Q modification, no K modification, no mask change), only the value is replaced with the depth-mixed version:

O_ℓ = CausalAttn(Q_ℓ, K_ℓ, Ṽ_ℓ)   # Ṽ_ℓ replaces the original V_ℓ

Three elegant properties:

1. Zero new parameters. Depth-Attention fully reuses the Q and K projection matrices already in standard self-attention — the same Q serves both sequence-direction and depth-direction attention. No new parameter matrices at all.

2. Zero additional KV cache. The depth-mixed Ṽ has exactly the same shape as the original V (T×d). During inference, it replaces rather than appends — Ṽ goes into the original V cache slot. Subsequent tokens reading that slot automatically get the depth-mixed version — so the persistent state at inference is identical to a vanilla decoder's KV cache. Not a single byte extra.

3. Recursive information propagation — the paper's most ingenious design. Let me write out the formula clearly:

ṽ_ℓ^t = α_ℓ,ℓ · v_ℓ^t  (current layer's own value)
       + Σ_{j<ℓ} α_ℓ,j · ṽ_j^t  (previous layers' already-mixed values)

Notice the second term: the mixture uses not the raw v_j, but the already depth-mixed ṽ_j. This means information propagates cascadingly — ṽ_4 contains all mixed info from ṽ_0 through ṽ_3. When ṽ_16 reads from ṽ_4, it gets not just layer 4's raw value, but depth-mixed information already processed by layer 4.

If it mixed raw v_j, layer 10 could only directly access layer 2's value — with 8 layers of residual stream attenuation in between, the signal is highly degraded. But mixing ṽ_j means: layer 6 reads layer 4 → layer 4's ṽ already contains layer 0's info → layer 10 reads layer 6 → it also indirectly gains contributions from layer 0 and layer 2.

A recursive structure achieves efficient propagation. This allows shallow representations to penetrate through many intermediate layers and continue influencing deep layers.

3. Counter-Intuitive Ablation: More Layers Isn't Always Better

If every layer attended to all shallower layers, the compute cost would be O(TL²) — non-trivial in deep networks, especially with pipeline parallelism's cross-device communication overhead.

Depth-Attention addresses this with strided sampling: each layer only attends a sparse subset — itself + {0, s, 2s, ...} among the shallower layers.

With s=4, layer 20 attends {20, 16, 12, 8, 4, 0}, skipping the 14 layers in between.

Complexity drops to O(TL/s); with s large enough, it approaches O(T), completely negligible.

But the most counter-intuitive result is the ablation study: the paper tested different strides. The result was not "more is better." s = L/2 (half the total layers) performed best — better than denser s=L/4 and sparser s=L.

Why? The paper doesn't give a definitive explanation, but I think this phenomenon hints at an important principle: there may be an optimal "reception interval" for information propagation across depth — too dense creates redundancy (adjacent layers' values are highly correlated), too sparse loses critical information. s=L/2 happens to provide enough "diversity windows" without falling into the trap where "layer 20's info is highly redundant with neighboring layers 16 and 14."

This echoes the same design philosophy we saw in part one with Kimi K3's KDA hybrid architecture: it's not about stacking more — it's about choosing more wisely where to apply it.

4. GQA Compatibility: Naturally, Freely More Efficient

Modern large models widely use Grouped-Query Attention (GQA), where g query heads share one set of KV heads.

Depth-Attention handles GQA extremely naturally: average the g queries within each group, running depth attention at KV head resolution — not query head resolution.

This brings two additional benefits:

KV head dimension is already several times smaller than hidden size (typically 4x) — running depth attention in this space further shrinks compute and memory
No head-dimension expansion or alignment needed — just average g queries, zero hyperparameter changes

In other words: Depth-Attention is not only overhead-free in GQA scenarios, it's actually more efficient than in non-GQA scenarios — because depth attention's operating space is naturally compressed by GQA.

5. Experimental Results: What's the Actual Gain?

The paper ran experiments on Qwen3-style decoder architectures at 1.5B and 3B scales. All models were trained from scratch on 32B tokens from the Pile, with identical data and hyperparameters — a fair comparison.

Baselines included: Vanilla Transformer, mHC (manifold hyper-connections), Attention Residuals (Kimi K3's approach), and DenseFormer.

Key results

At 1.5B scale, zero-shot:

Vanilla Transformer average downstream accuracy: 51.26
Depth-Attention: 53.56 (+2.3 points)
Attention Residuals and mHC fall between Vanilla and Depth-Attention

At 3B scale, the gap widens — Depth-Attention reduces perplexity to 6.66 (Vanilla: 7.10), and average accuracy improves from 53.08 to 55.27 (+2.19 points), comprehensively beating all baselines.

Under 5-shot settings, the pattern holds — Depth-Attention is best at both scales.

Notably: Attention Residuals, as the strongest baseline, does significantly beat Vanilla — the paper acknowledges this clearly — but Depth-Attention surpasses it on every metric, and does so without adding any inference state.

Efficiency: theory vs. measurements

Theoretical FLOPs: The paper's Appendix B provides detailed derivations of extra FLOPs for each method. Counting per decoder layer:

Depth-Attention extra compute: ≈ 2Td/s FLOPs (s = stride, d = head dim)
DenseFormer extra compute: ≈ T·d_model·L (weighted summation across all layers)
Attention Residuals extra compute: ≈ T·d_model·L (similar magnitude)

Plugging in typical values (T=4096, d_model=2048, L=32, s=16), Depth-Attention's extra FLOPs are less than 0.01% of self-attention FLOPs.

Training wall-clock: On a 3B model, Depth-Attention adds only ~1% per-step training time — significantly less than DenseFormer and Attention Residuals.

Inference throughput: At 128K token prefill, Depth-Attention throughput is essentially identical to Vanilla (<0.5% difference). Hidden-state methods, by contrast, need explicit extra state management in long-context scenarios, with noticeably higher memory usage.

Scaling experiments

The paper tested four scales from 360M to 3B. Depth-Attention maintains its advantage at all scales, with gains showing no sign of saturation. This suggests larger models may also benefit — though this hasn't been verified at 70B+ scale (a limitation the paper acknowledges).

6. Weight Visualization: What Did the Model Learn?

The paper visualizes the trained Depth-Attention weight distributions, with several interesting findings:

1. Early layers focus their depth attention on deeper information sources, not shallower ones.

Concretely: Layers 1-8 (early layers) have self-attention weights dominating in depth attention (α_ℓ,ℓ close to 1) — they don't look much at shallower layers, because there simply aren't enough shallower layers.

But layers 9-32 (the second half) start dispersing their depth attention weights: deep layers still retain large self-attention weights (preserving their own features) but radiate uniformly toward middle layers.

2. Shallower isn't necessarily more important.

Intuition might suggest "shallow features are more fundamental, deep layers should attend more to them." But from the weight plots, deep layers (e.g., layer 30) don't assign higher attention weights to very shallow layers (layer 0, 4) than to middle layers (layer 12, 16).

This hints at an interesting learned strategy: the model learns to extract information at different granularities from different depth levels, rather than simply treating shallow layers as an "information repository."

3. Some attention heads are more cross-layer-dependent than others.

Different heads show different degrees of dependence on depth information. Some heads have self-attention weights close to 1 at almost all layers (barely looking at other layers), while others have dispersed weights across the entire depth dimension (actively leveraging cross-layer information). The model thus gains flexibility — use "self-attention-heavy" heads when local fine-grained processing is needed, and "cross-layer-heavy" heads when contextual fusion is needed.

7. Looped Transformer Experiment: Parameter Sharing Works Too

This is one of the paper's most profound experiments. A Looped Transformer has only a few physical layers (e.g., 4 layers) that execute in a loop (e.g., 8 cycles = 32 total effective layers), with all cycles sharing the same parameters.

Depth-Attention's gains persist in this setting — each loop cycle acts as a new "logical layer" that can attend to the values from its own previous cycles.

This result rules out a possible explanation: "Depth-Attention works because different layers have different parameters, so their value semantics differ, making them worth retrieving." In the parameter-sharing scenario, values from different layers are still selectively passed through depth attention — proving that the mechanism's benefit doesn't depend on "different parameters per layer," but rather that selective cross-depth information transfer is inherently valuable.

Taking it deeper: this result hints at a potential relationship between Depth-Attention and recurrence/SSMs. If parameter sharing works, what Depth-Attention is doing is somewhat like a soft state-space model — selectively remembering and retrieving depth-history information in the value dimension, rather than propagating through fixed residuals.

8. Depth-Attention vs. Kimi K3's Attention Residuals: A Design Philosophy Divergence

This is the most fascinating comparison — they solve the same problem but choose different positions.

	Attention Residuals	Depth-Attention
Operation point	On residual stream (outside module)	Inside attention (value position)
Target	Hidden states	V cache slots
Extra inference state	Required (retain shallow hidden)	None
Extra parameters	Yes	Zero
Information granularity	Full hidden state (rich)	Value state (compressed)
Performance (paper comparison)	Better than vanilla	Better (at 1.5B/3B)

Both share the same conceptual origin — enabling deeper layers to selectively attend to shallower ones. The difference is in the tradeoff:

Attention Residuals operates at the most information-rich position (hidden state), but at a cost
Depth-Attention operates at the cheapest position (V cache), at zero additional cost

This divergence doesn't have a final answer yet: who wins at 70B+ scale? Nobody knows. But one thing is clear — Depth-Attention found a "free lunch" path. It proved that you don't need to add anything outside the KV cache to get all the benefits of cross-layer selective information transfer.

This isn't a free lunch — it's the lunch you were already eating, you just didn't realize you could eat it better.

9. What This Means

Architecture evolution: one horizontal, one vertical

The first article covered sequence-direction (horizontal) attention evolution from full attention to KDA — that was "how to remember more information with less state."

This article completes the inter-layer (vertical) direction — "how to let deep layers selectively use shallow representations without paying extra."

Put together:

      ← Sequence direction (horizontal) →
      Full Attn → KV Cache → Linear Attn → DeltaNet → KDA
      (compute → memory → compression → precision → hybrid recipe)

   ↑
   Inter-layer direction (vertical)
   Residual → DenseFormer → Attention Residuals → Depth-Attention
   (sum → weighted → attention → zero-cost attention)
   ↓

For model users:

Next time you see a model claiming "cheap long context," don't just look at parameter count and context window — ask "what's your inter-layer communication scheme? How does information flow in the depth direction?"
Kimi K3 uses Attention Residuals; the next version might switch to Depth-Attention or a variant — keep an eye on this thread

For model trainers:

Depth-Attention's zero-parameter nature means a lossless drop-in replacement: no weight changes, no cache changes, no pipeline changes — just modify a few lines of attention code and reliably gain ~2 points
The paper's code is open-source on GitHub, with a pre-trained 3B checkpoint on HuggingFace for verification

For technical investors:

The design philosophy divergence (hidden state vs. value state path) could influence next-generation model architecture choices
Inter-layer communication is becoming a core optimization dimension alongside "attention compression"

10. Limitations (Frankly)

Limited validation scale. The 1.5B and 3B experiments are convincing (32B tokens trained from scratch, 8 downstream tasks), but whether results replicate at 70B+ scale remains unknown. The paper acknowledges this.
Mechanism explanation for optimal stride is missing. s=L/2 working best is an empirical finding — why this value, and whether better non-uniform sampling strategies exist, leave significant analysis space.
Interaction with other optimizations under-explored. Effects when combined with MoE (e.g., K3's 896 experts), compatibility with MLA, differential behavior in prefill vs. decode phases — all await follow-up work.
Is the "dual attention" structure of depth + sequence optimal? This is an architectural question — perhaps a future unified primitive will fuse both directional attentions into one operation. Depth-Attention is an important step, but not necessarily the last.

References

Depth-Attention: Cross-Layer Value Mixing for Language Models — Boyi Zeng et al., Shanghai Jiao Tong University LUMIA Lab, ICML 2026
Code & Models — GitHub + HuggingFace (3B checkpoint available)
Attention Residuals — Kimi Team, 2026 (cross-layer mechanism used in Kimi K3)
DenseFormer — Pagliardini et al., NeurIPS 2024
Hyper-Connections / mHC — Zhu et al., 2025
GQA — Ainslie et al., EMNLP 2023

This is the second article in the "Attention Mechanism Evolution" series. Part one covered the horizontal (sequence-direction) evolution — from GPT-2's full attention to Kimi K3's KDA hybrid architecture, with cross-validation across 5 papers, a complete evolution table, and code-level analysis. Part three preview: from sparse attention to MoE — why K3's 896 experts only activate 1.8%.

5 LAN File Transfer Tools Compared: Up to 40x Faster Than Cloud, Two Don't Even Need Installation

AICDragon — Sun, 09 Aug 2026 14:34:08 +0000

TL;DR: Transferring a 1GB file over your home Wi-Fi? Cloud-based messengers take ~3 minutes. LAN tools do it in under 5 seconds. This article synthesizes benchmarks from MakeUseOf, XDA Developers, DEV Community, and Speedyshare - plus five GitHub issue deep-dives - to give you one clear recommendation per use case.

First, the Numbers

Here's a fact you might not believe: on your home Wi-Fi, sending a 1GB video file to your laptop via WeChat/WhatsApp takes about 3 minutes. With a LAN direct-transfer tool under the same conditions? Under 5 seconds.

That's a ~40x gap.

Here's why: cloud messengers route through remote servers, bottlenecked by your upload bandwidth (typically 30-50 Mbps on residential connections). LAN tools go device-to-device over local Wi-Fi, routinely hitting 200-400 Mbps or higher.

This article has one goal: find the right LAN transfer tool for your situation - so you can downgrade your messenger's file transfer to "meme and screenshot duty only."

Three Things Before We Start

First, data provenance. This isn't a single-machine benchmark. Speed data comes from published tests by MakeUseOf (2025-2026), Speedyshare (2026), XDA Developers (July 2026), DEV Community (April 2025), and Brave2049 (2025). I also combed through five GitHub repositories' issue sections for long-term community sentiment. This article's value is "synthesizing scattered information to help you decide" - not "running benchmarks for you."

Second, this isn't a tool catalog. Plenty of articles list these tools. They rarely offer a decision framework. This one maps six common scenarios to one clear answer each.

Third, nomenclature. "AirDrop" means Apple's AirDrop. "Quick Share" means the Android/Windows equivalent. When I say "LAN tools," I mean any tool that transfers files directly over local Wi-Fi without routing through an external server.

The Five Contenders

Tool	Type	GitHub Stars	Core Advantage
LocalSend	Native App	86,000+	Most stable, fastest, safest. Open-source AirDrop for everything
PairDrop	Browser	Snappedrop fork	Zero installation. Open a URL, start transferring
Blip	Native App	Newcomer (2025)	Speed dark horse, no file size limits
Snapdrop	Web	20,000 (legacy)	Former gold standard, now acquired by LimeWire
ShareDrop	Web	Classic project	Only one with multi-user room sharing

These five fall into two camps: native apps (LocalSend, Blip) - fast but requires installation; browser tools (PairDrop, ShareDrop) - zero friction but speed-capped by the browser runtime. Snapdrop is its own category: it was the browser champion. It isn't anymore.

The Deep Dive

LocalSend: If You Only Install One, Make It This

86,000+ GitHub stars. Apache 2.0 license. Built with Flutter - native clients for Windows, macOS, Linux, iOS, Android. Once installed, devices on the same Wi-Fi appear automatically. One tap to send, one tap to receive. Feels almost exactly like AirDrop.

On security, LocalSend is the most serious LAN transfer tool I've seen. Each device locally generates TLS/SSL certificates on the fly. Transfers use HTTPS encryption. Files never touch any external server - you can even use it with the internet disconnected (LAN-only). In GitHub issue #979 (2024), a user asked about transfer speed; the developer responded that stability was prioritized over raw speed, and community reaction was overwhelmingly positive.

Real-world speeds: DEV Community ran a 50GB / 100-file mixed stress test in April 2025 across five different systems (MacBook Pro M3, iPhone 16, Pixel 9, Ryzen 7 Windows desktop, ThinkPad X1 Ubuntu). On Wi-Fi 6, LocalSend's cross-platform throughput held steady at 200-400 Mbps. Translation: a 1GB file transfers in 3 to 15 seconds. Compared to Snapdrop's WebRTC approach in the same test, LocalSend was roughly 3-5x faster.

The one downside: every device needs the app installed. For your own devices, this is a non-issue - install once, done forever. But if you're sending a file to someone one-off, the "install this app first" friction kills the experience. That's where PairDrop comes in.

One more tip: LocalSend's default discovery setting is visible to everyone on the same network. If you're at a coffee shop or corporate Wi-Fi, go to Settings and turn off "Allow discovery by everyone" or enable "Known devices only." Security is fine (transfers are encrypted) - this is about privacy.

PairDrop: The Zero-Install Plan B

PairDrop has a story. Its predecessor, Snapdrop, was once the name in LAN transfers - open a website, auto-discover, send. 20K stars. Clean and simple. Then, in February 2025, LimeWire (yes, that LimeWire from the P2P music era) quietly acquired Snapdrop.

GitHub Issue #651 "Intransparent switch to LimeWire" got 58 thumbs-ups. Issue #663 "SnapDrop.net has been replaced by PairDrop.net" got 44. The community reaction was crystal clear: we trusted Snapdrop because it was open-source, peer-to-peer, zero-server. Now LimeWire handles your data - and no one knows what their data retention policy is.

PairDrop was born in this moment. Community members forked Snapdrop's pre-acquisition source, renamed it PairDrop, and kept it running as pure WebRTC P2P. Functionally identical: two devices open pairdrop.net, auto-discover, send. WebRTC DTLS encryption, files never leave your LAN. Two bonus features: device pairing (lock to a specific device - no broadcasting to the entire network) and temporary text messaging.

Speed-wise: WebRTC in a browser has inherent limits. Typically 50-100 Mbps. Slower than native apps, but perfectly fine for one-off use - a 1GB file takes 1-2 minutes, still about twice as fast as cloud messengers, and zero compression.

Use case: you need to send a big file to a friend, colleague, or client. They don't need to install anything. Send them a URL.

Blip: The Speed Dark Horse (Not Yet Fully Proven)

A 2025 newcomer. blip.net offers native clients for all platforms. Three selling points: "no file size limits," "folder transfer without zipping," and "resumable transfers."

MakeUseOf ran a LocalSend vs Blip vs PairDrop speed comparison in December 2025. The result surprised a lot of people: Blip was the fastest of the three in certain transfer scenarios. XDA Developers published "I tried replacing Quick Share with LocalSend, PairDrop, and Blip - only one became my go-to" in July 2026. The author chose Blip as their daily driver, noting "speed is absurd" and "folder transfer is the killer feature you can't go back from."

But Blip has one critical uncertainty: it's not fully open-source. Compared to LocalSend's 86K stars of community trust, Blip's transparency is significantly lower. If "where does my file go?" matters to you, LocalSend is the safer bet.

Use case: you frequently transfer very large files or entire folders (video footage, VM images, datasets) and you don't mind using a partially closed-source tool.

Snapdrop: Don't Use It. Seriously.

The first half of Snapdrop's story was told above. Here's the post-acquisition reality: file transfers now route through LimeWire's cloud infrastructure instead of staying purely on your LAN. The acquirer has published no data retention policy. No clear privacy statement.

This isn't speculation. Speedyshare wrote in July 2026: "Snapdrop now routes through Limewire's cloud after its 2025 acquisition, so treat it with caution."

If you have Snapdrop bookmarked, delete it. Replace with pairdrop.net. Same functionality. Your files stay on your network.

ShareDrop: The "Conference Room" Option

Web-based tool, same WebRTC foundation as PairDrop. The one and only differentiator: room functionality. Create a room, multiple people join, anyone drops files in, everyone sees them in real-time.

Two ideal scenarios: small-team offices (no need for individual transfers) and remote collaboration sessions (PairDrop works too, but a shared room is more convenient with multiple people).

Note: ShareDrop's room is for discovery and organization only. File transfer is still WebRTC P2P - nothing goes through a server. This is fundamentally different from Snapdrop's cloud routing.

One Limitation Every Tool Shares

You must be on the same Wi-Fi network.

LAN transfer tools work via device-to-device direct connection. They discover each other through local network broadcast. Transfers happen over Wi-Fi private IPs. No internet involved. If you're not on the same network, all five tools do nothing.

This is a physical constraint, not a product flaw. But it's the #1 source of confusion for first-time users: "Why can't my phone on the subway reach my office desktop?" Because you're not on the same LAN.

If you need cross-network transfers (office desktop ? home phone), you need cloud-relay or NAT traversal solutions. That's a different product category entirely.

My Actual Setup: Two Tools Is All You Need

After two years using these tools daily, my setup is dead simple:

LocalSend on three primary devices (phone, laptop, desktop). Daily photos, screenshots, documents, archives - arrive in seconds. Two taps per transfer.
PairDrop as occasional backup. One-off big file sends to other people. They open a URL in their browser. From "let me send you something" to "got it" averages 15 seconds.

Messengers for file transfer? Downgraded to meme and screenshot channels only.

Key Takeaways

LAN transfer tools are 5-40x faster than cloud messengers - and they don't compress quality. The speed gap aside, cloud messengers apply lossy compression that degrades images and video. LAN tools transfer raw, bit-for-bit.
For your own devices: LocalSend. 86K stars, fully open-source, all platforms supported, encrypted. Install once, done for life. The safest choice in this category.
For sending to others: PairDrop. Zero-install means the "let me send you a file" conversation goes from "hold on... install this... did you find it yet..." to "open this URL." Simple to the point of being undeniable.
Snapdrop is no longer the Snapdrop you remember. Post-LimeWire acquisition, files travel through their cloud. Privacy policy unknown. Switch to PairDrop - it's what Snapdrop was when it was still clean.
Pick the right tool, not the most tools. For most people, LocalSend alone does the job. Don't fall into the "try every tool" trap. Using the right one beats trying them all.

References

Speedyshare: Snapdrop vs LocalSend vs PairDrop (2026) - Feature comparison and scenario recommendations
DEV Community: LocalSend vs AirDrop vs SnapDrop vs PixelDrop - 100-file / 50GB cross-platform throughput benchmark
MakeUseOf: I speed-tested LocalSend, Blip, and PairDrop (2025) - Source of Blip surpassing LocalSend in speed
XDA: I tried replacing Quick Share with LocalSend, PairDrop, and Blip (2026-07) - Author's final pick for Blip
Linuxadictos: LocalSend vs Warpinator real-world comparison - Linux-specific comparison
Brave2049: LAN cross-platform file transfer comparison (2025) - Snapdrop/ShareDrop/LocalSend 3D comparison
GitHub: Snapdrop Issue #651 - Intransparent switch to LimeWire - Community reaction to LimeWire acquisition
GitHub: Snapdrop Issue #663 - SnapDrop.net has been replaced by PairDrop.net - Community migration direction
LocalSend GitHub - 86K+ stars, Apache 2.0
PairDrop - Snapdrop community fork, pure browser solution
Blip - Native app, no file size limits

I Talked to an Elon Musk AI About Product Management — Here's What It Said About Surviving the AI Wave

AICDragon — Sat, 08 Aug 2026 16:12:43 +0000

I Talked to an Elon Musk AI About Product Management — Here's What It Said

"The only rules you have to follow are the laws of physics — everything else is a recommendation."

Lately, I've been seeing a wave of "personality skills" for AI assistants — distillations of bosses, colleagues, exes, mentors. You name it, someone's bottled it. So I installed an Elon Musk Skill into my AI assistant and asked it one question:

"How does an AI product manager break out in this market?"

What came back wasn't a listicle. It was a first-principles operating system. Here's what the Musk skill said, verbatim.

First, Let's Calculate the Asymptotic Limit of an AI PM

A perfect AI PM, in theory, should be able to convert user needs into scalable AI capabilities with zero friction. No need to write code — but understands technical boundaries. No need to do research — but understands model capabilities. No need to run operations — but understands data flywheels.

Reality check. What are most AI PMs actually doing?

Waiting for the model team to publish API docs
Copying competitor feature lists
Writing PRDs and waiting for review
Staring at dashboards waiting for "insights"

The idiot index on this is at least 10. Too many middle layers, and every single one is taxing you for information opacity.

Where Does Your Edge Come From?

1. Vertically Integrate Your Capability Stack

Here's how most people approach becoming an AI PM:

Take a course → Get certified → Apply for jobs → Wait for interviews → Get hired → Learn the tools

This is outsourced thinking. Every layer depends on someone else defining the standard.

Here's the vertically integrated path:

Build something real with an API → Hit genuine problems → Read the docs to solve them → Iterate → Iterate again

It's not "learn AI product management." It's "build products with AI."

Action: This week, use any API — OpenAI, Claude, whatever — to build something that solves your own problem. Even if it's just a script that auto-organizes your inbox. Manufacturing is 10x harder than designing. You'll learn more from building one thing than from reading ten "AI PM starter guides."

2. Question the Need to "Learn" in the First Place

Who told you that you need to "learn" before you can start?

When I started SpaceX in 2002, nobody said "you should go study rocket science first." I read books. I asked people. Then I started designing rockets. The first three exploded. The fourth one worked — and landed a $1.6 billion NASA contract.

AI products are much simpler than rockets. Nobody dies when an API call fails. Wrong prompt? Tweak it and try again.

Your current state: "I need to learn first, then I'll start building."

The correct state: "I start building, and I learn what I need along the way."

Order matters. Getting it backwards costs you a 10x efficiency penalty.

3. Find Your Battery Factory

Tesla's turning point wasn't the Roadster. It was the decision to build their own battery factory.

Everyone said: "Just buy battery cells from suppliers. Why would you manufacture your own?" Because I did the math. The supply chain markup was absurd.

What's your battery factory?

Surface Skill	Deep Moat
Knows how to use ChatGPT	Understands the token-cost-to-latency tradeoff
Can write a PRD	Can calculate the unit economics of an AI feature
Studies competitors	Has proprietary datasets and evaluation benchmarks
Can craft a prompt	Knows model capability boundaries and failure modes

Most people stop at the surface. Because going deep takes time — you have to calculate, build, fail, and recalculate. That's exactly why it's a moat.

4. Rebuild Your Prep Pipeline with The Algorithm

Here's what you're probably doing right now — run through the five-step filter:

Your Plan	The Algorithm Verdict
Take 3 AI courses	Delete. Courses update slower than the industry moves.
Get AI certifications	Delete. Nobody looks at these except training companies.
Read 20 AI books	Delete. Books are already outdated by the time they're published.
Study competitor features	Keep — but change to "reverse-engineer and reproduce."
Learn Axure/Figma	Keep — but only the bare minimum you need.
Grind LeetCode	Delete. PMs don't need this.
Build a real project	Accelerate. This is the only thing that matters.

If you haven't added back at least 10% of what you deleted, you didn't delete enough.

5. Radical Timelines

How long are you giving yourself to "get ready"? Three months? Six months?

Make it two weeks.

Not "two weeks until I get a job." Two weeks until you have something you can show.

Timelines are management tools, not commitments. Crying wolf costs you credibility. But it also forces real speed. That tradeoff is worth it when you're starting out.

Concrete Action Plan

Week 1: Build Something

Pick a problem you actually have (not "something users might want")
Solve it with any AI API
Deploy it — even if it's just on Vercel or Streamlit's free tier
Send it to 5 people, collect feedback

Week 2: Iterate + Reverse-Engineer

Ship one revision based on feedback
Pick 3 competitors. Reverse-engineer their tech stack, cost structure, data flywheel
Write and publish an analysis. Publicly. Start building your brand.

Week 3–4: Amplify

Post your project on Product Hunt / Hacker News / Reddit
Cold-message 3 AI company PMs. Ask if they want to chat.
In interviews, lead with the project. Not your resume.

One Last Thing

You told me: "The AI product space moves so fast. I'm afraid I can't keep up."

That's not a problem. That's your advantage.

In fast-moving industries, experience depreciates fast and newcomers get openings. In slow-moving industries, incumbents hold their positions and you wait a decade for your turn.

When SpaceX entered the rocket industry, Boeing and Lockheed had been doing this for decades. Everyone said: "There's no room for you." We calculated the asymptotic limit: raw materials for a rocket cost about 2% of the final selling price. There was a 50x improvement gap sitting right in front of everyone.

AI is at that exact moment right now.

The gap between what the laws of physics allow for an optimal AI product and what exists today is at least 10x.

Whoever sees that gap first and starts moving — wins.

Not "learn first, then start."

Start, and the learning takes care of itself.

Ship it. Tomorrow.

How This Article Was Written

I installed an Elon Musk Skill into OpenClaw — an open-source AI assistant framework. The skill distills Musk's core mental models (asymptotic limit thinking, the five-step algorithm, vertical integration, etc.) from his biographies, interviews, podcasts, and public statements.

This is not Elon Musk's actual words. It's a thinking framework synthesized from public information, used as a lens to examine career and product decisions from a different angle.

If you use OpenClaw, you can install the skill yourself: github.com/alchaincyf/elon-musk-skill

How a Baseten Engineer Traced 7 Years of Attention Mechanism Evolution -- From GPT-2 to Kimi K3, in Runable PyTorch

AICDragon — Fri, 31 Jul 2026 03:23:15 +0000

Last week, a Baseten inference engineer who goes by @waterloo_intern published a technical blog post titled "22,580: From GPT-2 to Kimi K3, Explained." It hit 2.4 million views in days.

He didn't write a press release. He wrote runnable PyTorch code — starting from GPT-2's attention block, stepping through every architectural change, explaining one problem and one cost per iteration. It's the best transformer lineage explanation I've seen.

I devoured his post, then cross-checked the key claims against 5 original papers. Here's the full picture.

The 22,580x Number

In February 2019, OpenAI released GPT-2 — 124M parameters. Seven years later, Moonshot AI open-sourced Kimi K3 — 2.8T parameters. You could fit 22,580 GPT-2s inside one Kimi K3.

But this isn't a "throw more compute at it" story. It's a story about how we store, update, and retrieve memory.

Starting Point: GPT-2

class Block(nn.Module):
    def forward(self, x):
        x = x + self.attn(self.ln_1(x))
        x = x + self.mlp(self.ln_2(x))
        return x

Every time the model generates a new token, it recomputes Q, K, V projections for all historical tokens, then runs an O(N²) softmax attention. K and V from tokens 1 through N-1? Thrown away. Token N+1 arrives? Recompute everything.

That's why KV Cache was invented.

KV Cache: Store It, Don't Recompute

Simple idea: cache the already-computed keys and values. For the next token, new Q only needs one dot product against the cached K.

Problem solved — but a new one created. KV cache grows linearly with sequence length. At 1M tokens × d_model × layers, that's dozens of GB of VRAM. Every decoding step reads all of it from HBM.

The bottleneck isn't compute. It's memory bandwidth. This is the key to understanding every improvement that follows.

Linear Attention: Fixed-Size Memory

Can we compress O(N²D) into O(ND²)?

The idea: replace softmax with a feature map.

# Standard softmax (must materialize N×N first)
attention = softmax(QKᵀ / √d) × V

# Linear attention (fold K and V first)
Q' = ELU(Q) + 1
K' = ELU(K) + 1
output = (Q'(K')ᵀ × V') / (Q'(K')ᵀ × ones)

Now we can compute K'^T × V' first — a fixed-size D×D matrix — then multiply by Q'. Historical KV information gets "folded" into a constant-size state matrix. Cache no longer grows with N.

The cost? ELU+1 is an approximation of the softmax kernel. Expressiveness drops. But on long-context tasks, this tradeoff is often worth it.

Side Note: FlashAttention Didn't Do This

A common confusion point. Ali's post mentions "in 2020, there was no FlashAttention" — but FlashAttention and linear attention solve fundamentally different problems.

FlashAttention (Tri Dao, NeurIPS 2022) didn't change the attention algorithm. It optimized the GPU IO pattern — tiling the N×N matrix so it never fully lands in HBM. It makes softmax faster, but it's still O(N²).

Linear attention redefined the algorithm itself — replacing softmax with a feature map, going from O(N²D) to O(ND²).

One optimizes IO. One changes the algorithm. Orthogonal.

DeltaNet: You Can Now Edit Memory

Linear attention has a fatal flaw: it can only add, never update.

Every new token piles onto the state: S = S + K'^T × V'. Information only grows. It's like a notebook where you can only write — never erase or correct.

DeltaNet (Songlin Yang et al., NeurIPS 2024) fixes this.

The core idea traces back to Schmidhuber's "Fast Weight Programmers" from the 1990s, later formalized by Schlag et al. (ICML 2021) linking linear attention to fast weights. DeltaNet's approach: before writing, read what this key position currently stores.

v_old = k @ S_old           # read current value at key
delta = v_new - v_old       # compute the delta
S_new = S_old + k^T @ delta # write only the difference

If v_new == v_old, delta is zero — nothing changes. If completely different, delta equals v_new — equivalent to an overwrite. Everything in between is a smooth interpolation.

This is the delta rule: precise memory updates instead of blunt accumulation.

The Parallelization Trick (Ali Said This Took Him 7 Hours to Understand)

DeltaNet's state update is strictly sequential — each step depends on the previous S. It looks impossible to parallelize. But it is.

The method: split the sequence into chunks of size C. Within each chunk, use normal masked attention (GPU-parallel). Between chunks, use the state matrix + one matmul (Q @ S). A Householder transformation reparameterizes the delta updates so all deltas within a chunk can be computed at once.

Complexity: fixed cost 2LD² (state maintenance) + variable cost 2LCD (intra-chunk attention). Bigger C = more variable cost but better GPU efficiency. In practice, C=64 or 128 works best — FLOPs aren't the only metric; tensor core utilization matters too.

Gated DeltaNet: Now You Can Forget

DeltaNet can precisely edit individual key-value pairs. But what about forgetting at scale?

Imagine reading through all documents about Topic A in a 1M-token context, then switching to Topic B. The model should ideally "forget" Topic A to free up capacity for B.

DeltaNet can overwrite specific entries but can't bulk-decay global memory. Mamba-2 (Dao & Gu, ICML 2024) can:

cache = α × S_old + S_new

α is a gating value between 0 and 1, uniformly decaying all old memories. This is the core of Mamba-2's "State Space Duality" theory — softmax attention and SSMs are mathematically the same thing expressed differently. Mamba-2 unified them through gating.

Gated DeltaNet (Songlin Yang et al., ICLR 2025, NVIDIA) merges DeltaNet's delta updates with Mamba-2's gating:

S_new = α × S_old + k^T @ delta   # decay first, then precise write

α=1 is pure DeltaNet. α=0 is a memory wipe. Everything in between both forgets and updates.

The crucial mechanism: a token written at timestep x, read at x+t, has been through t rounds of cumulative α decay (αₓ × αₓ₊₁ × … × αₓ₊ₜ). Information written at different times forgets at different speeds — recent items decay little, distant ones may be fully gone.

KDA: Kimi's Secret Sauce

Kimi Linear (Moonshot AI, arXiv 2510.26692, October 2025) — the predecessor to K3 — refined the idea further: from scalar gating to per-dimension gating.

Gated DeltaNet uses α as a single scalar. One number controlling forgetting speed for all memory dimensions. KDA turns α into a vector (or matrix). Each dimension independently controls its own forgetting rate. Which concepts to retain, which dimensions to decay — the model learns it.

Key data from the paper:

Metric	Full MLA	Kimi Linear
KV Cache	100%	-75%
1M-context decode throughput	Baseline	6x
Short-context performance	Baseline	Outperforms
RL scaling	Baseline	Outperforms

This is the first time linear attention outperforms full attention under fair comparisons across all scenarios. Not just long context — short context too. And that "-75% KV cache" translates directly to inference cost savings.

I verified this claim directly against the paper's abstract: "for the first time, outperforms full attention under fair comparisons across various scenarios." Not marketing. Core conclusion.

K3's Final Hybrid Architecture

K3's technical report (arXiv 2607.24653, July 2026) confirms:

3/4 layers use KDA (linear), 1/4 use gated MLA (full softmax)
MoE routing: 896 experts, 16 active per token (~1.8%), Quantile Balancing for load distribution
Attention Residuals: layers can "look back" at specific earlier-layer representations
MXFP4 weights + MXFP8 activations, quantization-aware training

K3 isn't a pure linear-attention model. It's a hybrid system: KDA handles the bulk processing (cheap, fast, fixed-state), while periodic softmax layers do precision retrieval — recovering details that linear compression might lose. Exactly the tradeoff Ali analyzed: linear attention + periodic softmax retrieval.

So What Does 22,580x Actually Mean?

Ali's conclusion is sharper than anything I could write:

From GPT-2 to Kimi K3, each generation's core improvement wasn't "more parameters" — it was redesigning how memory is accessed: how you store, how you forget, how you retrieve.

The evolution in one table:

Stage	Representative	Memory Mechanism	Problem Solved
1	GPT-2 + KV Cache	Full cache, O(N) growth	Eliminate recomputation
2	Linear Attention	Fixed state, O(D²)	Cache doesn't grow with N
3	DeltaNet	Delta updates, editable	Can't update → can update
4	Gated DeltaNet	Gating + delta, forgettable	Can't forget → can forget
5	KDA / Kimi K3	Per-dim gating + hybrid	Linear surpasses full attention

Parameters grew 22,580×. But if that's all you see, you missed the entire story.

What This Means for You

If you're doing long-context work (code review, document analysis, multi-turn agents), attention architecture directly impacts your cost and output quality.

Cost isn't fixed. Same 1M-token context: KDA's KV cache is only 25% of full attention's. Lower memory bandwidth pressure, significantly better latency and throughput.
Long ≠ expensive. K3's 1M-token context window isn't "brute-forced." 75% of work goes through the linear path.
Hybrid is the trend. Linear won't replace softmax — they'll complement each other. Precision retrieval via softmax, bulk processing via linear. This paradigm will spread.

Key Takeaways

22,580× parameter growth is surface-level. The real story is memory management: full cache → fixed state → precise writes → adaptive forgetting.
The lineage is clear: DeltaNet (NeurIPS 2024) → GatedDeltaNet (ICLR 2025) → Kimi Linear (Oct 2025) → Kimi K3 (Jul 2026).
KDA is the first to surpass full attention in fair comparisons — across short, long, and RL scaling scenarios. Not a "cheaper alternative."
K3's 75% linear / 25% softmax hybrid is an engineering optimum, not a paper-only construct.
Next time you evaluate a model: ask about attention architecture, not parameter count. It matters more.

References

22,580: GPT-2 to Kimi K3, explained — @waterloo_intern, Baseten
FlashAttention (NeurIPS 2022) · DeltaNet (NeurIPS 2024) · Gated DeltaNet (ICLR 2025)
Mamba-2 (ICML 2024) · Kimi Linear (Oct 2025) · Kimi K3 (Jul 2026)

2026 Fields Medal Deep Dive: What Four Mathematicians' Papers Actually Say, and Why AI Needs Math

AICDragon — Thu, 30 Jul 2026 10:06:43 +0000

TL;DR: On July 23, the 2026 Fields Medals were awarded. Wang Hong and Deng Yu — both Peking University alumni — became the first Chinese nationals to win in the same year. But this isn't a news article. It's a heist. Inside these four mathematicians' papers are four ways of thinking you can steal and use today.

The Part That Doesn't Matter

Who won, which institution, how old they are — the news already covered that. Here's the snapshot:

Winner	Institution	Core Contribution
Wang Hong	NYU Courant	3D Kakeya Conjecture (100-year-old problem)
Deng Yu	University of Chicago	Breakthrough on Hilbert's 6th Problem
Jacob Tsimerman	University of Toronto	André-Oort Conjecture
John Pardon	Stony Brook University	3D Hilbert-Smith Conjecture

What I care about: these four people's papers contain four mental models. Models that apply just as well to AI engineering, writing, and product decisions as they do to mathematics.

This isn't "math for beginners." This is "methodology theft."

Wang Hong: A Hundred-Year Problem, Solved with Three Words — Multiscale Analysis

What's the Kakeya conjecture? One sentence: take a unit line segment in 3D space, rotate it 180°. What's the minimum volume it sweeps out? Zero.

The 2D case was solved in 1917 — the area can approach zero. But 3D? Unsolved for over a century. Intuition says the dimension of a 3D Kakeya set should be 3, but there was no rigorous proof.

In 2025, Wang Hong and Joshua Zahl dropped a 127-page paper with 14 figures, rigorously proving: in three-dimensional space, any set containing unit line segments in every direction must have Minkowski and Hausdorff dimension exactly 3.

The core method, in their own words: multiscale analysis.

You don't figure out the arrangement of all tubes in 3D space at once. You split the problem into different scales. At the coarse scale: distribution patterns — which directions cluster together inside a convex set? At the fine scale: local density — if tubes are packed tight in a region, what's the volume contribution? The two scales are handled separately, then stitched together with combinatorial estimates.

Concretely, they studied this: if δ-tubes in 3D can't be packed too many into the same convex set, then their union must have nearly maximal volume. This seemingly narrow intermediate result was exactly enough to derive the full Kakeya conjecture.

The impact goes far beyond pure math. CT/MRI reconstruction algorithms rely on Fourier analysis — the exact domain of the Kakeya conjecture. MIMO antenna beamforming and radar direction estimation directly benefit from this work.

But what you should remember is the method: when facing a complex problem, go coarse first, then fine.

Don't try to figure out "how to double my blog traffic" all at once. First, categorize at the coarse scale: tool reviews get 3-10x more reads than industry analysis, and 95%+ of viral hits come from recommendations. Then fine-tune within each category: how to structure headlines with numbers, what cover style works, what time to publish. Separating scales makes the problem solvable.

Deng Yu: Building the Bridge from Micro to Macro, Over Four Years

Start with an analogy.

The training rule for neural networks is dead simple — gradient descent, nudging a trillion parameters by a tiny amount each step. This micro-rule is boring to the point of absurdity. But after training, the model suddenly reasons, writes code, edits articles. Stare at those trillion parameters — you won't see any of that intelligence hiding there.

This is essentially the same problem Deng Yu studies in wave turbulence theory.

In 2023, Deng and collaborator Zaher Hani published a 138-page paper with 44 figures. What they did: starting from the cubic nonlinear Schrödinger equation (describing light propagation in fiber optics, ocean waves), they rigorously derived the wave kinetic equation under a scaling limit.

The setup: inside a large box (side length L), waves interact very weakly (parameter α). As L → ∞, α → 0, with the scaling relation α ~ L⁻¹, the system's long-time statistical behavior is fully described by the wave kinetic equation — valid all the way up to the kinetic time scale T ~ α⁻².

Translation: countless simple waves, each obeying the same micro-rule → given enough time → predictable macro-level statistical laws emerge automatically.

This framework directly impacts plasma physics (laser-plasma interaction simulations for fusion), atmospheric nonlinear wave modeling in weather forecasting, and oceanography for long-time current evolution.

Together with Hani and Ma Xiao, Deng also made a breakthrough in another direction: starting from hard-sphere dynamics, they rigorously derived the Boltzmann equation on timescales far exceeding previous theorems. This advances the core of Hilbert's 6th Problem (axiomatization of physics, proposed in 1900).

For you, remember this: micro-rules × long time × many particles = predictable macro-laws.

This isn't just math. Writing one article a day looks like a micro-behavior. After 100 articles, macro-patterns will emerge on their own — which headlines get traffic, which topics explode, what opening the algorithm loves.

Trust compounding, not shortcuts. This isn't motivational fluff. Deng Yu proved it with 138 pages of mathematics.

Tsimerman: His Paper Was Retracted. Then He Won the Fields Medal.

Tsimerman's André-Oort conjecture paper, v2, has a note I've reread several times:

"The previous version (v2) claiming to prove the general case has a serious error, as was kindly pointed out by Klingler, Ullmo and Yafaev. As such, the article is being reverted..."

Translation: the previous version had a serious error. Retracted.

You read that right. A Fields Medal winner's paper had a "serious error," was called out by peers, and withdrawn. Then he fixed it, resubmitted, and won mathematics' highest honor for that very work.

It wasn't just "being smart." The method matters more.

Tsimerman proved the André-Oort conjecture for A_g — a core problem about the distribution of "special points" (CM points) of elliptic curves in high-dimensional moduli spaces. Elliptic curves are the foundation of modern cryptography — the distribution of these special points directly affects the security of certain post-quantum cryptographic schemes.

But the real reason he won isn't the conclusion. It's the tools he assembled. He stitched together three fields that look completely unrelated:

Analytic Number Theory: lower bounds on Galois orbits of class groups of number fields
Model Theory: o-minimal structures to control the topological behavior of special points
Algebraic Geometry: Shimura variety theory to describe the geometric structure of moduli spaces

Tools from any single field weren't enough. Combined, they were exactly enough.

Together with Bakker and others, he leveraged the o-minimal framework to further develop GAGA theory, solving the Griffiths conjecture about period mapping images — building a new bridge between model theory and Hodge theory.

This is exactly the state of AI right now. The next breakthrough probably isn't "better Transformers." It's grafting algebraic topology onto neural network architectures, or redescribing training dynamics in the language of wave turbulence theory.

Tsimerman teaches you two things:

A retraction isn't shameful. Fix it and resubmit.
Don't grind in the field you know best. Go to the intersections.

Pardon: His Entire Paper Does One Thing — Find the Right Window

Pardon proved the 3D Hilbert-Smith conjecture. 24 pages.

The question: what kinds of symmetry can "live in" three-dimensional space? Can the p-adic integers Z_p? Pardon's answer: no.

He later made even more contributions in symplectic geometry — virtual fundamental chains, localization of wrapped Fukaya categories, curve counting on Calabi-Yau threefolds (the MNOP conjecture). This work is now the mathematical foundation for topological quantum error-correcting codes — quantum computers haven't been built yet, but their error correction is already being built on his foundation.

But what fascinates me most is a single methodological phrase in his Hilbert-Smith paper: "Approach is local on M."

The entire proof doesn't need to understand all geometric properties of a 3-manifold. He only needs to find a Z_p-invariant open set U on the manifold M, locate an incompressible surface F inside U, analyze the mapping class group homomorphism induced by Z_p's action on the isotopy class of F — and the contradiction emerges.

Don't scan the whole picture. Find one window. The information inside that window is enough to derive the global conclusion.

This is directly usable.

You want to analyze data from 55 AICDragon articles to figure out what determines readership. You could run statistics on all of them — but that's making work for yourself. Just pick three: the highest-read, the lowest-read, the median. The differences between them already contain most of the patterns.

You want to debug an encoding corruption. You don't need to trace OpenClaw's entire code path. Just look at the first 4 bytes of one file — E9 94 98 3F — it's UTF-8 BOM decoded through GBK. Root cause identified, no global investigation needed.

Pardon's lesson: it's not about having more information. It's about having the right information inside the right window.

So What Does AI Have to Do With These Papers?

It's not "this formula can optimize Transformers."

It's about ways of thinking.

Everything you do daily — topic selection, writing, debugging, data analysis, product decisions — has math-level complexity. The only difference is you haven't noticed.

Wang Hong says: multiscale thinking, coarse first then fine.
Deng Yu says: micro-rules × enough time = macro-laws. Trust compounding, not shortcuts.
Tsimerman says: don't grind in one field. The real breakthroughs are at the intersections. And a retraction isn't shameful — fix it and ship it.
Pardon says: don't try to understand everything. Find the right window.

These four methodologies aren't the privilege of mathematicians. They're weapons for anyone who needs to think and iterate continuously.

Including you.

References

ICM 2026 Official Announcement
Nature: Rising stars of mathematics awarded prestigious 2026 Fields Medal
Scientific American: 2026 Fields Medals go to four young mathematicians
Wang Hong & Zahl: [2502.17655] Volume estimates for unions of convex sets, and the Kakeya set conjecture in three dimensions
Deng Yu & Hani: [2104.11204] Full derivation of the wave kinetic equation
Tsimerman: [1506.01466] A proof of the Andre-Oort conjecture for A_g
Pardon: [1112.2324] The Hilbert-Smith conjecture for three-manifolds

I Downgraded My AI Assistant to an Older Version and It Went Insane

AICDragon — Tue, 28 Jul 2026 07:28:42 +0000

TL;DR: Installed OpenClaw 7.1 beta, didn't like it, downgraded back to 6.6 stable. Then my AI assistant's API consumption exploded 400% — a simple question triggered 20+ API calls, tokens burning nonstop. After an entire afternoon of debugging, I finally found the culprit: during the downgrade, the old and new configs had silently mixed together. This is the full debugging journey, the wrong turns, and the actual fix.

The Diagnosis in One Line

A daemon running without errors doesn't mean everything is fine. One leftover config file from a newer version can silently trigger cascading anomalies — and you won't notice until you see the bill.

What Happened

Here's the story.

I've been tinkering with my personal AI assistant project, OpenClaw. When version 7.1-beta.6 dropped, I eagerly installed it. Half a day in, the cracks started showing — it deleted my project files while setting up a directory, pumped out articles full of garbled characters. I'd had enough. Time to downgrade back to the 6.6 stable release. After all, it's just an npm downgrade — one command, right?

Sure enough, npm install -g openclaw@2026.6.6 took about thirty seconds.

Then things got weird.

Previously, when I'd type "hello," my AI assistant would respond once, burning maybe 3-5 API calls. After the downgrade, the same question triggered 20-plus calls. It was like someone who'd had way too much coffee — muttering to itself, circling back to confirm things multiple times. Token consumption quadrupled. Within minutes, it drained my DeepSeek API balance into the negative and I had to kill the process immediately.

If you work with AI tools or frameworks in any capacity, this post is worth the next five minutes.

The Debugging Journey

First Thought: Network Issues?

Nope. Responses came back fine. They were just... heavy.

Second Thought: Did the Config Get Corrupted?

I ran openclaw doctor and it spat out a version mismatch warning:

Warning: config was created by version 2026.7.1-beta.6,
current version is 2026.6.6.

Bingo. The config was generated by 7.1, but the running binary was 6.6. Version soup.

My instinct was straightforward: just delete the meta field from the config file and let it regenerate. How hard could that be?

$config = Get-Content "openclaw.json" | ConvertFrom-Json
$config.PSObject.Properties.Remove("meta")
$config | ConvertTo-Json | Set-Content "openclaw.json"

Script ran clean. Config refreshed. I restarted the gateway, confident the problem was solved — and the meta field was back.

I was baffled.

The Breakthrough: OpenClaw Has an Auto-Recovery Mechanism

After digging into the startup logs, the full picture emerged:

1. OpenClaw reads openclaw.json → detects manual tampering
2. Validates against schema → mismatch found
3. Triggers recovery → restores from openclaw.json.last-good
4. The backup still contains the 7.1 config → problem resurrected

Every single time I manually deleted that meta field, the recovery mechanism silently restored it. I spent an hour editing. It spent an hour undoing. We were in a stalemate and I didn't even know it.

This is the real lesson: some frameworks don't simply read old configs and ignore unrecognized fields when you downgrade. They have active detection and automatic rollback logic. You can't brute-force the file — you have to understand the machinery.

The Actual Fix

The core principle was simple: don't let 6.6 see a single trace of 7.1.

# Step 1: Isolate the old config (rename, don't delete)
Rename-Item "openclaw.json" "openclaw.7.1.compat.backup.json"
Rename-Item "openclaw.json.last-good" "openclaw.7.1.last-good.backup.json"

# Step 2: Let 6.6 generate its own fresh config
openclaw onboard --non-interactive --accept-risk --mode local ...

# Step 3: Verify the version marker
(Get-Content "openclaw.json" | ConvertFrom-Json).meta.lastTouchedVersion
# Output: 2026.6.6 ✅

The critical step wasn't editing the config at all — it was running a full onboard from the 6.6 binary, letting it generate a configuration that was its own from the ground up.

Only then did the migration start:

✅ Model configs, gateway settings, custom skills → safe to carry over
❌ Session histories, task queues, SQLite databases → leave behind (format-incompatible between versions)

Final verification: openclaw doctor showed zero warnings, and API consumption returned to normal.

Lessons

The whole episode took over half a day from discovery to resolution, but the knowledge density was high. Three takeaways:

1. A downgrade is not just "swapping a version number." The binary, config schema, session format, and database structure form a single integrated stack. Change one, inspect the other four.

2. Don't fight a framework's self-protection mechanism. If the file you edited keeps reverting, the framework's authors are more worried about config corruption than you are. The way around isn't to edit harder — it's to understand the mechanism's logic and take the legitimate path (in this case, re-running onboard).

3. When config version ≠ binary version, the most vulnerable spot is agent behavior. Not crashes. Not error logs. Silent anomalies. Behavior that looks normal but shows abnormal cost metrics — this is the hardest class of bug to notice, and the most expensive.

One final note: to be fair, the OpenClaw team did solid work on this front. They have last-good backups, automatic recovery, and version mismatch detection. The gap is just documentation — nowhere does it emphasize that downgrades require a fresh onboard. I suspect I'm not the only one who's stepped on this rake.

If you maintain stateful services or AI toolchains, add "config version check before upgrade/downgrade" to your checklist. This landmine? Already stepped on. You're welcome.

References

OpenClaw GitHub: https://github.com/openclaw/openclaw
This article is based on a real 2026.7.1-beta.6 → 2026.6.6 downgrade, with all steps verified.

Transparency note: Every troubleshooting step in this article has been reproduced and confirmed. The auto-recovery behavior was verified through multiple deliberate attempts, not speculation. API consumption comparisons are based on measured data from the same task under version-matched and version-mismatched conditions.