From Sparse Attention to MoE: Why K3's 896 Experts Only Activate 1.8% Per Token

This is the third installment in a series on the evolution of attention mechanisms.

Part 1 covered the sequence dimension: from GPT-2's full attention to KV Cache, linear attention, and KDA — seven years of progress.
Part 2 covered the inter-layer dimension: from simple residual summation to Attention Residuals and Depth-Attention — selective cross-layer communication at near-zero cost.

This article explores the third dimension: orthogonal computation — when Transformers face the demand of "make the model bigger without making inference slower," the answer is "most parameters stay idle." This is the story from sparse attention to MoE to Stable LatentMoE. Let's begin with one number: K3 has 896 experts, but at inference time each token only activates 16 — that's 1.8%.

1. The Origin of the Problem: The Curse of All-or-Nothing

Standard Transformers operate on an implicit assumption: every token at every layer must pass through all parameters.

Layer 0: Token -> (Attention + FFN) -> hidden state
Layer 1: Token -> (Attention + FFN) -> ...
...
Layer 92: Token -> (Attention + FFN) -> predict next token

Every layer's FFN processes every token. "Answer a primary-school math problem" and "write operating system kernel code" engage entirely different cognitive circuits — yet the Transformer is forced to use the same parameters for both.

This creates two fatal consequences:

1. Parameter count is constrained. More parameters means more operations per inference — compute cost scales linearly. Want a smarter model? Adding parameters = adding cost.
2. Wasted compute. The simplest token and the hardest token consume exactly the same FLOPs.

MoE's solution is brutally effective: prepare multiple sets of "cognitive circuits" (experts), and have each token call only a small subset.

Sounds like common sense. But turning that common sense into an industrially stable, inference-efficient model took several years.

2. Three Cards in MoE's Evolution

Card 1: Switch Transformer (2021, Google) — Moving MoE from Lab to Factory

MoE itself is an old idea (Jacobs et al., 1991), but before the LLM era it was stuck in the lab due to training instability.

Switch Transformer (Fedus et al., JMLR 2022) did two crucial things:

• Solved large-scale training stability. Used capacity factor to cap tokens per expert, and auxiliary loss to balance routing load.
• Trained a 1.6T-parameter model on C4, activating far fewer parameters at inference than the total count.

But Switch used Top-1 routing — each token goes to exactly one expert. The advantage is extreme sparsity (minimal activation rate). The problem is obvious: if that one expert fails (overloaded or dropped), the token produces no output — there's no backup.

Card 2: Mixtral 8x7B (2023, Mistral) — Top-2 Becomes the Industry Standard

Mistral's Mixtral 8x7B, released in late 2023, used Top-2 routing — each token activates 2 out of 8 experts.

This marked a key design philosophy shift. Top-1 is "all eggs in one basket." Top-2 begins to accept "spend a bit more compute for significantly better stability and performance." The 8-expert configuration is also clean enough to run on consumer-grade GPUs.

Mixtral, at 46.7B total parameters (12.9B active), matched or approached comparable dense models on multiple benchmarks while using substantially less compute. This proved that MoE isn't just for gigantic models — it pays off at medium scale too.

Card 3: DeepSeek-MoE (2024, DeepSeek) — Fine-Grained Experts + Shared Experts

DeepSeek-MoE (ACL 2024) introduced two of the most important architectural innovations in MoE history:

1. Fine-grained expert segmentation: Split N experts into mN smaller experts, activating mK per token. This enables more flexible combinations — instead of a few large experts handling broad knowledge domains, many small experts perform precise assembly.

2. Shared experts: Isolate K_s "shared experts" from the total expert pool, always activated. These handle foundational knowledge every token needs (grammar structure, common phrasing, basic reasoning frameworks), while routed experts focus on domain-differentiated knowledge.

This design solves MoE's biggest contradiction: "knowledge redundancy."

In early MoE, every expert could learn the same basics (e.g., "the" can be followed by a noun) because routing selects experts based on token features, and all experts see these high-frequency patterns. Isn't that just wasting parameters?

The shared expert idea: "Extract that shared part, let everyone use it, stop routed experts from redundantly learning fundamentals."

DeepSeek-MoE 16B achieved LLaMA2 7B performance at roughly 40% of the compute. DeepSeek-MoE 145B matched DeepSeek 67B using only 28.5% of the compute. This was the first paper to clearly demonstrate that the "fine-grained + shared expert" dual-track architecture has overwhelming advantages in compute efficiency.

This is also the direct predecessor of K3's MoE architecture.

3. K3's Stable LatentMoE: Pushing DeepSeek-MoE's Three Paths to the Limit

K3's technical report describes the Stable LatentMoE design in detail. Building on DeepSeek-MoE, it pushes three paths:

Path 1: Expert Count Jumps from 64 to 896

DeepSeek-MoE already made experts smaller and more numerous, but K3 pushes this to another order of magnitude.

DeepSeek-MoE 16B:   ~64 routing experts + shared experts
K3:                  896 routing experts + 2 shared experts

The key isn't "896 is a lot" — it's what 896 implies about how small each expert is.

Let's compute. K3 has 896 routing experts per layer, hidden dim = 7168, latent dim (the compressed dimension before entering routing experts) = 3584, each expert's intermediate dim = 3072.

One expert = two weight matrices (3584->3072 + 3072->3584) = roughly 22M parameters.

K3 has 2.78T parameters total. 896 experts x 93 layers = roughly 83,328 experts. At 22M each, the expert portion is... far below 2.78T. Because experts are just FFN replacements — the total parameter count includes embeddings, all layers' attention weights, shared experts, and other larger components.

But the core logic holds: when experts become small enough, routing ceases to be semantic assignment ("bring in the Python expert") and becomes feature assembly from a pool of fragments.

This leads to K3's most crucial design philosophy, and the most insightful analysis from the lilting channel —

The Router Is Not a "Classifier" — It's a "Compute Allocator"

The standard MoE understanding: "the router classifies each token to a domain expert." This understanding is wrong — at least for K3 with 896 experts.

K3's router is a single line:

scores = Sigmoid(W_r x x)   # W_r: 896 x 7168, x: hidden state
# Take Top-16, weighted combination by normalized scores

A linear projection -> Sigmoid -> Top-16. No complex operations whatsoever.

But this works not because the router is "smart," but because x itself has already been thoroughly processed by preceding layers' Attention and FFN. Just as BERT hidden states are good enough for a simple linear classifier to perform NLU tasks — K3's hidden state, after passing through dozens of layers before reaching the router, is already a highly structured representation space. A single linear projection is sufficient for the final "routing decision."

From another angle: with 896 experts, expecting a simple linear classifier to correctly assign each token to "exactly the right 16 experts" is clearly over-idealized. The router's real job isn't "find the best 16" — it's "under load-balancing constraints, give most tokens a reasonable set of 16."

This is why K3 invested massive engineering effort in load balancing —

Quantile Balancing: The Load-Balancing Challenge of 896 Experts

MoE routing training has a classic dilemma:

• Don't intervene in routing -> tokens flood a few experts -> GPU load imbalance -> some experts starve (insufficient training data)
• Force balance -> same input routed to different experts across training steps -> experts learn highly similar things -> pseudo-balance

K3's solution is Quantile Balancing: at each training step, compute quantile thresholds from the router score distribution — experts above this threshold receive a target number of tokens. Then update per-expert selection bias based on deviation from the target.

Crucially: this bias is only used during training. Fixed at inference.

This means K3 has already gained complete experience during training on "how to achieve load balance among 896 choices," and at inference it simply uses a fixed bias to prevent catastrophic load skew.

Path 2: Latent Compression + 2 Shared Experts

Another key design in Stable LatentMoE: before entering routing experts, compress the 7168-dimensional hidden state into a 3584-dimensional latent space.

x (7168) -> W_down -> z (3584) -> 16 routing experts -> u (3584) -> W_up -> RMSNorm -> output

Why? Two reasons:

1. Reduces per-expert parameter count and compute (input dimension halved)
2. Reduces the information dimension the router needs to handle

But compression loses information. So K3 retains 2 always-activated shared experts — they operate on the full 7168-dimensional hidden space, without latent compression.

Final output:

y = E1_shared(x) + E2_shared(x) + W_up(RMSNorm(u))

The 2 shared experts handle "foundational features everyone needs." The 16 routed experts handle "differentiated features specific to each token." Clean division of labor.

Path 3: 1.8% Activation Rate — It's Not a Bug, It's a Feature

Back to that number: 896 experts, only 16 activated. 1.8%.

Why not more? Say, 32 (3.6%)?

K3's technical report doesn't directly provide this ablation, but we can infer from the architecture:

1. Latent space capacity bottleneck. Latent dim = 3584. 16 experts already approach this dimension's information saturation. Adding more experts -> additional FFN capacity can't be effectively transmitted through the latent bottleneck -> diminishing marginal returns.

2. Cross-GPU communication cost. 896 experts are distributed across multiple GPUs. Each additional activated expert means more all-to-all communication across GPUs. K3's official docs explicitly recommend 64+ accelerators for deployment — when you're already doing all-to-all on a cluster that large, each extra expert adds real latency.

3. DeepSeek-MoE already validated: fine-grained + fewer activations > coarse-grained + more activations. Rather than activating 40 experts each learning a little, let 16 small experts each learn their part well.

1.8% isn't a design target — it's the optimal convergence point shaped jointly by the latent bottleneck, communication costs, and expert granularity.

4. The Evolution at a Glance: From Dense to 896-Expert

Year    Model                  Routing Strategy    Experts/Layer    Active    Shared Experts

2021    Switch Transformer     Top-1               64-2048          1         None
2023    Mixtral 8x7B           Top-2               8                2         None
2024    DeepSeek-MoE 16B       Fine-grained Top-K  ~64              ~6        K_s
2024    DeepSeek-MoE 145B      Fine-grained Top-K  ~192             ~18       K_s
2026    Kimi K3                Quantile Top-16     896              16        2

Clear trends:

1. Expert count is increasing, but active count stays in a stable range (1-16) — not because 16 is a magic number, but because the latent bottleneck and communication costs impose an invisible ceiling
2. Shared experts are a watershed — nobody did them before DeepSeek; K3 directly inherited the approach
3. Load balancing evolved from "aux loss assist" to "quantile direct control" — control precision went from "penalize imbalance" to "direct allocation"
4. Each expert is getting smaller — from a few large experts to hundreds of small ones, routing's meaning shifts from "domain classification" to "feature assembly"

5. K3 MoE's Real-World Performance

Data extracted from the official blog and technical report:

• Total params: 2.78T, active params: 104.2B (roughly 3.7%)
• Inference efficiency: Despite the routing overhead of 896 experts, K3's team claims 90%+ cache hit rate (in coding scenarios) through the Mooncake decoupled inference architecture, serving million-token contexts at $0.30/M tokens (cache-hit pricing)
• Training efficiency: Introduced fully balanced expert-parallel training with static shapes and no host-synchronization critical path, making 896-expert scale training feasible

More importantly, the synergy between MoE and other innovations: KDA (from Part 1) + Attention Residuals (mentioned in Part 1) + Stable LatentMoE (the protagonist of this article) -> Kimi K3 achieves roughly 2.5x overall scaling efficiency improvement over K2.

This isn't one component's magic — it's three orthogonal optimization dimensions stacked together: cheaper long contexts (KDA), smarter inter-layer communication (AttnRes), and compute concentrated on the most needed parameters (LatentMoE).

6. What's Next: MoE on the Road Ahead

K3 already does conditional computation at the FFN layer — each token uses a different FFN expert combination. This is just the first step:

Current K3 is still a 93-layer fixed-depth Transformer — every token, regardless of difficulty, must traverse all 93 layers. Possible evolutionary directions:

1. Dynamic depth: Simple tokens exit early, difficult tokens go deeper (ADEPT, 2026)
2. MoE-ified Attention: Not all tokens need full attention — SALSA uses a router to decide between attention and recurrent (2026)
3. Layer-level MoE: Different layers may have different expert architectures — input-adjacent layers use shared-type, output-adjacent layers use routing-type
4. From FFN routing to full-module routing: Not just choosing experts, but choosing "should this layer use attention, SSM, or skip" — turning the model into a token-level dynamic computation graph

If MoE's essence is "most parameters stay idle," these directions' essence is "most computation stays idle." The engineering implementations are entirely different, but the philosophy shares the same origin.

7. What This Means for You

Three "Don'ts":

1. Don't understand MoE through "domain classification." Thinking in terms of "math expert" and "code expert" misses the essence. Experts are FFN fragments — what the router allocates isn't semantic roles but compute resources. This understanding directly determines whether you can design efficient routing strategies.

2. Don't equate "more experts" with "better." K3 uses 896 not because it "needs 896," but because under the constraints of fine granularity + latent compression + load balancing, 896 is the optimal tradeoff for training and inference efficiency. Blindly adding experts -> load balancing collapse -> pseudo-balance -> wasted parameters.

3. Don't treat MoE as an isolated module. K3's 2.5x scaling efficiency improvement comes from the synergy of KDA + AttnRes + LatentMoE. The relationship between MoE and attention mechanisms isn't "this replaces that" — it's "simultaneous optimization across multiple dimensions": horizontal (sequence attention compression), vertical (inter-layer information flow), and orthogonal (selective parameter activation).

Three Numbers to Remember:

• 1.8% — K3's FFN activation rate per token. The current sparsity-efficiency ceiling for the strongest open-source model.
• 28.5% — The compute fraction DeepSeek-MoE 145B needs to match DeepSeek 67B Dense. The compute-efficiency baseline for fine-grained MoE.
• 90% — K3 API's cache hit rate in coding scenarios. Without this level of cache optimization, a 2.78T model could never achieve $0.30/M token pricing.

References

• Kimi K3: Open Frontier Intelligence — Kimi Team, July 2026, technical report + open weights
• Kimi K3 Tech Blog — Moonshot AI official blog
• K3's router picks 16 of 896 experts: an allocator, not a classifier — lilting channel, deep architecture analysis
• DeepSeekMoE: Towards Ultimate Expert Specialization — Damai Dai et al., ACL 2024
• Switch Transformers: Scaling to Trillion Parameter Models — Fedus et al., JMLR 2022
• Mixtral of Experts — Jiang et al., Mistral AI, 2024
• The Illusion of Specialization — ACL 2026, analysis of actual MoE expert behavior

---

This is Part 3 of the "Evolution of Attention Mechanisms" series and the first installment on the "orthogonal dimension." Together, the three parts form a complete coordinate system: horizontal (sequence attention) -> vertical (inter-layer communication) -> orthogonal (selective parameter activation). Coming next: if MoE decouples "parameter count" from "inference cost," what's the next dimension to be decoupled?

Comments

[TokenLat]

Really sharp write-up on K3's expert activation — the 1.8%/token figure is a great lens for why routing by scenario beats routing by size. On the 70%+ number you asked about: it's measured against list price of frontier models for the same task, not against other China models. Happy to share the exact methodology if useful.