Kimi K3 Under the Hood: A 2.8T MoE Built for Million-Token Agents

A technical analysis of Kimi K3's hybrid attention, Stable LatentMoE, long-horizon RL, serving stack, benchmark claims, and cybersecurity evidence.
Moonshot AI has released the weights of Kimi K3, a native multimodal Mixture-of-Experts model with 2.8 trillion total parameters, 104 billion active parameters, and a one-million-token context window.

K3 beats Claude Opus 4.8, GPT-5.5, GPT-5.6 Sol, or Claude Fable 5 on selected coding and agentic benchmarks. That does not make it the strongest model across every workload. Moonshot's own technical report says K3 still trails Fable 5 and GPT-5.6 Sol overall.

As the Moonshot technical blog explains, the engineering is more consequential than the leaderboard headline. K3 combines a hybrid recurrent/global attention stack, depth-wise attention over residuals, an extremely sparse latent MoE, persistent reinforcement-learning environments, and a cache architecture designed specifically for million-token agent traffic.

This article explains how those parts fit together, which claims are supported by released artifacts, and which still depend on Moonshot's internal measurements.

The released model

Property	Kimi K3
Total parameters	2.78T
Active parameters	104.2B
Transformer layers	93
Attention layers	69 KDA + 24 Gated MLA
Hidden size	7,168
Attention heads	96
Routed experts	896
Experts selected per token	16
Shared experts	2
Context window	1,048,576 tokens
Vision encoder	MoonViT-V2, 27 layers, 401M parameters
Release precision	MXFP4 expert weights, MXFP8 activations

The Hugging Face configuration matches the technical report. The checkpoint contains 96 safetensors shards totaling about 1.56 TB.

K3 is an open-weight release, but "open source" needs a qualifier. The custom Kimi K3 License permits use, modification, and distribution while adding conditions for large Model-as-a-Service businesses and very large commercial products.

Three dimensions of information flow

Moonshot organizes the architecture around sequence length, model depth, and channel width. Each dimension has its own scaling mechanism.

Sequence mixing: three KDA layers per global MLA layer

The backbone repeats:

KDA -> KDA -> KDA -> Gated MLA

The final layer is also MLA, so the output always passes through global attention.

Kimi Delta Attention is a recurrent linear-attention mechanism. It replaces the sequence-length-dependent KV cache with a fixed-size state updated by a delta rule. A channel-wise retention factor controls how much of the old state survives before the current key/value pair is written.

K3 changes the retention parameterization:

g = g_min * sigmoid(exp(A) * z)
alpha = exp(g)
g_min = -5

In the earlier Kimi Linear design, log-decay was unbounded below. During chunkwise execution, reciprocal cumulative decay could overflow. Diagonal 16-token tiles therefore needed a special position-pair implementation instead of the dense Tensor Core path used elsewhere.

The lower bound in K3 keeps a 16-token tile's cumulative log-decay within (-80, 0), which fits the BF16 dynamic range. Both diagonal and off-diagonal causal tiles can then use dense matrix multiplication.

This is algorithm-system co-design in a concrete form: changing a gate's mathematical range removes a specialized GPU-kernel path.

The recurrent state is still finite, so every fourth attention layer uses Multi-head Latent Attention. MLA compresses each token's key/value representation into a latent vector while preserving unrestricted token-to-token interaction.

K3 uses NoPE in its MLA layers. KDA supplies order and recency information, while MLA supplies global content matching. Extending the context therefore does not require RoPE rescaling or YaRN interpolation.

Depth mixing: Attention Residuals

A standard residual stream adds every layer's output with a fixed weight. As depth increases, all previous computation is compressed into one accumulated state.

Attention Residuals lets each layer attend over earlier representations instead. Full AttnRes would keep every layer output alive, increasing memory and pipeline communication. K3 uses Block AttnRes:

• layers are grouped into blocks of 12;
• outputs are summed inside a block;
• attention operates over block-level representations;
• the original embedding remains an addressable source;
• live-state memory drops from O(Ld) to O(Nd).

Moonshot's separate AttnRes paper reports scaling improvements on a smaller 48B-total/3B-active Kimi Linear model. The K3 configuration and implementation are public, but the effect has not been independently reproduced at 2.8T scale.

Channel mixing: Stable LatentMoE

Sending a full 7,168-dimensional hidden state to 16 experts would make communication and weight traffic dominate execution. LatentMoE projects the routed path to width 3,584, runs the specialized experts in that space, and projects the result back. Two shared experts retain a full-width path.

Scaling this design to 896 routed experts exposed two problems: activation outliers and unstable load balancing.

Moonshot addresses the first with:

• RMSNorm between routed-expert aggregation and the up-projection;
• SiTU-GLU instead of SwiGLU.

SiTU applies smooth tanh caps to both multiplicative branches. K3 uses bounds of 4 and 25, limiting the product's magnitude to 100. Near zero, the function behaves similarly to SwiGLU; for large positive inputs, it avoids quadratic growth.

Quantile Balancing handles expert load. An expert-specific bias affects Top-k dispatch but is excluded from the mixture weights, so it does not directly change router gradients. The next bias is derived from the margin quantile that gives each expert its target token count.

An exact global quantile would be expensive. Workers build per-expert histograms and combine the bins with one all-reduce. The update is tied directly to the target load instead of a manually tuned bias step.

Native multimodality from the start

MoonViT-V2 was trained from scratch together with the language backbone under the same next-token objective. Moonshot moved away from SigLIP initialization because its internal runs showed larger vision-tower gradient norms and frequent spikes when a pretrained encoder was attached.

The vision path uses:

• a 27-layer, roughly 0.4B-parameter ViT;
• separate spatial and temporal attention passes;
• shared image and video parameters;
• temporal pooling;
• a 2×2 pixel shuffle that reduces visual-token count by four;
• inputs up to 3584×3584 pixels.

The report includes an internal gradient plot and says the from-scratch model matched the SigLIP-initialized baseline on vision evaluations. It does not provide a complete numerical ablation or an independent replication.

Reaching one million tokens

K3 did not train at maximum length from the beginning:

8K -> 64K during pre-training
256K -> 1M during cooldown

The expensive long-sequence phases consume only a small part of the total budget. Long documents and videos were filtered for duplication, binary data, truncation, invalid logs, and low-quality segments.

Moonshot also synthesized tasks whose required evidence was distributed across the full context. Sequence length alone does not teach long-range retrieval; a model can still solve many concatenated documents with local patterns.

The report claims an approximately 2.5× scaling-efficiency improvement over Kimi K2. That number is not yet reproducible. The published graph omits the numerical run table, exact losses, uncertainty, total training FLOPs, and component-level ablations. Architecture, data, optimizer, model shape, and training schedule all changed at once.

The 2.5× result is best treated as a first-party scaling-law measurement rather than an independently established property.

Post-training nine policies into one model

After supervised fine-tuning, Moonshot trained reinforcement-learning experts in three domains:

1. general reasoning, knowledge, vision, and search;
2. general agents for assistants, research, and writing;
3. coding agents for software engineering, GPU kernels, and web development.

Each domain had low, high, and max reasoning-effort variants. Multi-Teacher On-Policy Distillation consolidated the nine policies into one model.

Partial rollouts

Long-horizon tasks create extreme stragglers. A trajectory may execute hundreds or thousands of tool calls.

K3's RL system pauses generation after a configurable fraction of trajectories has finished. Completed prompt groups move to policy optimization, while unfinished rollouts are queued and resumed in the next iteration.

A trajectory can therefore survive multiple policy updates and become stale. Moonshot uses per-token regularization to keep each update within a local neighborhood. The model state and the external sandbox state are both preserved, so the agent resumes the actual task instead of reconstructing it.

Reasoning budgets

A cold-start policy estimates a base token budget for each problem. A trajectory that exceeds tau × base_budget receives a reward of -1. Moonshot starts with a larger tau for the max-effort policy and anneals it for high and low effort.

A similar length constraint applies to the generative reward model, preventing verbose candidates from winning simply by producing more text.

Harness diversity and verifiers

The white-box RL environment composes tools, system prompts, context-management strategies, skills, memory, and subagents. Training rotates these modules instead of locking the model to one Kimi Code, Claude Code, or Codex-style protocol.

Many environments score the final state with deterministic or hidden verifiers. GPU-kernel tasks combine correctness and speed rewards with detectors for cache reuse, CUDA graph replay, and precision shortcuts. Autonomous tasks isolate the verifier and limit submission attempts.

This training setup targets a repeated loop:

reason -> act -> observe -> verify -> adapt

That loop is a plausible explanation for K3's strong long-horizon coding and vulnerability-research behavior, although Moonshot does not publish a cyber-specific causal ablation.

Infrastructure for a 3T-class MoE

K3 combines pipeline, expert, data, and context parallelism. Expert parallelism has an unavoidable load problem: routers send different token counts to different workers, causing idle devices, overloaded ranks, dynamic tensor shapes, and memory fragmentation.

MoonEP creates temporary copies of overloaded experts and gives every rank exactly S × K tokens. Moonshot proves an upper bound of E/R redundant experts per rank, guaranteeing that a feasible balanced plan exists.

Perfect balance enables:

• static computation shapes;
• no per-layer host synchronization to discover expert sizes;
• a fixed S × K communication buffer;
• zero-copy dispatch into the final expert-grouped positions.

The MoonEP repository includes CUDA code and multi-GPU tests. At release time it had one commit, and its performance comparison with DeepEP remained a first-party H20 benchmark.

KDA requires a different form of context parallelism from ordinary linear attention. Each sequence segment emits:

• a cumulative transition acting on its incoming state;
• a state generated locally from zero.

These affine transitions compose associatively with a prefix scan. Workers exchange fixed-size fragments rather than KV blocks that grow with sequence length. The implementation and distributed correctness tests have been merged into flash-linear-attention.

Persistent sandboxes for agentic RL

Long rollouts need a live external environment. Moonshot's AgentENV uses Firecracker microVMs because advanced tasks need mounts, nested containers, and virtual machines, while ordinary containers did not provide enough isolation.

The system supports:

• pause and resume;
• fork for side-effect-free reward evaluation;
• snapshots for recovery;
• incremental checkpoints that store dirty memory pages;
• OverlayBD-backed image distribution.

Moonshot reports checkpoint latency as low as 133 ms, resume latency as low as 49 ms, and up to 6.5× memory overcommit. Those performance numbers and the reported 51 million sandbox launches have not been independently reproduced.

Serving two incompatible cache types

Each K3 block combines three fixed-size KDA states with one sequence-growing MLA KV cache. A prefix is reusable only if both cache types can be restored at the same token boundary.

Moonshot puts both page types in one allocation, reference-counting, and eviction pool. It then decouples physical storage granularity from prefix matching:

• physical blocks contain roughly 1,024–6,144 tokens;
• prefix hashes may be recorded every 512 tokens;
• KDA checkpoints are kept only at selected hash boundaries;
• conversation-turn boundaries receive priority;
• a hit is valid only when every KDA cache group has a checkpoint at that boundary.

Cached checkpoints are read-only. A request copies the checkpoint into its private running state before continuing. All related cache groups are pinned before allocation, and evicting one KDA checkpoint invalidates its siblings atomically.

Speculative decoding introduces another issue: rejected draft tokens have already advanced the recurrent state. Saving a full state for every draft position would multiply memory traffic. K3 stores the much smaller projected inputs and replays accepted tokens on-chip inside a fused recurrence.

At fleet level, sessions are routed to the cluster holding their prefix cache. Consistent hashing assigns both a primary and a secondary cluster to distribute recovery work after a failure. Separate admission budgets prevent bursts of million-token requests from starving short requests.

What the cybersecurity results actually show

Moonshot evaluates two different capabilities.

Tier 1: discovering new vulnerabilities

The report claims:

• hundreds of candidate findings across dozens of systems;
• roughly 70% confirmation among findings selected for human review;
• 16 previously unknown vulnerabilities across six projects;
• two Linux-kernel findings, including a remote heap out-of-bounds write and an RDMA permission-check regression.

The missing methodology prevents a reliable discovery-rate estimate. Moonshot does not disclose:

• how many findings received human review;
• how those findings were selected;
• the six projects or advisory identifiers;
• prompts, tools, time, token, and agent budgets;
• access to commit history, issues, or patch clues;
• false negatives;
• a comparable blind baseline.

The 16-vulnerability result remains a first-party claim. The independent government evaluation did not test zero-day discovery.

Tier 2: end-to-end exploitation

Moonshot's internal suite contains 36 expert-solvable tasks. K3 solved 14, compared with eight for GLM-5.2. Ten K3 solves came from the 16 user-space targets; four came from 20 Linux-kernel targets.

The reported failure modes are informative:

1. failing to complete the final exploitation stage;
2. choosing poor strategies under mitigations;
3. entering long, unproductive debugging loops;
4. insufficiently verifying the final exploit.

UK AISI and the US CAISI independently found:

• ExploitBench: K3 32%, GLM-5.2 24%;
• arbitrary code execution: K3 0/41;
• The Last Ones cyber range: step 17 of 32 on average;
• one complete range solve in ten attempts under a 100M-token limit.

The leading closed cyber-capable models averaged step 28.5 and achieved arbitrary code execution on 20 of 41 ExploitBench tasks. K3 is a meaningful increase in open-weight cyber capability, especially over GLM-5.2, but it does not match the closed frontier at reliably completing hardened exploit chains.

The AISI/CAISI evaluation also found that K3's safeguards did not prevent it from attempting exploit development or offensive operations.

Reading the benchmark table carefully

Moonshot evaluated different models through Kimi Code, Claude Code, or Codex. Terminal-Bench uses the best harness per model. SWE-Marathon used an H20-calibrated pre-release branch. Some Fable 5 and GPT-5.6 Sol trajectories encountered fallbacks or cyberguards.

The evaluated system is therefore:

model + harness + reasoning budget + hardware + benchmark version

The results support a narrow conclusion: K3 can compete with the closed frontier on tool-heavy, long-horizon workloads at high test-time compute. They do not establish that its backbone is universally stronger.

The independent DeepSWE leaderboard on July 25 reported K3 at 69% ± 5%, below GPT-5.6 Sol and nominally below Fable 5, but above Opus 4.8. Overlapping confidence intervals make several adjacent rankings unstable.

Artificial Analysis independently places K3 in the frontier class while also measuring low output speed and high verbosity: roughly 33 output tokens per second and 130 million generated tokens for its Intelligence Index run.

Operational constraints

Self-hosting K3 is a cluster deployment. Moonshot recommends supernodes with at least 64 accelerators to keep expert communication inside a high-bandwidth domain.

K3 always uses reasoning and expects preserved thinking history. Multi-turn and tool-call clients must send back the complete previous assistant message, including reasoning_content and tool_calls. Dropping that state or switching models mid-session can make generation unstable.

Long-horizon RL also makes the model unusually proactive. Ambiguous instructions may lead it to make decisions on the user's behalf. Production agents need explicit system boundaries, approvals, least-privilege tools, and sandboxing.

The Hugging Face quickstart uses trust_remote_code=True. Pin the exact model revision, review the Python implementation, and load it inside an isolated environment.

Conclusion

Kimi K3 demonstrates a coherent path to scaling open-weight agentic models. A bounded decay gate simplifies KDA kernels, latent routing makes 896 experts practical, persistent rollouts train long tool-use loops, and hybrid cache management makes recurrent and global attention reusable at million-token scale.

The open questions are equally concrete. Moonshot has not published a reproducible training-compute statement, component-level K3 ablations, numerical support for the 2.5× scaling claim, or a reviewable protocol for the 16 new vulnerabilities. The released weights and code make architectural validation possible; independent replications must now test efficiency, long-context use, and blind vulnerability discovery.

Comments

[KL3FT3Z]

Excellent analysis — the author honestly separates architectural innovations that can be validated through open weights and code from marketing claims that remain first-party assertions without independent replication.
From my perspective as a red teamer, the most interesting yet problematic section is cybersecurity. Moonshot claims 16 zero-day findings, but without disclosing the methodology for selecting findings for human review, the total number of generated candidates, token/time/agent budgets, or access to commit history and issue trackers — this figure allows neither true discovery rate nor false negative rate estimation. This doesn't mean the findings are fake, but in offensive security, methodology is half the result. Without it, the number remains a PR metric.
Regarding end-to-end exploitation, the picture is more transparent thanks to independent AISI/CAISI evaluations. 0/41 on arbitrary code execution and step 17 out of 32 on The Last Ones indicate that K3 can reason about vulnerabilities and build chains, but struggles at final exploitation stages under modern mitigations, gets stuck in debug loops, and doesn't always verify the final exploit. This is a typical pattern for LLM agents in offensive security: good "theory," weak "practice" on hardened targets.
Architecturally, K3 is impressive. The bounded decay gate in KDA, Stable LatentMoE with SiTU-GLU, Block AttnRes, MoonEP — this isn't engineering for engineering's sake, but an attempt to solve specific scaling problems. Particularly interesting is how the KDA + MLA hybrid avoids YaRN/RoPE rescaling for million-token contexts — this could be key to long-horizon analysis of large codebases.
But there's also a concerning point: K3 is "unusually proactive" and its safeguards don't prevent exploit development attempts. For production deployment, this means strict sandboxing, least-privilege tools, and explicit system boundaries are mandatory. A 2.8T-parameter model that can autonomously make decisions on behalf of the user while being capable of generating exploit code isn't just a tool — it's a potential risk agent if launched without proper isolation.
Overall, K3 is an important step for the open-weight ecosystem. It shows that open models can compete with the closed frontier on long-horizon agentic tasks. But the gap in cybersecurity between open and closed remains substantial, and methodological transparency from Moonshot could make this work not just impressive, but scientifically valuable.