How DeepSeek-V4 Achieves Million-Token Contexts Without Quadratic Attention Costs

DeepSeek-V4, released in April 2026 under the MIT license, is a Mixture-of-Experts model that supports a one-million-token context window while using only 27% of the inference FLOPs that its predecessor DeepSeek-V3.2 required at the same context length. The key to that efficiency is a hybrid attention mechanism that replaces standard full attention with two complementary compression strategies. This post walks through how those mechanisms work, what else changed in the architecture, and what the numbers look like in practice.

The Problem: Attention Scales Quadratically

Standard transformer attention computes a similarity score between every pair of tokens. At 1,000 tokens, that's one million comparisons. At one million tokens, it's one trillion — and the KV cache grows proportionally. This is why most production models cap out at 128K or 256K tokens even when longer contexts would be useful.

The standard workarounds — sliding window attention, linear attention approximations, retrieval-augmented generation — each trade off some capability for efficiency. DeepSeek-V4 takes a different approach: it keeps full attention for a small fraction of the sequence and uses two levels of aggressive compression for the rest.

Two Compression Strategies: CSA and HCA

The hybrid attention system in DeepSeek-V4 interleaves two mechanisms across the model's layers:

Compressed Sparse Attention (CSA) compresses the KV cache by a factor of 4. Every four consecutive tokens are merged into a single KV entry using softmax-weighted pooling. A lightweight "lightning indexer" — running in FP4 precision — then scores the query against these compressed entries and selects the top-k most relevant ones (typically around 128) for the expensive softmax and matrix multiplication operations. A sliding window branch runs in parallel to preserve local dependencies. The result is that most of the sequence is represented at quarter resolution, and only the most relevant portions are attended to at full resolution.

Heavily Compressed Attention (HCA) applies more aggressive compression: 128 tokens are merged into a single KV entry, reducing a one-million-token sequence to roughly 8,000 entries — small enough for standard dense attention. HCA provides a coarse, document-level view of the entire context.

Both mechanisms share KV projections using Multi-Query Attention (MQA), apply query and KV normalization, and use Partial Rotary Positional Embedding (RoPE). The combination gives the model three views at any layer: a high-resolution local window, a medium-resolution sparse selection via CSA, and a low-resolution global summary via HCA.

The efficiency gains are substantial. At a one-million-token context, DeepSeek-V4-Pro requires 27% of the single-token inference FLOPs and 10% of the KV cache size compared to DeepSeek-V3.2 — the difference between a context length being theoretically possible and being economically viable to serve.

Manifold-Constrained Hyper-Connections

Beyond attention, DeepSeek-V4 replaces standard residual connections with Manifold-Constrained Hyper-Connections (mHC). In a standard residual block, the output is x + f(x). In mHC, the residual mapping is constrained to lie on the manifold of doubly stochastic matrices, which bounds the spectral norm to 1. The parameters controlling the connection are generated dynamically from the input at each layer rather than being fixed.

The practical effect is improved numerical stability during training and better signal propagation across depth. This matters more at scale: very deep models can suffer from gradient issues that mHC is designed to suppress.

The Muon Optimizer

DeepSeek-V4 was trained using the Muon optimizer, which orthogonalizes gradient update matrices using Newton-Schulz iterations before applying them. Standard Adam-style optimizers apply updates that can have large singular values, which can destabilize training. Muon constrains the update to be approximately orthogonal, which keeps singular values bounded and improves convergence stability.

The training run used more than 32 trillion tokens covering math, code, web text, and long documents. Additional stability measures included Anticipatory Routing (which decouples the routing update cycle from the main training step to prevent expert collapse) and SwiGLU Clamping (which clips activation values to prevent outliers from destabilizing the MoE gating).

Model Variants and Deployment

The V4 series ships in two sizes:

Model	Total Parameters	Active per Token	Context
DeepSeek-V4-Pro	1.6 trillion	49 billion	1 million tokens
DeepSeek-V4-Flash	284 billion	13 billion	1 million tokens

Both models support three reasoning modes: Non-Think (standard generation), Think High (chain-of-thought reasoning), and Think Max (extended reasoning). Unlike earlier DeepSeek models, V4 supports tool calls even while in thinking mode, which matters for agentic pipelines.

The models are available via the DeepSeek API at $0.435/1M input tokens for Pro and $0.14/1M input tokens for Flash (as of May 2026). Both are released under the MIT license, with weights available on Hugging Face. Self-hosting V4 Flash requires roughly 158GB of VRAM; V4 Pro requires a DGX H200 or equivalent.

Benchmark Results

On standard benchmarks, V4-Pro shows consistent improvements over V3.2:

Benchmark	DeepSeek-V3.2	DeepSeek-V4-Flash	DeepSeek-V4-Pro
MMLU (5-shot)	87.8	88.7	90.1
MMLU-Pro (5-shot)	65.5	68.3	73.5
HumanEval (Pass@1)	62.8	69.5	76.8
LongBench-V2 (1-shot)	40.2	44.7	51.5
SimpleQA (25-shot)	28.3	30.1	55.2

The LongBench-V2 result is the most directly relevant: a 28% relative improvement over V3.2 on long-context tasks, where the CSA/HCA hybrid attention does the most work. The SimpleQA jump for Pro (28.3 → 55.2) reflects improvements in factual recall from the larger active parameter count and the OPD post-training step.

Post-Training: On-Policy Distillation

The post-training pipeline uses On-Policy Distillation (OPD), which transfers knowledge from multiple domain-specific teacher models into the unified student using full-vocabulary logit distillation. Rather than training a single generalist model from scratch, OPD lets the team maintain specialized teachers for coding, math, and reasoning, then distill their combined knowledge into V4-Pro and V4-Flash.

What This Means for Developers

If you are building applications that need to process long documents — legal contracts, codebases, research papers, extended conversation histories — the CSA/HCA architecture offers a concrete path to doing so without the memory and compute costs that have historically made million-token contexts impractical.

A few practical notes:

1. The Flash variant is the realistic self-hosting option. At 13B active parameters and 284B total, V4-Flash is manageable on a cluster of H100s or H200s. V4-Pro at 49B active parameters requires significantly more infrastructure.

2. Reasoning mode selection matters for cost. Think Max mode uses substantially more tokens than Non-Think. For applications where you need the long context but not extended chain-of-thought, Non-Think mode at V4-Flash pricing is quite affordable.

3. The MIT license is genuinely permissive. Unlike some open-weight releases with restrictive commercial terms, MIT allows unrestricted use, modification, and redistribution. The weights and infrastructure details are documented in the technical report on arXiv.

The CSA/HCA framing — coarse global attention plus sparse high-resolution attention — is a general architectural pattern that is likely to appear in other long-context models as the field continues to push context lengths upward.

---

Primary source: DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence — DeepSeek AI, 2026

Supporting sources: DeepSeek V4 Complete Guide | Simon Willison's notes on DeepSeek-V4 | DeepSeek V4 API Documentation