ai maya

Posted on Oct 19

Ling-3.1-flash and the 7:1 hybrid: how much KV cache does a 560B model need at 1M tokens?

#ai #machinelearning #llm #inference

TL;DR

Ling-3.1-flash (MIT license, 560B total, 25B active, weights on Hugging Face since 2026-10-13) replaces most of its attention layers with linear attention: 63 Kimi Delta Attention (KDA) layers and only 9 gated Multi-head Latent Attention (MLA) layers, a 7:1 ratio. Only the 9 MLA layers keep a cache that grows with every token. Computing from the published config.json, that is roughly 10.4 KB of cache per token, or about 10.6 GB for a 1M-token sequence in bf16, against about 83 GB if all 72 layers used latent attention. The weights, not the cache, are now the main cost: the official FP8 repository is 573.8 GB. This post shows the arithmetic, the assumptions behind it, and what it means for serving long-context agents.

What did Ling-3.1-flash actually ship?

The model card and config.json on Hugging Face give these facts (all quoted from the card or read from the config):

Item	Value
Architecture	Hybrid-linear MoE (`bailing_hybrid`)
Parameters	560B total, 25B active per token
Layers	72 = 63 KDA + 9 gated MLA (7:1)
Experts	512 routed, 1 shared, 8 active per token; 4 dense layers first
Hidden size / heads	4096 / 64
MLA dimensions	`kv_lora_rank` 512, `qk_rope_head_dim` 64, `q_lora_rank` 1536
Context	1M tokens on the card; `max_position_embeddings` is 262144 in the config
Weights	FP8 (E4M3) official checkpoint, 54 safetensors shards, 573.8 GB measured through the Hugging Face API
License	MIT
Speculative decoding	1 next-token-prediction (MTP) layer, usable via SGLang or vLLM

Two details deserve a flag. First, the card advertises a 1M-token window while the config still says 262,144 positions with no RoPE scaling entry. That is not necessarily a bug (serving stacks can override it), but check what your engine actually allocates before you promise 1M to a user. Second, the card text lists benchmark names such as Terminal-Bench 4.0, SWE-bench Pro and GDPVal-AA V2.1, but the numbers sit in images. I do not quote scores here; read them on the card and treat vendor numbers as vendor numbers until someone reproduces them.

Why would a model keep only 9 full-attention layers?

Full attention stores a key and a value for every past token in every layer. That cache grows linearly with context and is read in full at every decoding step, which is why long-context decoding is bandwidth bound (I covered the mechanics in The KV Cache Is the Bottleneck).

Linear attention layers work differently. They fold the past into a fixed-size state matrix per head and update it as each token arrives. Memory per sequence stays constant no matter how long the context gets. The classic weakness is recall: a fixed state cannot hold an exact copy of everything, so pure linear models struggle at precise retrieval. Hybrids keep a few full-attention layers as the "exact lookup" path and let the linear layers do the cheap bulk of the work.

KDA is the linear layer from the Kimi Linear paper (Kimi Team, submitted 2025-10-30), which describes it as extending Gated DeltaNet with finer-grained gating. That paper used a 3:1 ratio of KDA to MLA layers and reported up to 75% less KV cache and up to 6x faster decoding at 1M tokens. Alibaba's Qwen3-Next-80B-A3B uses a similar 3:1 layout of Gated DeltaNet to gated attention. Ling-3.1-flash pushes the ratio to 7:1.

The trend is simple: each generation moves more layers to constant-memory linear attention and relies on fewer, wider full-attention layers for exact recall. Whether 7:1 loses recall quality relative to 3:1 is an empirical question the community will answer on long-context retrieval tests, not one that config arithmetic can settle.

How big is the cache per token, exactly?

Here is the calculation, using the config values. Assumptions are stated because they matter:

MLA layers store a compressed latent, kv_lora_rank + qk_rope_head_dim = 512 + 64 = 576 elements per token per layer. This is how latent-attention kernels in vLLM and SGLang cache it. A serving stack that expands keys and values would use far more.
bf16 cache, 2 bytes per element. An FP8 cache would halve it.
Each KDA layer keeps one 128 x 128 state per head across 64 heads, in fp32. This is my reading of the layer design (head_dim 128, num_attention_heads 64), not a number from the model card. Small convolution state (kernel size 4) is ignored.

# Cache arithmetic from Ling-3.1-flash config.json (inclusionAI/Ling-3.1-flash)
LAYERS, MLA, KDA = 72, 9, 63
LATENT = 512 + 64                       # kv_lora_rank + qk_rope_head_dim
BYTES = 2                               # bf16 cache

per_token_hybrid = MLA * LATENT * BYTES          # 10,368 bytes
per_token_all_mla = LAYERS * LATENT * BYTES      # 82,944 bytes (hypothetical)
kda_state = KDA * 64 * 128 * 128 * 4             # fp32, 264.2 MB, constant

def cache_gb(ctx, hybrid=True):
    if hybrid:
        return (per_token_hybrid * ctx + kda_state) / 1e9
    return per_token_all_mla * ctx / 1e9

for ctx in (32_768, 131_072, 262_144, 1_000_000):
    print(ctx, round(cache_gb(ctx), 2), round(cache_gb(ctx, False), 2))
# 32768     0.60   2.72
# 131072    1.62  10.87
# 262144    2.98  21.74
# 1000000  10.63  82.94

Three things fall out of the numbers:

At 1M tokens the hybrid layout needs about 10.6 GB per sequence versus 82.9 GB, a 7.8x reduction. That is slightly less than the naive 8x (72/9) because of the constant KDA state.
The constant state is not free. At 264 MB it equals the MLA cache of about 25,500 tokens. Below that length the linear layers cost more than they save; above it they win, and the gap widens linearly.
For scale, plain multi-head attention would be absurd. The config lists 64 key-value heads with a 192-wide key and 128-wide value. If all 72 layers cached full keys and values in bf16, that would be about 2.95 MB per token, or 2.9 TB at 1M tokens. Nobody serves that; it shows why latent attention was already a big step before the hybrid.

What does that mean for serving?

The vLLM recipe on the card uses tensor parallel 8, expert parallel, prefix caching, --gpu-memory-utilization 0.85 and MTP with 3 speculative tokens. The SGLang recipe targets 8 x H200 with TP8 and EP8, and 4 x GB300 with TP4 and EP4. Take 8 x 141 GB at 0.85 utilization and subtract the 573.8 GB of weights: about 385 GB is left for cache. That ignores activations, CUDA graphs, fragmentation and the MTP layer, so treat it as an upper bound.

Context	Hybrid layout	All-latent layout (hypothetical)
128K	237 sequences	35
262K	129	17
1M	36	4

This is illustrative arithmetic, not a measured throughput. Real concurrency depends on how many requests actually reach full length (most do not), on prefix-cache hits and on batch scheduling. But the direction is robust: a long-running agent that compacts its context every few hours holds many more live sessions per node with the hybrid layout.

Notice the other side of the ledger: the FP8 weights alone need 8 high-memory GPUs in the official recipe. The cache stopped being the dominant memory item at long context; the weights took over. That is why quantization became the next battleground.

How fast did the community react?

Searching the Hugging Face model API on 2026-10-19 for "Ling-3.1-flash" returns 13 repositories created between 2026-10-14 and 2026-10-17 besides the official one. Formats include GGUF builds (for example bartowski/Ling-3.1-flash-GGUF), EXL3 builds such as turboderp/Ling-3.1-flash-exl3 and NVFP4 variants. Third-party coverage (for example AlphaSignal) describes a mixed MXFP4 and block-FP8 recipe that cuts weights from 534 GB to 292 GB for 4-GPU Hopper serving. I could not find numbers on accuracy retention for that recipe, so treat the 292 GB figure as a size claim only.

Two practical cautions for anyone downloading community quants of a hybrid model:

Quantizers must understand the KDA layers. Linear-attention state and gates are sensitive to precision. The official config keeps router, gate and several attention projections out of FP8 conversion (see modules_to_not_convert). A quant that ignores that list can degrade quality in ways a short perplexity run will not show.
Run a long-context retrieval test on your own data before trusting any quant. Short benchmarks do not exercise the 9 full-attention layers that carry exact recall.

How should I decide whether a hybrid fits my workload?

A short checklist:

Context profile. If your p95 request is under 25K tokens, the constant KDA state eats the savings and a conventional model may be simpler. Long agent sessions and document pipelines are where hybrids pay off.
Recall sensitivity. Tasks that quote exact strings from far back (code references, contract clauses) lean on the full-attention layers. Test them.
Engine support. The vLLM recipe needs a dedicated branch and --mamba-cache-mode align, which tells you the linear state is managed like recurrent-model state. Prefix caching behaves differently for state-based layers than for KV caches. Verify cache hit rates in your own traffic.
Hardware budget. Count weights first (573.8 GB FP8), then cache, then headroom.

FAQ

What is Ling-3.1-flash?
An MIT-licensed hybrid-linear mixture-of-experts language model with 560B total and 25B active parameters, published on Hugging Face under the inclusionAI organization. Its repository was created on 2026-10-13.

What does a 7:1 hybrid ratio mean?
For every 8 layers, 7 use linear attention (KDA) with constant memory per sequence and 1 uses gated MLA with a cache that grows with context. Across 72 layers that is 63 KDA and 9 MLA.

How much KV cache does it need at 1M tokens?
From the config, about 10.4 KB per token for the 9 MLA layers, so roughly 10.4 GB at 1M tokens in bf16, plus a constant linear-attention state of about 0.26 GB under the assumptions listed above. The all-latent comparison (about 83 GB) is hypothetical, not a released model.

Can it run on a single GPU?
Not at the official FP8 size of 573.8 GB. The card's recipes use 8 x H200 or 4 x GB300. Community quantizations reduce the footprint, but check accuracy on your tasks.

Is the 1M-token window confirmed?
The model card states a 1M-token context window, while config.json lists 262,144 positions. Confirm the effective limit in your serving engine and test retrieval at the lengths you plan to use.

Does linear attention hurt accuracy?
It can on exact recall, which is why hybrids keep a few full-attention layers. Whether 7:1 is enough is a question for long-context benchmarks on independent hardware.

DEV Community