DEV Community

Cover image for Tencent Hy4 flash: 220B parameters, 12B active, and what the KV cache and license mean for self-hosting
ai maya
ai maya

Posted on

Tencent Hy4 flash: 220B parameters, 12B active, and what the KV cache and license mean for self-hosting

TL;DR

  • Tencent's Hy4 flash, released on Hugging Face on November 17, 2026, is a multimodal mixture-of-experts model with 220B total parameters, about 12B active per token, and a 1M-token context window. The weights are open, under the Tencent Hy License.
  • The headline size is misleading in a useful way: 63.5B of the 220B parameters are Engram lookup tables, which are fetched by hash and not multiplied through. The compute-relevant backbone is 155.5B.
  • Only 12 of its 42 layers attend over the full context. The other 30 use a 512-token sliding window. By my arithmetic, that cuts the KV cache at 1M tokens to roughly 6.9 GB per sequence at one byte per element, against about 24 GB if every layer were full.
  • The files are large: 440.5 GB in BF16, 287.1 GB in FP8, 168.3 GB for the Q4_K_M GGUF. The official recipe uses 8 GPUs.
  • The license has a revenue trigger. Past US$50 million in annual revenue, offering the model as an API needs a separate written license.

Hy4 flash download size by format

What did Tencent actually release?

Three checkpoints: the instruct model (tencent/Hy4-flash), an FP8 build (tencent/Hy4-flash-FP8), and a pre-trained base. A GGUF set from the AngelSlim team sits at AngelSlim/Hy4-flash-GGUF, which the model card describes as "for end-to-end testing purposes".

The model card gives these headline properties:

Property Value
Total parameters 220B (155.5B backbone + 63.5B Engram + 0.43B vision + 0.85B drafter)
Active per token 11.3B text, plus 0.43B for the vision encoder
Layers 42: one dense, 41 MoE
Experts 512 routed, 16 active per token, plus 1 shared
Context 1,048,576 tokens
Inputs Text and images

I checked the sum against the Hub's own metadata. The safetensors header lists 220,251,201,958 parameters in total, and 155.5 + 63.5 + 0.43 + 0.85 comes to 220.28B. The card and the files agree.

Why does a 220B model only read about 12B per token?

Two separate mechanisms are at work, and they are easy to conflate.

Sparse experts. Each MoE layer holds 512 routed experts with an intermediate size of 1,536, and the router picks 16 per token. The experts operate in a lower-dimensional latent space (1,536), which the card calls LatentMoE. Per-token compute scales with the 16 chosen experts, not the 512.

Engram. This is the part that makes the 220B number look bigger than the work done. The backbone is augmented with learned bigram and trigram embeddings, retrieved with multi-head hash lookups. The numbers in config.json let you rebuild the table size:

heads_per_order = 8
vocab_per_head  = 10_312_000   # approx, config lists primes near this value
dim_per_head    = 96           # 768 / 8
orders          = 2            # bigram, trigram
layers          = 4            # engram_layer_ids = [2, 12, 22, 32]

params = heads_per_order * vocab_per_head * dim_per_head * orders * layers
print(f"{params/1e9:.1f}B")    # 63.4B
Enter fullscreen mode Exit fullscreen mode

That reproduces the card's 63.4B embedding table. Lookups are table reads at hashed positions, so they add memory but almost no arithmetic. For capacity planning, treat Engram as a big, cheap-to-read table: it costs you VRAM or host memory, and not much decode time.

One small inconsistency worth knowing about if you script against the files: the README table lists the Engram layer indices as 1, 11, 21, 31, while config.json has engram_layer_ids of 2, 12, 22, 32. That looks like a zero-based versus one-based labeling difference, and the config is what the code reads.

How much memory do the weights need?

From the Hub, the used storage is 440.5 GB for the BF16 repo and 287.1 GB for the FP8 repo. The GGUF files add more reference points: Q4_K_M is 105.0 GB plus 63.4 GB across two shards (168.3 GB), Q3_K is 155.4 GB, and IQ2_AH is 135.0 GB.

The second shard in each GGUF set is the same size, 63.36 GB. That is close to the Engram table at roughly one byte per element, which suggests the lookup tables are kept at a fixed precision across these quant levels while the backbone shrinks. I am inferring that from file sizes, not from the conversion code, so treat it as a hypothesis.

Per-GPU arithmetic for the official 8-way tensor-parallel setup:

fp8_gb  = 287.1
bf16_gb = 440.5
for name, gb in [("FP8", fp8_gb), ("BF16", bf16_gb)]:
    per_gpu = gb / 8
    print(name, f"{per_gpu:.1f} GB/GPU", f"-> {80 - per_gpu:.1f} GB left on an 80 GB card")
# FP8  35.9 GB/GPU -> 44.1 GB left on an 80 GB card
# BF16 55.1 GB/GPU -> 24.9 GB left on an 80 GB card
Enter fullscreen mode Exit fullscreen mode

So FP8 on eight 80 GB cards leaves a comfortable margin for KV cache and activations. BF16 fits, but with far less room for long contexts. This is the weight footprint only, and real deployments also reserve memory for CUDA graphs and the drafter.

How big is the KV cache at 1M tokens?

This is where the architecture earns its keep. The backbone interleaves attention types in a fixed pattern: two sparse layers, then ten repeats of three sliding-window layers followed by one sparse layer. That gives 12 sparse layers and 30 sliding-window layers. Sparse layers pick up to 2,048 positions through a group-level selection. Sliding-window layers see a 512-token causal window.

Here is a back-of-envelope estimate. It assumes the cache stores the MLA latent, which is kv_lora_rank 512 plus 64 RoPE dimensions, or 576 elements per token per layer, at one byte per element:

latent = 512 + 64            # elements per token per layer
ctx    = 1_000_000
full_all  = 42 * latent * ctx / 1e9
hy4f_full = 12 * latent * ctx / 1e9
hy4f_swa  = 30 * latent * 512 / 1e9
print(round(full_all, 1), round(hy4f_full, 1), round(hy4f_swa, 3))
# 24.2 6.9 0.009
Enter fullscreen mode Exit fullscreen mode

Illustrative KV cache at 1M tokens

That is roughly 6.9 GB for the full-context layers plus 9 MB for the windowed ones, against 24.2 GB if all 42 layers cached the whole context. The layout alone gives about 3.5x. Tencent states that the combined effect of hybrid attention, LatentMoE and Engram is a KV cache of 1/24 that of Hy3 and 1/6 of Hy4, which is a comparison against its own earlier models and a different quantity from my illustration.

Two cautions. First, the estimate depends on the cache storing the compressed latent. The config also lists 32 key-value heads and a 256-dimension value head, and an implementation that cached expanded keys and values would need far more. The vLLM recipe pairs the model with the FLASHMLA_SPARSE attention backend, which points to a latent cache, but measure it on your own stack. Second, one-byte KV storage is my assumption. At BF16 the numbers double.

What does decoding cost per token?

A memory-bandwidth roofline is the quickest sanity check. At batch size 1, each generated token must read the active weights once:

active_b = 11.3  # billions, text path
for name, bytes_per in [("BF16", 2), ("FP8", 1), ("4-bit", 0.5)]:
    gb = active_b * bytes_per
    print(name, f"{gb:.2f} GB/token", f"-> {gb*150/1000:.2f} TB/s at 150 tok/s")
Enter fullscreen mode Exit fullscreen mode

Illustrative weight bytes per token

At 4-bit that is about 5.65 GB per token, or 0.85 TB/s for 150 tokens per second. Tencent's claim is that megakernels, 4-bit quantization and DFly2 together push decoding past 150 tokens/s. That is plausible against this arithmetic on eight modern accelerators, since their aggregate bandwidth is several times the requirement. It is a vendor claim, and the card does not state the hardware, batch size or context length behind it.

DFly2 is worth a closer look. It is a 0.85B, five-layer drafter for speculative decoding. Each draft layer shares the KV cache and the indexer's selected positions with a sparse attention layer of the target model. Three consequences follow from the card: each draft token attends to at most about 2,000 keys regardless of context length, the drafter needs no KV memory of its own, and it takes no part in prefill. The recipe sets five speculative tokens.

How do I serve it?

The model card points to prebuilt images for both major engines. The vLLM version:

docker run --gpus all -p 8000:8000 --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:hy4-flash tencent/Hy4-flash-FP8 \
    --tensor-parallel-size 8 \
    --speculative-config '{"num_speculative_tokens":5,"method":"dfly"}' \
    --attention-backend FLASHMLA_SPARSE \
    --tool-call-parser hy_v4 \
    --reasoning-parser hy_v4 \
    --enable-auto-tool-choice \
    --served-model-name hy4-flash
Enter fullscreen mode Exit fullscreen mode

Reasoning is controlled per request. The model supports "high" (the default, with thinking enabled) and "no_think":

client.chat.completions.create(
    model="hy4-flash",
    messages=[{"role": "user", "content": "Summarize this contract clause."}],
    temperature=0.9, top_p=1.0,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
Enter fullscreen mode Exit fullscreen mode

For the Transformers path, the card says support will be merged into the main branch soon and gives a pinned pull-request install in the meantime. If you want a plain release install, wait for the merge.

What do the benchmark and cost claims say?

The card compares the base model with Hy4, Hy3 preview, DeepSeek-V4-flash-Base and DeepSeek-V4-Pro-Base. A few rows from the few-shot table:

Benchmark Hy4 flash Base DeepSeek-V4-flash-Base
MMLU (5-shot) 88.7 89.7
MMLU-Pro 71.1 68.6
SuperGPQA 57.8 52.5
U-MATH 69.8 57.4
SimpleQA 37.5 31.3

The card also notes that its base model saw synthetic instruction data in mid-training and sometimes misses the expected answer format, which may distort few-shot scores. These are Tencent's own runs.

The cost story is the center of the release. On WorkIndex, an internal benchmark of 8 office-style task sets (slides, document editing, spreadsheet repair, audio and video processing, visual understanding), Hy4 flash scores 42.1 against 39.7 for Hy4 preview and 51.4 for Hy4. Average cost per task is stated as 1/13 of Hy4 preview, 1/20 of Hy4 and 1/8 of DeepSeek-V4.1-Flash-1017 at peak-hour Max-tier pricing. In a blind evaluation, user satisfaction was 2.79 out of 4 against 2.81 for Claude Haiku 5.5, with a median completion time of 6.7 minutes and RMB 0.26 per task.

Treat those as vendor-run numbers. WorkIndex is internal, and the blind test used 85 internal WorkBuddy users across 259 tasks in 27 categories. They describe the workload Tencent cares about and are not a substitute for your own task set. The card recommends Hy4 flash for everyday productivity work such as deep search, document processing, slides and computer use, and points to the larger Hy4 for long-horizon software engineering.

Can you use it commercially?

The Tencent Hy License is derived from Apache 2.0 with an added commercial section. Reading Section 10:

  • "MaaS" means giving third parties remote access to inference or fine-tuning, for example through an API.
  • If you run a MaaS business, you need a separate written license in two cases: aggregate worldwide gross revenue above US$50 million in any 12-month period, whether or not it comes from the model, or use in a product with more than 100 million monthly active users.
  • Internal use is exempt as long as neither the model, derivatives nor outputs are made available to third parties.
  • Derivatives explicitly include fine-tuned, quantized, distilled and merged weights.

Self-hosting for your own product features is generally the easy case. Reselling the model as an endpoint is where the revenue threshold matters. This is a summary of the text and not legal advice.

FAQ

How many parameters does Hy4 flash activate per token?
About 12B in the headline figure: 11.3B on the text path plus 0.43B for the vision encoder.

Does the 1M-token context fit on a single node?
On the arithmetic above, the KV cache for one 1M-token sequence is on the order of 7 GB at one byte per element, if the cache stores the latent. Weights dominate, at 287 GB in FP8.

What is Engram in this model?
Hashed bigram and trigram embedding tables totaling about 63.4B parameters across four layers. They add memory, not much compute.

How many GPUs does the official recipe use?
Eight, with tensor parallelism 8, FP8 weights and five speculative tokens through DFly2.

Is it open source?
The weights, config and fine-tuning scripts are public, but the license is the Tencent Hy License, which adds MaaS terms. It is not plain Apache 2.0.

Can I run it on a laptop?
No. The smallest GGUF in the reference set, IQ2_AH, is about 135 GB.

Links

Further reading

Top comments (0)