Hy4 Preview vs Hy3: 770B MoE, 1M Context & Deployment Guide

Released in August 2026, Tencent’s Hy4 Preview marks a major upgrade to its hybrid‑expert large‑model series. With 770B total parameters and 49B activated parameters, this model comes merely three days after the public release of Hy3. These two generations represent typical MoE‑based model iterations, covering architectural redesign, benchmark performance, hardware requirements, and practical production deployment guidance. This article dissects the core differences between Hy3 and Hy4 Preview, sorts out underlying optimizations, real‑world test metrics, deployment pitfalls, and provides practical suggestions for engineering teams.

1. Architectural Evolution: Comprehensive Restructuring for Sparse Attention and Multi‑Path Residual Connections

Superficially, Hy4 Preview only increases total parameters and context window. However, its core innovation lies in the complete rework of two Transformer building blocks: the attention mechanism and residual pathways. The mixture‑of‑experts routing system has also received substantial optimization.

1.1 MoE Layer: Advancements in Expert Count and Routing Logic

Both generations adopt a hybrid MoE architecture: dense feed‑forward layers occupy the front part of the network, while later layers switch to MoE design to balance capability and computational overhead.

Hy3 MoE Configuration

• Total layers: 80 layers; first 10 layers are dense FFN, layers 11‑70 adopt MoE structure
• Each MoE layer contains 19 static experts, 2 experts activated per token
• Shared experts: 1 shared expert per layer, cooperating with independent experts to balance load
• Total parameters: 295B, activated parameters: 21B

Hy4 Preview MoE Configuration

• Total layers: 78 layers; first 10 layers remain dense FFN, layers 11‑77 switch to MoE
• Each MoE layer holds 25 experts, 2 experts activated for every token
• Total parameters: 770B, activated parameters: 49B

The expansion of expert count brings stronger fitting capacity. Each token can select from a larger expert pool to handle specialized tasks such as logical reasoning and complex document analysis. A shared‑expert mechanism is retained. When token distribution shifts drastically, shared experts can stabilize partial computation and mitigate routing oscillation, which is a well‑known pain point for MoE models.

1.2 Attention Mechanism: From GQA to Gated DSA + IndexCache

The attention module constitutes the most critical architectural upgrade between Hy3 and Hy4 Preview.

Hy3 uses standard GQA (Grouped‑Query Attention). It compresses KV cache overhead and achieves good performance within its native 256K context window. Nevertheless, standard attention brings O(n²) computational complexity. When context expands to 1M tokens, raw attention computation imposes prohibitive pressure on inference latency and memory bandwidth.

Hy4 Preview implements Gated Decoupled Sparse Attention (Gated DSA) paired with IndexCache. Drawing from existing open‑source sparse‑attention research, Tencent introduces practical engineering tweaks for production scenarios.

Gated DSA runs in two phases. First, it executes coarse‑grained retrieval: each token selects top‑2048 most relevant context tokens. This step reduces the computation scope drastically. Second, it conducts precise attention calculation only on this filtered subset. Statistically, merely 0.2% of total token pairs participate in heavy‑duty attention calculation, delivering substantial latency reduction under long‑context conditions.

A gating control module is embedded inside Gated DSA. If critical information risks being filtered out during sparse retrieval, the gate triggers fallback full attention computation to avoid information loss. This dynamic switch runs automatically without manual intervention.

IndexCache further optimizes repeated index computation. Index results from upper layers are cached and reused by subsequent layers. Re‑computation only happens at certain intervals. This strategy cuts indexing overhead to a fraction of original cost and delivers obvious speed gains under 1M‑token prompts, especially for pre‑fill phases.

Combined, these mechanisms enable Hy4 Preview to support 1M context length while preserving competitive generation speed. According to official data, pre‑fill speed improves by more than one order of magnitude compared with vanilla dense attention.

1.3 Residual Pathways: iHC Identity‑Hook Connections

Hy3 applies conventional single‑stream residual connections. All information flows through one single pathway. When context grows extremely long, information attenuation and gradient degradation may emerge in deep layers.

Hy4 Preview introduces iHC (Identity‑Hook Connections). It expands residual flow from one lane to four parallel lanes. Multiple independent residual branches deliver identity mapping, ensuring original signal can propagate stably from bottom layers to top layers. This design improves model robustness for ultra‑long‑context reasoning and complex multi‑turn workflows.

1.4 MTP Layer: Multi‑Token Prediction Upgrade from 3.8B to 10B

Both models integrate native MTP (Multi‑Token Prediction) layers for speculative decoding acceleration. Unlike third‑party speculative decoding schemes, MTP is built natively inside the model weight, eliminating compatibility risks caused by mismatched assistant‑model output distribution.

• Hy3 MTP: 3.8B total parameters, 0.3B activated parameters, predicts up to 3 future tokens
• Hy4 Preview MTP: 10B total parameters, 0.7B activated parameters, predicts up to 3 future tokens

When MTP is enabled within vLLM inference frameworks, end‑to‑end generation throughput can increase by 30%‑50%. Native integration avoids style inconsistency and hallucination issues frequently observed in external speculative‑decoding implementations.

1.5 Core Architecture Parameter Comparison

Parameter	Hy3	Hy4 Preview
Architecture	MoE(Dense+MoE)	MoE(Dense+MoE)
Total Parameters	295B	770B
Activated Parameters	21B / token	49B / token
Layers	80 (10 dense +70 MoE)	78 (10 dense +68 MoE)
Expert Setup	19 experts, 1 shared expert	25 experts,1 shared expert
Attention	GQA (64Q /8KV)	Gated DSA + IndexCache
MTP Params	3.8B /0.38B activated	10B /0.7B activated
Max Context	256K	1M

2. In‑Depth Performance Evaluation: Capability Gains and Remaining Weaknesses

Architectural upgrades must be validated against real benchmarks and practical tasks. This section combines official published test results and observable practical feedback.

2.1 Comprehensive Benchmark: Advancing toward Top‑tier Open‑Source Models

Hy3 already achieved competitive scores among open‑source MoE models. Hy4 Preview lifts overall performance to a new level. According to official test data, its average score reaches 2.99. Compared with GLM‑5.3‑92 and Kimi K3‑94, win‑rate difference sits within 0.5%. For most general‑purpose tasks, Hy4 Preview is on par with mainstream top open‑source models.

Most performance improvements come from agent capability and tool invocation. Pure reasoning gain is relatively moderate. This indicates the iteration focuses more on practical engineering scenarios rather than pure mathematical reasoning benchmarks.

2.2 Long‑Context Capability: 256K to 1M Is Not Merely a Numerical Upgrade

Many open‑source models advertise million‑token context windows yet fail to retrieve key information accurately in practice. Hy3’s native 256K context already performs well, maintaining over 90% recall for key‑information lookup at 200K.

Hy4 Preview extends the theoretical window to 1M tokens. Thanks to Gated DSA and gating fallback mechanisms, key‑information recall can stay above 85% under real‑world 1M‑token inputs. It shows obvious advantages in large‑document processing scenarios. Typical applicable workloads include full‑repository code comprehension, multi‑thousand‑page legal‑document parsing, and long‑dialogue full‑history retention.

For most daily businesses, 256K‑320K is sufficient. Teams should evaluate real‑world demands before enabling full 1M context, as pre‑fill resource consumption rises significantly.

2.3 Code and Agent Capability: The Most Noticeable Productivity Breakthrough

Code and agent scenarios represent Hy4 Preview’s largest improvement compared with Hy3. Official benchmark data shows major gains on SWE‑Bench and SWE‑Marathon. These benchmarks simulate real‑world software‑engineering workflows: requirement understanding, multi‑file modification, debugging, and iteration, instead of simple code snippet completion.

Three factors drive this leap:

1. Architectural benefit: iHC multi‑path residual structure stabilizes long‑chain tool‑call state transmission.
2. Sparse‑attention cooperation: Intermediate states of multi‑step agent workflows can be retained efficiently.
3. Post‑training optimization: Massive agent‑oriented fine‑tuning covering tool invocation, self‑correction, and process planning.

Practical tests show Hy4 Preview can process medium‑size Spring‑Boot projects with tens of thousands lines of code. It completes architecture migration, configuration adjustment, and unit‑test modification with high completion rate. Hy3 tends to passively follow detailed human‑written instructions; Hy4 Preview can autonomously decompose objectives and complete multi‑step workflows.

2.4 Mathematical Reasoning: Progress with Persistent Limitations

Hy4 Preview obtains limited improvement on mathematical reasoning. It achieves decent scores on medium‑difficulty math problems, but still falls behind specialized reasoning‑focused models on Olympiad‑level hard problems. For scenarios dominated by mathematical derivation and scientific computation, teams still need to consider alternative specialized models.

2.5 Inference Speed and Cost: Larger Activated Parameters Do Not Equal Proportional Cost Inflation

Hy4 Preview raises activated parameters from 21B to 49B per token. Yet sparse‑attention mechanisms reduce effective computation under long prompts. In long‑context scenarios, actual token‑generation overhead may be merely 1.2‑1.8 times that of Hy3, instead of the 2.3x raw parameter ratio.

From the perspective of public API pricing, Hy4 Preview delivers stronger comprehensive capability with moderate cost growth. When deploying privately, hardware requirements rise sharply. BF16 inference for Hy4 Preview requires roughly 1.5TB‑1.7TB VRAM, demanding multiple high‑end A100 or H200 nodes.

When running multi‑model services, developers can route traffic through an API gateway such as 4sapi to unify access for Hy‑series and other open‑source models, reducing repetitive adaptation work.

3. Practical Engineering Deployment: Complete Guide from Environment Setup to Production

Advanced architecture brings extra deployment complexity. Below covers hardware requirements, sample code, inference framework notes, quantization trade‑offs, and frequent pitfalls.

3.1 Hardware Requirement Reference

Different quantization schemes create distinct hardware requirements. FP16/BF16 requires the largest resource footprint; FP8 can cut memory consumption with acceptable quality loss.
|Model|Precision|Minimum VRAM|Recommended Hardware|Generation Speed|
|---|---|---|---|---|
|Hy3|BF16|~350GB|8×H20‑80|30‑40 tok/s|
|Hy3|FP8|~150GB|4×H20‑80|20‑30 tok/s|
|Hy4 Preview|BF16|~1540GB|16×H200‑e /8×H100‑80|25‑35 tok/s|
|Hy4 Preview|FP8|~770GB|8×H200‑e /4×H100‑80|30‑40 tok/s|

Key notes:

1. Speed reflects batch‑processing throughput; single‑request interactive speed will be lower.
2. Enabling native MTP speculative decoding can boost generation speed by 30%‑50%.
3. FP8 quantization brings obvious memory reduction, but users must verify output quality for business scenarios.

3.2 Transformers Native Invocation

Transformers version ≥5.6.0 is required. Hy4 Preview needs special attention for sparse‑attention related parameters. If hardware does not support sparse‑attention kernels, users can disable Gated DSA and fall back to vanilla GQA, at the cost of long‑context performance and speed.

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "tencent/hy4-preview"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="bfloat16",
    trust_remote_code=True,
    device_map="auto",
    attn_implementation="gated_dsa_2"
)

3.3 vLLM Production Deployment

vLLM supports Hy‑series models, including MTP speculative‑decoding modules. Command‑line startup must specify custom attention and MTP parameters. After startup, services expose OpenAI‑compatible API endpoints for upper‑layer business integration.

vllm serve tencent/hy4-preview \
--tensor-parallel-size 8 \
--speculative-model tencent/hy4-preview \
--num-speculative-tokens 3 \
--attn-backend gated_dsa

3.4 Quantization Trade‑offs

• FP8 quantization: Most balanced option for production deployment; quality degradation stays within 5%‑10% for most tasks.
• 4‑bit quantization: Memory footprint drops significantly, yet code‑generation and long‑context recall performance degrade noticeably. Not recommended for agent‑heavy production workflows.

3.5 Common Deployment Pitfalls

1. trust_remote_code=True must be enabled. Both Hy3 and Hy4 Preview contain custom model implementations inside weight repositories.
2. Sparse‑attention kernels impose extra requirements on GPU compute capability. Old‑generation hardware may lack full support.
3. MTP speculative decoding works best under batch‑processing scenarios. Speed gain is limited for single‑request interactive workloads.
4. For 1M‑token inputs, pre‑fill phase consumes massive compute resources. Production systems should implement request throttling.

4. Selection Advice and Practical Application Scenarios

4.1 Choose Hy3 When

1. Service traffic is large, and cost per token is the primary constraint. Hy3 delivers acceptable general‑purpose capability at lower activation cost.
2. Business scenarios are concentrated within 256K context range, without heavy agent or complex multi‑file code modification requirements.
3. Existing hardware resources cannot satisfy Hy4 Preview’s massive multi‑GPU cluster requirements.

4.2 Choose Hy4 Preview When

1. Workloads involve million‑token long‑document comprehension, full‑code‑repository analysis, multi‑turn complex agent execution.
2. Code generation, tool invocation, autonomous multi‑step task decomposition are core business demands.
3. Hardware budget supports large‑scale multi‑GPU cluster deployment, and teams can accept higher inference costs for capability improvement.

4.3 Not Recommended Scenarios

1. Pure heavy‑duty mathematical reasoning and competitive‑level mathematical problem solving. Prefer reasoning‑specialized models.
2. Ultra‑high‑throughput simple chat services with strict cost constraints, where Hy4 Preview’s extra capability cannot generate business value.

5. Conclusion

Hy4 Preview represents a substantial architectural upgrade rather than a simple parameter expansion. MoE expert‑count expansion, Gated DSA plus IndexCache sparse‑attention system, multi‑path iHC residual connections, and enhanced native MTP speculative‑decoding work together to lift long‑context processing, code capability and agent performance to new heights.

Meanwhile, the model retains clear boundaries: mathematical reasoning still has room for improvement, and production deployment imposes high requirements on GPU resources and engineering capability. Engineering teams ought to balance capability gains against hardware cost, conduct targeted verification for core business cases, and select Hy3 or Hy4 Preview according to real‑world requirement instead of blindly chasing larger‑parameter versions. When mixing multiple model services, unified gateway access helps reduce repetitive adaptation overhead.

Learn more: https://4sapi.com