TL;DR
Baidu published ERNIE-5.5-Flash-Base on Hugging Face on November 5: a 280B-parameter mixture-of-experts base model with 14B active parameters per token, trained on 33.5T tokens. The FP8 repository is 288.4 GB (measured from the file listing), so it fits on eight 80 GB GPUs with room to spare and on four 80 GB GPUs only with about 32 GB of headroom for everything else. The attention design is the interesting engineering part: by my estimate from the published config, one 1,048,576-token sequence needs roughly 5 GB of KV cache at FP8, against about 185 GB for a conventional full-attention stack of the same depth.
The harder question is not memory. The license (ERNIE 5.5 Flash Base Model License Agreement v1.0) grants rights only in a "Territory" that excludes the European Union, the United Kingdom and South Korea, forbids using outputs to improve any other model, and counts the full revenue of any product that uses the model when it decides which commercial tier you are in. Read it before you plan a deployment.
Everything below marked "my estimate" or "illustrative" is arithmetic I did from the public config and stated assumptions. Benchmark numbers are Baidu's own, from the model card.
What did Baidu actually release?
Two repositories, both public and ungated: baidu/ERNIE-5.5-Flash-Base in FP8 and baidu/ERNIE-5.5-Flash-Base-BF16. Both are 128 safetensors shards. It is a text-only base checkpoint taken at the end of pre-training, before any mid-training or post-training, so it will complete text rather than follow chat instructions. The README says to call the completions endpoint, not chat.
| Item | Value (source: model card and config) |
|---|---|
| Total / active parameters | 280B / 14B per token |
| Layers | 43 (1 dense FFN, 42 MoE) |
| Experts | 512 routed (top-10) plus 1 shared per MoE layer |
| Hidden size / vocabulary | 4,096 / 201,216 |
| Pre-training | 33.5T tokens, up to 128K context |
| Inference context | 1,048,576 tokens enabled in the released config |
| License | custom, "license: other" on the Hub |
Per the Hub metadata the FP8 repo holds 286.2B parameters in total, almost all of them stored as FP8 (E4M3) with block scales. Adding up the stored dtypes gives 288.35 GB, which matches the 288.43 GB shard total within rounding.
How much GPU memory does it need?
Weights first, because they are the floor. This is the measured file size against the memory pools people actually own:
| Setup | Pool | Free after FP8 weights (288.4 GB) | Free after BF16 weights (572.9 GB) |
|---|---|---|---|
| 4 x 80 GB | 320 GB | 31.6 GB | does not fit |
| 8 x 80 GB | 640 GB | 351.6 GB | 67.1 GB |
| 4 x 141 GB | 564 GB | 275.6 GB | does not fit (short by 8.9 GB) |
| 8 x 141 GB | 1,128 GB | 839.6 GB | 555.1 GB |
The 4 x 80 GB row is where people will be tempted, because the README's launch commands use four-way parallelism. Keep in mind that 31.6 GB of free memory has to cover activations, CUDA graphs, communication buffers, the KV cache for every concurrent request, and per-GPU imbalance. It can work for short contexts and low concurrency, but it is not a comfortable production target. Eight 80 GB GPUs or four 141 GB GPUs are the first setups where the weights stop being the main constraint. The SGLang cookbook linked from the README preselects a B300 single-node FP8 low-latency recipe, so Baidu's own reference path is on newer hardware than H100.
Here is a short script that reproduces the weight number from Hub metadata, so you can rerun it for any checkpoint:
import json, urllib.request
def get(url):
return json.load(urllib.request.urlopen(url))
repo = "baidu/ERNIE-5.5-Flash-Base"
params = get(f"https://huggingface.co/api/models/{repo}")["safetensors"]["parameters"]
bytes_per = {"F8_E4M3": 1, "BF16": 2, "F16": 2, "F32": 4}
weights_gb = sum(n * bytes_per[d] for d, n in params.items()) / 1e9
print(f"{sum(params.values())/1e9:.1f}B params, {weights_gb:.1f} GB of weights")
Why is the KV cache so small at a million tokens?
This is the part of the design that changes serving cost. According to the model card, 37 of the 43 layers use Heavily Compressed Attention (HCA), which caches one entry per 128 tokens, and 6 layers (one in seven) use multi-head latent attention with sparse attention (MLA plus DSA) to keep full-context access. The config agrees: compress_ratios has 37 entries of 128, and the remaining entries mark the non-compressed layers. The config also lists kv_lora_rank of 512, a 64-dimension rope part, and a sparse-attention indexer with 128-dimension heads.
My estimate for one 1,048,576-token sequence, with the assumptions spelled out:
- MLA layers: (512 latent + 64 rope) values per token per layer, 6 layers, 1 byte each at FP8. That is 3.62 GB.
- Indexer keys: 128 values per token per MLA layer, 6 layers, 1 byte each. That is 0.81 GB.
- HCA layers: one KV entry per 128 tokens, which I assume is one head of dimension 512 for keys and 512 for values (1,024 bytes at FP8), across 37 layers. That is 0.31 GB.
- Total: about 4.7 GB at FP8, about 9.5 GB at BF16.
For scale, a hypothetical full-attention stack with the same 43 layers and a typical grouped-query layout (8 KV heads of dimension 128, BF16) would need 43 x 2 x 8 x 128 x 2 bytes = 176 KB per token, or about 185 GB for the same million tokens. I made that baseline up to give a yardstick, it is not any specific model. Against it, the ERNIE layout is roughly 20x smaller in BF16 and 39x smaller in FP8.
There are two caveats. First, my HCA entry size is a guess about the implementation, so treat the 0.31 GB as an order of magnitude. Second, I left out the sliding window of 128 full-resolution tokens the config lists, which adds only a few megabytes per sequence. Even if I am off by 2x on everything, the conclusion holds: the KV cache stops being the thing that limits concurrency at long context, and the weights do. The README's vLLM command also sets --kv-cache-dtype fp8.
How fast can it decode? A bandwidth ceiling
Decode on a large MoE is a memory-bandwidth problem, and the interesting variable is batch size. At batch 1 you read only the experts the token routed to, about 14 GB of FP8 weights. As more sequences share a step, the union of experts touched grows, and with 512 experts and top-10 routing it grows fast.
From the config, each routed expert has three 2,048 x 2,048 matrices (my assumption: a gated unit), so 12.6M parameters per expert and about 270.6B routed-expert parameters across 42 layers. That leaves roughly 15.7B parameters elsewhere, of which about 8.7B are read per token. The two pieces add up to the advertised 14B active. Under uniform random routing, the fraction of experts touched at batch B is 1 - (1 - 10/512)^B:
| Batch | Experts touched | Weights read per step | Step time (4 x H100, 13.4 TB/s) | Aggregate ceiling | Per-stream ceiling |
|---|---|---|---|---|---|
| 1 | 2% | 14.0 GB | 1.04 ms | 957 tok/s | 957 tok/s |
| 8 | 15% | 48.2 GB | 3.60 ms | 2,223 tok/s | 278 tok/s |
| 32 | 47% | 135.4 GB | 10.1 ms | 3,168 tok/s | 99 tok/s |
| 128 | 92% | 257.6 GB | 19.2 ms | 6,658 tok/s | 52 tok/s |
| 512 | 100% | 279.3 GB | 20.8 ms | 24,565 tok/s | 48 tok/s |
This is a ceiling, not a benchmark. It assumes perfect sharding across four GPUs with 3.35 TB/s each, no communication cost, no attention or KV reads, and uniformly random routing. Real routing is skewed, which touches fewer experts and helps, but all-to-all expert communication and kernel overheads hurt much more. The useful reading is the shape: per-stream speed collapses from about 960 to under 60 tokens per second between batch 1 and batch 128, while aggregate throughput rises about 7x. If you serve a latency-sensitive product, you are choosing a point on this curve, and you will want to measure on your own hardware with your own traffic.
How does it compare on benchmarks?
Baidu's table compares ERNIE-5.5-Flash-Base with DeepSeek-V4-Flash-Base (284B-A13B), Hy3-preview-Base (295B-A21B) and MiMo-V2.5-Base (310B-A15B), all run through the same pipeline, prompts and few-shot settings. These are the vendor's own numbers and I have not reproduced them.
| Benchmark | ERNIE-5.5-Flash-Base | DeepSeek-V4-Flash-Base |
|---|---|---|
| MMLU (5-shot) | 93.3 | 89.4 |
| MMLU-Pro (5-shot) | 78.4 | 69.2 |
| GPQA (5-shot) | 60.2 | 55.9 |
| HumanEval+ (1-shot) | 82.3 | 69.5 |
| LiveCodeBench (1-shot) | 40.1 | 31.9 |
| ChineseSimpleQA | 73.0 | 75.3 |
| HELMET-Rerank | 56.2 | 60.0 |
| InfiniteBench | 46.5 | 48.0 |
ERNIE leads on most rows, but not all of them: the DeepSeek base is ahead on ChineseSimpleQA, HELMET-Rerank, InfiniteBench and INCLUDE-lite (79.8 against 79.5). A base-model few-shot table also says little about how the model behaves after you fine-tune it, which is the intended use here.
What does the license actually allow?
I am not a lawyer and this is a plain reading of the text in the repository, so check it with counsel before shipping. These are the clauses that change deployment decisions:
- Territory (sections 1.11, 2, 5.3). Rights are granted only in "all geographical areas worldwide other than the European Union, the United Kingdom and South Korea." Using, running, or distributing the model or its outputs outside the Territory is outside the grant. For a company serving users in those regions, that is a blocker, not a footnote.
- Revenue tiers (section 4). Non-commercial use is free. Commercial use is free of model fees up to US$50 million in annual revenue, with a "Powered by ERNIE" notice and a one-time use notice by email. Above US$50 million you need prior written authorization, and fees are set by separate agreement.
- How revenue is counted (section 1.7). It is the revenue of every related product that uses or relies on the model, and if that product also uses other models, its entire revenue counts, not just a share. Affiliates under common control are aggregated. It is assessed continuously over trailing 12 months, not per calendar year.
- Hosting counts as commercial (section 1.8). API services, hosted inference and fine-tuning services are listed as commercial use.
- No improving other models (section 5.5). You may not use the weights or any outputs to improve any other AI model or ML system, except the model's own derivatives. That rules out using ERNIE outputs as distillation or synthetic training data for a different model family.
- Acceptable use highlights (Exhibit A). Includes a ban on military use, on high-stakes automated decisions in areas such as healthcare, credit, employment and law enforcement, and a requirement to clearly label machine-generated material published in public settings, including code and posts from automated accounts.
- Law and venue (section 9). Hong Kong law, exclusive jurisdiction of Hong Kong courts.
The practical consequence is that "open weights" here means something narrower than it does for permissively licensed models. If you are building on it, the license belongs in the same review as the memory budget.
How would you launch it?
These are the commands from the model card. I have not run them, so I cannot vouch for performance on your cluster.
# SGLang
sglang serve \
--model-path baidu/ERNIE-5.5-Flash-Base \
--tp 4 --attn-dp-size 4 \
--context-length 131072 \
--max-running-requests 1024 --port 30000
# vLLM (needs the ERNIE plugin)
uv pip install vllm==0.32.0
uv pip install git+https://github.com/baidu/vLLM-ERNIE.git
vllm serve baidu/ERNIE-5.5-Flash-Base \
--kv-cache-dtype fp8 --enable-expert-parallel \
--data-parallel-size 4 --max-model-len 131072 --port 30000 \
--moe-backend humming
Both examples cap context at 131,072 tokens even though the config enables 1,048,576. Given the KV estimate above, the cap looks like a conservative default rather than a memory necessity, but test long-context quality yourself before you rely on it: the training context was 128K, and the model card says 1M is supported in fine-tuning and inference.
FAQ
How much GPU memory does ERNIE-5.5-Flash-Base need?
The FP8 weights are 288.4 GB and the BF16 weights are 572.9 GB, measured from the Hub file listing. Add KV cache, activations and buffers on top. Eight 80 GB GPUs is the first comfortable FP8 setup.
Is ERNIE-5.5-Flash-Base an instruction model?
No. It is a pre-training checkpoint. The model card positions it as a starting point for domain adaptation, instruction tuning and reinforcement-learning research, and says to serve it through the completions endpoint.
Can companies in the EU, UK or South Korea use it?
Under the license text, rights are granted only outside those three regions, and use or distribution outside the Territory is not authorized. Confirm with a lawyer for your situation.
Is it free for commercial use?
Free of model license fees up to US$50 million in annual revenue of related products, with the "Powered by ERNIE" notice and a one-time use notice. Above that threshold you need prior written authorization from Baidu.
Why is the KV cache so small?
Per the model card, 37 of 43 layers use heavily compressed attention that caches one entry per 128 tokens, and only 6 layers keep full-context access. My estimate is about 5 GB at FP8 for a million-token sequence, which depends on implementation details I assumed.
Are the benchmark numbers independent?
No. They are reported by Baidu in the model card, run through one pipeline for four models, and I have not reproduced them.
Further reading
- Mistral Large 4 on your own GPUs: does 1 trillion parameters fit on 8 H200s
- Ling-3.1-flash and the 7:1 hybrid: how much KV cache does a 560B model need at 1M tokens?
Sources: model card, config.json, license, MarkTechPost coverage.
Top comments (0)