ERNIE 5.5 Flash Base won't chat with you. Baidu's new 280B open-weight model on Hugging Face is the raw end-of-pre-training checkpoint, it's served through /v1/completions, the smaller of its two repos is a 288 GB download, and just holding the weights takes two 288 GB GPUs or eight 80 GB ones. If you want to run ERNIE 5.5 Flash Base locally and point a chat UI at it, save your bandwidth.
If you fine-tune, it's worth a look: on Baidu's own table it beats DeepSeek V4 Flash Base on 26 of 30 benchmarks. DeepSeek's repo ships MIT, though, and Baidu's license forbids use in the EU, the UK and South Korea.
I haven't run this model. The technical details below come from the release page, the model card, Baidu's plugin README, the SGLang cookbook and the license text. Anything else is linked or named where I use it, and the couple of memory numbers I worked out myself are labeled as mine.
What Baidu released on Hugging Face
- 280B total parameters, 14B active per token, 43 layers (release page)
- 33.5T tokens of pre-training, with no mid- or post-training
- Trained on contexts up to 128K tokens, with up to 1M for inference and fine-tuning
- FP8 repo at 288 GB, BF16 repo at 573 GB
Chat, reasoning and tool calling belong to ERNIE-5.5-Flash, which the plugin README lists as "Coming soon".
Six of the 43 layers use MLA with DSA sparse attention, and the other 37 use Heavily Compressed Attention, which folds every 128 tokens into one cache entry plus a 128-token sliding window. Every layer but the first is MoE, with 512 routed experts (top-10) and one shared expert. ECHO replaces the multi-token prediction module by reusing the final layer to predict future tokens, and that same layer doubles as the speculative-decoding draft model. Baidu claims a higher draft acceptance rate than an auxiliary MTP module but publishes no number for it.
How much VRAM does ERNIE 5.5 Flash Base need?
The model card gives no VRAM figure and never mentions GGUF. Its examples just assume 4-way parallelism. The numbers are in the vLLM-ERNIE README and the SGLang cookbook:
| GPU memory | FP8 | BF16 |
|---|---|---|
| 288 GB (B300) | 2 GPUs | 4 GPUs |
| 192 GB (B200) | 4 GPUs | 8 GPUs |
| 80 GB (H100) | 8 GPUs | 2 nodes × 8 GPUs |
The README describes these as the smallest counts that hold the weights, so budget extra for KV cache. The cookbook says its single-node FP8 recipes are verified on every platform it lists, while BF16 is verified only on B300. The 1M context doesn't fit on 80 GB GPUs at all; the README tops out at --max-model-len 262144 there. The plugin only runs on NVIDIA GPUs, too.
14B active won't save you, because all the experts still have to live somewhere. As horsawlarway put it on HN: "Active parameter count really drives inference speed not ram requirements."
By my own arithmetic, a 4-bit quant of the 288 GB FP8 repo would be about 144 GB of weights, and 6-bit about 216 GB. The same HN commenter estimates 175 GB at 4-bit and about 244 GB at 6-bit with the full 1M context, and the cookbook's "tens of GB" of activation memory for a 1M prefill makes that plausible. So the 2×128 GB box Tepix mentioned in the thread works on paper at 4-bit, but I found no GGUF or 4-bit build to load on it. The README lists an FP4 build only for the unreleased ERNIE-5.5-Flash.
How to run ERNIE 5.5 Flash Base with vLLM
Skip the plain pip install vllm snippet Hugging Face shows you. The README says a missing plugin gets you Model architectures ['Ernie5_5_MoeForCausalLM'] are not supported. An unpinned install also gets you vLLM 0.33.0 today, and the plugin pins 0.32.0. Use the card's deployment steps:
uv venv --python 3.12 ernie5-venv
source ernie5-venv/bin/activate
uv pip install vllm==0.32.0
uv pip install git+https://github.com/baidu/vLLM-ERNIE.git
vllm serve baidu/ERNIE-5.5-Flash-Base \
--kv-cache-dtype fp8 \
--enable-expert-parallel \
--data-parallel-size 4 \
--max-model-len 131072 \
--port 30000 \
--moe-backend humming
Things to change for your setup, all from the README:
- The README exports
VLLM_ENGINE_READY_TIMEOUT_S=3600before every serve command and the card doesn't. I'd copy it. -
--moe-backend hummingis for Blackwell. Drop it on Hopper. - On 80 GB GPUs, use
--data-parallel-size 8and--gpu-memory-utilization 0.93. - For 1M context, set
--max-model-len 1048576, lower--gpu-memory-utilizationfrom 0.92 to 0.85 and reduce--max-num-seqs. The README assumes 184 GiB GPUs for those values and gives no number for the last flag. - ECHO drafting is on in the README's commands but not in the card's. Add
--speculative-config '{"method":"mtp","num_speculative_tokens":2}'to match. - Those README commands bind
0.0.0.0with no auth. Add--api-keyor bind127.0.0.1unless you trust the network.
How to run it with SGLang
The card's deployment section uses a dedicated image, while Hugging Face's generic snippet uses latest:
docker run -d -it --name ernie55 \
--gpus all \
--network host \
--ipc host \
-v ~/.cache/huggingface:/root/.cache/huggingface \
lmsysorg/sglang:dev-ernie-55-flash
sglang serve \
--model-path baidu/ERNIE-5.5-Flash-Base \
--tp 4 \
--attn-dp-size 4 \
--context-length 131072 \
--max-running-requests 1024 \
--host 0.0.0.0 \
--port 30000
That's the card's command. The cookbook's recipes use --max-running-requests 256, so size it to your workload. For 1M, the cookbook says to pass --context-length 1048576 and drop --max-running-requests to single digits on 2×B300. This command also binds 0.0.0.0 on the host network, so the same auth warning applies.
Calling /v1/completions: few-shot prompts instead of chat
For few-shot use, the cookbook says to put the examples in the prompt and set a stop sequence. The README adds that you should pass "temperature": 0 for greedy decoding, since vLLM doesn't apply do_sample. I built this sketch from those two notes and haven't run it:
curl http://localhost:30000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "baidu/ERNIE-5.5-Flash-Base",
"prompt": "Q: Capital of France?\nA: Paris\n\nQ: Capital of Japan?\nA: Tokyo\n\nQ: Capital of Canada?\nA:",
"max_tokens": 16,
"temperature": 0,
"stop": ["\n\n"]
}'
The README says the tokenizer prepends <|ERNIE_MASK|> (id 2) to completions prompts by itself, so don't add it to the prompt.
In Transformers, load the BF16 repo and keep add_special_tokens=True. The release page calls this the BOS token, but tokenizer.json prepends <|ERNIE_MASK|> there too, while tokenizer_config.json names <|ERNIE_BEGIN|> as bos_token. If you hand-roll inputs, match the tokenizer.
ERNIE 5.5 Flash Base vs DeepSeek V4 Flash Base, on Baidu's numbers
These are vendor results, with the same harness, prompts and judge across all four models. I didn't find an independent run. Most are few-shot scores, so a bare prompt won't reproduce them.
| Benchmark | ERNIE 5.5 Flash | DeepSeek V4 Flash | Hy3 preview | MiMo V2.5 |
|---|---|---|---|---|
| MMLU-Pro | 78.4 | 69.2 | 66.9 | 65.7 |
| GPQA | 60.2 | 55.9 | 50.2 | 51.4 |
| HumanEval+ | 82.3 | 69.5 | 78.0 | 74.4 |
| LiveCodeBench | 40.1 | 31.9 | 33.5 | 32.4 |
| ChineseSimpleQA | 73.0 | 75.3 | 73.2 | 68.8 |
| InfiniteBench | 46.5 | 48.0 | 33.0 | 46.7 |
ERNIE tops 24 of 30 rows overall. The 12.8-point HumanEval+ gap over DeepSeek is the number I'd want someone to reproduce first. It wins four of the six long-context rows and loses InfiniteBench and HELMET-Rerank, both to DeepSeek, which is a mixed result for a model advertising 1M tokens.
input_sh on HN: "They're comparing it to Deepseek-v4-Flash, not 4.1 and not the 4.1 1017 update". I think that complaint only half lands. Tepix points out that V4.1 Flash is twice the size: DeepSeek's card lists 552B backbone parameters plus a 196B Engram memory. That card benchmarks a DeepSeek-V4.1-Flash-Base, but I couldn't find a public repo for it. A 284B base model you can actually download is the fair opponent here. And halJordan's "benchmaxxed to hell" stays an open accusation until someone runs these independently.
The ERNIE 5.5 Flash Base license excludes the EU, UK and South Korea
Per the ERNIE license:
- The territory excludes the EU, the UK and South Korea, and the ban covers the model, derivatives and their outputs.
- Non-commercial use is free. Commercial use at or below US$50 million in related annual revenue needs a one-time notice and a "Powered by ERNIE" attribution. Above that, you need written authorization first.
- Fine-tuning and quantization both count as creating a Derivative Model. So does distilling from it or training another model on its synthetic output (§1.5).
- You can't use the model or its outputs to improve any AI system other than the model and its Derivative Models (§5.5).
Those last two clauses bother me most. As I read them, anything you train on ERNIE's outputs falls under Baidu's terms, territory limit included. I'm not a lawyer, but a base model is supposed to be a starting point, and this one follows you downstream. DeepSeek V4 Flash Base has an MIT LICENSE file in a 295 GB repo, about the same download.
My call: if you're outside the excluded regions and fine-tuning for internal use, ERNIE is the one to evaluate first, because those code and exam gaps are large enough to survive some vendor optimism. If you're in the EU, redistributing weights, or generating training data for anything else, take DeepSeek. If you just want a local assistant, wait for ERNIE-5.5-Flash.
Would you fine-tune under Baidu's license for those HumanEval+ points, or take DeepSeek's MIT checkpoint and give them up? And if you've squeezed this onto 2×128 GB without a GGUF, how did you quantize it?
Top comments (0)