ai maya

Posted on Oct 31

Mistral Large 4 on your own GPUs: does 1 trillion parameters fit on 8 H200s, and what does the license allow?

#ai #llm #inference #opensource

TL;DR

Mistral published the open weights of Mistral Large 4 on 2026-10-29: about 1 trillion parameters in total, 49 billion active per token, a 1M-token context window, and a modified MIT license. I did not run the model. I read the public config.json, summed the file sizes in the Hugging Face repositories and did the memory arithmetic. What came out:

The official FP8 checkpoint is 1,060.6 GB. On 8 H200 GPUs (1,128 GB of nominal HBM) the weights alone take 94 percent of memory. The generic serving command on the model card uses --gpu-memory-utilization 0.85, and by my arithmetic that cannot hold these weights on H200 hardware.
The KV cache is not the problem. One full 1,048,576-token sequence needs about 76 GB in BF16 or 38 GB in FP8, roughly 32 times less than a plain multi-head cache would need, because of MLA and some cross-layer cache sharing.
The 610.1 GB NVFP4 checkpoint changes the picture, but the card says it targets Blackwell GPUs. On 8 B200 it leaves room for roughly 8 sequences of 1M tokens.
The license is free for internal use and restrictive for resale. Operating a "Model as a Service" business needs a separate commercial license.
At $0.70 per million input tokens, the API wins unless your node stays busy. The break-even is on the order of 1 request per second sustained around the clock, under assumed GPU prices.

What exactly shipped with Mistral Large 4?

The model card describes a natively multimodal mixture-of-experts model trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs. Text and images go in, text comes out. From the model card and config.json:

Field	Value
Total / active parameters	about 1T / 49B (52B including embeddings and output layers)
Decoder layers	72 (first layer dense, 71 mixture-of-experts layers)
Hidden size	6,144
Experts	128 routed (4 active per token) plus 1 shared
Attention	Multi-head Latent Attention with a sparse indexer, `kv_lora_rank` 512, `qk_rope_head_dim` 64
Context	1,048,576 tokens
Vocabulary	204,800
Vision encoder	Pixtral-style, 30 layers
Speculative decoding	embedded DSpark draft head, 3 BF16 MLA layers, block of 3 tokens
Official checkpoints	FP8 (1,060.6 GB) and NVFP4 (610.1 GB)

Mistral reports 48 percent on the Artificial Analysis Coding Agent Index, effectively tied with GLM-5.3 while using 47 percent fewer tokens per task, and 26 on the Artificial Analysis Intelligence Index v5. The card itself says that leading open-weight models from China score higher on that aggregate index. The announcement page adds 19.7 percent on FrontierCode 2.0 and 41 percent on Terminal-Bench 4.0. All of these are vendor-reported, and I have not reproduced any of them.

How much of a "1 trillion parameter" model actually runs per token?

Almost all of the parameters sit in experts, and only a small slice runs for each token. From the config, each expert is a gated feed-forward block with three matrices of 6,144 by 6,144, which is 113.2 million parameters (this assumes the usual three-matrix SwiGLU layout, which the silu activation suggests).

H = 6144; FF = 6144; MOE_LAYERS = 71
per_expert = 3 * H * FF                         # 113,246,208
all_experts = per_expert * (128 + 1) * MOE_LAYERS
active_experts = per_expert * (4 + 1) * MOE_LAYERS
print(all_experts / 1e12, active_experts / 1e9)  # 1.037 T, 40.2 B

Experts account for 1.037 trillion of the roughly 1.05 trillion total, and the five experts that fire per token account for 40.2 billion of the 49 billion active. The rest is attention, the first dense layer and similar. A cross-check: the community BF16 GGUF is 2,098.9 GB, which at 2 bytes per weight implies about 1.049 trillion parameters, consistent with the config.

This matters for serving because every expert has to be resident in memory even though 97 percent of the routed experts are idle for any single token.

Does it fit on 8 H200 GPUs?

Not comfortably, and not at the card's generic setting. I summed every file in the FP8 repository: 1,060,613,131,585 bytes, or 1,060.6 GB (987.8 GiB). Split across 8 GPUs that is 132.6 GB per GPU, against 141 GB of nominal HBM on an H200.

Node	Nominal HBM	Weights	Left at 100% of HBM	Left at the 0.85 setting
8x H200, FP8	1,128 GB	1,060.6 GB	67.4 GB	-101.8 GB (does not load)
8x B200, FP8	1,440 GB	1,060.6 GB	379.4 GB	163.4 GB
8x B200, NVFP4	1,440 GB	610.1 GB	829.9 GB	613.9 GB
4x B200, NVFP4	720 GB	610.1 GB	109.9 GB	1.9 GB
8x H100, FP8	640 GB	1,060.6 GB	does not fit	does not fit

The last column is the budget the card's vLLM command allows (--gpu-memory-utilization 0.85 on 8 GPUs, tensor parallel size 8) minus the weights. For H200 it is negative: the weights alone need about 94 percent of the pool, before any KV cache, activation buffers, CUDA graphs or communication buffers. The Hugging Face launch post says a ready-to-deploy recipe exists for 8 H200 with FP8 weights split by tensor and expert parallelism. I have not seen its flags, but the arithmetic says such a deployment has to run at a much higher memory fraction than 0.85 and will have very little left for context and concurrency. Treat 8 H200 as the floor for "it loads", and test your own context length and batch size before you commit to it.

Three other notes from the numbers:

Hopper cannot use the NVFP4 build as intended. The card says it targets Blackwell GPUs, so Hopper owners are looking at FP8 or community GGUF quantizations.
Four B200 is the tight corner. NVFP4 weights fit in 720 GB, but at the 0.85 budget nothing is left over.
8x H100 needs two nodes. 16 H100 hold 1,280 GB, so FP8 weights fill 83 percent of the pool.

The community GGUF sizes, read from the repository file listings, give a sense of the low end: IQ2_XXS is 297.7 GB, Q2_K 380.7 GB and Q4_K_M 634.8 GB. The Hugging Face post says the 2-bit quants run on a single Apple Silicon Ultra machine with 512 GB of unified memory.

Why is the KV cache so small for a 1M-token model?

The cache is the one place where this architecture is genuinely frugal. Multi-head Latent Attention stores a compressed latent per token instead of full keys and values. In the config that latent is kv_lora_rank plus qk_rope_head_dim, which is 512 + 64 = 576 values per token per layer.

A plain multi-head cache with 64 KV heads would store 64 x (128 + 128) = 16,384 values per token per layer. That is 28 times more per layer.

There is a second saving. The config has a field called kv_source_layers. For 63 of the 72 layers it points at the layer itself, and for layers 39, 43, 47, 51, 55, 59, 63, 67 and 71 it points at layer 35. My reading is that those nine layers reuse layer 35's cache, so only 63 layers hold their own. The card does not document this, and an inference engine may allocate differently, so treat it as an upper bound on the saving.

latent = 512 + 64
layers_with_cache = 63
ctx = 1_048_576
bf16 = latent * layers_with_cache * 2 * ctx / 1e9   # 76.1 GB
fp8  = latent * layers_with_cache * 1 * ctx / 1e9   # 38.1 GB
mha  = 64 * (128 + 128) * 72 * 2 * ctx / 1e9        # 2,473.9 GB

Context	BF16 cache, 1 sequence	FP8 cache, 1 sequence
131,072	9.5 GB	4.8 GB
262,144	19.0 GB	9.5 GB
1,048,576	76.1 GB	38.1 GB

Per token that is 72,576 bytes in BF16 across 63 layers, about 71 KiB. The ratio to a plain multi-head cache is 32.5 times. Two caveats keep this honest. First, the sparse indexer (the Hugging Face post says it selects up to 2,048 key positions per query) cuts how much of the cache each token reads, not obviously how much is stored, and it keeps its own keys. If it holds one 128-value FP8 key per token on each of the 44 layers marked full, that is another 5.9 GB at 1M tokens. That is my assumption, not something the card states. Second, the numbers exclude activation memory, the vision tower and the DSpark draft layers.

With those limits in mind, how many full-context sequences fit in what is left at the 0.85 budget? On 8 B200 with FP8 weights, 163 GB leaves room for 2 sequences of 1M tokens with a BF16 cache or 4 with an FP8 cache. With the NVFP4 build on 8 B200, 614 GB leaves room for 8 or 16.

Is decoding bandwidth-bound on a node like this?

Mixture-of-experts decoding has a quirk: the more requests you batch, the more experts each step has to read. If routing were uniform and independent, a batch of B tokens would touch 128 x (1 - (1 - 4/128)^B) distinct experts per layer. Real routing is skewed, which means fewer distinct experts, so this is an upper bound on the weight traffic.

I turned that into a bandwidth-only ceiling for 8 GPUs at the 4.8 TB/s H200 datasheet bandwidth, reading FP8 weights once per step. It ignores compute, interconnect, KV reads and imperfect overlap, so it is a ceiling and not a prediction.

Batch	Experts touched per layer (of 128)	Weights read per step	Step time floor	Node tokens/s ceiling	Per-sequence tokens/s
1	4.0	63.6 GB	1.7 ms	604	604
8	28.7	262 GB	6.8 ms	1,171	146
32	81.7	688 GB	17.9 ms	1,786	56
128	125.8	1,043 GB	27.2 ms	4,713	37

Going from batch 1 to batch 128 multiplies throughput by 7.8 while per-sequence speed falls by 16 times. DSpark speculation changes this trade. The card reports up to 1.9 times higher throughput, measured by Mistral, and says gains depend on hardware and workload.

What does the license actually allow?

The Mistral Ecosystem Open-Weight License is a modified MIT license. I read the full LICENSE file, which is short. Here is my plain-language reading of it, and I am not a lawyer:

Allowed without asking: use, copy, modify, fine-tune, distribute and sell copies of the model, including commercially, with the copyright notice kept.
Condition: if you or any of your affiliates operate "a Model as a Service business", you must sign a separate commercial license with Mistral before using the model or its derivatives for any commercial purpose.
Definition: "Model as a Service" means giving a third party access to the model or its derivatives for inference or post-training, for instance through an API or hosted endpoints, in a way that lets them exercise meaningful control over inputs, parameters or training data.
Not counted as MaaS: end-user products where the model is embedded in specific features or harnesses, and merely relaying requests to models hosted by others.
Exemptions: internal use that does not make the model available to third parties, and any use through Mistral's own official products.

Two details are easy to miss. The condition is triggered by the business you operate and includes affiliates, so a company with a hosting arm can pull the whole group into scope. And a fine-tuned derivative is covered by the same wording as the base model. If your plan is to expose a fine-tune to customers through an endpoint, read section 2 with counsel.

When does self-hosting beat the API?

Mistral's announcement lists the API at $0.70 per million input tokens, $0.07 per million cached input tokens and $2.10 per million output tokens. I priced a request of 8,000 input tokens and 500 output tokens, the same shape I used in earlier posts:

Uncached: (8,000 x 0.70 + 500 x 2.10) / 1,000,000 = $0.00665
With a 6,000-token cached prefix: (2,000 x 0.70 + 6,000 x 0.07 + 500 x 2.10) / 1,000,000 = $0.00287

An 8-GPU node costs a fixed amount per hour whether it is busy or idle. GPU rental prices vary a lot by provider and contract, so the rates below are assumptions for illustration, not quotes. The table gives the sustained request rate at which the node bill equals the API bill.

Assumed price per GPU-hour	Node per hour	Break-even, uncached	Break-even, cached prefix
$2	$16	0.67 req/s	1.55 req/s
$3	$24	1.00 req/s	2.32 req/s
$4	$32	1.34 req/s	3.10 req/s
$6	$48	2.01 req/s	4.65 req/s

At $3 per GPU-hour, you must serve about 3,600 uncached requests every hour, around the clock, to break even. Is that rate reachable? On paper, yes. One uncached request per second means 500 output tokens per second, well under the bandwidth ceilings above, and 8,000 prefill tokens per second. At about 2 x 49 billion FLOPs per token, that prefill load is 0.78 PFLOPS, a small share of the roughly 2 PFLOPS of dense FP8 on one H200 datasheet. But I have no measured throughput for this model, so I would not plan capacity from a paper estimate. The break-even is a utilization question. A node that is busy 24 hours a day can beat the API. A node that serves a business-hours workload at 20 percent utilization cannot, and none of this counts engineering time, redundancy, or the second node you need for failover.

FAQ

Is Mistral Large 4 really open source?
The weights are public and free to download without approval, under a modified MIT license. Because of the Model-as-a-Service condition, it does not meet the usual open source definition for every use. Internal use and embedded products are fine, and reselling inference needs a separate license.

How much GPU memory does Mistral Large 4 need?
The official FP8 checkpoint is 1,060.6 GB, so a single 8x H200 node (1,128 GB) holds the weights with about 67 GB to spare. The NVFP4 checkpoint is 610.1 GB and targets Blackwell GPUs. Add KV cache, which is about 76 GB for one 1M-token sequence in BF16.

Can Mistral Large 4 run on a Mac?
The Hugging Face launch post says the 2-bit community quantizations run on one Apple Silicon Ultra machine with 512 GB of unified memory. I have not seen a quality benchmark at 2 bits.

How big is the KV cache at a 1M-token context?
About 76 GB in BF16 or 38 GB in FP8 for one sequence, by my arithmetic from the config. A plain multi-head cache of the same shape would need about 2,474 GB.

What does the API cost?
Mistral lists $0.70 per million input tokens, $0.07 per million cached input tokens and $2.10 per million output tokens as of 2026-10-31. Check the live price list before you budget, since prices change.

Sources

All repository sizes and config values were read on 2026-10-31. Calculations are mine and use assumed GPU prices; they are not measurements of the model running.

DEV Community