The model has 763B parameters, the server has two graphics cards, and the gap was about six gigabytes per card. DeepSeek-V4.1-Flash, even compressed to 3 bits, wanted 102.1 GiB of VRAM (the memory on the GPU itself) on each of two RTX PRO 6000 Blackwell cards, and each card offered 95.6 GiB. Two launches in a row died with out-of-memory errors, before any room for the KV cache, the memory that holds each conversation's context.
For anyone running a model like this on their own hardware, the useful part is which setting got it in, which one looked equivalent and failed, and what to measure afterwards. The full write-up of DeepSeek-V4.1-Flash on two 96 GB GPUs has every number, the method and the limits.
The decision: which bytes leave the GPU
In BF16 (two bytes per parameter, the uncompressed format) the weights would be about 1.5 TB. DeepSeek ships its checkpoint in FP4 and FP8 at 475.2 GiB. coolbho3k's 3-bit EXL3 pack is 309.3 GiB (EXL3 is a quantization format: fewer bits per weight, a smaller file). The cards add up to 192 GB. Shrinking the weights got close, not close enough.
The way out is that it is a mixture of experts: each of its 40 layers holds 384 routed experts, small sub-networks, and every token (a chunk of a word) passes through only 6 of them per layer. The idle ones do not need to live on the card. Diffbot's per-expert plan moves 4,608 of the 15,360 routed experts to host RAM, the server's ordinary memory, plus lookup tables (both Engram tables in int4 and the embedding). The GPU reads them where they sit, across the PCIe bus, through UVA (unified virtual addressing, which lets the card address host memory directly). Each card dropped to 72.4 GiB, which leaves space for context.
The trap: same bytes, different result
Your first instinct will be vLLM's own UVA offload (vLLM is the open source model server underneath). It looks like the same thing, but with an identical set of bytes on the host, it crashed during initialization with a KV block size of 64, and with 128 it started and then crashed on the first request. A stock vLLM install cannot serve this model on these two cards.
What works is a stack, and every layer has an author. DeepSeek published the weights. coolbho3k quantized them to 3 bits. turboderp wrote the EXL3 format and ExLlamaV3, the library that runs it fast. Victor Cruz built vllm-exl3, the bridge into vLLM. Diffbot designed the per-expert plan, the split launch and the kernels, on top of vLLM. InferenceKey assembled those pieces, got them running together and measured the result.
The other trap: PCIe topology
With experts in host RAM, the path from each GPU to that memory sets part of the speed. On one rented machine the GPUs sat on different NUMA nodes (groups of CPU cores, each with its own local memory), and it ran between 1.17 and 1.51 times slower. Check before measuring:
nvidia-smi topo -m
# Shows how each GPU connects to the other GPU and to the CPUs.
# GPUs on different NUMA nodes ran 1.17x to 1.51x slower here.
Otherwise you may blame the offload for a slow machine.
What to measure: per stream, not total
A developer feels the speed of their own session, not the box's total, so measure tokens per second per stream (one stream is one agent's session). The article sets the bar at 40 tokens per second for each person. These are the median and the 5th percentile (the slowest 5% of samples) on agent turns:
| Concurrent agents | Median tok/s | p5 tok/s |
|---|---|---|
| 1 | 186 | 134 |
| 4 | 59 | 52 |
| 6 | 40 | 34 |
Four streams at once, with a 10% margin, is the capacity. Long context hardly slowed generation: the median time per output token was 7.6 ms after 4K tokens of context and 8.3 ms after 256K. Reading a long prompt the first time is the expensive step, 18.33 s for 128K tokens, which is why prefix caching (keeping an already processed prompt so it is not read again) matters. With a cached 256K prefix and 4 users, 95% of requests got their first token within 1.66 s.
One developer's real week with Claude Code had the model generating for roughly 2.8 hours, 1.7% of the week. Treat each person as active with a 7.1% chance in an 8 hour window, independently of the others, and a team of 19 goes over 4 simultaneous streams no more than 1% of the time. Colleagues on a shared schedule are not independent, so the estimate leans generous, and it rests on one person's usage. The seat model behind the 19 developers per server also sets the hardware against 180 euro subscriptions.
Check that compression did not cost you answers
Going to 3 bits changes the model, so it was compared with the reference: fixed subsets of three benchmarks, thinking turned off, against DeepSeek's own API. GSM8K (math) came out at 0.960 on both sides. IFEval (instruction following) gave +0.013, interval [-0.013, +0.047]. HumanEval (code) gave -0.018, interval [-0.049, +0.012]. Both intervals include zero, and on roughly 150 questions a point or two is chance. The LiveBench agentic coding score (77.3, first of 69) is for the unquantized model at max effort.
What is still open
Every experiment ran on one boot only, and the agent turns were synthetic rather than captured from real sessions. There is also no baseline yet with the whole model in VRAM on a larger machine. That run is already written up as experiment E9. The method is in the paper (PDF).
Sizing a box for a team of coding agents? Start from the complete DeepSeek-V4.1-Flash measurements on inferencekey.com: the memory budget, per-stream speeds across six scenarios, the seat model and its limits.
Top comments (1)
the numa finding is the one i keep missing on my own box. two cards that look identical in nvidia-smi can sit on different cpu nodes, and the only difference between a fast offload and a slow one is the path to the same host ram. i have a little monitor that reads nvidia-smi per process, including the topology and the watts per card (github.com/SikamikanikoBG/homelab-monitor) — before i trust any per-stream number i check that both cards are on one node and that the power limit is not quietly throttling one of them.
the per-stream framing is also the right one. a team of 19 going over four streams less than one percent of the time is the kind of number that only makes sense when you measure the slowest five percent, not the median. that is the number to put in front of a hardware budget.