DEV Community

KeithVal
KeithVal

Posted on Edited on

DeepSeek V4.1 Flash at 42 tok/s with 256K Context on RTX 5070 Ti x2 + 5060 Ti — Full Benchmark Record

This article is based on my own benchmark records, written by me together with Muse (an AI assistant). The original Japanese version is on Zenn.

I run DeepSeek V4.1 Flash on a personal Linux machine. Generation reaches 42 tok/s across three GPUs, and around 38 tok/s in the two-GPU configuration I actually use every day — at a 256K context, with vision input enabled. Every number below is measured on my own hardware, and the full test conditions are in the table in the next section.

One caveat before the numbers, because it matters for comparison: these results come from an EXL3-quantized build running on a locally patched engine. They are not directly comparable to stock llama.cpp or Ollama numbers. As a distance marker, there is a public record of the same model running at just over 20 tok/s on a roughly ¥480,000 (~$3,200) workstation. The conditions don't line up exactly, but it gives a sense of where this setup lands: roughly twice that, on a machine in the same price class.

The headline numbers

Configuration Generation speed
2 GPUs (A + B, daily driver) ~38 tok/s (median 37.4 in real use)
3 GPUs (A + B + C) 42 tok/s
Reference: single GPU, early measurements 5070 Ti: 26.9 tok/s · 5060 Ti: 22.6–22.8 tok/s

Going from two GPUs to three only bought 38 → 42, about 10%. Adding a GPU does not scale speed linearly: the more you split the model, the more data has to move between GPUs. That pattern — transfer, not compute, as the ceiling — runs through everything below.

Generation speed is only half the story. Prefill (processing the prompt before the first token comes out) decides how the machine feels on long documents: in the production configuration, an 8K-token prompt takes about 10 seconds, and 32K takes about 26 seconds. I report both throughout, because a single tok/s figure hides the prefill cost.

These are not one-off peak numbers, either. When the current build was integrated into my agent setup, generation measured 38.5–39.0 tok/s — the same level as the benchmark table. Through an agent harness, a single step responds in 1.3–2.6 seconds (cold start of a new conversation was about 24 seconds, measured on the earlier configuration).

Full test conditions

Item Value
CPU Threadripper PRO 3975WX
RAM 256 GB
GPUs A: RTX 5070 Ti · B: RTX 5070 Ti · C: RTX 5060 Ti
OS Linux
Model DeepSeek V4.1 Flash ("1017" build)
Quantization EXL3 3.5 bpw
Engine Maya-family DS4 implementation (upstream v1.1.24) + 16 local patches
Context length 256K
GPU power limits A: 250 W · B: 250 W · C: 150 W

The current production build ("1017": EXL3 3.5 bpw, tensor parallel across GPUs A+B, 256K context, text + vision) has been running since October 24, 2026. It replaced the previous generation (EXL3 2.9 bpw on vLLM, GPUs A+C). All headline numbers above are from the current generation. Where a number below comes from the older 2.9 bpw generation, I say so explicitly — mixing generations would make the record worthless.

Why run this locally at all? Two reasons: some data can't go to a cloud API, and agent-style workloads call the model over and over, so per-call API pricing scales with use. The 38–42 tok/s figures matter because they were achieved inside those constraints.

Placement is the design

DeepSeek V4.1 Flash is an MoE model: for each token, only a subset of its "experts" is active. That structure drives every decision in this setup.

The experts don't fit in VRAM alone, so they spill into system RAM — the 256 GB is not decoration. The n-gram lookup table (Engram) sits on an Optane SSD: weights that are read constantly live in memory, lookup tables live on fast storage. Placement is the performance design here.

The split is deliberate: heavily used ("thick") experts go on the GPU; experts that only collect a few tokens run on the CPU. Put everything on the GPU and VRAM collapses; put everything on the CPU and the CPU becomes the bottleneck. Deciding what lives where is the core of this configuration.

256K context: verified, not just enabled

Setting a 256K context window is easy. Knowing the model actually reads all of it is a separate problem, so I verified it: I fed conversations up to 248,550 tokens and checked that information buried in the middle could be retrieved correctly. At 250K-class input, answers still came back without dropping content. A fresh 24K-token input took 53–59 seconds, but generation afterwards held at 29.9 tok/s (measured on the two-GPU configuration of that period). Many setups lose generation speed noticeably as context grows; the fact that this one didn't is a large part of why I committed to it as the daily driver.

Prefix caching is doing real work here. It reuses the shared parts of a conversation (system prompt, earlier turns) instead of recomputing them, in 128-token units, and I confirmed it works on V4.1. For agent workloads that resend the same preamble constantly, cache hits are the difference between "~1 minute for a fresh 24K prompt" and a light follow-up turn. When someone quotes context performance, it's worth asking which of those two numbers they mean.

The bottleneck was PCIe, not the GPU

The single most effective optimization in this project had nothing to do with GPU compute. It was about how experts are transported.

During prefill, experts were being fetched over PCIe (the bus between CPU/memory and GPU) in chunks of 2,048 tokens. Counting the traffic showed a heavy skew: 53–54% of token rows concentrated on "thick" experts — popular experts collecting more than 128 rows, about 18 of them per layer on average.

Reading a thick expert over PCIe cost 2.3–2.5 ms. Computing it from weights already on the GPU cost 0.5–0.7 ms — roughly a 4× difference. The compute wasn't slow; the delivery was.

So I changed the rule: thick experts are computed from on-GPU weights. Real-input prefill got 27–32% faster:

Input length Before After
2K tokens 5.1–5.6 s 3.90 s
8K tokens 18.1 s 13.01 s
32K tokens 68.7 s 46.9 s

Same day, same conditions. At 32K that's about 22 seconds less waiting — which, for document-heavy work, is the difference you actually feel.

The same class of stall showed up elsewhere. CPU-side expert computation was limited not by arithmetic but by a single core's instruction flow; removing one data-copy step made it 9–11% faster. In the three-GPU experiments, letting GPU C take part of each layer's compute produced 1.22–1.47× improvements in one phase. And not everything "obvious" worked: running the EXL3 kernel from two streams on the main GPU deadlocks. The lesson is plain — for MoE inference, start by counting which experts are thick and how much data moves, before trying to make everything faster.

Did the speed cost quality?

Speed is useless if the answers get thinner, so I measured quality in parallel with a home-made yardstick: 38 business-style questions and 12 math questions, scored against the API version of the model.

Yardstick This machine API version
Business, 38 questions 28 28
Math, 12 questions 9 12

Business tasks tied the API. Math did not — that gap is real, and I'm not going to round it away. Quantization plus local execution is not free. But tying on the business set is what justified using this as a daily work tool.

Two measurement traps are worth sharing. First, at one point answers came back empty — the model appeared to "fail" questions, but the cause was a cap on thinking tokens: hitting the cap truncated the reasoning and the answer. Fixing the cap fixed the "capability problem." Second, the yardstick itself was flawed: 20 of the 38 business points scored identically no matter which model I ran, so they discriminated nothing. I rebuilt the questions and re-measured; on the rebuilt set the model scored 155/156 — but that figure was measured on the API version via OpenRouter, so it tells you about the model's ceiling, not about my local build, and I keep it out of the table above. Fixing the measuring instrument is unglamorous work, but it has to happen before the speed numbers mean anything.

Vision input didn't slow generation down

The production build accepts image input. I evaluated it with 12 test images, mostly counting tasks (count the shapes, count the stars):

Condition Correct
Official bias + thinking (this machine) 11/12
Official bias, no thinking (this machine) 10/12
Reference: Qwen XS, no thinking 12/12

The miss was a star-counting image, dropped even with thinking on. The reference model took the full set — on this yardstick, it wins, and I'll say so plainly. Speed flips the result, though: Qwen XS needed about 10 seconds per image, while first response here was 1.6–3.3 seconds depending on image token count (1.62 s at 402 tokens, 3.3 s at 1,018 tokens). Accuracy and responsiveness are different yardsticks with different winners.

The cost of enabling vision was also measured: generation ran at 32.14 tok/s with vision support in the build versus 32.05 tok/s for the text-only build of the same code — effectively no penalty, so vision stays on rather than being a benchmark-only toggle. The long-context check also passed with images in the mix: in a conversation extended to 248,550 tokens including image turns, the buried-information retrieval test still answered correctly.

Generation note: these vision figures are from the previous 2.9 bpw generation, measured on October 6–7, 2026, and shipped in that era's production build. Vision input is enabled in the current 1017 build (3.5 bpw), but I have not yet re-run the 12-image yardstick on it. Read these numbers as belonging to the older generation.

Why the 3-GPU setup is not the daily driver

The 42 tok/s figure is real, but day to day I run the two-GPU configuration at ~38 tok/s. Two reasons.

First, GPU sharing: in the three-GPU layout, the LLM occupies every card. The same machine also does image generation, and GPU C (the 5060 Ti) needs to stay available for that. A configuration that coexists with the rest of my work is worth more than 4 extra tok/s.

Second, startup. From model load to ready takes about two minutes — measured at 114.0–135.2 seconds of loading, ready at 136 seconds, on the day the build went into production. I shaved what I could (pre-reading the weight manifest, batching pinned-memory allocation, raising the kernel-compile cache limit — all "avoid re-doing work" fixes rather than "load faster" fixes), but ~2 minutes remains. So the operational compromise is: start on first request, auto-stop after 60 minutes idle. It splits the difference between standby power cost and wait time.

Heat is part of the same picture. During long prefill runs, CPU-side expert computation ran hot; changing the wait strategy (sleep instead of spin) brought the median CPU temperature during input processing from 71.4 °C down to 62.75 °C, at a cost of 2–3% of generation speed. A configuration that thermally throttles is not a daily driver, so that trade was worth it. The power caps (250 W / 250 W / 150 W) are set with the same mindset: this is a tool I run every day, not a benchmark rig chasing a single number. (The machine itself cost about ¥580,000; the build breakdown and the mistakes made along the way are documented on my blog.)

Next: the anatomy of the "50 tok/s wall"

After reaching this plateau, I didn't add hardware. I audited where the remaining time goes, working only from production logs and the code (the production setup untouched), on November 15, 2026.

The conclusion was unglamorous: within changes that don't alter the model's outputs, the total headroom is roughly +5–10%. Two GPUs will not reach 50 tok/s.

The estimated breakdown of one token's generation time: PCIe transfer ≈ 50%, main expert compute plus CPU waiting 16%, attention and accumulation 14%, speculative-decoding draft experts 11%, expert routing 6%, the rest other. The link itself measures 25.3 GB/s on copies against a 27.8 GB/s practical ceiling (PCIe Gen4 x16). When half the time is transfer, faster compute barely moves the total.

A second angle produced a more interesting hypothesis. If a selected expert already sits in VRAM, no transfer happens; only misses ride the PCIe bus. In the current production configuration, the observed expert hit rate is 60.3% under real usage — meaning roughly 40% of expert selections are being fetched, every token.

I modeled how the hit rate grows with VRAM. Adding 14 GB (one 16 GB card's worth) would fit 786 more experts and lift the hit rate to an estimated 73–81%, putting generation at an estimated 44–51 tok/s. Adding 30 GB: hit rate 84–94%, generation 54–68 tok/s. These are model estimates, and the assumptions lean optimistic — they presume added VRAM is usable the same way as the current two cards, and that GPU-side compute for the extra experts doesn't become the new bottleneck. Still, the hypothesis is now concrete: the "50 wall" may be a placement wall, not a compute wall.

Prefill has its own ceiling: at inputs of 8K and below, the link is already near saturation (22.9 GB/s measured on the receiving side), so the remaining prefill headroom is in larger inputs (32K+) and in avoiding re-reads — an estimated 10–20%. I also evaluated replacing the engine wholesale with the newer upstream release and rejected it after inspection: the new version lacked the PCIe handling for EXL3 tensor parallelism that this setup depends on. Meanwhile, a small change I call "flow" is confirmed by measurement to give +1–3% with byte-identical outputs. Nothing dramatic — but the arrival point above was built out of exactly this kind of accumulation.

Next step is to test the hypothesis with real measurements: extract the true hit-rate curve from production usage records, then measure whether added VRAM actually reaches 50 tok/s — whether the modeled 44–51 is real, or collapses by exactly the optimism baked into the assumptions.

The measurement templates behind this record

This record exists because every run was logged the same way: a conditions table (hardware, engine build, quantization, context, power limits, flags), generation and prefill measured separately, medians instead of best runs, and a note on which generation of the setup each number belongs to. After ~215 local-LLM experiments, I packaged that logging format — condition sheets, a prefill/generation measurement protocol, a long-context verification procedure, a quality-yardstick starter set, and a bottleneck-breakdown worksheet, with this DeepSeek run included as a fully worked example — as the Local LLM Benchmark Field Kit:

👉 https://minamiwave06.gumroad.com/l/idjiba

If you're publishing local inference numbers, it may save you the measurement mistakes I made first — including a yardstick where 20 of 38 points couldn't discriminate between models at all.

Top comments (0)