Two local inference projects shipped fixes on 1 August for the same problem: DeepSeek V4 Flash 0731 was emitting valid tool calls that local agent runners could not parse, so agents stopped mid-task. llama.cpp release b10217 added support for tool calls issued inside the model's reasoning block, and KoboldCpp v1.118 separately fixed multi-turn prompt processing for the same model. The day-two story for the year's biggest open-weight release is not benchmarks. It is plumbing.
Key facts
- llama.cpp's b10217 release lists "enable tool call in thinking for DS4" among its changes, shipped 1 August 2026.
- KoboldCpp v1.118 lists a "Fix for multi-turn deepseek v4 flash prompt process issues" the same day.
- The model is a 671-billion-parameter mixture of experts with 37 billion active per token; published quantisations run from about 91 GB to 162 GB.
- Primary sources: the llama.cpp release notes and the official model card.
The root cause is unusual and worth understanding, because it will recur. Most open-weight releases ship a Jinja chat template - a small text file telling any runner how to assemble a conversation into the exact token sequence the model was trained on. DeepSeek's 0731 release ships no generic template. Instead it supplies custom Python encoding code that converts ordinary tool messages into result blocks inside user content, reorders results to match the original calls, and interleaves schemas, calls and thinking state.
That means a runner cannot just read a template and comply. It has to reimplement a protocol. And when the model does something the reimplementation did not anticipate - like emitting a tool call before it closes its reasoning block - the parser gives up and the agent quietly ends its turn having done nothing.
The analogy is a restaurant that stops printing tickets and starts shouting orders in its own shorthand. The food is fine. The kitchen is fine. But every new server has to learn the shorthand by ear, and the first week is chaos. The merged llama.cpp work, in pull request 26269, taught the parser the shorthand; the reviewer confirmed both ordinary and in-thought tool calls working, with tool use combined with JSON schema output still noted as an edge case.
Behind the plumbing sits a more interesting reality check about what "frontier model at home" means in practice. This is a mixture-of-experts model: only about 37 billion parameters activate for any given token, but all 671 billion still have to be somewhere the machine can reach. Unsloth's published quantisation inventory makes the arithmetic unavoidable - its two-bit variant is about 91 GB, its three-bit variants run 116 to 128 GB, and its lossless eight-bit build is 162 GB. A 24 GB graphics card is not running this model. It is accelerating a hundred-plus-gigabyte model that lives in system RAM, which is why local inference is memory-bound long before it is compute-bound.
The first-hand reports from 1 August are consistent with that. One user with a Ryzen 9900X, a 24 GB RTX 3090 and DDR5-5600 memory ran a three-bit quant by pushing 39 layers of experts into system RAM, and averaged about 12 tokens per second across a 21,000-token generation; the loader estimated roughly 136 GB required. A user with two RTX 3060s and 96 GB of RAM needed a two-bit quant and got about 4.5 tokens per second - a 4,338-token answer took sixteen minutes - and disclosed that one card was on a PCIe 3.0 x1 link. A three-3090 benchmark improved generation from about 7.7 to 13.9 tokens per second by fully offloading experts to GPU, but prompt ingestion fell from 116 to 72 tokens per second in the process. That last trade is the one nobody mentions: optimise for interactive typing speed and you can make long prompts slower.
There is also a capability the day-two reports mostly are not getting. The 0731 checkpoint ships with a speculative-decoding module called DSpark, but llama.cpp's DSpark implementation was not yet DeepSeek-V4-capable as of late July. Most local numbers circulating today are ordinary decode, missing the speed-up the release was designed around.
The honest caveat is about quality, not speed. Fitting is not the same as performing. This model is already FP4 for its routed experts and FP8 for much else, so squeezing it to two or three bits is a second compression on top of an aggressive first one, not a clean conversion from full precision. One widely shared demo running a three-bit quant on a 96 GB workstation card produced an impressive interactive panel but needed four follow-up correction passes to fix visible defects. Several experienced local users with far better hardware still say they switch to smaller Qwen models for multi-file coding and tool-call consistency. Frontier-class open weights are now locally ownable. They are not yet locally effortless - and, as this site noted when Flash first hit 32 tokens a second on one desktop, the headline number always belongs to somebody's best case.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)