Synchopate’s llama-modchip (v2.4.0) claims to run DeepSeek-V4-Flash — 284.33B total parameters, 13.17B active — on a pure 64 GB CPU system with zero GPUs. The compressed GGUF is 56.56 GiB (1.71 bpw via SwiGLU-H2 + Fractal-64), peak VmRSS stays under 59 GiB at 256K context, and they report 44.36 tok/s prefill + 9.1–14.7 tok/s decode on a 16-core Zen 5. Smaller 30B MoE variants hit 25–111 tok/s. The catch is a custom tensor type that requires their patched llama.cpp fork; stock Ollama or llama.cpp rejects it with unknown tensor type 40.
That is a legitimate systems win for local inference. The model + KV cache now fits in RAM that a lot of workstations already have. Speculative decoding (their Stockfish-PVS L3) and 13 ms KV restore from /dev/shm make the numbers less theoretical. For offline coding agents or long-context research this moves the needle.
It does not move the needle for the latency budget that actually matters when an agent has to listen to a live call.
The capture path is still the bottleneck
Even if you have a 284B model sitting in RAM and answering in ~70–110 ms per token on CPU, the end-to-end path from speech to usable transcript (or to the first tool call) is dominated by Windows audio.
Typical real-world stack on a modern Windows laptop or desktop:
- Endpoint device (USB headset, Bluetooth HFP/A2DP, built-in mic, or WASAPI loopback of the render endpoint).
- Windows Audio Engine (audiodg.exe + audioeng.dll): shared-mode mixing, communications enhancements, spatial audio, vendor APOs.
- Your capture client buffer (10–40 ms practical range; lower is fragile under load or with Bluetooth).
- Resample + format conversion (48 kHz float32 → 16 kHz int16 is common) + lightweight VAD.
- Streaming STT first partial / final (even a fast local or cloud model is 35–150 ms time-to-final on recent Coval numbers).
- Overlay or UI update (<16 ms if you paint on a dedicated thread).
Add them up and a “fast model” still lands in the 80–200 ms range before the agent sees clean text. Bluetooth and shared-mode contention make it worse. The llama-modchip numbers are impressive for generation; they do not appear in this timeline.
Core Audio path. Shared mode goes through the engine mixer; exclusive mode bypasses it but is exclusive. Loopback is an input stream on a render device. Diagram from Microsoft Learn.
Dual-track capture is non-negotiable
For any call or interview assistant you need two independent streams:
- Microphone (your voice)
- Loopback of the render endpoint (the other side)
Mixing them early destroys speaker labels and forces the recognizer to handle two languages or two accents in one beam. The correct architecture is two lock-free SPSC ring buffers, two consumer threads that resample once and run a cheap energy/VAD gate, and only then open or reuse the streaming WebSocket (or local model) with 20–40 ms chunks.
WASAPI exclusive mode on the capture side (and on the render side if you are doing local TTS) is the only reliable way to keep the buffer period under ~10 ms and avoid the engine’s enhancements. Shared mode is easier and more compatible; it also adds 10–40+ ms of engine latency and any “communications” processing the OEM decided to enable.
A practical skeleton (C++ / WASAPI, simplified):
// Exclusive-mode capture, 10 ms period, 48 kHz float32
WAVEFORMATEX wfx = { WAVE_FORMAT_IEEE_FLOAT, 1, 48000, 48000 * 4, 4, 32, 0 };
// AUDCLNT_SHAREMODE_EXCLUSIVE, AUDCLNT_STREAMFLAGS_EVENTCALLBACK
// then IAudioCaptureClient::GetBuffer in the event thread
// copy into SPSC ring (power-of-two, no alloc on writer)
// consumer does one resample to 16 kHz int16 and VAD
Do the resample and VAD off the audio callback. Allocation or locking on the capture thread is how you get dropouts.
Local models help after the audio is clean
Once the audio is in a clean 16 kHz mono buffer, a small local STT (Qwen3-ASR 1.7B Q4_K_M is ~1.3 GB and measures in the 35 ms time-to-final range on recent public benchmarks) can run inside the same process as the overlay. A larger model like the llama-modchip DeepSeek can sit in another process or even another machine and only see the text. The CPU-only 284B is useful for the agent reasoning step; it is not useful for the first 100 ms of the pipeline.
Tradeoffs worth stating:
- Fractal-64 / 1.7 bpw is aggressive. The project shows kernel exactness against the unpatched runtime on the same file; it does not prove the model is as capable as the API version. For tool-calling or precise extraction you still want to measure on your actual prompts.
- Custom tensor type means you are pinned to their binary or you re-implement the kernels. Stock GGUF tooling will not load it.
- 9–15 tok/s decode is fine for non-interactive research; for a live agent that needs to emit a tool call or a short reply while the other person is still talking, you still want a smaller model or a cloud endpoint for the hot path.
- Memory is tight. 57–59 GiB VmRSS on a 64 GB box leaves almost nothing for the OS, browser, or the capture process itself. On Windows the working set and commit charge behave differently than Linux VmRSS; test it.
What actually ships for real-time
The projects that make local agents usable on Windows right now are the ones that treat the capture path as the primary constraint:
- Exclusive WASAPI or a carefully tuned shared-mode period.
- Lock-free rings and zero-copy or single-copy into the STT buffer.
- Early, single resample to the model’s native rate.
- Separate pipelines for mic and loopback.
- Overlay painted on a dedicated thread so the audio thread never waits on UI.
- Optional local residual echo cancellation if the agent is also speaking through the same device.
Tools that ignore this and just throw a big model at the problem still feel laggy on a real call. The llama-modchip numbers are real for what they measure. The Windows audio path is also real, and it has not gotten faster because a new quantization format appeared.
If you are building anything that has to listen to a live Windows call (interviews, support, meetings, language practice), start with the capture graph. The model size is the easy part once the samples are clean and on time.
I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.
Top comments (2)
The number that matters for 284B-on-CPU is not whether it runs but $/M output tokens at the speed you actually get. A 64GB DDR5 box at 1-2 t/s produces 3.6-7k tokens an hour; the electricity for that is pennies, so the energy is cheap and the machine is not. Spread two years of ownership across ~50M tokens and you land around $20-30/M - roughly 50-100x a hosted small model. CPU inference at this scale is a capability demonstration, not a cost strategy, unless the workload is batch and the data cannot leave the building.
The two independent ring buffers also need a shared timing contract. Keeping microphone and loopback audio separate preserves the source labels, but independently resampling the streams does not by itself keep their transcripts aligned over a long call. Small differences in their capture timelines can accumulate.
I would retain capture timestamps through resampling and test a replay with a slight clock-rate mismatch between tracks. Checking alignment near the beginning and end would expose drift that a short latency benchmark misses. That matters before merging partial transcripts into the single conversational context sent to the reasoning model.