Last winter I put NVIDIA's DGX Spark on my desk. GB10 Grace Blackwell chip, 128 GB of unified memory, about 4,700 euros, roughly the size of two stacked paperbacks. I run an automation consultancy, my clients are German and allergic to sending documents into US clouds, so "the model runs in my office" is a feature I can literally sell.
This is the write-up I couldn't find before buying: what this box is actually good at, why the spec sheet misled me for an entire evening, and the config that turned it from a disappointment into the machine that now handles most of my production workloads.
The number that matters isn't on the marketing page
Here's the trap. The headline specs scream capacity: 128 GB unified memory, a petaflop of FP4 compute. Capacity determines what fits. It says nothing about what's usable.
Token generation is memory-bandwidth-bound. For every single token, the engine streams essentially all active model weights through the chip. So your ceiling is simple division:
text
max tokens/sec ≈ memory bandwidth / bytes of active weights
DGX Spark bandwidth: ~273 GB/s
Dense 70B @ Q4 (~42 GB): 273 / 42 ≈ 6.5 tok/s ceiling
Dense 70B @ FP8 (~70 GB): 273 / 70 ≈ 3.9 tok/s ceiling
Real-world numbers land below the ceiling. NVIDIA's own Ollama benchmark shows about 2.7 tok/s for a 70B dense model at FP8 on this box. My first evening, I loaded exactly such a model, typed a prompt, and watched tokens drip out like a leaky faucet. For comparison, an M3 Ultra with its 819 GB/s does the same model at 25-30 tok/s. The Spark has triple the Mac's practical model capacity and a third of its bandwidth. Nobody puts that sentence in a product video.
So is the box bad? No. I was holding it wrong, in two specific ways.
Fix one: stop running dense models
Mixture-of-experts models only activate a fraction of their parameters per token. The bandwidth equation only counts active bytes:
text
MoE, ~110B total / ~12B active @ Q4 (~7 GB active):
273 / 7 ≈ 39 tok/s ceiling
Suddenly the box makes sense. The 128 GB holds the full expert set, which smaller machines simply cannot fit, while the per-token traffic stays small. Big MoE models are precisely the workload this hardware was shaped for, and the current open-weight generation (the larger Qwen MoE variants, gpt-oss-120b, and friends) delivers 30-80 tok/s here depending on model and engine. That's faster than anyone reads.
The practical rule I now give everyone evaluating this class of hardware: never trust a tokens-per-second number without three qualifiers. Which model architecture (dense or MoE), which quantization, which inference engine. The same box produces a 2.7 and a 60 depending on those choices, and both numbers are honest.
Fix two: don't use the default stack for the big stuff
The convenience tools (Ollama out of the box) are wonderful for getting started and noticeably behind on this specific ARM+Blackwell platform. The optimized path, TensorRT-LLM or a recent llama.cpp built with CUDA support, plus NVFP4 quantized weights where available, is worth real percentages. NVIDIA has been shipping steady software updates for the platform since launch and the gains have been meaningful, which is a polite way of saying the launch-day software left performance on the table.
My setup settled into three layers.
Layer 1: llama.cpp servers, one per model. Compiled from source on the box (it runs DGX OS, an Ubuntu derivative, so this is unexciting). Each model gets a llama-server process exposing an OpenAI-compatible endpoint.
Layer 2: llama-swap as the traffic cop. I don't want five models resident at once, and I don't want to SSH in to switch them. llama-swap is a small proxy that lazily starts and stops model servers based on the model field of incoming requests:
`yaml
config.yaml (abridged)
models:
"qwen-big":
cmd: >
/opt/llama.cpp/llama-server
-m /models/qwen3-moe-q4.gguf
--port ${PORT} -c 32768 -ngl 999
ttl: 900 # unload after 15 min idle
"gemma-fast":
cmd: >
/opt/llama.cpp/llama-server
-m /models/gemma3-27b-q4.gguf
--port ${PORT} -c 16384 -ngl 999
ttl: 900
Request model: "qwen-big" and the proxy spins it up, drains the old one, and routes. From the outside it behaves like one endpoint hosting many models, exactly like a cloud provider.
Layer 3: a LiteLLM proxy in front of everything. This is the piece I'd defend in a knife fight. All my n8n workflows and scripts speak to one API. Behind it, LiteLLM routes by model name: local names go to the Spark, frontier names go to cloud APIs. Which means any workflow can be moved between local and cloud by changing one string, no code changes:
`
yaml
model_list:
- model_name: local-workhorse litellm_params: model: openai/qwen-big api_base: http://spark.local:8080/v1
- model_name: heavy-thinking litellm_params: model: anthropic/claude-sonnet-4-6``
The migration path this enables is the actual killer feature. I prototype a workflow against a cloud model, and once the prompt is stable I point it at local-workhorse and see if quality holds. It does more often than I expected, roughly for everything that is extraction, classification, summarization or transformation. It doesn't for genuinely hard reasoning, and pretending otherwise would cost me clients.
What actually runs on it in production
Concretely, after six months of settling:
A mid-size dense model (27B class) stays hot for latency-sensitive tasks: classification in my email pipelines, extraction from documents, anonymization before anything leaves the building. Effectively instant for single users.
A large MoE model gets swapped in for quality-sensitive batch work: report generation, first-draft briefings, transcript summarization. Comfortable reading speed, zero marginal cost, and I've stopped hesitating before running experiments across thousands of documents, because the meter isn't running.
An embedding model runs permanently for search across my own document archive. This workload is so light it's almost free, and it's the one where "the data never leaves the room" matters most to me.
Monthly electricity for all this lands somewhere around a tank of fuel, and my cloud API bill dropped visibly. Full honesty though: at solo-consultant volume, the box amortizes over years, not months. I bought data sovereignty and free experimentation, and the cost savings are a side dish.
Should you buy one?
Buy it if: you need big-model capacity in a small quiet box, your workloads are MoE-shaped or batch-shaped, CUDA compatibility matters to you, or "local" is a compliance requirement you can invoice against.
Skip it if: you mainly want fast dense 70B chat (a Mac Studio with fat bandwidth beats it embarrassingly), you've never touched Linux (it's Linux, fully, forever), or your real goal is learning, in which case an 8B model on your current laptop teaches you the same lessons for free.
And whatever you buy: do the bandwidth division before the purchase, not after, on the first evening, with a sinking feeling, like a certain consultant I could name.
Comments are open, and I'm specifically curious what tokens/sec others are getting on this platform with recent llama.cpp builds versus TensorRT-LLM. My numbers keep improving with every software update and I've half stopped trusting my own benchmarks.
Top comments (0)