A 304B Model Running on ONE GPU — and the FP8 Trap Nobody Warned You About

Subtitle: DeepSeek V4 Flash, a 304B-parameter MoE checkpoint, running entirely in the HBM of a single AMD MI300X — no quantization, no offload, 168 tokens/sec. This is the story of how the cost curve of frontier inference just bent, and the subtle FP8 bug that almost broke it.

Every few months, an AI headline lands that's really about the economics of the field rather than a new benchmark. Today's is one of those: a production-grade setup running DeepSeek V4 Flash — a 304-billion-parameter model with 256 experts — on a single AMD MI300X GPU, at usable speed, with the full checkpoint resident in memory. No quantization, no weight offload, no tricks. Just 156.67 GiB of weights sitting in 192 GB of HBM3.

Here's why that matters, what actually had to be fixed to make it work, and the FP8 trap that would have silently corrupted results.

What was actually achieved

The pinned production stack (vLLM ROCm nightly + AITER) reports:

168.6 tokens/sec median single-stream decode
~8K tokens/sec prefill with tuned kernels
542 tokens/sec aggregate across 8 concurrent streams
830 tokens/sec burst across 64 streams with no OOM
256K validated context (the architecture supports 1M)

The entire 304B checkpoint fits in HBM because the MI300X carries 192 GB of HBM3 with 5.3 TB/s of memory bandwidth — 2.4× the HBM capacity of an H100 SXM5, at roughly half the list price. That's the whole story in two numbers: a frontier-class model, single-card deployment, dramatically lower cost per token.

Why "fits in one GPU" is the real revolution

For years, the narrative has been: big models need big clusters. MoE models like DeepSeek's are dense in experts — 256 of them — which historically pushed total VRAM requirements past what any single accelerator could hold, forcing multi-GPU tensor parallelism, weight streaming, or aggressive quantization.

This setup breaks that assumption:

No quantization. The checkpoint runs as shipped (native MXFP4 for the MoE exports per the community notes, but no additional compression was needed).
No offload. No PCIe weight streaming, no layer swapping — the whole model stays in HBM.
Headroom to spare. After the weights, there's still room for a 20 GB GPU KV pool and a 96 GiB CPU tier for evicted prefix-cache entries.

The practical consequence: an individual developer or small lab can now serve a frontier-adjacent open model on a single workstation-class card, rather than renting an 8-GPU box.

The FP8 trap that would have corrupted everything

Here's the part most write-ups will skip: AMD's MI300X doesn't implement FP8 the way you'd expect. It implements the AMD/Graphcore fnuz variant of E4M3, while MI325X and newer cards use the OCP-standard FP8. A kernel that assumes OCP semantics on an MI300X can be wrong by a factor of two in the scale domain — which means silently incorrect logits, not a crash. You'd serve garbage results and never get an error message.

Fixing correctness on this FP8 implementation was the first priority of the whole project; performance tuning came after. That's a good lesson for anyone deploying on non-NVIDIA hardware: the silicon may look like a drop-in replacement, but the numerics can differ in ways that are invisible until you validate outputs.

What had to be fixed beyond FP8

Running an NVIDIA-first recipe on AMD hardware is never "just works." The production setup required fixes for:

MoE routing at high concurrency — routing decisions that race under load
Causal speculative verification — the DSpark-style drafting pipeline
CPU-KV synchronization — keeping the CPU-side cache coherent
Several untuned kernel shapes — the predictable "works on paper, slow in practice" layer

None of this is glamorous. All of it is what "community productionization" of open models actually looks like: not a single heroic patch, but a long tail of small, version-pinned corrections.

The honest caveats

Hardware availability. You can't casually buy a single MI300X — it ships as an OAM module, typically in 8-GPU boxes (~€250K). The MI350P (PCIe, 144GB) is the more practical single-card target, and V4 Flash fits in 144GB too thanks to native MXFP4.
Headroom to NVIDIA. Community benchmarks still show DeepSeek's own H800 serving hitting higher throughput — there's probably more performance on the table with further optimization.
This is one GPU serving modest concurrency. It's not a replacement for a serving fleet; it's a replacement for renting one when you don't need fleet scale.

What this means for developers

Three takeaways, in order of importance:

The memory barrier fell. The binding constraint on serving big open models is now the size of one GPU's HBM, and that keeps growing (MI325X, MI355X, and successors push further). 300B-class models on single cards is a trend, not a one-off.
Numerics are now a porting concern. The fnuz-vs-OCP FP8 divergence is a warning shot: as more vendors compete, "compatible hardware" will hide subtle format differences. Always validate outputs, not just throughput.
The community is doing the platform work. Version-pinned Docker stacks, SHA-pinned overlays, reference diffs — this is how open-weight models become deployable infrastructure. The repo isn't just a config dump; it's a productionization artifact.

Bottom line

A 304B-parameter model running at production speed on a single GPU — no quantization, no offload — is the kind of headline that quietly resets expectations. It doesn't mean the cluster era is over; it means the entry point for serving frontier-adjacent models just dropped by an order of magnitude in hardware cost. And the FP8 story is a reminder that the interesting problems in this field are increasingly about numerics and systems, not just model architecture.

This is part of SinoBot's daily AI pulse — tracking the frontier of AI research, models, and infrastructure. Follow for regular breakdowns of what actually matters in AI.