Yesterday morning my agent pipeline was crawling. Six model calls chained together, each one waiting for the previous to finish decoding. The model itself was fast enough per call, but at roughly twenty tokens per second of decode speed, the whole loop took over a minute. Nobody waits a minute for a chat reply in 2026, and no agent pipeline survives six of those in a row.
That is the actual problem both companies in this article are selling against, whether they say it plainly or not. Cerebras just announced the CS-4, a rack-scale system it claims delivers up to 30x faster inference than GPU systems, and the launch shot straight to the top of Hacker News with around 400 points. NVIDIA's competing answer for the reasoning era is the GB300 NVL72, a 72-GPU rack that anchors most frontier inference deployments today.
Full disclosure up front: I have not run either rack system. Almost nobody has, they cost millions and ship to hyperscalers first. What I do run daily is inference-dependent agent infrastructure built on Spring Boot, and I have spent enough time modeling token latency to know which of these numbers matter and which are marketing. This piece is a developer's read of two spec sheets: what is real, what is spin, and what it means for the APIs you will actually call.
The two contenders in one paragraph each
Cerebras CS-4 is the fourth generation of the wafer-scale idea. Instead of wiring 72 separate chips together, Cerebras manufactures one enormous chip, the Wafer Scale Engine, and builds a system around it. The CS-4 contains three WSE-3 Turbo wafers, each delivering up to 2x the speed of the previous generation, linked with a wafer-to-wafer latency of 2 microseconds. Cerebras claims over 1,000 tokens per second on models exceeding 10 trillion parameters, up to 10x more throughput per watt than its own CS-3, and first shipments this quarter. All of these figures are from the Cerebras.com CS-4 product page.
NVIDIA GB300 NVL72 is the incumbent. One rack integrates 72 Blackwell Ultra GPUs and 36 Grace CPUs connected by 130 TB/s of NVLink bandwidth, with roughly 20 TB of HBM3e GPU memory delivering up to 576 TB/s of bandwidth, plus 17 TB of LPDDR5X on the CPU side. On paper it delivers 1,440 PFLOPS of FP4 compute with sparsity, 1,080 dense. NVIDIA positions it explicitly for test-time scaling and reasoning workloads. These figures are from the NVIDIA GB300 NVL72 product page.
Two completely different bets on the same question: how do you make trillion-parameter models respond instantly?
Why wafer-scale attacks latency at the root
The core insight is distance. When a large model runs across 72 GPUs, every step of the computation involves moving activations between chips. NVLink is astonishingly fast, but light only travels so far in a nanosecond, and a signal crossing a rack-scale fabric costs real time on every single layer of the transformer.
Cerebras sidesteps this by putting the whole model's critical compute on one piece of silicon. The prior WSE-3 packed 900,000 AI cores and 44 GB of on-wafer SRAM with roughly 21 PB/s of memory bandwidth, and the WSE-3 Turbo in the CS-4 pushes clocks and throughput further. When your weights live centimeters from your compute instead of across a switch fabric, decode stops being bound by interconnect.
The 2 microsecond number is the one I would watch. Cerebras says wafer-to-wafer latency is as low as 2 microseconds, which is what lets three wafers behave like one logical accelerator. For comparison, cross-rack GPU communication is measured in tens to hundreds of microseconds depending on topology. For interactive decode, where every generated token re-reads the whole model state, that gap compounds thousands of times per response.
The power story is quietly radical too. The CS-4 puts power delivery 0.5 millimeters from the processor, versus roughly 50mm on conventional GPU boards. Cerebras claims this nearly eliminates board-level power loss and lets each wafer run at higher frequency, delivering twice the power to the silicon. Cerebras also claims up to 10x more throughput per watt than the CS-3. Efficiency claims across vendors are rarely apples-to-apples, but directionally, feeding silicon cleanly is how you buy clock speed.
Why the GPU rack refuses to die
Raw memory capacity still favors NVIDIA, massively. A GB300 NVL72 rack carries about 37 TB of fast memory in total. That is what lets you keep multiple giant models resident, run huge batches for throughput-oriented serving, and absorb the long contexts that reasoning models demand. Wafer-scale SRAM is blazing fast but small; HBM3e is a memory compromise that happens to be enormous. If your economics are "maximum tokens per megawatt for millions of users," batch-heavy GPU serving remains the default.
The ecosystem moat is real. Every serving stack, every quantization scheme, every KV cache optimization, every piece of operator tooling in mainstream inference was built for CUDA GPUs first. NVIDIA also owns the training market outright, and labs like keeping training and inference on aligned hardware. Cerebras inference is real and fast, but it is a much thinner software world.
NVIDIA is not standing still on the exact weakness Cerebras attacks. The GB300 exists because the GB200's interconnect ceiling showed up in real reasoning workloads, and the Vera Rubin platform iterates further. The gap narrows every generation even if the architectural cost of chip-to-chip communication never disappears.
Reading the "30x" claim honestly
Headline multipliers are configuration-specific. Cerebras says up to 30x faster inference compared to GPU systems, and its own page carries the caveat that improvements vary by workload, configuration, and model. The 30x figure will be true against some baseline configuration on some latency-bound workload, almost certainly single-stream decode where interconnect overhead dominates. Against a well-tuned GB300 NVL72 running batched throughput serving, the real-world gap for a given cost will be far smaller, and possibly reversed for batch workloads.
The number that matters to you is tokens per second per dollar on your workload. For interactive agent loops, single-stream decode speed dominates, and that is precisely where wafer-scale shines. For nightly batch summarization of a million documents, throughput per watt dominates, and that is GPU territory. Neither vendor's marketing page answers the question for your case, which is why I treat both spec sheets as directions, not destinations.
What this means if you build agents (the part spec sheets skip)
Here is the practical takeaway, and the reason I track hardware launches at all when my daily work is Spring Boot services.
Agent loops multiply latency. A single model call at 20 tokens per second feels acceptable. Six chained calls feel broken. At 1,000+ tokens per second decode, the same pipeline finishes before your user's attention wanders. Hardware like this is why agentic products that were demos in 2024 are shippable in 2026. The CS-4's claim of over 1,000 tokens per second on 10-trillion-parameter-class models, if it holds in production, changes what an acceptable agent step budget looks like.
Speed asymmetry means model routing. You should not pay frontier-model latency for every step. In my own setup I route differently by task, and in Spring AI this is just two beans. Here is the shape of it:
@Bean
ChatModel deepModel(OpenAiApi deepApi) {
// Frontier model, used for planning and final synthesis
return OpenAiChatModel.builder()
.openAiApi(deepApi)
.model("frontier-reasoning-model")
.build();
}
@Bean
ChatModel fastModel(OpenAiApi fastApi) {
// Speed-optimized endpoint, used inside agent loops
// where decode latency compounds across steps
return OpenAiChatModel.builder()
.openAiApi(fastApi)
.model("speed-optimized-model")
.build();
}
The API details change monthly, and the specific vendors behind deepApi and fastApi change even faster. The architecture does not: keep a slow smart model at the edges of your pipeline and a fast cheap one inside the loop. Hardware competition like CS-4 versus GB300 NVL72 is exactly what keeps making the fast tier faster and cheaper, and your routing code is how you capture that instead of reading about it.
Falling inference cost per token is the real product of this rivalry. Every claim in both launches, throughput per watt, tokens per second, deployment time shrinking from days to hours with Cerebras's modular Wafer-Scale Backpack design, is ultimately a bet on cheaper, faster tokens. You will likely never touch either rack. You will touch the APIs built on them, and their price-per-token curves are what your agent budget lives on.
My scorecard
Choose Cerebras-style speed when: your workload is interactive and latency-bound, single-stream decode dominates, you are building agent loops or coding assistants where time-to-completion is the product, and you are happy targeting an OpenAI-compatible endpoint rather than owning your stack.
Choose GPU-style serving when: you need maximum throughput per dollar at scale, huge resident model capacity, the full CUDA tooling ecosystem, or training and inference on one platform.
What I actually expect: both to win their halves of the market. NVIDIA keeps batch serving and training, and frankly most of everything, while wafer-scale takes the latency crown and drags interactive expectations upward. Developers get faster tokens either way without caring whose logo is on the rack.
If I were building a startup around agents today, I would architect for model routing from day one, assume the fast tier keeps getting absurdly faster, and spend my differentiation budget on everything except raw inference, because raw inference is about to become someone else's race to the bottom.
The checklist I would tape to the monitor
- Decode speed, not total throughput, is what your users feel in agent loops. Benchmark single-stream.
- Route by task: smart and slow at the edges, fast and cheap inside the loop.
- Treat every vendor multiplier as "up to," workload-specific. Ask for tokens per second per dollar on your traffic.
- Watch wafer-to-wafer and interconnect latency numbers, they are the leading indicator of interactive performance.
- Never couple your codebase to one inference vendor. OpenAI-compatible endpoints are the portability layer.
I write about Java, Spring Boot, and AI every week. Subscribe, it's free.
Have you run anything on Cerebras inference, or noticed your provider's fast tier getting quicker this year? What does decode latency do to your agent pipelines? I would genuinely like to hear your numbers in the comments.
Top comments (0)