TL;DR — The token-price collapse you keep celebrating is a hardware war, not a software miracle. NVIDIA's near-monopoly is under attack on two flanks: Cerebras and Groq are racing on raw speed (thousands of tokens per second), while NVIDIA's own next-gen Rubin is counterpunching on cost-per-token. Whoever wins that fight — not your prompt engineering — sets your real inference bill and your latency floor.
Every few weeks someone publishes a chart of plummeting LLM prices and credits it to "efficiency" — better quantization, smarter batching, leaner prompts. That story is comforting because it's the part you control. It's also mostly wrong. The dominant force pushing your cost-per-token down isn't a clever decoding trick; it's a brawl over the silicon that the tokens run on. The price you pay is, to a first approximation, a hardware number with a software wrapper.
That matters because it changes what you should be paying attention to. If your inference economics are set by prompt golf, you optimize prompts. If they're set by which accelerator wins the next 18 months, you watch the foundries, the power contracts, and the architecture roadmaps — and you build so you can move when the floor drops. Here's the actual war.
The incumbent: NVIDIA owns the ground you're standing on
Start with the fact everyone knows and few price correctly: NVIDIA isn't winning inference because its chips are the fastest at generating tokens. It's winning because it owns training, and because CUDA is the most expensive switching cost in computing. Every serious model is trained on NVIDIA. Every framework, kernel, and serving stack assumes CUDA first. That gravity well means most inference lands on NVIDIA hardware by default, not by benchmark.
The result is a quasi-monopoly on the plumbing of AI. And monopolies don't usually cut their own prices — until someone makes them. The interesting development of 2026 is that NVIDIA is now being attacked from two directions at once, and it is responding by attacking its own margins before anyone else can.
Flank one: the speed war (Cerebras and Groq)
The first flank is latency and throughput, and it is not being fought with GPUs at all. Two companies decided the GPU was the wrong shape for inference and built something else.
Cerebras builds a wafer-scale processor — the CS-3 system is a single chip the size of a dinner plate, the largest commercial chip ever made. The whole point is to keep the model's weights in enormous on-wafer SRAM instead of shuttling them across a network of smaller GPUs. Memory bandwidth is the thing that throttles token generation, and wafer-scale attacks it head-on. On the open GPT-OSS-120B model, Cerebras reports roughly 2,700 tokens/sec with a time-to-first-token around 280ms. For comparison, Cerebras's own benchmark puts a Blackwell GPU at roughly 650 tokens/sec on the same model — call it a 4x gap. (That figure is vendor-sourced and Cerebras has every incentive to flatter it, so treat the exact multiple as marketing-adjacent. The order of magnitude, though, is real and repeatable across independent third-party measurements.)
Groq takes a different route to the same goal with its LPU — a "Language Processing Unit" built around deterministic, software-scheduled dataflow instead of the GPU's dynamic, cache-driven execution. No speculative scheduling, no memory-hierarchy guessing; the compiler lays out exactly when every value moves. That determinism is what lets Groq sustain very high tokens/sec at low, predictable latency. On price, Groq sits in the $0.15/M input, $0.75/M output class for open models (secondary figures — confirm against the live price sheet before you build a budget on them).
The trade is explicit and worth saying plainly: both architectures buy speed by sacrificing flexibility. Wafer-scale and LPU systems are tuned for a curated set of models. You don't get the GPU's "run literally anything, including the training run you'll do next month" generality. You get a blisteringly fast appliance for serving the models it supports. For a lot of production inference, that's exactly the trade you want — but it is a trade, not a free lunch.
Why thousands of tokens per second is a product feature, not a flex
It's tempting to read 2,700 tok/s as a vanity number. It isn't. There's a UX phase change that happens somewhere north of human reading speed, and these systems blow past it.
At 50 tok/s — typical streamed GPU output — you watch the model think. You stream tokens because you have to; the latency is the experience. At 2,000+ tok/s, a multi-paragraph answer materializes effectively instantly, which kills the entire premise of streaming and unlocks workloads that were previously impossible. Agentic loops are the big one: an agent that makes ten sequential model calls to plan, act, and reflect is dead on arrival at 50 tok/s and snappy at 2,000. Speculative decoding, long chain-of-thought, multi-pass self-critique — all the techniques that trade more tokens for better answers — only become economical when each token is nearly free in time. Speed doesn't just make the same product faster; it changes which products are buildable.
The OpenAI–Cerebras bet: speed at industrial scale
If you thought the speed flank was a niche play, the numbers say otherwise. In January 2026, Cerebras signed a roughly $10 billion cloud-inference contract with OpenAI — about 750 MW of capacity, which works out to something like 32,768 CS-3 systems, deploying from Q1 2026 through 2028 (reported by The Next Platform, January 15, 2026).
OpenAI committing ten billion dollars to non-NVIDIA inference silicon is the loudest possible signal that the latency war is real, well-funded, and aimed squarely at the GPU's most profitable workload.
On pricing, Cerebras's cloud sits around $0.25/M input and $0.69/M output for the open models it serves — competitive with GPU clouds on dollars while being multiples faster on tokens/sec. That combination — comparable price, far better latency — is precisely the wedge a challenger needs to pry workloads off the incumbent. You don't have to be cheaper if you're dramatically faster at the same price; you just have to be fast enough to enable products the incumbent can't.
Flank two: the efficiency war (NVIDIA vs. NVIDIA)
The second flank is cost-per-token, and here NVIDIA is the one swinging — at its own installed base. At CES in January 2026, NVIDIA announced Vera Rubin NVL72, the successor to the Blackwell generation. The headline claims (NVIDIA's own, so read them as vendor targets, not benchmarks): up to 5x inference performance and up to 10x lower cost-per-token versus Blackwell, with volume ramp in the second half of 2026.
Sit with that 10x. Even if the real-world number lands at half the claim, NVIDIA is telling its customers that the chips they bought last year will be roughly an order of magnitude more expensive per token to run than the chips shipping this year. That is not a company resting on a monopoly. That is a company that has looked at Cerebras, Groq, and the hyperscalers' in-house silicon (Google's TPUs, Amazon's Trainium/Inferentia) and concluded the only safe move is to obsolete itself on a schedule before a competitor does it for them. The CUDA moat protects the platform; it does not protect any single generation's pricing.
This is the part the "it's just software efficiency" crowd misses. A 10x cost-per-token improvement at the silicon layer swamps anything you can squeeze out of a prompt. When the token-price chart drops a notch in late 2026, the cause won't be your batching strategy. It'll be Rubin entering the fleet.
Power is the wall everyone eventually hits
Underneath both flanks sits the constraint that increasingly decides the whole game: electricity. The Stanford AI Index 2026 (via IEEE Spectrum) pegs AI datacenter power at roughly 29.6 GW — the scale of a large national grid, devoted to running models. At that scale, efficiency stops being an environmental footnote and becomes the binding cost input.
The same data offers an illustrative spread worth internalizing: the least-efficient inference setups can emit more than 10x the carbon of the most efficient for comparable work, with one model drawing on the order of 23W for a medium prompt where another draws around 5W (these are illustrative single-model figures, not a universal law). The lesson generalizes cleanly: when you're power-bound, the accelerator that delivers more tokens per watt wins on cost and on how much capacity you can physically stand up behind a fixed power contract. Tokens-per-watt is quietly becoming the metric that decides which of these architectures can actually scale — because at 750 MW deployments, you can't buy your way past the substation.
What this means if you serve models at scale
Strip away the vendor theater and the operating reality for an infra team is blunt: your serving cost is dominated by hardware dollars-per-token and your utilization, not by which model you picked. A few practical consequences follow.
Latency is now a procurement decision, not a code decision. If your product needs real-time agents, no amount of streaming polish gets you what a Cerebras or Groq endpoint gives you out of the box. Buy the latency floor; don't try to engineer around physics.
Don't hard-couple to one accelerator's quirks. The same lesson as model-swapping applies one layer down. The vendor with the best dollars-per-token will rotate — Blackwell, then Rubin, then whatever Cerebras and the TPU teams answer with. Build your serving layer so the accelerator behind the endpoint is a config value, not an assumption baked into forty call sites.
Track tokens-per-watt, not just dollars-per-token. In a power-bound world the two converge, and the watt number is the one that tells you whether you can grow.
Utilization is the silent margin killer. A faster, pricier chip running at 80% beats a cheaper one idling at 20%. The hardware war only helps you if your scheduling actually keeps the silicon hot.
Where this lands by 2027
Here's my read. NVIDIA keeps the throne — CUDA and its lock on training are not falling in eighteen months, and Rubin's cost-per-token claims, even discounted, are enough to defend the bulk of the market. But the monopoly on inference specifically gets carved up at the edges, and the edges are the high-value parts: ultra-low-latency, agent-heavy, real-time workloads peel off to wafer-scale and LPU systems, with OpenAI's $10B Cerebras bet as the template others copy. The hyperscalers' in-house chips quietly eat another slice for their own first-party traffic.
The net effect for everyone serving models: cost-per-token keeps falling fast through 2027, and almost none of that win will come from your prompts. It'll come from Rubin ramping, from challengers forcing price discipline, and from power efficiency becoming the real ceiling. The teams who profit are the ones who treated the accelerator as a swappable component and built the eval and serving harness to move the moment a cheaper, faster floor appears. The token-price collapse is real — just don't flatter yourself about who's causing it. It's being decided in a fab, not in your prompt.
Top comments (0)