For years the AI hardware story was one story: training, and NVIDIA's GPUs won it. The interesting shift this month is that inference has become its own separate race with its own separate chips, and NVIDIA just made the loudest possible statement about it. Groq 3 LPX, a dedicated inference accelerator, is in full production. It comes from the roughly $20 billion deal NVIDIA closed at the end of 2025 to absorb Groq, the startup whose LPU architecture had been beating GPUs at exactly this job.
When the training champion pays twenty billion dollars to acquire the team that was better at inference, that tells you where the next fight is.
Why inference gets its own chip
A GPU is a generalist built to do enormous parallel math, which is what training needs. Generating tokens one after another for a running agent is a different shape of problem. The bottleneck is the decode phase, the step-by-step production of each next token, and it rewards low latency far more than raw throughput. Groq's LPU was designed around that specific job, and the LPX is the productized version slotting into NVIDIA's Vera Rubin racks, up to 256 accelerators per rack, aimed squarely at fast token generation.
I wrote a couple of weeks ago that when a model's quality converges with everyone else's, the differentiation moves into how it is served. This is that idea cast in silicon. The same weights, run on hardware built for decode latency instead of training throughput, become a faster and cheaper product. The chip is the serving layer.
Why this matters if you never touch a data center
Most of us rent inference, we do not build it, so the read-through is about economics and design rather than procurement. Two things follow.
First, the fast-inference tier is going to get cheaper and more available, because the biggest player just vertically integrated the technology that made it fast. The 750-tokens-per-second previews and the aggressive price cuts of the last month are not a promotion. They are early signs of a hardware shift that lowers the floor under inference cost, and that floor keeps dropping as this chip ships in volume.
Second, and more useful for a builder, cheap fast inference changes which agent designs are worth attempting. Loops that felt too slow or too expensive a year ago, like generating several candidates and picking the best, re-planning after every step, or running a verification pass before showing a user anything, all get more reasonable as the per-token cost of decode falls. The teams that win the next round will be the ones whose agent loops were designed to spend tokens freely, because the tokens are about to be cheap.
The part worth keeping an eye on
There is a concentration story here I would be careless to skip. NVIDIA already dominated training. Buying the leading inference challenger and folding its technology into its own racks means the company now has a strong position in both halves of the AI compute stack. That is efficient, and it also means the cost and availability of the inference you rent increasingly trace back to one company's decisions. Good for prices in the short run, worth watching in the long run.
Still, the direction is clear and it favors people who build. Inference is now a first-class hardware category with real competition and real money behind making it fast. Plan your agents on the assumption that fast, cheap token generation is arriving broadly, because the largest company in the industry just spent twenty billion dollars to make sure it does.
If you run inference-heavy workloads and have watched your cost-per-token move this year, I would like to hear which way and by how much. Those real numbers are the best evidence of where this is actually going.
Top comments (0)