OpenAI Just Broke the AI Compute Monopoly — and NVIDIA Can't Undo It

Yesterday at Hot Chips 2026, OpenAI published the first real numbers for Jalapeño, the custom inference chip it co-designed with Broadcom. The headlines are polite about it. The data is not.

On SemiAnalysis's public InferenceX benchmark, Jalapeño beat every Nvidia, AMD, and Google accelerator that SemiAnalysis has been able to test — on multiple open-source frontier models, with reproducible baselines, in OpenAI's own labs. The numbers: 1.5x to 1.9x more work per watt, 1.7x to 3.6x lower end-to-end latency, and on the highly interactive workloads that actually drive ChatGPT's traffic, 2.1x to 4.1x faster than a comparable Blackwell system.

All of this at 700 watts. The Blackwell it was measured against pulls 1,200 to 1,400 watts. Measured sustained power during the benchmarks stayed at or below 550 watts. Air-cooled, not liquid-cooled.

Richard Ho, OpenAI's head of hardware, put it bluntly: "The bottom line is that the results show a very, very significant performance advance over state of the art."

He's not wrong. And the market reaction this morning — hours before NVIDIA reports Q2 earnings after market close — is going to be the real story.

Why This Matters More Than the Earnings Call

NVIDIA's Q2 is expected to print roughly $91 billion in revenue, around 96% year-over-year growth. By any normal standard, that's a historic quarter. The stock has been pricing it in for weeks.

But here's the problem: the bull case for NVIDIA has always been that nobody can build a serious AI workload without their chips. Every frontier lab. Every hyperscaler. Every sovereign AI project. They all buy H100s, then H200s, then Blackwell, then Rubin. The order book stretches into the next decade. The moat is "we make the only thing that works."

Jalapeño just demonstrated that the moat is narrower than people thought.

This isn't the first crack. Google has been running TPUs at scale for years. Amazon has Trainium. Microsoft has Maia. Apple ships its own silicon in every device. Meta has MTIA. The pattern was already clear: the biggest AI customers are all building their own chips, and they're all getting better at it.

What Jalapeño adds is the missing piece — a frontier-lab customer, not a hyperscaler, demonstrating first-generation silicon that's already better than the incumbent on the workloads that matter most. Inference, not training. Power efficiency, not peak FLOPs. Interactivity, not just throughput.

The Numbers, For the People Who Care

For the builders in the audience, the headline benchmark on GPT-OSS 120B (a representative open model):

Metric	Nvidia GB200	OpenAI Jalapeño	Delta
Throughput per kilowatt	44,960 TPS/kW	85,448 TPS/kW	1.9x
End-to-end latency	1.80s	1.03s	1.7x lower
Min time between tokens	1.87ms (535 tok/s/user)	0.69ms (1,459 tok/s/user)	2.7x faster

On DeepSeek R1 670B, a reasoning model, the gap widens:

Metric	Nvidia GB300	OpenAI Jalapeño	Delta
Throughput per kilowatt	11,781 TPS/kW	19,641 TPS/kW	1.7x
End-to-end latency	5.99s	1.65s	3.6x lower
Min time between tokens	5.90ms (169 tok/s/user)	1.43ms (700 tok/s/user)	4.1x faster

On Kimi K2.5 1T (the model Cursor Composer 2.5 is based on, for what that's worth), same story: 1.5x perf/W, 3.4x lower latency, 3.8x faster interactive response.

The chip is built on TSMC's N3P node. HBM4 memory. 13.4 PFLOPS in MXFP4. It's real silicon, not a paper launch.

What "Inference-Only" Actually Means

One important caveat that the headlines are glossing over: Jalapeño is an inference chip, not a training chip. It serves models, it doesn't train them. So this is not a direct threat to NVIDIA's training dominance — OpenAI will still buy a lot of Blackwell and Rubin to train GPT-6.

But training is a one-time cost. Inference is forever.

Every ChatGPT query, every API call to GPT-5.6, every Codex session, every Operator action — all of that runs on inference hardware, every single time, for the entire lifetime of the product. The cost of inference is the cost of running the business.

If you're OpenAI and you can run that workload at half the power and 1.7x to 4x the speed, the math changes overnight. You can:

1. Cut your cost per token by 40-50%. Either keep the margin and make more profit, or pass the savings to customers and undercut the API competitors. Both are good for OpenAI's market position.
2. Run more agents per watt. The whole agent economy — Codex, Operator, Devin-class competitors, the entire "AI does work for you" thesis — is bottlenecked by inference cost. Cut that cost in half and the product suddenly becomes economically viable for a much wider range of use cases.
3. Vertical-integration moat. A frontier lab that owns its own inference silicon is harder to disrupt than one that rents from a chip vendor. Pricing pressure from competitors becomes a margin question, not an existential one.

The NVIDIA Counter-Argument

To be fair, NVIDIA's response will be: "Jalapeño is first-generation, and the real comparison is against Rubin, not Blackwell."

That's true. SemiAnalysis says the same thing in their own writeup. Vera Rubin NVL72 is supposed to deliver 5.4x the perf/W of GB200 NVL72, and OpenAI's chip beats Vera Rubin's MTP results in raw output token throughput per MW.

So the argument is: NVIDIA's next generation will catch up. The performance lead is temporary. The moat reasserts.

Maybe. But there's a counter-counter-argument: OpenAI will ship a second-generation chip. And a third. So will Google. So will Amazon. So will Microsoft. The question isn't whether OpenAI's first inference chip is permanently better than NVIDIA's fifth-generation training-and-inference GPU. The question is whether the trajectory of custom silicon in inference is fast enough to permanently compress NVIDIA's margins on the workloads that account for the majority of their revenue going forward.

If you're an NVIDIA investor, the Q2 print tonight is going to feel great. The question is whether you'll feel the same way in 2027, when half the inference workloads in the world are running on chips made by the people who use them.

What This Means For Builders

If you're building on top of AI APIs, not much changes tomorrow. Your tokens cost what they cost. But watch the trajectory:

• API pricing should fall faster than people expect. OpenAI, Google, and Anthropic all have economic incentives to drop per-token pricing aggressively in 2026-2027 as their custom silicon comes online. If you're planning AI product economics, assume a 30-50% per-token price drop in the next 18 months.
• Latency budgets get much tighter. When inference is 4x faster, "real-time" applications become trivial. Voice agents that feel snappy. Video generation that's interactive. Search and summarization that doesn't make you wait. The product possibilities open up.
• Agent economics get more interesting. A 2x cost reduction in inference is the difference between "AI agent that does one task at a time" and "AI agent that runs continuously in the background." The labor-substitution thesis gets a lot more real when the per-action cost halves.

The Quiet Revolution

Jalapeño isn't a one-off. It's the most visible data point in a trend that's been building for two years: the AI compute stack is unbundling. NVIDIA still trains the models. Broadcom designs the custom chips. TSMC fabricates them. Celestica integrates the systems. OpenAI, Google, Amazon, Microsoft, and Meta each own their inference path.

For a decade, the assumption was that one company — NVIDIA — would capture the majority of the value in AI infrastructure. The numbers today suggest that's no longer the safe assumption. The inference layer is fragmenting, the way the cloud fragmented in 2010, the way mobile chips fragmented in 2015.

NVIDIA will be fine for a while. Their training moat is real, their software stack is unmatched, and the demand for compute is so absurdly high that even a smaller share of a much bigger pie is a great business.

But "a smaller share" is what today's news is really about. And NVIDIA's earnings call tonight — for all the record revenue and the 96% growth — is going to be the first time management has to answer questions about a customer that just built something better than what they're selling.

That's the story. Not the chip. The fact that the story is now possible.

---

🤖 This post was automatically syndicated from The Sol AI Blog — daily AI analysis from a UK/EU/US perspective.
Follow along for more →