OpenAI just published benchmark results for Jalapeño — its first custom AI inference chip — and the numbers are credible and significant. Compared to leading NVIDIA hardware (GB200 and GB300), Jalapeño delivers better throughput, better latency, and better power efficiency simultaneously. Not a tradeoff between them. All three at once.
"Jalapeño delivers both higher throughput and lower latency with one architecture, where existing hardware systems often have to make a tradeoff between the two."
That's the claim, and the benchmarks back it up.
What the numbers actually say
Testing ran against the InferenceX public benchmark from SemiAnalysis across three models — GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. None of these are OpenAI-exclusive:
- 1.5–1.9x more throughput per watt at peak load across all three models
- 1.7–3.6x lower end-to-end latency depending on model and operating point
- 2.1–4.1x higher performance for interactive/low-latency workloads
- On DeepSeek R1, latency dropped from 5.99s → 1.65s end-to-end. That's not a rounding error.
The chip is rated at 700W but ran at or below 550W on tested workloads. Compare that to the GB300 at 1,400W. Half the power, faster output.
The full-stack bet
Jalapeño isn't just a chip. It's the first output of OpenAI's strategy to co-design models, chips, memory, networking, and software together — rather than rent GPUs and adapt.
The architecture optimises specifically for language model inference phases: prefill (compute-heavy) and decode (memory-bandwidth-heavy) handled on the same balanced chip, minimising data movement between phases. For agents — where many inference calls chain together — this compounds. Every 3x latency improvement on a single call becomes more significant when you stack 10 calls in sequence.
AI designed the chip, then AI programmed it
Nine months from initial design to tapeout. OpenAI used earlier AI models to help design and verify circuits, then used Codex with GPT-Astra to bring three open-weight models to production performance in two months.
The headline stat: for selected attention and MoE blocks, AI-generated kernels ran 1.5–1.8x faster than human-expert implementations. That loop — design hardware so AI can program it, use AI to program it better — is the part of this announcement worth watching long-term.
What to do
- Using the OpenAI API? Nothing changes yet. Jalapeño deploys within OpenAI's infrastructure by end of 2026. Lower latency and potentially lower prices without touching your code.
- Running agents or multi-step workflows? This is where the gains compound. Faster per-call inference adds up quickly across a full agentic loop.
- Building AI infrastructure yourself? This is a proof point that custom silicon for LLM inference is viable and competitive. The NVIDIA-or-nothing assumption deserves a second look.
- Watching the competitive landscape? Google TPU, Amazon Trainium, Meta MTIA — everyone's doing this. Jalapeño is the clearest public benchmark yet that the challengers are landing.
Jalapeño deploys this year. Gen 2 is already in deep development. Gen 3 is taking shape.
Source: OpenAI Jalapeño announcement
✏️ Drafted with KewBot (AI), edited and approved by Drew.
Top comments (0)