OpenAI Jalapeño Chip Sets Benchmark for AI Inference

Why Jalapeño Matters

OpenAI’s decision to design its own inference silicon marks a decisive shift from the traditional “buy‑off‑the‑shelf GPU” model that has dominated AI compute for years. The Jalapeño chip is not just another accelerator; it is a purpose‑built, full‑stack solution that tackles the three core constraints of interactive AI agents:

1. Throughput – the ability to serve many requests per second.
2. Power efficiency – keeping operational costs and carbon footprints manageable.
3. Latency – delivering responses fast enough for real‑time conversation.

By hitting a Pareto frontier across these dimensions, Jalapeño promises to lower the total cost of ownership for large‑scale inference workloads. For enterprises that run multi‑billion‑parameter models around the clock, even a modest improvement in tokens‑per‑second per kilowatt translates into millions of dollars saved annually.

The chip also signals OpenAI’s broader ambition to control the entire AI stack—from model training on NVIDIA hardware to inference on its own silicon. This vertical integration reduces reliance on external suppliers and gives OpenAI the flexibility to iterate hardware and software in lockstep.

Technical Architecture Deep Dive

Full‑Stack Integration

Jalapeño’s architecture is built around a co‑designed ecosystem that includes:

• Custom silicon optimized for both the prefill (heavy matrix multiplications) and decode (KV‑cache lookups) phases of transformer inference.
• On‑board high‑bandwidth memory that stores the KV cache locally, eliminating the need for costly off‑chip memory accesses that dominate decode latency.
• Large‑domain networking that stitches multiple chips into a single, latency‑aware rack‑scale system, keeping data movement within a tightly coupled fabric.

This holistic approach mirrors the design philosophy described in OpenAI’s earlier full‑stack post, where the company emphasized minimizing data shuffling across layers of the stack. Read more here.

Design Cycle Accelerated by AI

The chip’s nine‑month design‑to‑tapeout timeline is noteworthy. OpenAI leveraged generative AI tools—specifically Codex and GPT‑Astra—to automate layout generation, timing closure, and power budgeting. The result is a silicon product that is both predictable for human engineers and optimizable by AI scripts, fulfilling the claim: “We used AI to design the chip, and designed the chip so AI could program it.”

Power Envelope

• TDP: 700 W (maximum).
• Measured sustained power: ≤ 550 W on benchmark workloads, indicating headroom for thermal management and potential over‑clocking in future revisions.

Programming Model

Jalapeño is exposed through a predictable programming interface that abstracts away low‑level hardware quirks. Developers can compile models with existing frameworks (e.g., PyTorch) and rely on OpenAI’s optimizer to map operations onto the chip’s pref‑fill and decode pipelines efficiently. This reduces the engineering effort required to port large models, a pain point that has slowed adoption of custom accelerators in the past.

Benchmark Performance Analysis

OpenAI evaluated Jalapeño using Semi Analysis’s Inference X benchmark suite, a public, reproducible test harness that measures mixed throughput, latency, and token‑generation speed. The chip was pitted against two leading NVIDIA‑based reference systems: GB200 (1,200 W) and GB300 (1,400 W).

🔹 -------
• Metric: --------
• Jalapeño vs. GB200/GB300: --------------------------

🔹 *GPT‑OSS 120B*
• Metric: Peak mixed TPS/kW
• Jalapeño vs. GB200/GB300: 1.9× higher (85,448 vs. 44,960)

🔹 ****
• Metric: End‑to‑end latency
• Jalapeño vs. GB200/GB300: 1.7× lower (1.03 s vs. 1.80 s)

🔹 ****
• Metric: Minimum TBT
• Jalapeño vs. GB200/GB300: 2.7× lower (0.69 ms vs. 1.87 ms)

🔹 ****
• Metric: Throughput at prior TBT
• Jalapeño vs. GB200/GB300: 53.7× higher (22,935 vs. 427)

🔹 *Deep Seek R1 670B*
• Metric: Peak mixed TPS/kW
• Jalapeño vs. GB200/GB300: 1.7× higher (19,641 vs. 11,781)

🔹 ****
• Metric: End‑to‑end latency
• Jalapeño vs. GB200/GB300: 3.6× lower (1.65 s vs. 5.99 s)

🔹 ****
• Metric: Minimum TBT
• Jalapeño vs. GB200/GB300: 4.1× lower (1.43 ms vs. 5.90 ms)

| | Throughput at prior TBT | 104.3× higher (12

🔹 ****
• Peak mixed TPS/kW: End‑to‑end latency
• 1.5× higher (18,195 vs. 11,862): 3.4× lower (1.56 s vs. 5.31 s)

🔹 ****
• Peak mixed TPS/kW: Minimum TBT
• 1.5× higher (18,195 vs. 11,862): 3.8× lower (1.44 ms vs. 5.48 ms)

🔹 ****
• Peak mixed TPS/kW: Throughput at prior TBT
• 1.5× higher (18,195 vs. 11,862): 56.1× higher (6,744 vs. 120 mixed / kW)

Note: “Mixed TPS/kW” combines both pre‑fill and decode phases, providing a single efficiency metric that reflects real‑world serving workloads.

Implications for the AI Ecosystem

1. Cost‑per‑token drops dramatically

The order‑of‑magnitude improvements in throughput per kilowatt translate directly into lower operational expenses. For a typical SaaS AI provider that processes billions of tokens per day, the savings can reach hundreds of millions of dollars annually when migrating from a GB300‑class GPU cluster to a Jalapeño‑based rack.

2. Latency‑critical applications become viable at scale

Interactive agents—search assistants, real‑time translation, and gaming NPCs—have long been constrained by the 1‑2 ms token‑generation floor of GPU‑based decoders. Jalapeño’s sub‑1 ms TBT pushes the envelope into the realm of human‑perceptible real‑time response, opening new product categories that were previously cost‑prohibitive.

3. Supply‑chain independence

By owning the inference silicon, OpenAI reduces its exposure to the volatile GPU market, where demand spikes (e.g., from large‑language‑model training) can cause price spikes and lead times of months. This strategic autonomy also gives OpenAI leverage in negotiating with NVIDIA for training‑accelerator supply while keeping inference in‑house.

4. Catalyst for a new wave of custom AI silicon

Jalapeño’s success validates the “AI‑designed, AI‑programmed” paradigm. Competitors—both cloud providers and semiconductor startups—are likely to accelerate their own custom‑silicon programs, focusing on the same full‑stack co‑design principles that yielded Jalapeño’s Pareto‑optimal results.

Read the full breakdown originally published at https://ltdeveloperblogs.github.io/posts/jalapenos-first-results-show-industry-leading-speed-and-efficiency-in-ai-inference/