The New Front in the AI Infrastructure War
For most people using artificial intelligence, the experience still feels weightless. A question is typed into a chat window, an image prompt becomes a polished illustration, a coding agent silently rewrites a module, and a voice assistant responds as if intelligence had become just another utility. But behind every apparently effortless answer is a vast industrial machine: racks of accelerators, memory stacks, liquid cooling loops, networking fabrics, power contracts, supply agreements, and a global scramble for the hardware that can turn trained models into real-time services. That machine has been built, more than any other company’s machine, around Nvidia GPUs. From the early deep learning boom to the current age of large language models, Nvidia’s hardware and software ecosystem became the default platform for training frontier AI models and serving them to billions of users. Now OpenAI, the company whose products helped make AI infrastructure one of the defining economic stories of the decade, is moving deeper into that machine with its own silicon.
The chip is called Jalapeño, and the name is playful enough to sound like a lab joke that escaped into a press release. The numbers behind it are less playful. OpenAI has published more detailed inference benchmark results for the custom AI accelerator, developed with Broadcom, claiming that Jalapeño delivers between 1.5 and 1.9 times more AI work per watt and 1.7 to 3.6 times lower end-to-end latency than comparison systems based on Nvidia’s GB200 and GB300 platforms, depending on the model and workload. OpenAI says the measurements were made on InferenceX, a public benchmark from SemiAnalysis that evaluates the full process of serving an AI request rather than only reporting isolated chip-level peak figures. The company also says Jalapeño is rated at 700 watts, while the tested Nvidia comparison systems used published package power ratings of 1,200 watts for GB200 and 1,400 watts for GB300 in the benchmark appendix.
The practical significance is not that Nvidia is suddenly finished. It is not even that GPUs are about to disappear from AI data centers. OpenAI itself says it will continue to widely deploy Nvidia and other partner accelerators for both training and inference workloads, and Axios reported that OpenAI’s VP of hardware, Richard Ho, described Jalapeño as one part of a broader compute mix that also includes Nvidia, AMD, and other systems. The real meaning is subtler and potentially more important: the largest AI service providers are no longer satisfied with renting or buying generic acceleration capacity at whatever price the market demands. They are trying to bend the economics of inference—the cost of answering each query, running each agent step, generating each token—by building ASICs that match their own workloads with brutal specificity.
That shift matters because inference is becoming the long-term financial center of gravity in AI. Training a frontier model is spectacularly expensive, but it is episodic. Inference never stops. Every user prompt, enterprise workflow, code completion, agentic browser action, voice interaction, and multimodal request has to be served again and again, across the day and across the globe. As AI systems become more useful, they also become more interactive and more persistent. A simple chatbot response may require one model call; an AI agent planning travel, reading documents, writing code, checking tools, and revising its own work may require dozens or hundreds of sequential inference steps. In that world, latency is not merely a comfort metric. It becomes part of product quality, user trust, and cost structure. A chip that can return useful tokens faster while consuming less power per unit of AI work does not just make a benchmark chart look good. It changes how many users can be served from a data center, how expensive premium AI plans need to be, and how much room a company has to build more capable models without letting serving costs overwhelm revenue.
Why OpenAI Wants Its Own Silicon
The history of AI hardware is partly a history of accidental fit. GPUs were not originally designed for neural networks. They were built to render graphics, pushing enormous numbers of similar mathematical operations through parallel pipelines. That made them oddly perfect for the matrix multiplications that powered deep learning. Nvidia’s great achievement was not simply having chips that could do parallel arithmetic. It was building CUDA, libraries, developer tools, networking, systems, and a software ecosystem that allowed researchers to translate mathematical ambition into running code. By the time transformers and large language models began to dominate AI research, Nvidia’s GPUs were already the default medium in which modern AI was expressed.
But a GPU is a general-purpose accelerator, and generality always carries a price. It can train vision models, simulate physics, render graphics, run recommendation systems, execute scientific workloads, and serve language models. That flexibility is enormously valuable, especially in research environments where model architectures change quickly and no one knows what next year’s dominant workload will look like. Yet once a company reaches the scale of OpenAI, Google, Amazon, Microsoft, or Meta, some workloads become predictable enough to justify specialization. If a large fraction of total compute demand comes from serving transformer-like language models to users, the temptation to build a narrower, more efficient inference engine becomes hard to resist.
Jalapeño appears to be a product of that realization. OpenAI described it as its first custom inference chip and said it was designed specifically for large language model serving, not as a universal GPU replacement. In its original announcement with Broadcom, OpenAI presented the chip as part of a multi-generation platform and emphasized that OpenAI models helped accelerate the design process, including the path to tape-out. In the later benchmark publication, OpenAI expanded that argument, saying Jalapeño’s gains come from co-designing the chip, memory, network, software, and rack-scale system around real language-model workloads. That phrase—co-designing the full stack—has become almost obligatory in AI infrastructure, but in this case it points to a real architectural divide between general acceleration and workload-specific silicon.
The most important part of OpenAI’s claim is not just peak throughput. It is the combination of throughput, energy efficiency, and end-to-end latency. AI hardware companies often highlight the largest number available: floating-point operations per second, tokens per second, memory bandwidth, or accelerator-to-accelerator interconnect speed. Those numbers matter, but inference services live in a more complicated world. A model server has to accept requests of different lengths, process prompts, generate tokens, manage caches, schedule batches, communicate across chips, and meet latency expectations that users actually feel. A system that looks powerful at full batch utilization can feel sluggish when serving interactive users. Conversely, a system tuned for ultra-low latency may waste capacity if it cannot keep enough work flowing through the hardware. OpenAI’s Jalapeño results are framed as an attempt to win both sides of that trade-off.
That is why the benchmark choice matters. OpenAI says it used InferenceX to measure the full serving process at matched user experience, comparing systems across operating points from high-throughput serving to more interactive, low-latency use. The company reported that Jalapeño performed strongly on GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T, and said that across those three models it delivered 1.5 to 1.9 times more AI work per watt at peak throughput and 1.7 to 3.6 times lower end-to-end latency than the comparison systems. These are company-published results, not a neutral declaration that the chip has permanently surpassed Nvidia in every relevant dimension. Still, if the results hold under broader deployment, they suggest that OpenAI has identified an inference niche where a custom ASIC can do exactly what ASICs are supposed to do: sacrifice flexibility in exchange for efficiency.
The incentive is enormous. The marginal cost of inference is shaped by hardware acquisition, power draw, cooling, memory, data center space, utilization, networking, software overhead, and operational reliability. Power efficiency is not only about electricity bills, although those are increasingly material. It is also about the number of requests that can be served within a fixed power envelope. A hyperscale data center may be constrained less by floor space than by available megawatts. In that environment, more AI work per watt can become the difference between launching a new model widely and throttling demand behind waitlists, usage caps, or premium pricing. For a company like OpenAI, which must serve consumer products, enterprise APIs, coding tools, voice systems, image and video models, and agentic workflows, the economics of inference can shape product strategy as much as model research does.
Inside the Inference Problem
To understand why Jalapeño’s benchmark claims have attracted attention, it helps to look at what language-model inference actually does. When a user submits a prompt, the system first performs a phase often called prefill, in which the model processes the input context. This phase is compute-intensive because the model must transform the prompt into internal representations across many layers. Once the prompt has been processed, the model enters decode, generating the response token by token. Decode behaves differently. Each new token depends on previous tokens, and the system repeatedly reads model weights and uses a growing key-value cache, often called the KV cache, to avoid recomputing attention over the entire sequence. OpenAI’s benchmark article explicitly describes this split: prefill is more compute-intensive, while decode is more constrained by memory bandwidth, and communication delays can leave processors idle while data moves across chips.
This is where the old mental picture of AI hardware as merely “more math” begins to fail. Modern inference is as much about data movement as arithmetic. The energy cost and time cost of moving data between memory, cores, accelerators, and networked systems can dominate the actual multiplication work. A trillion-parameter model cannot simply be dropped onto one chip like an app on a phone. It must be partitioned, scheduled, cached, streamed, and synchronized. Each decision about where weights live, where attention state is stored, how requests are batched, and how tokens are routed has a cost. A chip designed for this world has to care about locality, memory hierarchy, interconnect topology, compiler behavior, kernel efficiency, and failure recovery, not just raw tensor performance.
OpenAI says Jalapeño was designed to minimize data movement and communication delays by allowing model state, including the KV cache, to be explicitly placed and kept local while the system activates the right combination of compute, memory, and networking for each inference phase. It also says the network is integral to the architecture and that the system’s large connected domain helps keep workloads inside one connected system, reducing movement and latency. In plain English, the chip is trying to make the geography of inference more favorable. Instead of constantly hauling intermediate state across an expensive digital city, it tries to put the most important data closer to the engines that need it, then coordinate those engines with less waiting.
That architectural emphasis is especially relevant to agents. A conversational chatbot can tolerate a little delay; a coding or research agent that makes many calls in sequence suffers from compounded latency. If each model call takes several seconds, the total workflow becomes slow enough to break the illusion of assistance. If each step becomes faster, the agent can explore more possibilities, check more sources, revise more code, or recover from mistakes before the user loses patience. OpenAI’s benchmark article says interactive agents need many sequential steps and that delays compound across an entire task, which is why it evaluates performance at a matched user experience rather than only reporting isolated throughput. That is a crucial point: the future of inference is not only more users asking one question, but more software systems making chains of model calls on behalf of users.
The OpenAI appendix gives specific examples of how the company wants readers to interpret the chip. On GPT-OSS 120B, it reports about 1.9 times higher peak mixed tokens per second per kilowatt than GB200, with end-to-end latency of 1.03 seconds versus 1.80 seconds. On DeepSeek R1 670B, compared with GB300, it reports about 1.7 times higher peak mixed tokens per second per kilowatt and 1.65 seconds of end-to-end latency versus 5.99 seconds. On Kimi K2.5 1T, also compared with GB300, it reports about 1.5 times higher peak mixed tokens per second per kilowatt and 1.56 seconds versus 5.31 seconds of end-to-end latency. These details matter because they suggest the advantage is not uniform but workload-dependent. The chip’s strongest story is not a single universal victory number. It is that across several large models, OpenAI claims Jalapeño sits on a better efficiency-latency frontier for the operating points it cares about.
The caveat is equally important. Benchmarks are arguments, not reality itself. They depend on model choice, precision format, sequence lengths, batching policy, software maturity, power accounting, comparison system configuration, and whether both sides have been equally optimized. OpenAI is transparent that Jalapeño’s results are measured on selected public models and that production qualification, software maturation, scale validation, and broader model testing are still underway. Nvidia’s systems, meanwhile, are not static targets. The company has spent years improving both hardware and software for inference, and its newest platforms are designed at rack scale, not merely chip scale. A fair reading is that Jalapeño has shown enough promise to be taken seriously, not that the infrastructure war is over.
The Nvidia Benchmark Shadow
Nvidia’s position in AI is often described as a monopoly, but that understates the depth of the moat. It is not only a chip company in this market. It is a platform company, a systems company, a networking company, and a software company whose GPUs became the lingua franca of machine learning. Blackwell and its Grace Blackwell systems extended that strategy from individual accelerators to rack-scale AI factories. Nvidia describes GB200 NVL72 as a system that connects 36 Grace CPUs and 72 Blackwell GPUs in a rack-scale design with a 72-GPU NVLink domain, essentially allowing a large cluster of accelerators to behave like a tightly connected machine for massive AI models. Its GB300 NVL72 platform is similarly positioned for AI reasoning and inference, with Nvidia emphasizing fully liquid-cooled rack-scale architecture and high network connectivity per GPU.
That system-level approach is precisely why Jalapeño cannot be judged like a small startup accelerator trying to beat one GPU on a synthetic test. OpenAI is challenging Nvidia on Nvidia’s own newer terrain: the rack, the network, the software stack, and the economics of useful output per unit of power. Nvidia has spent years convincing the market that the relevant product is no longer a discrete chip but a complete AI factory. OpenAI’s Jalapeño strategy accepts that premise and asks whether the factory can be more efficient if its machinery is designed by the owner of the workload rather than the supplier of a general platform.
The answer may differ by workload. Nvidia’s GPUs remain extraordinarily valuable because they can do many things well. Training, fine-tuning, reinforcement learning, simulation, multimodal processing, batch inference, experimental architectures, custom kernels, and rapidly changing research all reward flexibility. When OpenAI says Jalapeño is an inference chip and not a training chip, it is acknowledging a boundary. Axios reported that OpenAI is still highly dependent on Nvidia and others for training new models, and OpenAI’s own benchmark article says the company will continue deploying Nvidia and other accelerators widely. This is not a clean substitution story. It is a segmentation story.
Segmentation, however, can still be economically painful for Nvidia if it grows large enough. The most profitable parts of a platform business are often protected by ubiquity. When every AI company needs the same scarce accelerator, the supplier has pricing power. But if the largest customers carve away their highest-volume inference workloads and run them on internal ASICs, they do not need to abandon Nvidia to reduce Nvidia’s leverage. They only need credible alternatives. Google’s TPU program already proved that a hyperscaler could build a long-running custom AI accelerator family and use it internally and through cloud services. Google Cloud says TPUs are custom accelerators designed for AI workloads across training, reasoning, and inference, and Google has described its TPU history as a decade-long effort to co-design chips, software, and systems for its own AI needs. Amazon has Inferentia for inference, AWS says it is designed to deliver high performance at low cost for deep learning and generative AI inference, and Microsoft has introduced Maia accelerators for Azure workloads. OpenAI’s entry makes the pattern harder to dismiss.
The most dangerous thing for Nvidia is not a single OpenAI chip beating a single Nvidia configuration in a benchmark. It is the possibility that hyperscale AI demand fragments into many internal silicon islands. Google runs more on TPUs. Amazon steers inference to Inferentia. Microsoft places Copilot or Azure inference on Maia. OpenAI serves more ChatGPT and API traffic on Jalapeño. Anthropic explores in-house chip work. Meta continues building infrastructure optimized for its own models. Nvidia would still sell enormous volumes of accelerators, especially for training and for customers without the scale or expertise to design chips. But the market would become less monolithic. Nvidia would remain central, yet less unavoidable.
There is another reason Jalapeño matters symbolically. OpenAI is not a cloud provider in the traditional sense. It does not own the same kind of vertically integrated infrastructure footprint as Google, Amazon, or Microsoft, and its compute supply has historically depended heavily on strategic partners. By building a custom inference ASIC with Broadcom, OpenAI is behaving more like a hyperscaler because its demand profile has become hyperscale-like. The company has enough predictable inference volume to justify a custom chip program, enough software control to optimize around its own models, and enough economic pressure to care deeply about tokens per watt. In that sense, Jalapeño is not only a piece of hardware. It is a sign that AI model companies are becoming infrastructure companies whether they wanted to or not.
Broadcom, ASICs, and the Return of Specialization
The word ASIC can sound dry, but it carries a simple industrial logic. An application-specific integrated circuit is a chip built for a particular job. It does not try to be universal. It tries to be efficient.
Top comments (0)