DEV Community: Amit

Most Enterprises Are Chasing AI Autonomy Before Defining the Outcome

Amit — Thu, 24 Sep 2026 18:00:35 +0000

TL;DR

Enterprises are adopting agents faster than they are redesigning the work those agents will perform.
The unit of autonomy is the delegation envelope: one actor, one goal, one allowed action set, one evidence contract, and one recovery boundary.
Capability, delegated authority, and operating maturity are separate scorecards. A system that can perform an action has not automatically earned permission to take it.
The goal is the minimum authority needed to improve a defined outcome, expanded only when the workflow can prove and recover from its work.

IBM found that 64% of 2,000 surveyed CEOs said fear of falling behind drives technology investment before value is clear. Another 61% were adopting AI agents and preparing to scale them, while only 25% of AI initiatives had delivered the expected return. Enterprises are chasing autonomy faster than outcomes.

That is why How autonomous should our AI be? is the wrong opening question. An enterprise does not have one autonomy level. A software-maintenance workflow can complete bounded work while a payment workflow requires approval for every consequential action. Customer service can resolve a routine request while legal review remains advisory.

The useful question is:

What outcome are we trying to achieve, how does the work produce it today, and what authority has each part of that workflow earned?

Start With the Outcome, Not the Agent

“Deploy an agent” is not a business outcome.

Reduce account-opening time. Resolve routine service requests without repeat contact. Cut invoice exceptions. Detect fraud before money moves. These are measurable outcomes. An agent is one possible actor alongside deterministic software, analytical models, policies, systems of record, and human judgment.

Task productivity is not workflow performance. An agent may draft an answer in seconds while the case waits two days for approval, or generate more code while review becomes the bottleneck.

McKinsey’s 2025 global survey found that 62% of respondents’ organizations were experimenting with agents, but only 23% were scaling an agentic system somewhere in the enterprise. In any individual function, no more than 10% reported scaling agents. Adoption, workflow penetration, and business value are different scorecards.

The outcome defines what “done” means, which decisions matter, what the agent may change, which errors are tolerable, when a person intervenes, and how success is measured. Without that sequence, autonomy becomes a technology objective detached from the work.

The Delegation Envelope Is the Real Unit

Authority lives in a delegation envelope: one actor, one goal, one allowed action set, one evidence contract, and one recovery boundary. Change any part and you have a different envelope. This is the unit that can be designed, audited, graduated, and revoked.

A workflow contains multiple envelopes at different levels. The same agent may hold advisory authority for a legal conclusion, bounded authority for assembling its evidence, and no authority to send the result externally. Calling the whole system “L3” hides the consequential boundaries.

Three scorecards must stay separate:

Capability: What can the system accomplish, and with what reliability?
Delegated authority: What effects may it cause without further approval?
Operating maturity: Can the organization observe, constrain, audit, interrupt, and recover that delegation?

“The model can do it” answers only the first question. The levels below describe envelope scope, not corporate progress. There is no universal L1–L5 standard for enterprise agents; this is my design model.

Level	What the envelope permits	Human responsibility
L1 — Advise	Analyze, draft, predict, or recommend. No material external action.	Performs the work and makes the decision
L2 — Per-action approval	One discrete action whose exact occurrence and consequential parameters receive explicit approval	Directs the action and reviews consequential parameters
L3 — Pre-authorized bounded workflow	A multi-step workflow whose class, boundaries, and evidence contract were approved in advance; individual runs are not separately approved	Approves the workflow class and its exceptions—not each step
L4 — Coordinating envelope	Sequences and selects among several already-bounded L2/L3 envelopes toward a defined outcome. Composed authority, not new authority.	Sets constraints, owns material deviations and cross-boundary exceptions
L5 — Continuing objective	Creates and revises its own plans toward a standing objective under policy, budget, and time limits	Sets strategy, prohibited actions, and revocation authority

The L2/L3 distinction is when approval happens: each occurrence at L2, the workflow class at L3. L4 composes bounded envelopes; if the coordinator gains authority none of them had, the boundary has leaked. L5 is a horizon, not a target. The public sources reviewed for this article did not document a standing-objective envelope with self-revised plans under material consequence.

Enterprise Workflows Are Landing at Different Levels

The ranges below are my assessment of public evidence available as of September 2026—not a market standard or benchmark. I use ranges because the evidence does not support point estimates, and I label vendor claims, regulatory boundaries, and thin evidence.

Enterprise workflow	Typical range	Basis, and its limits
Software engineering	L2–L3	Vendor product claim: GitHub documents that its coding agent implements bounded changes and opens pull requests; humans retain merge authority. Documented capability, not measured adoption depth.
IT and employee service	L2–L3	Vendor product claim: ServiceNow's platform announcement describes routine service workflows completing inside existing approvals. Marketing material; no independent outcome data.
Customer service	L2–L3	Routine resolution is the most commonly reported bounded-completion case in the surveys cited above; exceptions stay human-owned. Vendor deflection figures are not comparable across firms; I do not rely on them.
Sales	L1–L2	Weak evidence. No function-specific study cited. Inference from the ≤10% per-function scaling ceiling in McKinsey's survey plus the structural point that pricing and commitments are contractually binding. Hypothesis.
Marketing	L1–L2	Observed adoption: BCG reports agent-led workflows remain a minority; campaigns operate inside approved briefs and budgets.
HR	L1–L2	Observed adoption: employee service and onboarding advance. Regulatory boundary: employment decisions face EU AI Act high-risk obligations—a legal ceiling, not a measurement of practice.
Finance and accounting	L1–L2	Observed adoption: reconciliation and close preparation advance; assurance readiness and material approvals remain limiting (KPMG).
Procurement	L1–L2	Observed adoption: supplier analysis and low-value transactions advance; trust in autonomous decisions is the leading reported barrier (BCG).
Supply chain and logistics	L1–L2	Observed adoption: agents generate and execute narrow plans; cross-network trade-offs typically remain planner-approved (BCG).
Risk, legal, and compliance	L1	Observed adoption: research, review, monitoring, drafting, and evidence collection dominate (Thomson Reuters). Also subject to professional-responsibility limits.
Physical operations	L2–L3 in engineered environments only	Weak evidence. No study cited here covers robotics. Inference from the engineered-environment argument below: bounded robots operate where state and recovery are physically constrained. General-purpose physical agents are earlier. Treat as hypothesis.

Three patterns matter. Engineering leads where evidence and rollback are executable. Regulated functions move more slowly because their actions carry legal or financial consequence. Maturity varies within every function because workflows expose different boundaries.

Why Software Engineering Moved First

Software engineering already had the control system for bounded delegation: machine-readable state, tests, sandboxes, granular permissions, reviewable diffs, independent approval, and rollback.

GitHub's coding-agent documentation—a vendor product description—maps cleanly onto the envelope. The agent evaluates an issue, changes code, runs tests, and opens a pull request. The pull request is the evidence contract; merge authority is the boundary. The agent completes a pre-authorized workflow without inheriting authority over the software outcome.

Other functions need equivalent controls: simulation instead of a branch, deterministic business rules instead of tests, action diffs instead of code diffs, task-scoped credentials instead of deployment access, and compensating transactions instead of rollback.

The ICLR 2026 Measuring Agents in Production study found that across 306 practitioners and 20 case studies, 68% of production agents executed at most ten steps before human intervention, 74% relied primarily on human evaluation, and reliability remained the leading challenge. Production maturity currently comes from bounded authority, not unrestricted action.

A Worked Example: Invoice Exceptions

The outcome is not “process invoices with an agent.” It is “resolve valid exceptions within one business day without an unauthorized posting or payment.”

An L1 envelope classifies the exception and assembles evidence. L2 requests missing documentation under a pre-approved rule. L3 resolves named exception classes below a financial threshold, while payment release remains separately gated. Each step records the source data, rule, and proposed change. Recovery means reopening the exception, reversing a provisional posting, and revoking authority when error thresholds are exceeded.

The envelope earns wider scope only when its goal, action set, evidence contract, authority history, and recovery boundary strengthen together. Widening authority without strengthening those controls is weak governance wearing a level number.

What Executives Are Actually Deciding

Deloitte’s 2026 State of AI research found that 74% of surveyed leaders expected moderate or extensive agent use within two years. Yet only 21% reported mature governance for autonomous agents, 30% were redesigning key processes around AI, and 84% had not redesigned jobs around it.

The gap is concrete: organizations want class-level autonomy before they have evidence contracts that make class-level approval safe. L4 compounds the problem across decision rights, data definitions, permissions, budgets, and accountability. The composition itself needs an owner.

McKinsey found AI high performers were 2.8 times more likely to report fundamental workflow redesign than other respondents—55% versus 20%. That is an association, not proof of causation, but it is a stronger maturity signal than agent count.

The operating sequence is simple: observe the workflow, let the system advise, approve individual actions, pre-authorize proven workflow classes, then coordinate bounded workflows. At each step, collect outcome and exception evidence, monitor drift, and demote or revoke authority when the evidence deteriorates.

The NIST AI Risk Management Framework asks organizations to map an AI system’s intended purpose, context, affected actors, objectives, and legal requirements. NIST’s February 2026 draft on agent identity asks how standards can support identification, authorization, auditing, and non-repudiation. My architectural implication is that agents should receive scoped, auditable delegation rather than silently inherit a person’s full access.

What’s Missing

The market has plenty of agent frameworks. It has fewer mechanisms for operating authority.

Business workflows still lack standard equivalents for tests, action diffs, merge protection, and rollback. Evaluations score answers while production risk lives in tool use, permissions, side effects, and exceptions. Weak human review becomes an approval button without enough context or power to refuse.

The missing layer is not another autonomy label. It is an authority lifecycle that defines the envelope, gathers evidence in simulation and production, graduates scope within explicit thresholds, monitors drift, and revokes authority when performance deteriorates.

So What

Ask which envelope you are designing: whose goal, which actions, what evidence, what recovery, and whether approval happens per action or per workflow class. Some envelopes should remain advisory permanently. Maturity is knowing which can move—and having a mechanism for moving them back.

Here is a bet that can be checked. By September 2028, enterprises publicly reporting durable financial outcomes from agents will be distinguished by workflow-level evidence infrastructure—action diffs, scoped credentials, compensating transactions, and named envelope owners—rather than by the breadth or autonomy level of their deployments. I also expect no credible public case of a sustained L5 envelope operating over material consequence by that date.

I am wrong if enterprises scale agent authority broadly and profitably without those controls, or if a standing-objective agent runs a consequential business objective with self-revised plans and holds up under audit.

The question I have not resolved is how much evidence a workflow should accumulate before it moves up one rung—and who should set that threshold when the benefits and the cost of failure fall on different functions.

Wiring Codex and ChatGPT Desktop to OpenAI Models on Amazon Bedrock Runtime

Amit — Wed, 23 Sep 2026 04:54:39 +0000

Codex CLI and the ChatGPT desktop app can use the same native Amazon Bedrock Runtime configuration. The working pattern has three parts: the built-in Runtime provider, an AWS profile and Region, and a global model profile ID.

I verified this pattern on September 11, 2026 with four OpenAI model profiles: GPT-5.6 Sol, Terra, Luna, and GPT-6 Astra. All four completed headless runs through the native Runtime provider in the desktop app's bundled Codex 0.153.4. Astra also passed with Codex CLI 0.154.0 using the same saved configuration.

TL;DR

Codex CLI and the ChatGPT desktop app read the same local Codex configuration, so one provider definition can serve both surfaces.
The endpoint remains regional, while a global.openai.* model ID selects global cross-Region inference.
The built-in amazon-bedrock-runtime provider uses an AWS profile directly; no static API key, token generator, or local proxy is required.
amazon-bedrock and amazon-bedrock-runtime are different providers. The first selects Mantle in the tested clients; the second selects Bedrock Runtime.

The working architecture

OpenAI documents shared local configuration across Codex surfaces. Amazon Bedrock documents the Runtime endpoint as the recommended endpoint for new applications, including OpenAI-compatible Responses and Chat Completions paths under /openai/v1.

The resulting request path is direct:

Codex CLI or ChatGPT desktop
        |
        | reads ~/.codex/config.toml
        v
Native amazon-bedrock-runtime provider
        |
        | uses AWS profile and Region
        v
bedrock-runtime.<region>.amazonaws.com/openai/v1
        |
        v
global.openai.<model>

There is no gateway in this path. AWS credentials stay in the normal local profile chain, and Codex sends Responses API traffic directly to Bedrock Runtime.

The word global applies to the model profile, not the hostname. The network endpoint still contains a Region such as us-west-2; the global.openai.* model ID tells Bedrock to use its global cross-Region inference profile. The Bedrock Responses API documentation shows the Runtime base URL and global profile naming pattern.

Prerequisites

The setup needs:

A current Codex CLI installation
The ChatGPT desktop app with Codex support
An AWS profile with permission to invoke the selected Bedrock models

Confirm the AWS session before changing Codex:

aws sts get-caller-identity --profile YOUR_AWS_PROFILE

The shared Codex configuration

Add the following provider to ~/.codex/config.toml. Replace YOUR_AWS_PROFILE and select a Bedrock Region where the models are available.

model = "global.openai.gpt-6-astra"
model_provider = "amazon-bedrock-runtime"
model_reasoning_effort = "medium"

[model_providers.amazon-bedrock-runtime.aws]
region = "us-west-2"
profile = "YOUR_AWS_PROFILE"

The Codex configuration sample documents AWS profile and Region settings for built-in Amazon Bedrock providers. This configuration references the selected profile rather than embedding access keys. The profile can still use AWS IAM Identity Center, role assumption, environment credentials, or another supported AWS credential source.

Global OpenAI model profiles

These are the four global profile IDs I verified:

Model	Codex model ID	Verified result
GPT-5.6 Sol	`global.openai.gpt-5.6-sol`	Native Runtime passed
GPT-5.6 Terra	`global.openai.gpt-5.6-terra`	Native Runtime passed
GPT-5.6 Luna	`global.openai.gpt-5.6-luna`	Native Runtime passed
GPT-6 Astra	`global.openai.gpt-6-astra`	Native Runtime passed

AWS introduced Sol, Terra, and Luna on Amazon Bedrock in July 2026 and announced GPT-6 Astra availability in September 2026. Model access still depends on the account, selected Region, and current Bedrock catalog, so discovery in one account is evidence for that environment rather than a universal availability guarantee.

Set Astra as the default in the configuration, or select a model for an individual Codex run:

codex -m global.openai.gpt-5.6-sol
codex -m global.openai.gpt-5.6-terra
codex -m global.openai.gpt-5.6-luna
codex -m global.openai.gpt-6-astra

Verify the Runtime path

A headless run provides a clean test because it removes the interactive interface from the request path:

codex exec \
  --ephemeral \
  --skip-git-repo-check \
  -m global.openai.gpt-5.6-sol \
  "Reply with exactly SOL_RUNTIME_OK"

Repeat the probe with the other model IDs and a distinct expected string. My four probes returned:

SOL_RUNTIME_OK
TERRA_RUNTIME_OK
LUNA_RUNTIME_OK
ASTRA_RUNTIME_OK

This verifies more than model discovery. It confirms that Codex can load the provider, resolve the named AWS profile, reach the Runtime endpoint, select the global profile, and receive a model response.

For the desktop app, fully quit and reopen the application after changing ~/.codex/config.toml. The model picker should then expose the same provider catalog. Existing conversations may retain their original model selection, so use a new task when validating a newly selected model.

The provider name selects the endpoint

The two built-in Bedrock provider names do not select the same endpoint.

Provider	Endpoint selected in the tested clients
`amazon-bedrock`	Bedrock Mantle
`amazon-bedrock-runtime`	Bedrock Runtime

This distinction is visible in an actual call. With Codex 0.153.4, amazon-bedrock sent Astra to a bedrock-mantle URL and received a model-not-found response. The same binary, model ID, AWS profile, and Region succeeded after changing only the provider to amazon-bedrock-runtime.

The native provider passed across both tested Codex versions:

Desktop-bundled Codex 0.153.4 returned SOL_NATIVE_OK, TERRA_NATIVE_OK, LUNA_NATIVE_OK, and ASTRA_NATIVE_OK.
Standalone Codex CLI 0.154.0 returned CLI_NATIVE_RUNTIME_OK from the same saved provider configuration.

Inspect the versions when diagnosing behavior:

codex --version

The desktop app's bundled version is visible in its diagnostic output. Version differences still matter for model catalogs and client features, but they do not require a custom provider for this Runtime path.

Direct Runtime first, gateway when required

Direct Bedrock Runtime is the lower-complexity path when AWS identity, model access, logging, and billing controls meet the requirement. An AWS reference architecture for Codex with LiteLLM positions a gateway as an additional layer for centralized routing, budgets, or policy controls—not as a requirement for basic Codex connectivity.

That distinction matters. A local proxy or hosted gateway adds another credential boundary, another failure point, and another component to operate. Add one when it supplies a control the direct path does not provide.

What remains open

The native Runtime provider passed direct Codex execution in both binaries. The desktop app also restarted with successful configuration and model-catalog reads. I have not yet tested every desktop-only helper action against all four global model profiles; those flows can select models independently from the main conversation.

The configuration boundary is the part to keep under test: one file, two client surfaces, four global model profiles, and one AWS profile. When any client version changes, rerun one headless probe and one fresh desktop task before treating the wiring as stable.

So what: Bedrock model access is only one layer of the setup. The complete contract is endpoint, API format, authentication, model profile, and client configuration—and all five need to work together.

Alternative AI Hardware Is a Systems Problem

Amit — Wed, 23 Sep 2026 04:54:04 +0000

Alternative AI hardware is not one product category.

Huawei Ascend is becoming a complete compute, interconnect, systems, and software platform. FuriosaAI RNGD is a power-conscious inference accelerator that fits into standard servers. Rebellions REBEL starts with a PCIe NPU and scales into multi-card and rack-level systems.

All three reduce dependence on the dominant GPU stack. They do it at different boundaries, with different software commitments and different evidence. This is an architecture comparison; public evidence is not yet consistent enough for a normalized performance ranking.

TL;DR

Huawei competes at the platform boundary: accelerator, scale-up fabric, clusters, compiler, framework, and cloud infrastructure.
FuriosaAI narrows the problem to efficient transformer inference in a familiar PCIe server form factor.
Rebellions combines a data-center NPU with PCIe, CXL-oriented memory expansion, scale-out Ethernet, and rack-level system designs.
The strategic test is not whether an NPU has competitive peak arithmetic. It is whether the surrounding system can run supported models reliably, at the required quality, with enough software and operational evidence.

Three architectures, three ownership boundaries

The phrase “alternative accelerator” hides the most important distinction: who owns the system around the silicon?

Platform	Primary boundary	Physical unit	Software commitment	Strategic objective
Huawei Ascend	Full platform	Chip, server, SuperPod, SuperCluster, and cloud	CANN, MindSpore, framework adapters, cluster software	Build an integrated sovereign compute stack
FuriosaAI RNGD	Efficient inference accelerator	PCIe card, server, and partner system	Furiosa SDK, compiler, runtime, and model support	Add efficient inference capacity to standard data centers
Rebellions REBEL	Scalable inference NPU	PCIe card, multi-card system, rack, and cloud service	RBLN SDK, compiler, runtime, serving integration	Scale transformer inference from card to rack

These systems should not be collapsed into one benchmark row. Huawei is attempting to reproduce much of the platform surrounding a GPU. Furiosa and Rebellions can coexist beside GPU pools because their first deployment boundary remains a server accelerator.

flowchart TB
    APP["Models and applications"]

    subgraph H["Huawei platform boundary"]
        HC["Ascend accelerator"]
        HF["UnifiedBus / system fabric"]
        HS["SuperPod and SuperCluster"]
        HSW["CANN, MindSpore, cluster software"]
        HC --> HF --> HS
        HSW --> HS
    end

    subgraph F["Furiosa deployment boundary"]
        FC["RNGD PCIe cards"]
        FS["Standard inference server"]
        FSW["Furiosa SDK and runtime"]
        FC --> FS
        FSW --> FS
    end

    subgraph R["Rebellions deployment boundary"]
        RC["REBEL PCIe NPUs"]
        RS["Multi-card server or rack"]
        RSW["RBLN SDK and serving stack"]
        RC --> RS
        RSW --> RS
    end

    APP --> HSW
    APP --> FSW
    APP --> RSW

The operator accepts the largest platform commitment with Huawei. The smaller vendors ask for a narrower model and serving commitment, but depend more heavily on host CPUs, merchant networking, server partners, and external orchestration.

Huawei is building the whole stack

Huawei’s architecture has moved far beyond a single Ascend card.

The company’s Atlas 900 A3 SuperPoD combines Ascend compute, high-speed interconnect, storage, management, and liquid-cooled cabinets into a scale-up system. Huawei describes a 384-card configuration with 300 TB of HBM and 122.8 TB/s of scale-up bandwidth.

The published SuperCluster architecture expands that boundary again. Multiple SuperPods connect through the company’s UnifiedBus fabric, with separate scale-up and scale-out layers. Huawei announced Atlas 950 and Atlas 960 SuperPods based on future Ascend generations and described SuperClusters containing hundreds of thousands to more than one million cards.

Those figures are roadmap and vendor architecture claims. They establish direction and intended scale, not measured production efficiency.

The software boundary is equally broad. Huawei’s CANN platform provides compilers, libraries, runtime components, development tools, and framework adapters for Ascend hardware. MindSpore supplies a Huawei-backed framework path, while PyTorch and other frameworks rely on integration layers.

That creates an integrated stack:

Model and framework
  → CANN graph compilation and kernels
  → Ascend runtime and collectives
  → UnifiedBus scale-up fabric
  → SuperPod
  → SuperCluster and cloud operations

The benefit is one architecture across silicon, fabric, server, and software. The cost is a larger migration and validation boundary. Kernel coverage, numerical behavior, framework compatibility, tooling, and distributed execution all need to match the model portfolio.

Huawei is therefore the closest of these three to a sovereign platform strategy. The objective is not merely efficient inference. It is control over the complete compute system and its supply path.

FuriosaAI keeps the facility boundary familiar

RNGD takes a narrower approach.

The second-generation FuriosaAI accelerator is a 180-watt PCIe card with 48 GB of HBM3. The company positions it for large-language and multimodal inference, with two cards fitting inside a conventional server power and cooling envelope.

RNGD’s Tensor Contraction Processor uses a programmable tensor engine rather than exposing a GPU-style collection of fixed matrix units. Furiosa argues that the architecture can map a wider set of tensor shapes while keeping control overhead and data movement low.

The data-center contract remains recognizable:

PCIe attachment;
standard x86 host servers;
air-cooled operation;
HBM on each card;
Kubernetes and container-based deployment;
a vendor compiler and runtime behind supported frameworks.

That matters because adopting a specialist involves two separate changes. The first is physical: power, cooling, network, server qualification, and sparing. The second is logical: model compilation, operator support, quantization, serving, and observability. A 180-watt PCIe card reduces the first change. It does not remove the second.

Furiosa’s software stack includes a compiler, runtime, model tooling, and serving integration. The company publishes supported models and performance material, but the usable boundary remains model-specific. An architecture can execute transformers in principle while still lacking a production-ready path for a particular operator, attention variant, quantization method, or serving engine.

Vendor and partner evidence now includes RNGD volume shipments, a live Samsung SDS NPU-as-a-Service offering, and announced data-center installations. These sources establish commercial activity, but they are not independently normalized production measurements. The key missing figures are sustained utilization, failure rates, model onboarding time, power at the complete server boundary, and independently comparable service cost.

Rebellions scales a PCIe NPU toward the rack

REBEL also starts as a standard data-center accelerator.

The card provides 144 GB of HBM3E and connects through PCIe Gen5. Rebellions describes native support for several low-precision formats and positions the design for large-model inference. A REBEL-Quad system combines four accelerators behind one host, creating a larger local memory and compute domain.

The architecture then extends through merchant data-center interfaces:

PCIe for host attachment;
CXL-oriented memory expansion in system designs;
Ethernet for scale-out;
liquid-cooled rack configurations for higher density;
a Kubernetes-based operational path.

Rebellions’ RBLN software stack provides the compiler, runtime, framework integration, model zoo, and serving tools required to turn checkpoints into executable artifacts. As with Corsair and RNGD, that compiler is part of the product. The hardware cannot be evaluated separately from supported operators, model conversion, quantization, and distributed-serving behavior.

The company has announced cloud and telecom infrastructure work, including a 2026 collaboration with SK Telecom and Arm, along with server-system partnerships. These announcements show an operating ecosystem. They do not yet create a common benchmark boundary across REBEL, RNGD, Ascend, and GPUs.

Rebellions occupies the middle of the three strategies. It does not own a full sovereign software and fabric stack like Huawei, but it aims beyond a single add-in card by defining multi-card and rack-scale inference systems.

Efficiency claims need a service boundary

Each vendor presents performance-per-watt or cost advantages. Those claims can be true within their test boundary and still fail to answer a production question.

The useful unit is not peak TOPS. It is a completed unit of model service:

Measurement	Required boundary
Latency	Prompt length, output length, batch policy, streaming behavior, and percentile
Throughput	Concurrent requests, quality target, scheduler policy, and complete system
Accuracy	Model revision, dataset, quantization, and numerical tolerance
Power	Card, host server, network, cooling, or facility boundary
Cost	Hardware utilization, software work, support, spares, and facility life
Portability	Time and engineering required to qualify another model
Availability	Card, server, rack, compiler, and service failure rates

A PCIe NPU can draw less power than a high-end GPU and still produce a worse service if the model does not compile cleanly, the cards remain underutilized, or requests spill back to a costly compatibility pool.

The reverse is also true. A specialist does not need universal model coverage to create value. It needs enough stable demand for supported models to keep the pool utilized.

Two meanings of sovereignty

These architectures separate two meanings of sovereignty.

One is supply sovereignty: control over silicon, manufacturing access, networking, system design, and software. Huawei is pursuing that broader boundary.

The other is operational choice: adding accelerators from more than one supplier so stable workloads are not tied to one runtime and pricing model. Furiosa and Rebellions contribute to that boundary without replacing the rest of the data-center stack.

What remains unproven

The vendors have demonstrated supported transformer models through published systems, model catalogs, partner deployments, or cloud endpoints. That establishes execution capability for those documented configurations, not broad model coverage or equal production economics.

The gap is whether the surrounding systems can sustain a broad production workload with predictable onboarding time, quality, utilization, and availability. Public material still lacks a consistent comparison of:

the same current model and serving software;
the same quantization and accuracy target;
identical latency percentiles and batch policy;
card, server, rack, and facility power;
compiler and model-porting effort;
system availability and replacement time;
software support across successive model generations.

That missing evidence affects each architecture differently. Huawei must prove the breadth and operability of an integrated platform. Furiosa must prove that a narrow, efficient pool can stay busy across enough models. Rebellions must prove that its card-to-rack expansion preserves the efficiency and simplicity promised at the card boundary.

Alternative AI hardware becomes credible when the complete operating system around the chip is visible. The unresolved question is who will publish enough production evidence to compare that system honestly—without reducing architectural choice to a peak-performance table or treating every alternative as the same kind of accelerator.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

SambaNova Compiles Models Into a Rack

Amit — Wed, 23 Sep 2026 04:53:28 +0000

SambaNova's most important product is not its accelerator chip. It is a compiled model deployment that spans the chip, three memory tiers, a 16-RDU rack, and the software routing requests across them.

That system boundary separates SambaNova from a conventional accelerator card. A model configuration becomes a Processor Executable Format binary, its weights occupy a planned memory hierarchy, and SambaStack deploys the result as a model service rather than treating every request as an arbitrary program arriving at a general-purpose processor.

The thesis: SambaNova compiles models into a rack. The Reconfigurable Dataflow Unit, or RDU, is the execution engine, but the useful commercial unit is the complete inference system around it.

TL;DR

The RDU maps a model's dataflow graph onto programmable compute, memory, and communication resources instead of repeatedly launching isolated kernels.
SN40L combines distributed SRAM, HBM, and attached DDR; the memory system is part of model placement, not merely storage behind the processor.
The mature deployment boundary is SambaRack SN40-16 plus SambaStack on-premises, or the same architecture exposed through SambaCloud.
SambaNova's software compiles specific model, sequence-length, batch-size, and speculative-decoding configurations into deployable binaries and validates whether a deployment fits available RDU memory.
SN50 extends the thesis toward larger scale and disaggregated inference, but its strongest public evidence remains preview benchmarks and vendor roadmap claims rather than broad production measurements.

Dataflow changes what the compiler owns

CUDA's programming model presents a GPU as a device executing kernels. Each kernel performs a bounded operation, while software schedules launches and moves tensors through the memory hierarchy. Fusion can reduce that traffic, but the execution model remains kernel-oriented.

SambaNova makes a different cut. Its current Dataflow Architecture description shows a grid of programmable compute units and SRAM-based programmable memory units connected as a streaming pipeline. While one operator computes, the system can fetch data for the next. Intermediate activations stay near the operators consuming them.

The underlying idea predates the current product naming. SambaNova's architecture material describes an RDU as a tiled array of compute and memory units joined by a programmable communication fabric. The SN40L paper presented at MICRO 2024 adds the current system details: dataflow cores contain compute units, memory units, address-generation units, and an on-chip reconfigurable network.

This does not mean the RDU executes without instructions or memory movement. It means the compiler can plan more of the communication path and keep a longer section of the model graph resident as a spatial pipeline.

flowchart TD
    A[Model checkpoint] --> B[Conversion and compilation]
    B --> C[PEF: compiled model configuration]
    A --> D[Model weights]
    C --> E[ModelProfile: supported runtime shapes]
    D --> F[Model: checkpoint and tokenizer]
    E --> G[Model and profile pair]
    F --> G
    G --> H[Optional ModelBundle]
    G --> I[ModelDeployment]
    H --> I
    I --> J[Memory-fit legalizer]
    J --> K[Inference router and runtime]
    K --> L[Attached DDR: model catalog and cache]
    L --> M[HBM: weight streaming]
    M --> N[Distributed SRAM: active data]
    N --> O[PCU and PMU dataflow pipeline]
    O --> P[OpenAI-compatible API response]

The diagram exposes the adoption trade. A familiar checkpoint enters the system, but it does not remain an arbitrary framework program at runtime. The current SambaStack deployment model defines a PEF as a compiled model binary for an RDU. A ModelProfile groups compatible PEFs and the runtime shapes they support. A Model identifies checkpoints and tokenizers, while a ModelDeployment creates replicas and a routable endpoint. ModelBundle is now optional: it groups several model/profile pairs when they need to be deployed, validated, or shared as one unit.

That compilation boundary is where SambaNova gains control over data movement. It is also where model support, graph coverage, compiler maturity, and artifact lifecycle become commercial constraints.

Three memory tiers make the rack the machine

The RDU's second architectural choice is capacity placement. The current SambaRack SN40-16 datasheet specifies 16 SN40L RDUs with three memory tiers:

Memory tier	SN40-16 published capacity	System role
Distributed on-RDU SRAM	8 GB total	Keeps active operands and intermediate data close to compute
HBM	1 TB total	Streams model weights at high bandwidth
RDU-attached DDR	4 TB total	Holds checkpoints, prompt cache, and a catalog of model configurations

The same rack contains host CPUs, NVMe storage, management components, and 400/200 GbE networking. SambaNova publishes 10 kW typical and 16.4 kW maximum inference power for the air-cooled rack. Those are rack specifications, not measured energy per token.

This hierarchy supports a different capacity strategy from placing every active model entirely in HBM. Large DDR capacity can hold more weights and configurations close to the RDUs. HBM provides the high-bandwidth path, while SRAM feeds the active graph. SambaNova's claim is that this enables fast switching among resident models and reduces repeated transfers from external storage.

The software reveals how tightly memory and deployment are coupled. SambaStack runs a legalizer before deployment to verify that the selected PEFs and checkpoints fit the node's memory constraints. A deployment can include multiple models, sequence-length profiles, batch sizes, and a target/draft pair for speculative decoding. The runtime then selects among compiled configurations according to the request.

The deployable object is therefore not "a model on an RDU." It is a memory-valid deployment distributed across a rack.

The commercial boundary has three forms

SambaNova now exposes the same architectural thesis through different operational boundaries.

Offering	Customer operates	SambaNova supplies	Practical boundary
SambaStack	Data center, network, Kubernetes integration, identity, storage, and service policy	RDU rack, runtime, management software, compiled artifacts, and serving stack	Customer-operated private or sovereign inference
SambaManaged	Facility boundary and application integration	Fully managed inference infrastructure installed in the customer's data center	Dedicated capacity without operating the full stack
SambaCloud	API client, model selection, and application	Infrastructure, model deployment, routing, and service operation	Multi-model inference API

The on-premises path is not a simple appliance with one network cable. SambaNova's current installation guide requires a front-end data network, a private inter-rack network, a management network, customer identity and DNS services, storage, load balancing, and a Kubernetes control plane. SambaStack installs through Helm and exposes deployed models through an OpenAI-compatible API.

At the other end, SambaCloud hides the compiled artifacts and rack operations. Its July 2026 release added prompt caching and continued expanding API compatibility, tool use, structured output, and model support. That surface is easier to adopt, but it offers the curated models and configurations SambaNova has prepared rather than arbitrary code execution.

This is the commercial read: the buyer chooses how much of the system boundary to own. Infrastructure operators can run a private RDU system, let SambaNova operate dedicated capacity in their facility, or consume the architecture as an API without seeing the compiler boundary.

SN40L is deployed; SN50 is the next claim

The generation boundary matters because SambaNova's public material now discusses SN40 and SN50 together.

SN40L is the fourth-generation RDU behind the documented SambaRack and the current on-premises software guides. Independent deployment evidence exists. The Argonne Leadership Computing Facility opened its Metis system to researchers in October 2025 with two 16-RDU SambaRacks. Its current user guide describes a larger six-rack SN40L cluster serving multiple models through OpenAI-compatible endpoints.

Earlier systems also operated alongside conventional supercomputers. Lawrence Livermore National Laboratory integrated SambaNova dataflow hardware into its computing environment for cognitive simulation research. That establishes real installation history, although the older system used an earlier RDU generation and does not prove current LLM-serving economics.

SN50 is SambaNova's fifth-generation inference processor. The company says a 16-chip SambaRack averages 20 kW, offers five times the compute and four times the network bandwidth of SN40, and can connect as many as 256 RDUs across racks. Its February 2026 announcement said customer shipments would begin in the second half of 2026.

The most interesting SN50 direction is not a larger standalone rack. It is disaggregated inference. SambaNova demonstrated NVIDIA GPUs handling compute-heavy prefill while RDUs handled memory-bound decode. A July 2026 demonstration used four H200 GPUs for prefill and one 16-RDU SN50 rack for decode on MiniMax M2.7.

The evidence has advanced beyond first-party performance charts, but it remains early. SambaNova's summary of a SemiAnalysis-run benchmark reports roughly 800 tokens per second at the highest-interactivity point and roughly 400 tokens per second as throughput approached the fastest B200 configuration tested on the same model. The available public summary does not provide the power, utilization, availability, or economic data required for a complete rack comparison.

That changes the competitive framing. SambaNova does not need to replace every GPU to become useful. It can become a specialized decode tier behind a serving layer that sends each phase to different hardware.

Where the architecture fits

Workload	Architectural fit	Evidence boundary
Interactive LLM decode	Strong: sequential generation rewards memory bandwidth, locality, and planned execution	Artificial Analysis independently measures high output speed on several SambaCloud models, but endpoint tests do not reveal rack utilization or energy
Multi-model and agent workflows	Strong in principle: large DDR capacity and deployments keep multiple models and configurations resident	Fast switching and model bundling are documented primarily by SambaNova
Private or sovereign inference	Strong: complete air-cooled racks can run on-premises and expose familiar APIs	Installation requirements still include Kubernetes, networking, identity, storage, and vendor-provided artifacts
Disaggregated prefill and decode	Promising: GPUs retain prefill while RDUs specialize in decode	Public SN40/SN50 demonstrations exist; broad production measurements do not
Custom fine-tuned models	Supported through checkpoint conversion, PEFs, and custom bundles	Support depends on compiler coverage and validated configurations
Large-scale model training	Historically supported by earlier SambaNova systems	Current SN40/SN50 positioning, software, and evidence concentrate on inference
Arbitrary or rapidly changing research graphs	Weaker fit	Compilation and validated artifacts add friction when operators change models or shapes frequently

Artificial Analysis currently reports SambaCloud output rates above 700 tokens per second for some gpt-oss-120b configurations and above 400 tokens per second for MiniMax M2.7. Those measurements establish that the public service can deliver fast single-request generation. They do not establish aggregate rack throughput, sustained concurrency, power efficiency, availability, or total service cost.

That distinction matters. The architecture is optimized around planned model execution, but a production service also contains queues, routers, caches, network paths, replicas, failures, and model-specific quality choices. Tokens per second from one endpoint are evidence about the service experience—not a complete system benchmark.

What's missing

SambaNova publishes enough material to understand the shape of the system. It does not publish enough comparable data to close the economic argument.

There is no current, public evidence set that places SN40 or SN50 beside competing systems with the same model, precision, context distribution, batch policy, concurrency, accuracy target, measured wall power, and availability window. The SN40L MICRO paper is detailed, but its headline comparisons focus on SambaNova's Composition of Experts workload rather than a neutral industry benchmark. The SN50 material is earlier still.

Compiler coverage is the other open boundary. SambaStack documents stable, preview, deprecated, and removed PEF and checkpoint versions. That is healthy operational machinery, but it also shows that model support is a maintained product matrix. A framework checkpoint being available does not mean every shape, feature, quantization, or serving optimization is immediately deployable on an RDU.

So what

SambaNova is best understood as a system for controlling model movement.

The RDU turns a model graph into a spatial dataflow pipeline. The memory hierarchy decides what remains resident and what streams into active execution. The compiler turns framework artifacts into hardware-specific configurations. SambaStack packages those configurations into a rack-level service, while SambaCloud hides the machinery behind an API.

That full boundary is the product. Comparing one RDU with one GPU misses it.

The unresolved question is whether SambaNova can keep compiler and artifact delivery ahead of changing model architectures while proving that phase-specific routing still wins after network, utilization, power, and operational costs are included.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

The GPU Stopped Being the Product

Amit — Wed, 23 Sep 2026 04:52:53 +0000

NVIDIA's most important architectural change is not a faster GPU. It is the expansion of what counts as the computer.

The Tesla P100 arrived as an accelerator inside a server. The A100 made the eight-GPU system a repeatable unit. Hopper joined the CPU and GPU more tightly. Blackwell added a top-end configuration whose scale-up boundary is a liquid-cooled rack. Vera Rubin extends the design across compute racks, networking, power, cooling, and inference orchestration.

At the high end, the product is no longer only a chip. It is a large part of the AI data center assembled as one machine: Tensor Cores change the math, HBM feeds it, NVLink expands the scale-up domain, networking connects racks, and the software stack makes each generation usable.

TL;DR

P100 established the modern foundation with HBM2, NVLink, and the first DGX system; V100 added Tensor Cores.
A100 turned the GPU into elastic infrastructure through TF32, structured sparsity, Multi-Instance GPU, faster NVLink, and a standardized eight-GPU NVSwitch system.
Hopper optimized transformers directly through FP8 and Transformer Engine, while Grace Hopper introduced coherent CPU-GPU coupling.
Blackwell introduced a top-end 72-GPU liquid-cooled rack with 130 TB/s of summed bidirectional NVLink endpoint bandwidth, while B200 also remained available in server configurations.
Vera Rubin doubles NVLink bandwidth again and raises HBM bandwidth to 22 TB/s per GPU. Groq-derived LPX racks add specialized inference paths, while Rubin remains in production ramp rather than broad availability.

Start before A100

The useful starting point is the Tesla P100 and Pascal GP100 architecture in 2016. P100 introduced three pieces that still define NVIDIA's systems:

HBM on the GPU package for high-bandwidth access to model data.
NVLink for GPU-to-GPU communication beyond PCIe.
DGX-1, an integrated eight-GPU server with a tested software stack.

P100 did not have Tensor Cores. Its AI acceleration came from running FP16 arithmetic at twice the rate of FP32 on ordinary CUDA cores. The Tesla V100 Volta architecture made the next decisive move in 2017: 640 first-generation Tensor Cores dedicated to matrix multiplication.

Pascal made the GPU a better parallel processor. Volta began shaping its numerical machinery around neural networks.

flowchart LR
    P["P100 Pascal<br/>2016<br/>HBM2 + NVLink + DGX-1"] --> V["V100 Volta<br/>2017<br/>Tensor Cores"]
    V --> A["A100 Ampere<br/>2020<br/>TF32 + sparsity + MIG"]
    A --> H["H100/H200 Hopper<br/>2022–2024<br/>FP8 + Transformer Engine"]
    H --> B["B200 Blackwell<br/>2024 architecture<br/>dual-die GPU + NVL72"]
    B --> BU["B300 Blackwell Ultra<br/>2025 architecture<br/>288 GB HBM3e"]
    BU --> R["Vera Rubin<br/>2026 production ramp<br/>HBM4 + NVLink 6"]
    R -. roadmap .-> RU["Rubin Ultra<br/>2027"]
    RU -. roadmap .-> F["Feynman<br/>2028"]

The names hide overlapping product lines. H200 is a Hopper memory upgrade, while Blackwell Ultra expands Blackwell's memory and low-precision throughput. Grace Hopper, GB200, and Vera Rubin are platforms spanning CPUs, GPUs, memory, interconnects, and systems.

The generation map

The table uses representative high-end data-center configurations. SXM and rack-scale modules run at different power and bandwidth levels from PCIe cards with the same architecture. Peak Tensor Core figures also change numerical format across generations, so they are not one continuous performance benchmark.

Platform	Architectural step	Representative memory	Scale-up fabric per GPU	Deployment unit	Status on Aug. 30, 2026
P100 Pascal	HBM2, NVLink 1, high-rate FP16	16 GB, 0.72 TB/s	0.16 TB/s bidirectional	Eight-GPU DGX-1 with a direct NVLink mesh	Legacy
V100 Volta	First Tensor Cores, NVLink 2	16–32 GB, 0.9 TB/s	0.30 TB/s bidirectional	DGX-1 mesh; DGX-2 introduced NVSwitch	Legacy
A100 Ampere	TF32, BF16, FP64 Tensor Cores, 2:4 sparsity, MIG	40–80 GB, up to 2.0 TB/s	0.60 TB/s bidirectional	Eight-GPU DGX/HGX NVSwitch baseboard	Mature production
H100 Hopper	FP8, Transformer Engine, Tensor Memory Accelerator	80 GB, 3.35 TB/s	0.90 TB/s bidirectional	Eight-GPU HGX/DGX; external NVLink Switch systems	Mature production
H200 Hopper	Hopper compute with larger, faster HBM3e	141 GB, 4.8 TB/s	0.90 TB/s bidirectional	H100-compatible system generation	Mature production
B200 Blackwell	Two reticle-sized dies act as one GPU; FP4; NVLink 5	180 GB, up to 8 TB/s	1.8 TB/s bidirectional	DGX/HGX server or GB200 NVL72 rack	Production
B300 Blackwell Ultra	288 GB HBM3e, more NVFP4 and attention throughput	288 GB, up to 8 TB/s	1.8 TB/s bidirectional	GB300 NVL72 rack	Production
Vera Rubin	HBM4, sixth-generation Tensor Cores, NVLink 6, Vera CPU	288 GB, 22 TB/s	3.6 TB/s bidirectional	Vera Rubin NVL72 and pod-scale platform	Production ramp; shipments scheduled for fall 2026
Rubin Ultra and Feynman	Larger scale-up domain, then next named architecture	Not fully disclosed	Not fully disclosed	2027 and 2028 roadmap	Roadmap

The status boundary matters. On August 27, AWS received its first Vera CPU server and Rubin GPU for deployment work. NVIDIA still describes production shipments as beginning in fall 2026. Early component delivery is not general availability of complete NVL72 systems.

Three caveats matter:

NVIDIA's NVLink numbers add transmit and receive bandwidth. They are not one-way payload rates.
Sparse throughput assumes the required structured sparsity pattern. It is not dense performance.
FP16, TF32, FP8, FP4, and NVFP4 trade precision, range, storage, and compute differently. A larger headline FLOPS number at a narrower format does not mean every workload runs proportionally faster.

Component 1: Compute changed from general arithmetic to model-aware arithmetic

P100 accelerated FP16 on CUDA cores. V100 added dedicated Tensor Cores that multiplied FP16 matrices and accumulated in FP32, giving neural-network matrix operations their own hardware.

A100's third-generation Tensor Cores added TF32, BF16, IEEE FP64, INT8, INT4, and 2:4 structured sparsity. One matrix engine now covered training, inference, and scientific computing.

The H100 architecture added FP8 and the first Transformer Engine, coordinating lower- and higher-precision execution by transformer layer.

Blackwell extended that direction to FP6 and FP4 with a second-generation Transformer Engine. Rubin's third-generation Transformer Engine adds sixth-generation Tensor Cores and up to 50 petaflops of NVFP4 inference compute according to NVIDIA. That is a vendor peak, not a cross-generation application benchmark.

Generation	Numerical shift	What changed architecturally
Pascal	FP16 on CUDA cores	AI uses the general parallel processor more efficiently
Volta	FP16 Tensor Cores with FP32 accumulation	Matrix multiplication receives dedicated hardware
Ampere	TF32, BF16, FP64 Tensor Cores, INT8/4, 2:4 sparsity	One matrix engine spans training, inference, and HPC
Hopper	FP8 plus Transformer Engine	Precision selection becomes part of model execution
Blackwell	FP6, FP4, microscaling	Low-precision formats become finer-grained and model-aware
Rubin	Expanded NVFP4/NVFP6 and third-generation Transformer Engine	Precision, attention, sparsity, and communication are co-optimized

Component 2: Memory became as important as compute

Neural networks repeatedly move weights, activations, and cache state. More arithmetic units do not help when those units wait for data.

P100's HBM2 placed stacked memory beside the GPU on a silicon interposer. It delivered 16 GB at 720 GB/s. V100 raised bandwidth to 900 GB/s. The original A100 reached 1.555 TB/s with 40 GB, and the later A100 80 GB exceeded 2 TB/s.

H100 increased bandwidth to 3.35 TB/s, but H200 demonstrates why capacity and bandwidth must be separated. H200 retained Hopper compute while moving to 141 GB of HBM3e at 4.8 TB/s. On memory-intensive large-model inference, that change can matter without a new Tensor Core generation.

Blackwell increased the representative B200 package to 180 GB at up to 8 TB/s. Blackwell Ultra reached 288 GB. Rubin keeps 288 GB but replaces HBM3e with HBM4 and raises the vendor-published peak to 22 TB/s.

xychart-beta
    title "Selected vendor-published peak HBM bandwidth"
    x-axis ["P100", "V100", "A100 80GB", "H100", "H200", "B200", "Rubin"]
    y-axis "TB/s" 0 --> 24
    bar [0.72, 0.90, 2.04, 3.35, 4.80, 8.00, 22.00]

The chart keeps one physical quantity on the axis, but it combines product introductions, later memory variants, and Rubin's production-ramp specification. Every value is a published peak rather than achieved application bandwidth.

Rubin's published HBM bandwidth is more than 30 times P100's. Power and cooling rose with it: P100 SXM was rated at 300 watts, A100 SXM at 400 watts, H100 SXM at up to 700 watts, and B200 at up to 1,000 watts per GPU.

Component 3: Packaging and CPU coupling expanded the processor boundary

P100, V100, A100, and H100 were large monolithic GPU dies. Each generation pushed close to the practical manufacturing limit for one piece of silicon.

Blackwell changed the package. Two reticle-limited dies connect through a 10 TB/s chip-to-chip interface and present themselves as one CUDA GPU. NVIDIA could increase transistor count to 208 billion without requiring one manufacturable die of that size.

Rubin continues the dual-reticle approach with 336 billion transistors. NVIDIA calls the inter-die connection NV-HBI. From the programming layer, the goal remains one GPU abstraction. Underneath it, the package has become a tightly integrated multi-die system.

Once a "GPU" contains multiple compute dies, HBM stacks, high-speed die links, NVLink interfaces, and control engines, the package is already a system.

Grace Hopper tightened that relationship. A Grace CPU and Hopper GPU communicate over coherent NVLink-C2C, giving the CPU and GPU a higher-bandwidth path and a coherent memory relationship inside one superchip.

Blackwell's GB200 platform connects one Grace CPU to two Blackwell GPUs. The GB200 NVL72 rack contains 36 Grace CPUs and 72 Blackwell GPUs. Vera Rubin replaces Grace with the Vera CPU while retaining the 36-to-72 relationship.

This creates two different fabrics:

NVLink-C2C connects the CPU and GPU inside a superchip.
NVLink and NVSwitch connect GPUs across the scale-up domain.

PCIe remains the host and peripheral fabric, while NVLink-C2C and NVLink carry the tightly coupled processor paths.

flowchart TB
    APP["Models and applications"]
    ORCH["Inference orchestration<br/>Dynamo, routing, cache coordination"]
    SERVE["Serving and model runtimes<br/>Triton, TensorRT-LLM"]
    LIBS["Accelerated libraries<br/>cuDNN, cuBLAS, NCCL, NVSHMEM"]
    CUDA["CUDA platform<br/>compiler, runtime, graphs, memory APIs"]
    CPU["Grace or Vera CPU<br/>system memory and control"]
    GPU["GPU package<br/>Tensor Cores, cache, HBM"]
    SCALEUP["NVLink + NVSwitch<br/>scale-up domain"]
    NODE["Server or rack I/O domain"]
    SCALEOUT["InfiniBand or Spectrum-X<br/>scale-out fabric"]
    OPS["NGC, GPU Operator, Mission Control<br/>deployment and operations"]

    APP --> ORCH --> SERVE --> LIBS --> CUDA --> GPU
    CPU <-->|"NVLink-C2C"| GPU
    GPU <-->|"NVLink"| SCALEUP
    SCALEUP --> NODE
    NODE <-->|"ConnectX / BlueField"| SCALEOUT
    OPS -. manages .-> SERVE
    OPS -. manages .-> CUDA
    OPS -. manages .-> SCALEOUT

Component 4: NVLink turned multiple GPUs into one scale-up computer

PCIe connects devices to a host. NVLink exists because tightly coupled GPU workloads need a different path.

P100 supported four first-generation NVLinks for 160 GB/s of aggregate bidirectional bandwidth per GPU. DGX-1 arranged eight GPUs in a hybrid cube mesh, so not every pair had a dedicated direct connection.

V100 increased per-GPU NVLink bandwidth to 300 GB/s. DGX-2 then introduced NVSwitch, replacing the fixed mesh with a switched 16-GPU fabric.

A100 doubled per-GPU bandwidth to 600 GB/s and standardized an eight-GPU fully connected NVSwitch baseboard. H100 reached 900 GB/s, while external NVLink Switch systems stretched the scale-up fabric beyond one server.

Blackwell moved the fabric across an entire rack. GB200 and GB300 NVL72 connect 72 GPUs through fifth-generation NVLink at 1.8 TB/s per GPU, producing NVIDIA's rounded 130 TB/s aggregate endpoint figure.

Rubin doubles the per-GPU figure to 3.6 TB/s and the 72-GPU aggregate to 260 TB/s.

xychart-beta
    title "Published aggregate bidirectional NVLink bandwidth per GPU"
    x-axis ["P100", "V100", "A100", "H100", "B200", "Rubin"]
    y-axis "TB/s" 0 --> 4
    line [0.16, 0.30, 0.60, 0.90, 1.80, 3.60]

The directionality deserves emphasis. A100's 600 GB/s means 300 GB/s in each direction. Blackwell's 1.8 TB/s and Rubin's 3.6 TB/s are also bidirectional endpoint totals. The 130 TB/s and 260 TB/s rack numbers multiply those endpoint figures by 72 GPUs. They do not state the fabric's one-way bisection bandwidth.

NVLink generation	Introduction	Per-GPU published bandwidth	Representative scale-up domain
NVLink 1	P100	160 GB/s bidirectional	Eight-GPU direct mesh
NVLink 2	V100	300 GB/s bidirectional	Eight-GPU mesh; 16 GPUs with NVSwitch
NVLink 3	A100	600 GB/s bidirectional	Eight-GPU fully switched baseboard
NVLink 4	H100	900 GB/s bidirectional	Eight-GPU HGX; larger Grace Hopper switch systems
NVLink 5	Blackwell	1.8 TB/s bidirectional	72-GPU NVL72 rack
NVLink 6	Rubin	3.6 TB/s bidirectional	72-GPU NVL72; larger 2027 roadmap domain

The scale-up domain supports frequent, low-latency communication inside one model execution. Scale-out networking joins systems and racks into a larger cluster.

Component 5: Networking and cooling expanded the deployment boundary

NVIDIA's 2020 acquisition of Mellanox brought the network into the platform.

Mellanox added several distinct components:

ConnectX network adapters move data between GPU systems.
Quantum InfiniBand provides a purpose-built scale-out fabric for large compute clusters.
Spectrum-X Ethernet combines Ethernet switches, adapters, congestion control, and software for AI traffic.
BlueField DPUs handle infrastructure work including networking, storage, isolation, and security.

Successive NVIDIA systems used 100 Gb/s EDR, 200 Gb/s HDR, 400 Gb/s NDR or Spectrum-X, and then 800 Gb/s Quantum-X800 and Spectrum-X800. Rubin adds ConnectX-9 while retaining InfiniBand and Ethernet scale-out paths.

NVLink and InfiniBand are not competing names for the same connection:

Fabric	Primary job	Typical boundary
PCIe	Host and peripheral attachment	CPU, GPU, NIC, storage inside a system
NVLink-C2C	Coherent processor coupling	CPU to GPU inside a superchip
NVLink + NVSwitch	High-bandwidth GPU scale-up	GPUs inside a server, rack, or tightly coupled domain
InfiniBand / Spectrum-X	System scale-out	Servers and racks across a cluster

The deployment unit grew with the fabric. The first DGX-1 was a 3U, air-cooled server rated at 3.2 kW. DGX A100 grew to 6U and 6.5 kW, while DGX H100 reached roughly 10.2 kW.

GB200 NVL72 changes the boundary: 72 GPUs, 36 CPUs, nine NVSwitch trays, power shelves, networking, and direct-to-chip liquid cooling form one rack-scale system. NVIDIA documents approximately 120 kW for that rack, while GB300 NVL72 raises the published envelope to about 142 kW.

Those figures do not compare efficiency because the measured boundary grows from one eight-GPU server to an entire 72-GPU rack. They show the architectural transition: electrical delivery, cooling, switching, and compute are now specified together.

Component 6: CUDA expanded into an operating stack for AI infrastructure

CUDA is the longest-running source of continuity. The CUDA programming model, drivers, compilers, and libraries let applications survive multiple hardware generations without starting over.

The stack around CUDA kept expanding:

Layer	Representative components	Role
Programming platform	CUDA	Kernels, memory management, graphs, compilation, runtime
Accelerated libraries	cuBLAS, cuDNN, NCCL	Matrix math, neural-network operations, multi-GPU collectives
Precision layer	Transformer Engine	FP8 and newer precision recipes for transformer models
Model optimization	TensorRT, TensorRT-LLM	Graph optimization, quantization, kernels, KV cache, distributed LLM execution
Serving	Triton Inference Server	Model serving, batching, ensembles, multiple backends
Fleet orchestration	Dynamo and NIXL	Request routing, disaggregated prefill and decode, cache and tensor movement
Packaging and operations	NGC containers, Container Toolkit, GPU Operator, Mission Control	Tested software combinations, deployment, drivers, scheduling, lifecycle management
Resource isolation	MIG and vGPU	Partitioning and sharing accelerator capacity

Volta exposed Tensor Cores through CUDA 9 and accelerated libraries. Ampere added TF32, sparsity, CUDA Graphs, and MIG. Hopper paired FP8 hardware with Transformer Engine. Blackwell paired FP4 with TensorRT-LLM, NeMo, and rack-scale NVLink. Dynamo moved optimization above the model server into routing, KV-cache placement, prefill/decode separation, and worker-to-worker data movement.

CUDA compatibility is valuable, but it is not magic portability. The CUDA compatibility model has minimum driver requirements and version boundaries. Native cubins target specific compute capabilities. TensorRT engines can be tied to library versions and GPU assumptions. MIG profiles differ by architecture. An optimized FP8 or NVFP4 execution recipe may need rebuilding for a different generation.

The application logic may travel. The highest-performance execution artifact often does not.

Groq adds a specialized inference path

The Groq relationship adds one important direction to NVIDIA's architecture: the GPU does not need to execute every phase of inference.

This was not an acquisition of Groq. In December 2025, Groq announced a non-exclusive license of its inference technology to NVIDIA. Several Groq leaders and team members joined NVIDIA, while Groq remained an independent company and continued operating GroqCloud.

NVIDIA used the licensed technology in the Groq 3 LPX rack, which it positions beside Vera Rubin NVL72. One representative pattern separates prefill from decode: Rubin processes the prompt and creates the KV cache, LPX generates output tokens, and NVIDIA Dynamo coordinates routing and data movement.

The pattern matters more than the product pairing. NVIDIA is extending its platform around heterogeneous inference—general-purpose GPUs for broad model execution, specialized processors for latency-sensitive phases, and software that decides where each phase runs.

What measured performance can actually tell us

Peak specifications describe hardware ceilings. Vendor platform claims describe selected workloads and configurations. Neither is a substitute for a controlled benchmark.

The methodologically cleanest comparison in the available results is MLPerf Training v0.7. NVIDIA submitted an eight-GPU V100 DGX-1 and an eight-GPU DGX A100 to the same BERT benchmark under the same release. Applying MLPerf's rule of discarding the fastest and slowest runs and averaging the remaining eight gives roughly 166.2 minutes for V100 and 49.0 minutes for A100—about 3.4 times faster.

The result combines hardware and generation-appropriate software rather than isolating the chip—which matches how users experience the system.

Later comparisons are harder:

NVIDIA reported up to 6.7 times higher H100 performance than its first A100 submission in MLPerf Training v2.1, but "up to" selects the strongest workload and spans different submission vintages.
In MLPerf Inference v4.0, NVIDIA reported about 28% greater H200 performance than H100 at the same 700-watt envelope and up to 45% at 1,000 watts. The equal-power result is the cleaner evidence for HBM3e's contribution.
Blackwell and Blackwell Ultra have measured MLPerf submissions, but accelerator count, numerical format, model, power, and software must match before calculating a ratio.
Rubin had no comparable public MLPerf submission by August 30, 2026. Its performance and cost claims remain NVIDIA measurements until reproducible benchmark systems appear.

There is no honest single line for "GPU performance over time." Workload, precision, sparsity, GPU count, power, and software all move.

What's missing

The stack still lacks a neutral system benchmark that matches how current AI infrastructure is purchased and operated.

MLPerf is the strongest public mechanism available, but the product boundary keeps expanding faster than the benchmark boundary. A useful rack-scale comparison would hold all of these constant:

model and quality target;
prompt and output-length distribution, including concurrency;
time to first token and inter-token latency;
sustained throughput under concurrency;
training or inference precision;
dense versus sparse execution;
rack, network, cooling, and facility power;
software engineering required to reach the result;
failure recovery and degraded-operation behavior;
acquisition price and cost per completed unit of work.

Without that boundary, a vendor can compare one Blackwell rack with many Hopper servers, or FP4 with FP8, or projected utilization with measured throughput. The numbers may all be correct while the comparison remains unhelpful.

Rubin makes the gap larger. NVIDIA now describes benefits from GPU architecture, HBM4, NVLink 6, Vera CPUs, Spectrum-X, power smoothing, liquid cooling, Dynamo, and workload scheduling in one platform claim. That is a reasonable systems argument. It also makes independent reproduction much harder.

So what

NVIDIA's evolution from P100 to Rubin is the story of bottlenecks moving outward.

P100 attacked memory and PCIe. V100 attacked matrix math. A100 attacked precision, utilization, and multi-tenancy. Hopper attacked transformer execution. Blackwell attacked the server boundary. Rubin attacks the rack and pod as operating systems for continuous inference.

That changes how to evaluate the company and its competitors. Comparing accelerator FLOPS is now the narrowest possible cut. The real comparison is the amount of useful model work a complete system produces within its power, latency, reliability, and software constraints.

The GPU did not disappear. It became the center of a much larger machine.

The open question is whether this integration remains one durable platform advantage or creates enough cost, power, and operational complexity for the stack to split again—between general-purpose GPUs, specialized inference processors, open interconnects, and software that can schedule work across all of them.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

There Is No Universal AI Runtime—So Build a Portable Control Plane

Amit — Wed, 23 Sep 2026 04:52:17 +0000

AI accelerators are not interchangeable computers. A CUDA kernel does not become a Trainium executable because both systems run PyTorch, and an inference server supporting two GPU families does not erase their different compilers, libraries, memory models, collectives, or performance envelopes.

The portable layer sits higher.

Applications can call one stable service contract. A model registry can govern several hardware-specific artifacts. A router can place each request on a qualified pool. Observability can compare the resulting service, and policy can decide when moving a workload is economically justified.

That is a portable control plane—not a universal runtime.

TL;DR

Kernels, compiled artifacts, collectives, quantization paths, and performance tuning remain tied to an accelerator stack.
Serving frameworks can present common APIs across several backends, but backend support does not guarantee identical model coverage, behavior, or economics.
The durable architecture keeps NVIDIA, AMD, TPU, Trainium, and specialist accelerators in separate pools behind one service contract.
Model governance, workload qualification, routing policy, service-level telemetry, and cost policy are more portable than execution.
Portability is an operating model: maintain a broad compatibility pool and move stable workloads only after another pool passes correctness, performance, and cost gates.

The runtime boundary is real

CUDA defines a GPU programming model, runtime, compiler toolchain, and libraries for NVIDIA hardware. CUDA C++ device code is compiled by nvcc, which can produce architecture-specific binary images and PTX for later compilation by the CUDA runtime. That mechanism provides compatibility within the NVIDIA platform; it does not produce an executable for an unrelated accelerator.

HIP deliberately resembles CUDA and includes tools that translate many CUDA API calls. The same documentation also states that HIP code must be compiled for a specific AMD GPU architecture and that the resulting binaries contain code for that target. Source translation reduces migration work. It does not turn a CUDA binary into an AMD binary.

The boundary becomes clearer with cloud accelerators. The Neuron compiler transforms framework graphs into a Neuron Executable File Format artifact that the Neuron runtime loads on Neuron devices. PyTorch/XLA lets applications retain familiar PyTorch APIs, but it traces operations into an intermediate graph and sends HLO to XLA for compilation. Changes in graph shape can cause additional compilations, and hardware-specific optimization still happens below the framework.

OpenXLA is a useful portability layer precisely because it separates a versioned operation set from target-specific backends. StableHLO can carry a model graph between framework and compiler, while each backend still performs its own optimization and code generation. The graph can travel farther than the executable.

This produces a simple hierarchy:

Layer	What can be shared	What remains hardware-specific
Application	Request schema, authentication, streaming contract, error model	Model behavior under different precision and implementations
Model source	Framework checkpoint, tokenizer, configuration, evaluation set	Converted, sharded, quantized, or compiled artifacts
Graph representation	ONNX or StableHLO where the model and operators are supported	Unsupported operators, custom extensions, lowering, graph partitioning
Serving	Endpoint shape, admission control, rollout mechanism	Engine build, kernels, cache layout, batching, parallelism
Scheduling	Workload intent, pool labels, quotas, priority	Device discovery, drivers, topology, allocation mechanism
Execution	Nothing universal enough to assume interchangeability	Runtime, compiler, kernels, libraries, collectives, memory placement
Measurement	Correctness, latency, throughput, availability, cost per outcome	Vendor counters, power boundaries, utilization definitions

The mistake is not using abstraction. The mistake is placing the abstraction below the level where the evidence supports it.

One contract, separate hardware pools

The stable unit should be a service, not a device. Each accelerator family operates as a separately qualified pool with its own software image, driver, runtime, serving engine, model artifacts, deployment process, and failure domain.

flowchart TB
    A[Applications and agents] --> B[Stable inference or training contract]
    B --> C[Identity, quotas, admission control]
    C --> D[Model catalog and artifact registry]
    D --> E[Workload qualification and policy router]

    E --> N[NVIDIA pool<br/>CUDA artifacts and kernels]
    E --> M[AMD pool<br/>ROCm and HIP artifacts]
    E --> T[TPU pool<br/>XLA-compiled artifacts]
    E --> R[Trainium pool<br/>Neuron-compiled artifacts]
    E --> S[Specialist pool<br/>Vendor runtime and artifacts]

    N --> O[Normalized service telemetry]
    M --> O
    T --> O
    R --> O
    S --> O

    O --> P[Cost, reliability, and placement policy]
    P --> E

This design does not schedule one container indiscriminately across every device. It selects a pool that already has a tested artifact and a known operating envelope for the requested model, precision, context length, batch profile, and service objective.

The model catalog therefore needs more than a model name. A useful entry binds a logical model version to a set of qualified implementations:

logical model: reasoning-model-v3
evaluation policy: eval-suite-2026-08

qualified implementations:
  - pool: nvidia-hopper
    artifact: fp8-tensor-parallel-8
    status: production
  - pool: amd-mi300
    artifact: fp8-tensor-parallel-8-rocm
    status: canary
  - pool: trainium
    artifact: neuron-compiled-seq4096
    status: batch-only

The logical model gives applications a stable name. Each implementation retains its own artifact identity, software bill of materials, evaluation results, supported limits, and rollback history. Promotion happens per implementation, not once for the abstract model.

Serving frameworks widen the interface, not the binary

Modern serving projects make this architecture easier, but none removes the qualification work.

vLLM supports hardware plugins outside its main repository. SGLang documents support for multiple accelerator families and offers an API compatible with common language-model request patterns. These are valuable common surfaces. Their hardware paths can still differ in kernel availability, quantization support, distributed execution, model coverage, and release maturity.

Triton Inference Server makes the separation visible: every model is associated with a backend, and the backend is the implementation that executes it. Triton can wrap TensorRT, PyTorch, ONNX Runtime, vLLM, or custom logic, but its own documentation warns that not every backend is supported on every platform.

ONNX Runtime follows a similar pattern through Execution Providers. The application uses a consistent API while a provider claims supported nodes or subgraphs and hands them to a hardware-specific library. That is portable application integration with specialized execution underneath. It is not proof that every model will partition, compile, or perform equally on every provider.

KServe can standardize how a model service is declared and operated on Kubernetes. Its serving runtimes still select concrete backends, images, resource requests, and model formats. The operational object becomes portable; the implementation remains qualified for a particular pool.

The practical contract should cover:

logical model and version;
task type and request schema;
streaming and cancellation behavior;
context, input, and output limits;
safety and data-handling policy;
latency and availability objectives;
error categories and retry rules;
response metadata identifying the implementation used.

It should not promise identical tokenization, numerical output, latency, or cost unless those properties were explicitly tested.

The scheduler places workloads; it does not port them

Cluster schedulers already understand heterogeneous resources. Kubernetes device plugins let vendors advertise specialized devices to the kubelet. A cluster can expose resources such as nvidia.com/gpu and amd.com/gpu, then use labels and affinity to place a workload on the intended node type.

Slurm GRES similarly models GPU types as generic resources and lets a job request a specific type and count. Ray defines accelerator resources for NVIDIA GPUs, AMD GPUs, Neuron cores, TPUs, and other devices, then assigns tasks or actors to nodes with the requested resource.

These systems solve allocation. They do not determine whether a model is correct on that accelerator, whether the serving engine has the required kernel, or whether the workload meets its latency target. A portable control plane must make placement eligibility a product of qualification:

eligible pool =
  supported model
  + approved artifact
  + compatible request limits
  + passing evaluation
  + healthy capacity
  + acceptable economics

Only after those gates pass should the router compare eligible pools.

Route on workload shape and evidence

A lowest-cost accelerator rule is too crude. Inference cost changes with context length, prefill-to-decode ratio, output length, concurrency, batch formation, cache reuse, quantization, and the amount of idle capacity held to protect latency.

The routing decision needs a qualification matrix:

Decision input	Why it changes placement
Model and revision	Operator and architecture support change between releases
Precision and quantization	Available kernels and acceptable accuracy loss differ
Context length	Memory consumption, compilation shapes, and cache pressure change
Prefill versus decode	The two phases stress compute and memory differently
Latency objective	A cheap saturated pool can violate the service contract
Batchability	Offline work can use capacity that interactive traffic cannot
Geography and data policy	Some pools may be ineligible before performance is considered
Current queue and warm capacity	A nominally efficient pool can be expensive after delay and cold start
Measured cost per accepted output	Hardware-hour price alone omits utilization and failed work

Routing also occurs at more than one level. Ray Serve’s routing documentation distinguishes model-level ingress routing from replica selection. A cross-hardware control plane needs both: first select the qualified implementation and pool, then let the local serving system select a replica using queue, cache, and locality signals.

The broad compatibility pool plays an important role. New models, unusual operators, rapidly changing context requirements, and workloads without enough evaluation evidence stay there. Specialist pools earn traffic when the workload becomes stable enough to compile, measure, and govern.

Normalize outcomes, preserve native telemetry

Observability is portable only at the service layer. Time to first token, inter-token latency, total response time, requests per second, tokens per second, error rate, evaluation score, queue delay, and cost per accepted output can be defined consistently.

The lower-level evidence remains native. GPU occupancy, NeuronCore utilization, compiler-cache behavior, HBM counters, collective timing, and device power are not one universal metric family. Flattening them into a single “accelerator utilization” number destroys the details needed to diagnose a pool.

The clean design keeps two views:

Normalized service telemetry compares the outcome delivered to the application.
Backend-native telemetry explains why one implementation produced that outcome.

Economic policy should use both. A pool that has a lower device-hour price but compiles frequently, rejects more requests, requires excess standby capacity, or produces lower-quality outputs can have the higher service cost.

Compare service cost, not accelerator price

A portable control plane needs one economic denominator across every pool:

service cost per accepted unit =
  reserved accelerator and host capacity
  + network, storage, and energy
  + amortized model-porting and software work
  + operator and support cost
  + failed, retried, and quality-rejected work
  ------------------------------------------------
  accepted outputs that met the service objective

The denominator can be accepted tokens, images, training steps, or completed jobs, but it must include the quality and latency objective. Utilization belongs inside the calculation because reserved capacity accrues cost while idle. Availability belongs inside it because failed work and standby capacity reduce the accepted output produced by the same reservation.

This illustrative monthly comparison shows why hardware price alone is misleading:

Monthly component	Broad compatibility pool	New specialist pool
Reserved compute, hosts, and fabric	$72,000	$45,000
Energy and facility allocation	$8,000	$4,000
Amortized porting and software work	$6,000	$15,000
Operations, support, and standby capacity	$8,000	$12,000
Failed, retried, or rejected work	$4,000	$6,000
Accepted output meeting the service objective	140 billion tokens	90 billion tokens
Service cost per million accepted tokens	$0.70	$0.91

The numbers are hypothetical; the mechanism is not. The specialist has lower compute and energy cost but loses the comparison because it carries porting work, idle capacity, and failed output across a smaller qualified workload. If model coverage grows, utilization rises, and software work is amortized across more accepted output, the same pool can later become the cheaper path.

The control plane should therefore store both the current service cost and the evidence behind it: measurement interval, reservation model, utilization, failure rate, quality threshold, energy boundary, labor allocation, and software version. Routing on a number without that provenance recreates the same false portability problem at the economic layer.

What is still missing

The software projects above provide pieces of this architecture, not one finished control plane. Schedulers allocate devices. Serving engines execute models. Model registries track artifacts. Gateways route requests. Telemetry systems collect measurements.

The missing layer is a durable qualification record connecting all of them:

which logical model maps to which hardware-specific artifact;
which evaluation approved that artifact;
which request shapes it supports;
which software and driver versions produced the result;
which service and economic thresholds it currently meets;
which policy moved traffic and how to reverse that decision.

Without that record, “multi-accelerator” becomes a collection of independent deployments sharing an endpoint. With it, hardware diversity becomes governable.

So what

Do not wait for a universal AI runtime. The compilers, kernels, collectives, and memory systems are moving in the opposite direction—toward deeper specialization.

Build portability where it holds: the application contract, model identity, evaluation policy, workload router, service telemetry, and economic decision. Keep execution pools separate, qualify every implementation, and route only across proven choices.

The open thread is how much control-plane standardization is possible before the policy itself becomes coupled to the dominant serving engine. If every backend exposes different cache state, queue semantics, compilation behavior, and power evidence, the router may remain portable in name while its best decisions depend on vendor-specific signals.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

The Interconnect Determines How Much of the Chip You Can Use

Amit — Wed, 23 Sep 2026 04:51:41 +0000

An accelerator can finish its arithmetic and still leave the system waiting.

That is the architectural shift behind rack-scale AI. More processors do not automatically produce a faster computer. They produce a larger distributed system whose useful performance depends on moving weights, activations, gradients, expert tokens, and cache state between the right processors at the right time.

At rack and pod scale, data movement can constrain useful accelerator performance more than arithmetic. The chip still matters. But the interconnect increasingly decides how much of the chip the system can use.

TL;DR

Scale-up fabrics connect accelerators into one tightly coupled compute domain; scale-out networks connect servers, racks, and pods. They solve different communication problems.
NVLink and NVSwitch are a mature vertically owned scale-up stack. UALink and Ethernet-based scale-up proposals create open alternatives, but a published specification is not a deployed ecosystem.
InfiniBand and Ethernet/RoCE both support large AI clusters. Their useful performance depends on NIC placement, topology, congestion control, collectives, and operations—not the link rate alone.
NCCL, RCCL, and related libraries turn collective operations into traffic patterns. They are part of the performance architecture, not a thin software wrapper.
Copper remains efficient over short electrical reaches. Pluggable optics dominate longer links. Co-packaged optics and optical I/O move conversion closer to the switch or accelerator as electrical reach and front-panel power become harder constraints.

Scale-up and scale-out are different systems

The cleanest way to understand an AI fabric is to ask where the tightly coupled compute domain ends.

Scale-up connects accelerators that exchange data frequently enough to behave like one larger machine. Latency, bandwidth, memory semantics, ordering, and collective behavior all matter. The fabric may remain inside a server, cross a rack, or eventually span several racks.

Scale-out connects those compute domains into a larger cluster. It carries distributed training collectives, inference traffic, model and checkpoint movement, storage access, and control traffic across a broader topology.

flowchart LR
    subgraph R1["Rack-scale compute domain"]
        A1["Accelerator"]
        A2["Accelerator"]
        A3["Accelerator"]
        SU1["Scale-up switches<br/>NVSwitch, UALink, or another fabric"]
        A1 <--> SU1
        A2 <--> SU1
        A3 <--> SU1
    end

    NIC1["NIC or SuperNIC"]
    SO["Scale-out network<br/>InfiniBand or Ethernet"]
    NIC2["NIC or SuperNIC"]

    subgraph R2["Second compute domain"]
        SU2["Scale-up fabric"]
        B1["Accelerator"]
        B2["Accelerator"]
        SU2 <--> B1
        SU2 <--> B2
    end

    SU1 <--> NIC1
    NIC1 <--> SO
    SO <--> NIC2
    NIC2 <--> SU2

The distinction is architectural, not merely physical. NVIDIA's NVLink carries tightly coupled GPU traffic, while its Quantum InfiniBand and Spectrum-X Ethernet platforms connect systems across the scale-out network. A cable does not become scale-up because it is fast, and Ethernet does not become scale-out-only because that is its conventional role.

That last boundary is now moving. The UALink Consortium defines an open 200 Gb/s-per-lane scale-up specification for accelerator pods. Broadcom's Scale-Up Ethernet framework applies Ethernet technology to the scale-up domain. AMD's UALink over Ethernet, or UALoE, uses that broader direction to connect 72 MI455X GPUs in the Helios rack. The physical and link technologies can converge while the workload contract remains different.

The scale-up contest is also an ownership contest

NVLink's advantage is not only its published bandwidth. NVIDIA owns the GPU interface, NVSwitch silicon, firmware, topology, collective library, rack design, and much of the qualification process.

Blackwell NVLink 5 provides 1.8 TB/s of aggregate bidirectional bandwidth per GPU. Rubin NVLink 6 doubles that published endpoint figure to 3.6 TB/s. In NVL72 systems, NVSwitch extends the tightly coupled domain across 72 GPUs in one rack.

Those figures are endpoint totals, not achieved application throughput or one-way fabric bisection bandwidth. Blackwell systems are deployed, while NVIDIA describes Vera Rubin as ramping into full production. Both belong to an integrated product stack—not an interface specification waiting for implementations.

UALink takes a different path. Its 200G 1.0 specification separates the fabric definition from any single accelerator vendor and supports scale-up domains as large as 1,024 accelerators. That creates room for accelerators, switch silicon, cables, management components, and systems from different suppliers.

The UALink specification family now also separates common, data-link, physical-layer, chiplet, and manageability documents. The open boundary moves integration responsibility outward. Compatibility requires more than matching electrical lanes: memory semantics, error handling, management, firmware, topology discovery, collectives, and system qualification must converge across companies.

Fabric or initiative	Primary boundary	Ownership model	Status by Aug. 30, 2026	What the label does not prove
NVLink + NVSwitch	NVIDIA GPU scale-up	Vertically controlled by NVIDIA	Mature across several generations; NVL72 in production	Application bandwidth, availability, or compatibility with non-NVIDIA accelerators
UALink specifications	Open accelerator scale-up	Consortium specifications and multi-vendor implementations	200G 1.0 plus newer component specifications published; implementation ecosystem emerging	Shipping interoperable systems at NVLink maturity
Broadcom Scale-Up Ethernet	Ethernet-based accelerator scale-up	Merchant switch silicon plus ecosystem	Framework and products published; ecosystem integration under way	That ordinary data-center Ethernet automatically meets scale-up semantics
AMD UALoE	72-GPU Helios scale-up domain	AMD implementation using Ethernet switch silicon	Helios architecture introduced in 2026	Independent interoperability or NVLink-equivalent operational maturity
Ultra Ethernet 1.0.3	AI/HPC scale-out transport	Consortium specification	Current specification published; products and software arriving by vendor	A replacement for scale-up fabrics or an installed multi-vendor fleet

These rows are not benchmark peers. NVLink is a deployed proprietary stack. UALink is an open scale-up specification. Ultra Ethernet is a scale-out transport specification. Scale-Up Ethernet is a family of merchant-Ethernet approaches aimed at a new boundary.

The strategic question is whether an open ecosystem can coordinate those layers quickly enough to behave like one system.

Link speed is not collective performance

Distributed models do not send a uniform stream of independent packets. They execute collective operations.

An all-reduce combines values across workers and returns the result to each worker. An all-gather assembles shards. A reduce-scatter combines and redistributes them. Mixture-of-experts models can produce all-to-all traffic as tokens move toward different experts.

NCCL implements these collectives for NVIDIA GPUs across PCIe, NVLink, InfiniBand, and Ethernet paths. RCCL provides the corresponding collective library in AMD's ROCm stack. Intel maintains oneCCL for distributed communication across its supported compute environments.

The library chooses algorithms and routes based on topology, message size, available transports, and system configuration. A ring can use links efficiently for large transfers. A tree can reduce latency for other message sizes. Hierarchical algorithms can reduce data inside a server or rack before crossing the scale-out network.

flowchart TB
    OP["Model operation<br/>all-reduce, all-gather, all-to-all"]
    LIB["Collective library<br/>NCCL, RCCL, or oneCCL"]
    PLAN["Algorithm and topology choice<br/>ring, tree, hierarchy, channels"]
    LOCAL["Local movement<br/>HBM, PCIe, scale-up fabric"]
    NIC["NIC path<br/>RDMA, GPU memory access, transport offloads"]
    FABRIC["Scale-out fabric<br/>switches, routes, congestion control"]
    RESULT["Useful synchronized work"]

    OP --> LIB --> PLAN
    PLAN --> LOCAL
    PLAN --> NIC --> FABRIC
    LOCAL --> RESULT
    FABRIC --> RESULT

This stack explains why a nominally faster network can deliver less useful performance. A job can lose time through a poor rank-to-NIC mapping, oversubscribed links, uneven paths, head-of-line blocking, retransmission, collective imbalance, or one slow worker.

The accelerator metric that matters is not peak FLOPS. It is the fraction of execution time spent doing useful compute rather than waiting for communication or synchronization.

InfiniBand and Ethernet are systems, not cable choices

InfiniBand entered AI clusters with an integrated remote-memory and lossless-fabric model. NVIDIA's Quantum-2 platform combines switches, ConnectX adapters, RDMA, congestion management, telemetry, and its SHARP in-network-reduction capability. In supported Quantum deployments, the fabric can participate in selected collective operations instead of treating every reduction as endpoint-only traffic.

Ethernet starts with a broader multi-vendor base. Running AI collectives over Ethernet commonly uses RoCE, which carries RDMA over Ethernet. That removes repeated CPU and kernel involvement from the data path, but it does not remove the need to engineer congestion behavior.

Traditional RoCE deployments can combine priority flow control, explicit congestion notification, data-center quantized congestion notification, adaptive routing, and careful buffer configuration. NVIDIA's Spectrum-X packages Ethernet switches, BlueField SuperNICs, telemetry, congestion control, and adaptive routing as one AI-networking platform.

The Ultra Ethernet Consortium's current 1.0.3 specification goes further by defining an AI/HPC transport that includes packet spraying across paths, selective retransmission, congestion management, and collective-communication support. It is an attempt to make large Ethernet fabrics less dependent on every deployment recreating the same behavior through proprietary combinations.

Layer	InfiniBand example	Ethernet example	Operational owner
Link and switching	Quantum switches and InfiniBand links	Ethernet switches and links	Network platform team
Endpoint	ConnectX HCA	NIC or SuperNIC	Server and network teams
Remote-memory transport	InfiniBand RDMA	RoCE or Ultra Ethernet transport	NIC, driver, and fabric stack
Congestion behavior	InfiniBand controls and adaptive routing	PFC/ECN/DCQCN, vendor controls, or UEC mechanisms	Fabric engineering
Collective layer	NCCL/RCCL plus optional in-network operations	NCCL/RCCL/oneCCL plus transport integration	Accelerator platform team
Job placement	Topology-aware scheduler and rank placement	Topology-aware scheduler and rank placement	Cluster platform team

This is why “InfiniBand versus Ethernet” is an incomplete comparison. The real unit is endpoint hardware, switching, transport, congestion control, collectives, topology, and operations.

NICs and DPUs own different work

A NIC terminates the network path. A modern AI NIC can also expose RDMA, access accelerator memory, steer traffic across rails, timestamp packets, and offload transport work.

NVIDIA uses the term SuperNIC for a high-performance network endpoint optimized for accelerator traffic. It positions BlueField-3 SuperNIC around GPU-to-GPU networking, while its DPU configuration adds programmable Arm cores and infrastructure offloads.

A DPU has a broader ownership boundary. It can run networking, storage, security, isolation, and management functions away from the host CPU. AMD's Pensando infrastructure products follow that broader offload model.

Neither device fixes a poor fabric by itself. A faster NIC attached through the wrong PCIe root, mapped to distant GPUs, or connected to an oversubscribed leaf still creates a slow communication path. Hardware topology and software placement have to agree.

Copper reaches a wall before optics does

The physical medium is becoming an architectural choice.

Passive copper is attractive inside a tray or across short rack distances because it avoids optical conversion and can deliver low power and low latency. Active electrical cables extend that reach by adding signal conditioning. As lane rates rise, electrical loss increases and the practical reach shrinks.

Pluggable optical modules move the electrical-to-optical conversion to the switch faceplate. They provide longer reach and replaceable modules, but every port consumes front-panel area and conversion power. Linear pluggable optics remove part of the digital signal processing from the module to reduce power, while placing tighter signal-integrity requirements on the host system.

Co-packaged optics moves the optical engines beside the switch ASIC. Broadcom's co-packaged-optics portfolio and NVIDIA's silicon-photonics switch platforms shorten the high-speed electrical path and increase optical bandwidth density. They also couple optics more closely to switch packaging, cooling, yield, and service procedures.

flowchart TB
    NEED["Required reach, bandwidth density,<br/>power, and service model"]
    TRACE["On-board traces<br/>inside board or package"]
    COPPER["Passive or active copper<br/>tray and short rack links"]
    PLUG["Pluggable optics<br/>conversion at faceplate"]
    CPO["Co-packaged optics<br/>conversion beside switch ASIC"]
    OIO["Optical I/O<br/>conversion beside compute or memory"]

    NEED --> TRACE
    NEED --> COPPER
    NEED --> PLUG
    NEED --> CPO
    NEED --> OIO

Optical I/O pushes the boundary closer to compute. Ayar Labs' TeraPHY packages optical chiplets for chip-to-chip communication. Lightmatter Passage combines optical connectivity with packaging intended to link processors and memory. Celestial AI's Photonic Fabric targets optical scale-up and memory connectivity.

These products do not share one maturity level or deployment model:

Optical boundary	Public maturity by Aug. 30, 2026	Evidence boundary
Pluggable optics	Shipping infrastructure across data-center networks	Mature module ecosystem; power and faceplate density remain constraints
Broadcom CPO and NVIDIA silicon photonics	Commercial portfolios and announced switch platforms	Product availability varies by platform and configuration
Ayar Labs optical I/O	Chiplet products and customer sampling programs	Not yet a broadly interchangeable accelerator interface
Lightmatter Passage and Celestial AI Photonic Fabric	Product programs and partner integrations	Deployment scale and production economics remain vendor-reported

The direction is consistent even though the maturity is not: when electrical reach, connector density, and SerDes power become system limits, designers can move the optical conversion point inward. That choice is driven by distance and packaging boundary, not one universal copper-to-optics migration sequence.

The interconnect changes who owns the computer

NVIDIA's model places the scale-up link, switch, NIC, network, collective library, and rack qualification under one platform owner. That reduces the number of boundaries a system integrator must reconcile.

An open stack distributes those responsibilities. An accelerator vendor can implement UALink. A merchant-silicon supplier can provide scale-up and scale-out switches. A NIC vendor can own RDMA and congestion functions. An OEM can assemble the rack. A collective library can adapt the topology to the framework.

That structure creates supplier choice. It also creates a harder acceptance test. A specification match does not establish that the assembled system sustains collectives under congestion, survives link failures, maps jobs correctly, or exposes one support boundary when performance falls.

The practical evaluation unit is therefore not the accelerator card. It is:

the tightly coupled scale-up domain;
the scale-out fabric joining those domains;
the collective software translating model operations into traffic;
the topology and congestion mechanisms preserving progress;
the physical medium carrying the required bandwidth within the power and service envelope.

What remains unproven

Vendor bandwidth figures are abundant. Comparable useful-work evidence is not.

A credible system comparison needs the same model, precision, collective mix, accelerator count, topology, message sizes, failure policy, and power boundary. It also needs achieved bandwidth, communication-to-compute overlap, tail behavior under congestion, job completion time, and recovery behavior—not only a clean microbenchmark.

Open fabrics face a second proof burden: interoperability. UALink and Ultra Ethernet can publish complete specifications before multiple vendors ship independently qualified systems. That is normal standards work, but it creates a gap between protocol maturity and operational maturity.

So what

The next AI platform decision cannot stop at chip throughput. The fabric determines the size of the useful computer, the software determines how traffic enters it, and optics determines how far that computer can extend before power and signal integrity pull it apart.

The open question is whether the industry can create a genuinely interchangeable scale-up and scale-out ecosystem—or whether every high-performing AI cluster will remain a platform-specific computer whose compatibility ends exactly where its collective software, congestion controls, and optical packaging diverge.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

Groq Makes the Compiler Part of the Processor

Amit — Wed, 23 Sep 2026 04:51:06 +0000

Groq's architectural bet is that an inference processor should execute a plan, not discover one while the model is running.

The Language Processing Unit removes much of the reactive machinery found in conventional processors: hardware caches, dynamic arbitration, speculative execution, and out-of-order scheduling. Its compiler decides where operations run, where data lives, and when each value moves. The processor then follows that schedule across a large on-chip SRAM system.

That changes the unit worth evaluating. One LPU does not hold a modern large language model. The useful machine is the connected system of LPUs, its compiler-generated schedule, and the cloud service around it.

TL;DR

Groq moves instruction scheduling, tensor placement, and much data movement into the compiler, leaving simpler hardware to execute a predetermined program.
SRAM supplies high bandwidth and predictable access time, but limited capacity forces model weights and execution across many connected LPUs.
Deterministic chip execution reduces one source of latency variation; it does not remove API networking, admission control, queueing, or shared-service contention.
Groq is strongest where one user's token rate matters: voice, coding loops, agents, and other interactive inference. That is different from maximizing total batched throughput.
NVIDIA did not acquire Groq. It licensed Groq's inference technology non-exclusively and hired Groq leaders and staff; Groq remains independent and continues to operate GroqCloud.

The processor begins in the compiler

Modern GPUs are also compiler-driven machines. Framework graphs are lowered into kernels, kernels are compiled, and serving engines plan memory and batch work. The distinction is what remains unresolved at runtime.

A GPU retains hardware mechanisms that react to events while a program executes: caches answer memory requests, arbiters resolve contention, and schedulers choose from ready work. Groq's original Tensor Streaming Processor paper describes the opposite cut. It eliminates reactive elements such as caches and arbiters so the compiler can reason about execution timing precisely.

The hardware is functionally sliced. Separate regions perform matrix operations, vector arithmetic, memory access, data movement, and instruction control. Values stream between those regions through a software-controlled interconnect. The compiler maps the model graph onto that physical machine and emits a cycle-level schedule.

Groq describes this as a programmable assembly line: each functional unit receives an instruction, takes data from a named stream, performs its operation, and places the result onto another stream. Hardware does not wait to discover which path wins arbitration because the compiler has already assigned the path and time.

System decision	Conventional GPU-oriented execution	Groq LPU execution
Instruction readiness	Hardware and runtime mechanisms select ready work	Compiler places work into a predetermined schedule
Data locality	Registers, caches, HBM, and runtime-managed movement	Compiler assigns tensors across distributed SRAM
Resource contention	Hardware arbitration resolves competing requests	Compiler attempts to prevent conflicts before execution
Timing model	Performance can vary with cache behavior and contention	Scheduled accelerator execution is predictable cycle by cycle
Optimization boundary	Kernels, runtime, serving engine, and hardware	Model compiler, LPU fabric, and serving system

This is not “software instead of hardware.” The compiler works because the hardware was built to expose timing and data movement directly. Groq makes the compiler part of the processor's effective control plane.

flowchart LR
    GRAPH["Model graph"]
    COMPILER["Groq compiler<br/>partition + place + schedule"]
    ARTIFACT["Compiled execution plan"]

    subgraph SYSTEM["The useful compute boundary"]
        LPU1["LPU<br/>compute + SRAM"]
        LPU2["LPU<br/>compute + SRAM"]
        LPUN["More LPUs<br/>compute + SRAM"]
        FABRIC["RealScale interconnect<br/>scheduled chip-to-chip movement"]

        LPU1 <--> FABRIC
        LPU2 <--> FABRIC
        LPUN <--> FABRIC
    end

    API["GroqCloud or system API"]
    REQUESTS["Inference requests"]

    GRAPH --> COMPILER --> ARTIFACT --> SYSTEM
    REQUESTS --> API --> SYSTEM
    SYSTEM --> API

SRAM makes speed possible—and scale necessary

Groq places the primary working memory on the processor in SRAM. SRAM is faster and more predictable than fetching weights from off-chip DRAM or HBM, but it occupies much more silicon area per stored bit.

The first-generation chip described in the architecture literature contains approximately 220 MB of SRAM. Groq's current eight-accelerator GroqNode lists 1.76 GB of aggregate on-die memory, consistent with eight chips at that capacity. A 70-billion-parameter model requires roughly 35 GB merely to store weights at four bits per parameter, before accounting for scales, activations, code, and key-value cache.

One chip is therefore the wrong capacity boundary. Groq partitions model weights and operations across multiple LPUs, then streams activations through the connected machine. GroqNode exposes 88 chip-to-chip connectors and is positioned for multi-server and multi-rack scaling without a separate external switch in the LPU fabric.

The result resembles a pipeline more than a pool of interchangeable accelerators. Each stage owns part of the model and passes intermediate values to the next stage according to the compiled schedule. Adding LPUs expands both SRAM capacity and execution resources, but it also makes partitioning, communication, and pipeline utilization part of the performance result.

Groq's latest rack specification makes that system boundary explicit. Its inference platform page lists 256 LPUs, 128 GB of on-chip SRAM, and 40 PB/s of aggregate SRAM bandwidth per rack for the Groq 3 generation. Those are vendor specifications, not independently measured application results.

The architectural trade is clear: spend silicon and system complexity on low-latency SRAM and predictable movement, then recover model capacity by joining many processors into one scheduled machine.

Deterministic execution is not deterministic service

Groq uses “deterministic” precisely at the accelerator layer. Given the same compiled program, the LPU schedule specifies when instructions execute and when values move. The hardware does not introduce cache misses or arbitration delays that change the cycle count.

A cloud request crosses a larger system:

client network
  → API gateway
  → admission control and queue
  → model replica selection
  → prompt processing
  → scheduled LPU execution
  → streamed response

Only one part of that chain is the deterministic machine. Request bursts can still create queues. Prompts have different lengths. Tool calls introduce external dependencies. Network paths vary. A shared service can also reserve, batch, or route work differently as load changes.

This distinction matters because latency has several meanings:

Metric	What it measures	What it can hide
Time to first token	Delay before generation begins	Prompt length, queueing, and network transit
Output speed	Tokens delivered per second after generation starts	Queueing before the first token and aggregate capacity
End-to-end response time	User wait for the complete answer	Different answer lengths and reasoning-token behavior
Aggregate throughput	Total work completed by the serving system	Whether any individual user receives tokens quickly
Tail latency	Slow requests at the 95th or 99th percentile	Average results that look healthy while some users wait

Groq's strongest public evidence is output speed. Its GroqCloud model catalog publishes model-specific rates ranging into hundreds of tokens per second, with smaller models reaching about 1,000 tokens per second. These are service specifications and vary by model.

Artificial Analysis independently measures Groq's public endpoints. At the August 30, 2026 cutoff, it reports the fastest tracked Groq configuration near 980 output tokens per second. For the same Llama 3.3 70B model, its provider comparison measures Groq at roughly 298 output tokens per second—more than 20 times one of the slower tracked providers.

That is useful independent evidence for interactive speed. It is not a rack-throughput benchmark, a power measurement, or a total-cost comparison. Artificial Analysis measures hosted APIs; it does not expose how many chips, replicas, queued users, or reserved capacity sit behind each endpoint.

The distinction also explains Groq's workload fit. Fast sequential generation is valuable for conversational voice, coding agents waiting inside a loop, interactive search, and workflows that call a model repeatedly. Offline jobs with large batches care more about total tokens per dollar and total tokens per rack than the token rate of one stream.

SemiAnalysis argues that SRAM-heavy systems trade capacity available for weights and key-value cache against high single-user token rates, while GPUs can win aggregate throughput and cost through batching. That is the right comparison boundary: latency-first and throughput-first inference are different products even when both report “tokens per second.”

GroqCloud is the complete product

The LPU does not arrive as a general accelerator card with a CUDA-like programming surface. GroqCloud presents supported models through an API that is largely compatible with the OpenAI client pattern. The compiler, model artifacts, partitioning, and machine topology remain behind the service boundary.

That lowers adoption work for a supported model: change the endpoint and test behavior. It also moves control to the provider. Model availability, context limits, rate limits, pricing, quantization, and rollout schedules follow the GroqCloud catalog rather than the customer's own accelerator fleet.

The fit is strongest when:

the model and operators are supported by Groq's compiler and service;
low per-user generation latency changes the product experience;
the workload is stable enough to justify a compiled deployment;
API consumption is preferable to owning the hardware and compiler lifecycle.

The fit weakens when:

the workload requires custom or rapidly changing operators;
the model is absent from the supported catalog;
large dynamic key-value caches dominate memory demand;
maximum batched throughput matters more than individual stream speed;
infrastructure ownership or artifact control is mandatory.

GroqCloud therefore sells more than processor time. It packages a compiler-controlled distributed machine behind a familiar inference contract.

NVIDIA licensed the technology; it did not acquire Groq

The NVIDIA relationship is easy to overstate because technology and senior personnel moved together.

On December 24, 2025, Groq announced a non-exclusive inference technology licensing agreement with NVIDIA. Founder Jonathan Ross, president Sunny Madra, and other Groq staff joined NVIDIA. The same announcement says Groq would remain an independent company under CEO Simon Edwards and that GroqCloud would continue operating.

Reuters described the structure as a technology license plus executive hiring that stopped short of a formal acquisition. Reported deal values remain reporting, not disclosed transaction terms from either company.

The distinction now appears in two product paths:

GroqCloud remains Groq's independently operated inference service.
NVIDIA Groq 3 LPX uses licensed Groq technology inside NVIDIA's Vera Rubin platform.

NVIDIA's Groq 3 LPX architecture description specifies a 256-LPU rack with 128 GB of SRAM, 40 PB/s of SRAM bandwidth, and 640 TB/s of rack-scale communication. NVIDIA announced the product in full production in August 2026, with initial cloud availability planned later in the year.

LPX does not establish that every model should run entirely on an LPU rack. NVIDIA presents a heterogeneous design coordinated by Dynamo:

Vera Rubin GPUs can process prefill, attention, and key-value cache.
LPX can execute latency-sensitive feed-forward or mixture-of-experts decode stages.
The two systems exchange intermediate activations according to the selected serving pattern.

NVIDIA calls these patterns prefill-decode disaggregation, attention-FFN disaggregation, and external-drafter speculative decoding. The published numbers—including 3,431 output tokens per second on one 100K-context test—come from NVIDIA-operated LPX systems and vendor-selected configurations, even when Artificial Analysis supplied the test harness.

The evidence floor is narrower than “Groq replaces GPU inference.” LPX shows that a latency-specialized SRAM machine can become one stage inside a larger GPU inference system. GPUs retain the memory-heavy and general-purpose work; LPUs handle selected decode paths where deterministic scheduling and SRAM bandwidth matter most.

What's missing

Groq has credible evidence for high output speed and a coherent explanation for why the architecture produces it. The public record remains thinner on aggregate system economics.

A complete comparison needs the same model, quality target, prompt distribution, context length, concurrency, and service-level objective across systems. It also needs measured rack power, total output throughput, tail latency, compiler coverage, failed-compilation rate, capacity reservation, and cost under sustained load.

No current public MLPerf result establishes a standardized Groq-to-GPU comparison across those boundaries. GroqCloud API benchmarks prove that users can receive tokens quickly. They do not reveal whether the underlying rack delivers the best aggregate tokens per watt or tokens per dollar.

So what

Groq's contribution is larger than a fast inference endpoint. It redraws the processor boundary around the compiler and the connected machine.

SRAM makes each operation fast and predictable. The compiler turns that memory and compute into a timed execution plan. The interconnect lets many small memory domains behave like one model-serving pipeline. GroqCloud turns that system into an API product, while NVIDIA's licensed LPX path places the same architectural idea inside a heterogeneous GPU platform.

The open question is operational: as inference splits into prefill, attention, feed-forward, speculative decoding, and cache management, does the winning system use one broad accelerator—or does the compiler become the place where several specialized processors are assembled into one service?

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

d-Matrix Moves the Math Into Memory

Amit — Wed, 23 Sep 2026 04:50:30 +0000

Inference is becoming a memory-placement problem.

A modern accelerator can perform arithmetic faster than it can repeatedly move model weights into the compute engines. Autoregressive decoding makes the mismatch visible: every generated token requires another pass through the model, often at a batch size small enough that raw matrix throughput is not the limiting resource.

d-Matrix Corsair attacks that boundary by placing digital arithmetic beside a large SRAM tier. The system still needs off-chip capacity memory, PCIe, Ethernet, compilation, quantization, and distributed execution. The architectural bet is narrower and more interesting than “replace the GPU”: keep the latency-critical working set near the arithmetic and use the rest of the memory hierarchy deliberately.

TL;DR

Corsair combines digital in-memory compute with 2 GB of high-bandwidth SRAM per card and up to 256 GB of off-chip capacity memory.
The useful deployment boundary expands from a paired PCIe card to an eight-card server and then a 64-card rack because larger models must be partitioned across multiple SRAM pools.
d-Matrix reduces physical adoption friction by using standard PCIe and Ethernet, but model compilation, compression, partitioning, and numerical validation remain accelerator-specific.
The clearest near-term pattern may be heterogeneous inference: use a specialist for a memory-bound phase while GPUs retain compute-heavy phases.

The bottleneck is the trip to memory

The arithmetic in transformer inference is not uniform.

Prefill processes many prompt tokens in parallel and can keep a large compute engine busy. Decode generates tokens sequentially. At small batch sizes, the accelerator repeatedly reads model weights to produce relatively little new work. The service becomes sensitive to memory bandwidth, placement, and communication latency.

Corsair puts a digital in-memory compute engine beside what d-Matrix calls Performance Memory. One card provides 2 GB of that SRAM at a published 150 TB/s, plus as much as 256 GB of Capacity Memory off chip. A two-card pair doubles those figures and connects through the company’s DMX Bridge.

Those two tiers serve different jobs:

Memory tier	Published card capacity	Architectural job	Main constraint
Performance Memory	2 GB SRAM	Keep latency-sensitive weights and activations next to the compute engines	Capacity
Capacity Memory	Up to 256 GB	Hold larger models and support throughput-oriented execution	Lower bandwidth and greater movement cost
Distributed rack memory	128 GB aggregate SRAM and up to 16.4 TB capacity in the 64-card reference rack	Partition larger models and requests across cards and servers	Fabric latency, partitioning, and software coordination

This is not one flat pool. The compiler and runtime must decide what fits in the fast tier, how the model is divided, and when data moves between cards or servers.

flowchart LR
    MODEL["Model checkpoint"]
    COMP["Aviator compiler<br/>compression and partitioning"]
    CAP["Capacity Memory<br/>larger model storage"]
    PERF["Performance Memory<br/>SRAM beside compute"]
    DIMC["Digital in-memory<br/>compute engines"]
    FABRIC["DMX Bridge, PCIe,<br/>or Ethernet"]
    NEXT["Next card or<br/>inference stage"]

    MODEL --> COMP
    COMP --> CAP
    COMP --> PERF
    CAP --> PERF
    PERF <--> DIMC
    DIMC --> FABRIC
    FABRIC --> NEXT

The diagram captures the trade: less distance between arithmetic and the fast memory tier, followed by more responsibility for placement and distributed execution.

A card is not the complete product

Corsair uses an industry-standard full-height, full-length PCIe Gen5 form factor. That makes it physically familiar, but the deployment unit changes with the model.

The company describes three reference boundaries:

Boundary	Configuration	Published memory boundary	Intended use
Card pair	Two connected Corsair cards	4 GB Performance Memory; up to 512 GB Capacity Memory	Smaller models and low-latency stages
Server	Eight cards	16 GB Performance Memory; up to 2 TB Capacity Memory	Multi-card inference within one host
Rack	Eight servers, 64 cards	128 GB Performance Memory; up to 16.4 TB Capacity Memory	Larger models and higher request volume

The aggregate SRAM determines which model weights can remain in the fastest tier. d-Matrix describes smaller models as an appliance-level fit and larger models as a rack-level problem. That is why the system architecture matters more than the card specification.

Within a node, PCIe and DMX Bridge connect the accelerators. Across nodes, the design uses Ethernet. d-Matrix’s JetStream NIC is intended to let accelerators coordinate transfers without routing every step through the host control path. The company says each JetStream connects four Corsair cards to a leaf switch and extends accelerator-to-accelerator execution across servers.

That is still a distributed system. The architecture reduces some host-mediated handshakes; it does not remove switches, congestion, topology, failure handling, or the need to keep model partitions synchronized.

The software stack carries the memory policy

In-memory compute does not make a PyTorch model automatically executable.

The Aviator software stack contains model templates, compression tools, an MLIR-based compiler, a distributed inference engine, host runtime, device firmware, and operational tooling. It integrates with PyTorch and Triton DSL at the upper boundary, then produces artifacts for Corsair’s spatial architecture and numerical formats.

Software layer	Responsibility
Model Factory	Provides distributed model templates
Compressor	Converts weights into supported block floating-point formats
Compiler	Places and schedules operations for the accelerator
Inference Engine	Coordinates execution across multiple cards
Host Runtime	Manages interaction between server CPUs and cards
Chip Runtime	Launches device work and manages on-chip execution
Operational tools	Monitor cards and analyze workload performance

This is the portability boundary. A framework checkpoint may be shared with another accelerator, but the compiled artifact, quantization choices, partitioning plan, and performance profile are hardware-specific.

The numerical choice matters. Corsair supports microscaling block floating-point formats, including MXINT8 and MXINT4 modes. Lower precision reduces storage and movement, but every model still needs accuracy evaluation. Peak low-precision throughput is not proof that a particular model preserves quality or meets a production latency target.

Standard racks reduce one kind of risk

d-Matrix does not require a proprietary wafer-scale chassis or a new liquid-cooled megawatt rack. Corsair is a PCIe card, and the company positions air-cooled server and rack designs that use Ethernet for scale-out.

That can reduce facility integration work:

standard server mechanics;
familiar PCIe enumeration;
Ethernet-based rack networking;
incremental deployment beside existing accelerator pools;
independent scaling of specialist and GPU capacity.

It does not remove system integration. A 64-card model partition still needs validated servers, network topology, firmware, cooling, power, observability, and replacement procedures. The operator also needs a serving layer that knows which model artifact can run on which pool.

The deployment pattern is therefore less “one new universal accelerator” and more “one specialized pool behind a stable inference contract.”

Heterogeneous inference is the stronger first use

The most useful evidence for Corsair is not a vendor comparison claiming that one rack replaces a GPU fleet. It is a narrower experiment that gives the specialist one phase of the request.

A partner engineering study from Gimlet Labs evaluated Corsair for speculative decoding while leaving prefill and verification on GPUs. The target model was gpt-oss-120b; a 1.6-billion-parameter draft model ran on two Corsair cards. Gimlet reported a 2–10× improvement in interactivity over its GPU-only speculative-decoding configuration at matched energy-efficiency points.

The boundary matters:

the analysis combined measured and modeled data;
it modeled an 8K-input, 1K-output coding workload;
token-acceptance assumptions materially affected the result;
the exact comparison GPU was not disclosed;
the test evaluated one inference phase, not complete replacement of the target-model hardware.

That caveat does not weaken the result. It makes the result useful. A memory-bound draft phase fits SRAM-centric hardware differently from compute-heavy prefill and batched verification. The experiment supports a heterogeneous architecture where each phase runs on the hardware that matches its bottleneck.

flowchart LR
    REQ["Request"]
    PREFILL["Prefill<br/>GPU pool"]
    DRAFT["Draft tokens<br/>Corsair pool"]
    VERIFY["Verify tokens<br/>GPU pool"]
    ACCEPT{"Accepted?"}
    OUTPUT["Stream output"]

    REQ --> PREFILL
    PREFILL --> DRAFT
    DRAFT --> VERIFY
    VERIFY --> ACCEPT
    ACCEPT -->|yes| OUTPUT
    ACCEPT -->|continue| DRAFT

This is disaggregated inference in concrete form. It adds network hops and orchestration, so the specialist must save more time than the handoff consumes.

The evidence is still uneven

d-Matrix publishes large claims for Corsair, including tens of thousands of tokens per second and substantial improvements in latency, power efficiency, and total cost. Its launch announcement labels the performance and cost estimates preliminary and subject to change.

The product page also mixes several boundaries:

card-level peak arithmetic;
server and rack aggregate memory;
model-specific projected performance;
preliminary comparisons against H100;
capacity-mode and performance-mode configurations.

Those numbers are not interchangeable. A useful independent comparison needs the same model, precision, quality threshold, prompt and output lengths, batching policy, service-level objective, power boundary, and complete system cost.

No Corsair result appeared in the MLCommons Inference results available during this August 30, 2026 review. This is a bounded research finding, not proof that a submission cannot exist elsewhere. Public partner and integration announcements establish ecosystem activity, but they do not provide the same evidence as measured production utilization, availability, or cost per accepted token.

The roadmap adds another evidence boundary. d-Matrix says its Pavehawk 3DIMC test silicon is operating in the lab and that a future Raptor architecture will use stacked memory. Those are development milestones and targets, not shipping-product measurements.

What d-Matrix changes in the landscape

d-Matrix is not trying to recreate the entire NVIDIA stack.

It attacks one expensive path: inference workloads where moving weights dominates useful arithmetic. The physical interface remains PCIe and Ethernet. The differentiated layer is the memory-compute complex, followed by the compiler and runtime required to use it.

That makes d-Matrix easier to combine with other accelerators than a closed, facility-scale replacement architecture. It also means the surrounding control plane becomes critical. The model registry needs multiple compiled artifacts. The router needs to understand workload phases. Observability needs to compare latency, throughput, quality, energy, and handoff overhead across dissimilar pools.

The strategic question is not whether every token should run on in-memory compute. It is whether enough stable, memory-bound work can be isolated to keep the specialist busy without making the serving system harder to operate than the savings justify.

If heterogeneous inference becomes the normal design, the unresolved boundary moves upward: who owns the optimizer that decides when a request should cross from a GPU into a specialist—and what evidence proves that decision still wins after network, compilation, capacity, and operational costs are included?

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

When the Platform Operator Designs the Chip

Amit — Wed, 23 Sep 2026 04:49:55 +0000

When the operator of a cloud, model platform, or captive fleet designs the accelerator, the chip stops being the product. The product becomes a vertically integrated compute system: silicon, memory, interconnect, compiler, scheduler, and service assembled behind one operating boundary.

The thesis: when the platform operator owns those layers together, its infrastructure becomes the computer.

This is a different competitive model from selling an accelerator card. AWS Trainium, Google TPU, Microsoft Maia, Meta MTIA, and OpenAI Jalapeño all pursue it, but they are not at the same maturity or aimed at the same customer. Trainium and TPU are external cloud platforms. Maia is entering Azure infrastructure. MTIA is a captive production system. Jalapeño is working silicon still moving through qualification.

TL;DR

The real unit of platform-designed silicon is a managed system, not a chip: the operator controls compilation, topology, scheduling, capacity, and service delivery.
Trainium exposes the cleanest cloud hierarchy—NeuronCore to chip to UltraServer to UltraCluster—while TPU has the deepest published history of co-designing matrix engines, XLA, optical fabrics, and pod scheduling.
Maia, MTIA, and Jalapeño validate the same vertical-integration thesis at different stages, but their public evidence does not yet support treating them as interchangeable cloud products.
The trade is architectural: tighter co-design can improve utilization and economics, while moving portability upward from device binaries to models, frameworks, APIs, and workload-routing policy.

The platform is the system boundary

A merchant accelerator has to fit into systems that other companies assemble. A platform-designed accelerator starts with a different constraint: it can assume the operator also controls the host, network, rack topology, compiler fleet, workload scheduler, and service API.

That changes where optimization happens. A compiler can target known memory layouts. A scheduler can place a job into a known accelerator topology. The network can expose collectives that match the chip. Capacity planning can treat thousands of devices as one managed pool. The customer requests instances, slices, reservations, pods, or a managed model service rather than buying the underlying board.

flowchart TB
    A["Model and framework<br/>PyTorch, JAX, service API"] --> B["Compiler and graph optimizer<br/>Neuron or XLA-class layer"]
    B --> D["Capacity scheduler<br/>selects an available topology"]
    D --> E["Selected cluster, pod,<br/>or accelerator slice"]
    E --> C["Device runtime and<br/>collective libraries"]
    C --> F["Scale-up server or<br/>accelerator domain"]
    F --> G["Accelerator + HBM +<br/>local interconnect"]

    H["Cloud control plane"] --> B
    H --> D
    H --> E
    H --> F

The diagram is the important part of the argument. The platform operator is not merely substituting one arithmetic engine for another. It is closing the control loop from model graph to physical placement.

Trainium turns EC2 into the machine

Trainium3 makes the hierarchy unusually explicit. One chip contains eight NeuronCores and 144 GB of HBM3e with 4.9 TB/s of memory bandwidth. A trn3.48xlarge instance contains 16 chips. A Trn3 UltraServer joins nine instances—144 chips—through NeuronSwitch-v1 and a 28.8 TB/s NeuronFabric scale-up domain. UltraCluster 3.0 then connects UltraServers through Elastic Fabric Adapter networking.

That hierarchy matters more than any isolated TOPS number:

NeuronCore
  → Trainium3 chip
  → EC2 instance
  → Trn3 UltraServer
  → UltraCluster 3.0
  → managed training or inference service

The software path is equally deliberate. The AWS Neuron SDK connects PyTorch and JAX to Neuron compilers, runtimes, collective communication libraries, profiling, and monitoring. Neuron Kernel Interface gives developers a lower-level route for custom kernels without exposing the hardware as a generic CUDA-compatible device.

Trainium3 also shows why headline generational numbers need careful reading. AWS reports 2.52 PFLOPS of FP8 dense compute and 10.1 PFLOPS at FP4, alongside 1.2 PFLOPS of BF16 dense compute. The largest gains come from lower precision, sparsity, memory bandwidth, and a larger connected system—not an equal multiplier for every model and numerical format. The usable gain depends on whether the compiler maps a workload onto those features without losing model quality or creating excess communication.

The cloud integration reaches beyond instances. Trainium is exposed through EC2 and integrated into managed paths including Amazon SageMaker AI and Amazon EKS. AWS also offers dedicated AI Factories that combine accelerator infrastructure, networking, services, and customer-selected data-center locations.

The maturity signal is customer access: Trainium2 and Trainium3 are purchasable cloud infrastructure rather than lab prototypes. The evidence gap is comparability. AWS publishes specifications, workload claims, and customer adoption, but the public record offers less standardized third-party accelerator benchmarking than the NVIDIA and Google ecosystems. Trainium is a real cloud platform whose strongest evidence is production availability and disclosed usage, not a broad set of independently normalized chip measurements.

TPU makes the pod programmable

Google’s TPU architecture has followed this systems thesis for a decade. The original in-datacenter TPU paper described a domain-specific processor built around matrix multiplication rather than the instruction and cache machinery of a general-purpose CPU or GPU. The architectural through-line is systolic execution: data flows through an array of multiply-accumulate units while software arranges computation and movement around the array.

The more consequential transition arrived at pod scale. TPU v4 combined 4,096 chips with optical circuit switches that could reconfigure the interconnect topology. Optical switching let Google route around unavailable components, change topology for different workloads, and improve system availability without wiring every connection through a fixed electrical fabric.

Ironwood, or TPU7x, extends that model for inference. Each chip provides 192 GiB of HBM3e and 7.38 TB/s of memory bandwidth. A cube contains 64 chips, and a full pod reaches 9,216 chips through the Inter-Chip Interconnect and optical fabric. Google exposes smaller slices through its multislice software, so the physical pod becomes a capacity pool rather than one indivisible machine.

The software boundary is XLA. Cloud TPU system architecture places the user program on a TPU virtual machine while XLA compiles framework graphs into TPU executables. JAX is the native-feeling path because its transformations and array semantics align with XLA, while PyTorch/XLA lets PyTorch programs target the same compiler stack.

The scheduler is part of the architecture. queued resources let users request TPU capacity that Google Cloud provisions when the topology is available. flex-start VMs add a managed queue for workloads that can wait for capacity. Reserved capacity and calendar modes serve workloads that need predictable start times. These are not billing decorations; they decide whether a distributed executable receives the physical shape it was compiled to use.

The next generation makes workload specialization more explicit. Google describes TPU 8 as two designs: TPU 8t for training and TPU 8i for inference. TPU 8t pairs two compute dies with eight HBM3e stacks and introduces an expanded optical scale-up domain. TPU 8i uses four HBM3e stacks and targets inference economics. At this cutoff, TPU 8 remains an early-access system, while Ironwood has the clearer public cloud operating record.

Google also has stronger standardized evidence than the other cloud ASIC programs. Its published MLPerf Training v6.0 results cover TPU v6e, TPU7x, and NVIDIA systems. That does not make every TPU claim independently proven—vendor submissions remain optimized submissions—but it creates a defined benchmark boundary that other stacks often lack.

Five vertically integrated stacks, five maturity levels

Stack	Real system boundary	Compiler and software path	Availability by Aug. 30, 2026	Public evidence boundary
AWS Trainium3	NeuronCore → chip → instance → 144-chip UltraServer → UltraCluster	PyTorch/JAX → Neuron compiler, runtime, collectives, NKI	Customer-accessible EC2 infrastructure	Detailed vendor specifications and production adoption; limited standardized cross-vendor results
Google TPU	TensorCore → chip → cube/slice → optically connected pod	JAX/PyTorch → XLA → TPU runtime and ICI collectives	Ironwood available; TPU 8 in early access	Architecture papers, cloud documentation, and MLPerf submissions; TPU 8 evidence remains early
Microsoft Maia 200	Accelerator → Azure server and network → regional inference fleet	PyTorch models → Maia SDK, compiler, Triton language, firmware and runtime	Deployed in Azure regions; SDK preview	Microsoft specifications and workload claims; external customer and benchmark evidence remains narrow
Meta MTIA 300	Accelerator → eight-chip host → Ethernet scale-up/scale-out fleet	PyTorch → in-house compiler, runtime, PyTorch Distributed, TorchTitan	Production use inside Meta	Concrete production workload and fleet design; not offered as a general cloud service
OpenAI Jalapeño	Accelerator → server → rack → OpenAI inference fleet	PyTorch → Triton-based kernels, compiler, runtime, observability	Engineering samples; deployment targeted for late 2026	Vendor-published silicon measurements; production qualification still underway

The table is not a ranking. Availability, public programmability, and benchmark disclosure answer different questions.

Maia, MTIA, and Jalapeño narrow the target

Maia 200 is an inference accelerator with 216 GB of HBM3e and 7 TB/s of memory bandwidth. Microsoft says it is deployed in its U.S. Central data-center region, with a second region planned, and that its workloads include models from the Microsoft Foundry catalog. The Maia SDK includes a Triton compiler, PyTorch integration, kernel libraries, a low-level programming language, and simulation tools.

That is credible system construction, but the public service boundary is still forming. Maia 200 is not documented as a broadly selectable Azure VM family comparable to established GPU instances or current TPU and Trainium offerings. The evidence establishes deployed hardware and an SDK preview, not an open production ecosystem.

MTIA 300 is farther along as captive infrastructure. Meta reports production use training a recommendation model with more than 150 billion parameters. Each accelerator includes two network interfaces, and an eight-accelerator host uses Ethernet for both scale-up and scale-out communication. The system plugs into PyTorch Distributed and TorchTitan while Meta’s compiler and runtime manage the device-specific path.

MTIA therefore supplies strong evidence for vertical integration and weak evidence for external portability. It is optimized around Meta’s models, data centers, network, and fleet software. That is the point. A captive accelerator does not need to become a universal platform if it can remove cost or capacity pressure from a large, stable workload.

OpenAI Jalapeño takes the same idea into the model-provider layer. OpenAI and Broadcom co-designed an inference accelerator with 128 GB of HBM3e and 3.6 TB/s of bandwidth, then built a PyTorch-oriented compiler, runtime, kernel library, observability layer, server, and rack around it. OpenAI reports first-silicon measurements from engineering samples and plans deployment in its inference fleet by the end of 2026.

Jalapeño is the least mature system in this comparison. The silicon is operating and the software stack exists, but production qualification, manufacturing scale, and fleet behavior remain open. Its significance is strategic rather than proven economic superiority: a model provider is designing the execution substrate around the distribution of its own workloads.

Portability moves above the device

Platform-designed silicon relocates portability rather than eliminating lock-in. Device binaries, kernels, collectives, topology assumptions, and compiled artifacts remain hardware-specific even when several stacks accept PyTorch.

The portable boundary sits at model identity, evaluation, service contracts, and outcome telemetry. There Is No Universal AI Runtime—So Build a Portable Control Plane develops that pattern; the important point here is that vertical integration makes execution less interchangeable while making the complete service easier for one operator to optimize.

What remains unproven

Vertical integration creates a measurement problem. The provider can optimize hardware, software, placement, and pricing together, but outsiders rarely observe every boundary. Peak arithmetic says little about model quality. Chip power says little about facility energy. Tokens per second say little without batch size, latency distribution, context length, model version, and service utilization.

Trainium needs more standardized independent results. TPU 8 needs production evidence beyond early access. Maia needs a clearer customer-facing service boundary. MTIA needs no external ecosystem to succeed, but that makes cross-platform comparison difficult. Jalapeño needs manufacturing and fleet evidence after qualification.

The strongest evaluation unit is therefore the complete service: supported model, required code changes, compilation reliability, useful throughput, latency, accuracy, availability, energy boundary, and total cost over a sustained workload. Any comparison that stops at the chip cuts the system at the wrong layer.

So what

Platform-designed silicon changes the question from “Which chip is fastest?” to “Which operator can turn its workload, compiler, network, scheduler, and capacity into the best service?”

Trainium and TPU show the mature forms of that strategy. Maia and MTIA show how large operators narrow the architecture around workloads they already understand. Jalapeño pushes the boundary one layer higher: the model provider starts designing the machine that serves the model.

The open thread is control. If the cloud becomes the computer, how much of that computer must remain visible—topology, compilation, queueing, power, failure domains, and pricing—for builders to make an informed infrastructure choice rather than accept a black-box service?

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

Cerebras Moved the Cluster Boundary Onto a Wafer

Amit — Wed, 23 Sep 2026 04:49:19 +0000

Cerebras makes one of the largest boundaries in computing disappear: the edge of an individual accelerator package.

The Wafer-Scale Engine keeps an entire silicon wafer intact, connecting 900,000 compute cores and 44 GB of distributed SRAM through an on-wafer mesh. Computation that would cross packages, boards, switches, and cables on a conventional cluster can remain on silicon.

That does not make distributed systems disappear. It moves their first boundary outward.

The useful Cerebras hierarchy is now WSE → CS system → Nexus rack → multi-system deployment. Training adds MemoryX and SwarmX outside the wafer. Large inference deployments add Direct Wafer Links, Ethernet, rack power, liquid cooling, and sometimes a separate prefill engine. The wafer becomes a larger processor, but the processor still lives inside a system.

TL;DR

Wafer-scale integration replaces thousands of package-level connections with a 2D mesh spanning 900,000 cores, but the 44 GB SRAM capacity still forces a memory hierarchy above the wafer.
Cerebras training stores model weights in MemoryX, broadcasts them through SwarmX, and keeps activations on the wafer; this simplifies model placement without eliminating cluster communication.
CS-4 puts three WSE-3 Turbo processors into an active-power, liquid-cooled Nexus rack and connects racks through Direct Wafer Links or RoCE v2 Ethernet.
The compiler is part of the architecture because it maps a model onto distributed cores, SRAM, routes, and data movement rather than targeting a conventional shared-memory accelerator.
Independent evidence supports high output speed on specific hosted inference models and useful scientific-computing results. It does not establish universal training cost, aggregate throughput, or energy leadership.

Start with the boundary Cerebras removes

A conventional accelerator begins as a die, becomes a package with external memory, joins other packages on a board, and then communicates across servers through network adapters and switches. Every boundary adds serialization, protocol overhead, distance, and power.

Cerebras uses the wafer as the compute package. The current WSE-3 architecture contains four trillion transistors across 46,225 mm² of silicon. Its 900,000 cores each sit beside local SRAM and a router. The routers form a 2D mesh that extends across the wafer.

The important architectural difference is not transistor count. It is locality.

The company’s WSE architecture material describes 48 KB of independently addressed SRAM per core and explicit communication through the fabric. There is no large shared memory behind a cache hierarchy. Data moves through compiler-configured routes between small compute-and-memory tiles.

That creates an unusual trade:

Layer	What Cerebras collapses	What remains outside
WSE-3	Package-to-package links among 900,000 cores	Model weights larger than 44 GB, host I/O, power, cooling
CS-3 system	Wafer, control, and I/O in one appliance	MemoryX, input servers, management, multi-system fabric
Training cluster	Model placement across accelerators through weight streaming	Weight storage, gradient reduction, data ingestion, scheduling
CS-4 on Nexus	Three wafer processors, rack power, cooling, and I/O	Other racks, facility power and water, heterogeneous prefill
Hosted inference	Hardware exposed through a familiar API	Model availability, queueing, concurrency, service economics

Wafer scale removes the smallest expensive boundaries. It does not remove memory capacity, network scale-out, or data-center operations.

SRAM changes the memory problem

HBM-based accelerators place large, fast DRAM stacks beside the processor. Cerebras takes the opposite path: distribute SRAM across the compute fabric and place every small memory bank close to its core.

HBM and SRAM solve different problems. HBM is the large warehouse: it holds far more data, but cores reach it through shared memory controllers and an off-die interface. SRAM is the workbench: it consumes much more silicon area per byte, but puts the active data beside the arithmetic that uses it.

GPUs already use both. NVIDIA calls each repeated compute neighborhood a streaming multiprocessor, or SM; it contains compute cores, registers, cache, and programmable shared SRAM. The Blackwell tuning guide lists up to 228 KB of shared memory per B200 SM and 126 MB of shared L2 cache on GB200, backed by as much as 180 GB of HBM. The distinction is not SRAM versus no SRAM. It is how much SRAM exists, where it sits, and whether software can keep the active working set there.

Cerebras gives each of its 900,000 cores 48 KB of private SRAM. One bank cannot hold a model or even a large layer, but it can hold the small tile of instructions, activations, and intermediate values that its core is processing now. Multiplied across the wafer, those small banks become 43.2 GB—marketed as 44 GB—and can all serve their nearby cores in parallel. The Cerebras SDK description makes that locality explicit: each processing element owns its 48 KB bank rather than sharing it with the rest of the wafer.

That is why a small amount beside each core creates a large system effect. The compiler can move a tile into local SRAM once, reuse it across many operations, and exchange nearby values through the mesh instead of repeatedly returning to HBM. The advantage depends on locality: when the active data does not fit or cannot be reused, external memory and communication become the constraint again.

The result is high local bandwidth and low movement distance. The WSE-3 publishes 21 PB/s of memory bandwidth and 44 GB of on-wafer SRAM. The newer WSE-3 Turbo inside CS-4 doubles the vendor-published figures to 43.2 PB/s and 250 sparse FP16 PFLOPS per wafer by running the existing architecture at a higher operating point.

Those are peak hardware specifications, not application results. They still reveal the architecture’s bet: spend silicon area on distributed memory and communication rather than attach a larger external HBM pool to each compute die.

SRAM solves bandwidth and creates a capacity constraint. Forty-four gigabytes cannot hold the weights of a frontier model, optimizer state, activations, and temporary data at once. Cerebras therefore separates two kinds of state:

Activations and intermediate values remain in distributed SRAM on the wafer.
Model weights live in external MemoryX storage and stream through the system layer by layer.

The wafer is not a self-contained memory island. It is a compute surface fed by a separate weight-memory system.

MemoryX and SwarmX bring the cluster back

The Cerebras Wafer-Scale Cluster documentation makes the distributed architecture explicit.

MemoryX stores model weights and streams them toward the compute systems. During training, updated gradients travel back toward the weight-memory layer. SwarmX broadcasts weights to multiple CS systems and reduces gradients in the opposite direction. Input servers prepare training data, while management servers schedule and coordinate the cluster.

This avoids one class of complexity. Developers do not manually split individual layers across a mesh of accelerator memories in the same way required by many tensor- and pipeline-parallel GPU configurations. Cerebras presents weight streaming and data-parallel scale-out as the main abstraction.

The communication has not vanished. It has become more structured:

flowchart TB
    CODE["PyTorch model or CSL program"]
    COMPILER["Cerebras compiler<br/>graph placement + routes + memory"]
    MANAGER["Cluster management<br/>scheduling + input services"]

    subgraph TRAIN["Training path"]
        MX["MemoryX<br/>weights + optimizer state"]
        SX["SwarmX<br/>weight broadcast + gradient reduction"]
        C1["CS-3<br/>WSE-3 + SRAM"]
        C2["CS-3<br/>WSE-3 + SRAM"]
        C3["Additional CS systems"]
        MX --> SX
        SX --> C1
        SX --> C2
        SX --> C3
        C1 -->|"gradients"| SX
        C2 -->|"gradients"| SX
        C3 -->|"gradients"| SX
    end

    subgraph INFER["Current rack-scale inference path"]
        PREFILL["Optional external prefill<br/>GPU or ASIC"]
        NEXUS["CS-4 Nexus rack<br/>power + liquid cooling + I/O"]
        W1["WSE-3 Turbo"]
        W2["WSE-3 Turbo"]
        W3["WSE-3 Turbo"]
        SCALE["Direct Wafer Links<br/>or RoCE v2 Ethernet"]
        PREFILL --> NEXUS
        NEXUS --> W1
        NEXUS --> W2
        NEXUS --> W3
        NEXUS --> SCALE
    end

    CODE --> COMPILER
    COMPILER --> MANAGER
    MANAGER --> MX
    COMPILER --> NEXUS

The processor boundary grows from one die to one wafer. The training boundary still includes storage, fabric, preprocessing, and management.

The compiler is part of the machine

A wafer containing 900,000 small cores is not useful if software treats it as a large conventional CPU or GPU.

The high-level path begins with supported PyTorch models and the Cerebras software stack. The Cerebras Graph Compiler converts the model into an executable placement across cores, local SRAM, and communication routes. The lower-level Cerebras Software Language compiler exposes the wafer as a programmable fabric for custom kernels and scientific workloads.

This compiler role is deeper than translating arithmetic instructions. It determines where tensors live, how data flows, and which paths connect computation across the wafer. Hardware and compiler share responsibility for making the wafer behave like one processor.

The cost is a narrower compatibility boundary. A PyTorch model is source material, not proof that every operator, dynamic graph, custom kernel, numerical mode, and training recipe maps without changes. The relevant portability question is not “Does it support PyTorch?” It is “Does this exact model compile, converge, and meet its service target on the supported stack?”

That distinction matters for both training and inference. A broad GPU runtime can absorb rapidly changing model architectures through a large library and kernel ecosystem. Cerebras can win when the compiler has a good mapping and the wafer’s locality matches the workload. Unsupported or inefficient mappings remain a software problem rather than a silicon problem.

CS-4 turns wafer scale into rack scale

The August 2026 CS-4 system changes the deployment unit. It is not a new WSE-4 chip. CS-4 uses three WSE-3 Turbo processors inside the first implementation of the Nexus rack-scale platform.

Each wafer sits in a rear-mounted “Wafer-Scale Backpack” containing local power conversion, direct liquid cooling, I/O, and control electronics. The rack provides active power infrastructure. Cerebras publishes 750 sparse FP16 PFLOPS, 129.6 PB/s of memory bandwidth, and 7.2 Tbit/s of I/O for the three-wafer system. These are vendor peak specifications.

Nexus supports two scale-out paths:

Direct Wafer Links connect Cerebras systems without an intervening switch and target low-latency wafer-to-wafer communication.
RoCE v2 RDMA over Ethernet connects the rack to standard data-center and heterogeneous infrastructure.

CS-4 also formalizes disaggregated inference. An external GPU or ASIC can process prefill—the prompt-heavy first phase—then transfer model state to Cerebras for latency-sensitive token generation. The CS-4 announcement names AMD and Trainium as potential prefill platforms.

This is an important correction to the “cluster became a processor” thesis. Cerebras can make decode behave like a tightly coupled wafer-scale computation while still depending on another processor, a state-transfer protocol, Ethernet, and service-level routing.

Power and cooling are the least transparent boundary

Wafer scale shortens communication distance, but it concentrates heat.

CS-4 is an active-power rack with three direct-liquid-cooled compute backpacks. Cerebras says moving power conversion within 0.5 millimeters of the processor reduces board-level loss and permits twice the power delivery to WSE-3 Turbo. The public CS-4 specifications do not publish rack input power, coolant temperatures, flow requirements, or facility heat-rejection assumptions.

That missing number matters. Cerebras claims up to ten times more throughput per watt than CS-3, but the comparison combines a new operating point, system design, workload selection, and projected results. Without measured rack input, cooling overhead, model, precision, concurrency, and quality target, it cannot support a facility-level efficiency conclusion.

The deployment comparison is therefore architectural rather than economic:

Deployment	Compute unit	Memory and fabric	Facility boundary	Evidence available
CS-3 appliance	One WSE-3	44 GB SRAM; external MemoryX and SwarmX for training	System plus external memory, fabric, input, and management components	Shipping systems, public docs, customer installations
Multi-CS-3 training cluster	Multiple WSE-3 systems	MemoryX weight storage; SwarmX broadcast and reduction	Multiple systems, input and management servers, network and storage	Vendor scaling claims; selected research publications
CS-4 Nexus rack	Three WSE-3 Turbo processors	Direct Wafer Links and RoCE v2	Active rack power and direct liquid cooling; public power draw absent	Announced Aug. 2026; shipments stated to begin during the quarter
Cerebras cloud inference	Provider-operated wafer systems	API-hidden placement and service fabric	Operator-owned power, cooling, capacity, and queueing	Independently observable endpoint speed; limited facility data

The rack is easier to reason about mechanically than economically. The modules, cooling approach, and I/O modes are visible. The watts, water, sustained utilization, and failure behavior are not.

Training and inference are different evidence problems

Cerebras has credible evidence, but it is fragmented across different workload boundaries.

For hosted inference, Artificial Analysis tracks Cerebras endpoints and has measured high output speeds on specific available models. This is stronger than a vendor slide because an external evaluator sends requests to a live service. It still does not establish aggregate throughput at high concurrency, queueing behavior during capacity pressure, power consumption, or performance on unsupported models.

For scientific computing, a 2025 Nature Communications paper reports large molecular-dynamics simulations on the WSE. That result demonstrates that the fabric and local memory can accelerate structured nearest-neighbor computation outside neural networks. It does not translate directly into transformer training economics.

For large-model training, Cerebras publishes cluster scale, parameter capacity, and time-to-train results. Those claims are useful architecture evidence when labeled as vendor measurements. Public, independently reproduced comparisons with matched model, token count, convergence target, precision, complete cluster power, and software effort remain scarce.

The honest evidence ladder is:

Strongest: externally tested live inference endpoints and peer-reviewed workload-specific papers.
Useful but bounded: customer or laboratory deployment announcements confirming that systems exist and run real work.
Vendor measured: model speed, scaling, and throughput claims with a disclosed configuration.
Forward-looking: CS-4 extrapolations, projected efficiency, and announced capacity not yet demonstrated in sustained operation.

No single rung proves that Cerebras is faster or cheaper for AI as a category.

What the architecture gets right

Cerebras attacks a real cost in distributed computing: moving data among small processors because manufacturing and packaging force the machine to be divided.

The wafer-scale approach buys a larger low-latency domain. Distributed SRAM keeps the active working set from crossing an HBM interface for every operation. The compiler can treat hundreds of thousands of tiles as one placement surface. Weight streaming allows models larger than the wafer’s SRAM to use that compute without slicing every layer across device memories.

The design also makes its remaining boundaries clearer. MemoryX owns weights. SwarmX owns training broadcast and reduction. Nexus owns rack power, cooling, and I/O. Direct Wafer Links or Ethernet own scale-out. An optional heterogeneous processor owns prefill.

That is not the elimination of distributed systems. It is a different decomposition of one.

So what

Cerebras should not be evaluated as an oversized GPU. Its architecture changes which links are on silicon, which state remains local, which state streams from outside, and which decisions move into the compiler.

The useful comparison is therefore not wafer versus chip. It is complete system versus complete system:

How much model state can remain local?
Which communication occurs on silicon, inside the rack, and across the network?
What compilation and model changes are required?
How does performance change under concurrency rather than one fast stream?
What is the measured rack and facility power for completed work?
How does the system recover when a wafer, memory service, network path, or prefill engine becomes unavailable?

Cerebras has shown that a wafer can behave like a processor. The unresolved question is whether the layers above it—weight memory, heterogeneous inference, rack power, scale-out networking, and software compatibility—can become as repeatable and measurable as the wafer itself.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

AMD Is Rebuilding the GPU Stack in the Open

Amit — Wed, 23 Sep 2026 04:48:43 +0000

The series' read is that AMD is the closest merchant alternative to NVIDIA because it is rebuilding more than a GPU. Instinct accelerators now sit inside eight-GPU platforms, liquid-cooled racks, Ethernet fabrics, open scale-up standards, and a software stack that reaches from PyTorch kernels to cluster validation.

The architecture is deliberately less vertically owned. AMD supplies critical pieces, but ROCm, UALink, UALoE, Ultra Ethernet, OEM systems, merchant switches, optics, and facility integration cross organizational boundaries. That creates choice at each layer—and makes the operator responsible for proving that those layers work as one system.

TL;DR

Instinct evolved from a 32 GB accelerator in MI100 to 288 GB of HBM3E in MI355X, while the deployment unit expanded from a PCIe card to eight-GPU platforms and a planned 72-GPU Helios rack.
AMD's current shipping scale-up path still centers on Infinity Fabric inside an eight-GPU platform; Helios shifts the rack-scale domain toward the open UALink ecosystem and AMD's Ethernet-based UALoE fabric.
ROCm is open and increasingly production-oriented, but openness does not remove qualification work. AMD documents exact operating-system, driver, firmware, container, collective-library, and network requirements.
AMD has credible evidence: production availability through major cloud and OEM channels, MLPerf submissions, and named deployments. That evidence still does not establish interchangeable performance across models, precisions, software versions, or rack designs.

The accelerator became a memory system

The clearest Instinct trend is not a single compute multiplier. It is the steady expansion of the memory domain surrounding the accelerator.

Generation	Public status by Aug. 30, 2026	HBM capacity	Peak memory bandwidth	Board or module power	System boundary
MI100	Previous generation	32 GB HBM2	1.23 TB/s	300 W	PCIe accelerator
MI250X	Previous generation	128 GB HBM2e	3.2 TB/s	560 W peak	Dual-die accelerator used in multi-GPU systems
MI300X	Shipping	192 GB HBM3	5.3 TB/s	750 W	Eight-GPU platform with 1.5 TB aggregate HBM
MI325X	Shipping	256 GB HBM3E	6.0 TB/s	1,000 W	Eight-GPU platform with 2 TB aggregate HBM
MI355X	Shipping	288 GB HBM3E	8.0 TB/s	1,400 W	Liquid-cooled eight-GPU platform with 2.3 TB aggregate HBM
MI455X in Helios	Launched; shipments planned for the second half of 2026	432 GB HBM4 per GPU	23.3 TB/s per GPU	Rack specification is the relevant boundary	72-GPU rack with 31 TB aggregate HBM

MI200 made chiplets part of the accelerator design. MI300X pushed the idea further: eight compute dies sit over four I/O dies, connected through Infinity Fabric and surrounded by eight HBM stacks. AMD's CDNA 3 architecture documentation describes the package as one logical accelerator even though compute, I/O, and memory are physically disaggregated.

MI350 continues that direction with CDNA 4 and denser low-precision formats. The more durable change is 288 GB of HBM3E per accelerator and 2.3 TB across an eight-GPU platform. Large models can retain more weights and key-value cache close to compute, reducing how often capacity pressure forces work across slower boundaries.

Helios changes the boundary again. AMD's current public design pairs 72 MI455X GPUs with EPYC host processors, Pensando networking, 31 TB of HBM4, and 1.7 PB/s of aggregate HBM bandwidth. Those are announced system specifications, not a claim about generally available production capacity or measured application performance.

Scale-up is moving from a platform to a rack

MI300X, MI325X, and MI355X platforms connect eight GPUs through Infinity Fabric. At that scale, AMD controls the accelerator package and much of the GPU-to-GPU communication path. The OEM still owns the server implementation, firmware bundle, power delivery, cooling, and qualification.

Helios expands the scale-up domain to 72 GPUs. Its architecture uses AMD's UALoE—an Ethernet-based fabric using UALink-aligned semantics—rather than treating Ethernet only as the network between independent servers. AMD develops this path alongside the broader UALink Consortium effort, where accelerator, switch, CPU, and systems companies define an open scale-up interconnect.

flowchart TB
    app["Models and frameworks"]
    rocm["ROCm: HIP, kernels, RCCL, tools"]
    rack["72-GPU Helios rack"]
    gpu["AMD Instinct MI455X"]
    scaleup["UALink software model + UALoE fabric"]
    scaleout["Ultra Ethernet / RoCE scale-out"]
    network["Pensando NICs and merchant Ethernet switches"]
    oem["OEM / ODM rack integration"]
    facility["Operator: power, liquid cooling, firmware, validation"]

    app --> rocm --> rack
    rack --> gpu
    rack --> scaleup
    rack --> scaleout --> network
    gpu --> oem
    scaleup --> oem
    network --> oem
    oem --> facility

The diagram exposes the trade. An open interface can let multiple switch, accelerator, cable, and system suppliers participate. It does not create one owner for congestion control, firmware compatibility, optics, cabling, collectives, telemetry, or failure recovery.

Scale-out follows the same pattern. AMD is a founding member of the Ultra Ethernet Consortium, while Pensando supplies NIC and DPU technology. The resulting cluster can use familiar Ethernet operations and merchant switching, but the final behavior depends on the complete path: GPU memory, PCIe attachment, NIC firmware, switch configuration, routing, loss management, RCCL, and the model's communication pattern.

ROCm makes portability possible, not automatic

ROCm is no longer accurately described as a compiler shim around a GPU. It includes the HIP programming model, math and kernel libraries, RCCL collectives, profiling and debugging tools, container images, model recipes, validation utilities, and integrations with major frameworks.

Software layer	AMD path	What remains operator-owned
Framework	PyTorch for ROCm, TensorFlow, JAX	Pinning a supported framework, Python, driver, and ROCm combination
Portability	HIP porting model	Reworking CUDA-specific extensions, intrinsics, build logic, and performance assumptions
Kernels	Composable Kernel, hipBLASLt, MIOpen, rocBLAS	Selecting or tuning kernels for the exact model, shape, precision, and GPU
Distributed execution	RCCL	Fabric configuration, topology, collective tuning, fault handling, and version alignment
Inference	vLLM on ROCm and SGLang on ROCm	Testing model features, quantization paths, custom operators, latency, and capacity
Cluster acceptance	ROCm validation and system-health tooling	Establishing a known-good baseline and revalidating it after firmware, driver, network, or container changes

HIP can translate a large amount of CUDA-shaped source code and preserve a familiar kernel programming model. It cannot guarantee that a CUDA extension compiles unchanged, that the same fused kernel exists, or that identical scheduling choices remain optimal. Portability is strongest at the framework and model-contract layers; it weakens around custom kernels, collectives, memory placement, quantization, and compilation.

AMD's own compatibility matrices make this responsibility explicit. Supported configurations pair specific GPU architectures with operating systems, kernel drivers, firmware, libraries, and framework releases. The Instinct cluster deployment guides add validated NIC, RoCE, topology, and collective requirements. A container freezes user-space dependencies, but it does not freeze host drivers, GPU firmware, NIC firmware, switch behavior, or cooling performance.

That is the operational difference between open source and an integrated product boundary. Source access improves inspection, contribution, and supplier choice. It does not eliminate systems engineering.

Deployment evidence is real but uneven

AMD has moved beyond reference designs. MI300X and later Instinct platforms are available through major server vendors and cloud providers. Microsoft's Azure ND MI300X v5 virtual machines expose eight MI300X GPUs with 1.5 TB of aggregate HBM. Oracle has published MI355X cluster results and deployment details. Microsoft has separately announced that Azure will deploy Helios for inference, with AMD stating that shipments begin in the second half of 2026.

The evidence ladder still matters:

Evidence level	What AMD has	What it establishes
Product specification	Datasheets for Instinct accelerators and platforms	Capacity, interfaces, rated bandwidth, and thermal boundaries
Vendor benchmark	AMD model and platform results	Performance under the vendor's disclosed configuration
Standard benchmark	MLPerf Training results from AMD and partners	Reproducible benchmark submissions within one MLPerf version and workload
Cloud or OEM availability	Azure, Oracle, Dell, HPE, Lenovo, Supermicro and others	A purchasable or consumable deployment path
Production disclosure	Named operator announcements and engineering reports	Workload use at a stated boundary, when configuration detail is sufficient

None of these rows alone proves lower service cost or better application performance. A useful comparison must align model, precision, sequence length, batch policy, software version, system count, networking, power boundary, and availability target. A result from eight MI300X GPUs cannot be casually compared with a rack-scale result, and an announced 72-GPU design is not equivalent to a measured production fleet.

The open stack changes the buying decision

AMD's advantage is not that every layer is interchangeable. It is that more layers can be selected, inspected, and sourced independently: accelerator, host CPU, NIC, switch, rack integrator, framework build, inference engine, and cloud or on-premises operator.

That changes the evaluation from a GPU comparison into an ownership map:

Which party certifies the complete server and rack?
Who owns failures that cross GPU, NIC, switch, firmware, and collective-library boundaries?
Which model features depend on backend-specific kernels?
Can the operator reproduce benchmark results after changing containers or firmware?
Does supplier choice reduce cost enough to fund the added integration and qualification work?

The thesis is therefore narrower than "open beats closed." AMD is building the closest merchant alternative because it now spans credible accelerators, large HBM domains, scale-up and scale-out networking, rack designs, and a usable software ecosystem. Its structure also places more responsibility at the interfaces between those layers.

The unresolved question is whether the UALink and UALoE ecosystem can make a multi-company rack behave like one supported product without converging on a single de facto integrator. If operators still need one company to certify every firmware, fabric, cooling, and software combination, how much of the architectural openness will survive contact with production?

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.