DEV Community: Amit

Structured Outputs on Amazon Bedrock: Make the Model's Response a Contract

Amit — Tue, 13 Oct 2026 06:28:32 +0000

Structured outputs let you send a JSON schema with a request and get back a response that matches it. The model's generation is constrained to the schema, so the response parses on the first try, contains the fields you defined, and uses only the values you allowed.

Here is one classification written two ways. The first is for a person:

This is a billing problem, it is high priority, and someone should look at it.

The second is for a program:

{"category": "billing", "priority": "high", "needs_human": true}

Structured outputs are how you ask for the second one and get it every time.

How it works

You pass a JSON schema in the request. Bedrock compiles it into a grammar and applies constrained sampling during generation, so each token the model emits has to keep the output valid against the schema. The first request with a new schema pays a compile cost, and compiled grammars are cached for 24 hours.

The Bedrock documentation describes two mechanisms, usable alone or together:

A JSON schema output format controls the shape of the model's text response.
Strict tool use (strict: true) applies a schema to the arguments the model passes to a tool.

Why it matters

A model asked for JSON in the prompt can return output that breaks the program reading it. The AWS launch post gives an example that parses fine and still fails: a booking function that needs passengers: int receives "two" or "2". Anthropic's documentation lists what goes wrong without a schema: parsing errors from invalid JSON syntax, missing required fields, inconsistent data types, and schema violations that need error handling and retries.

A schema removes these failure modes. The shape becomes a contract between the model and the code that reads its output, and the code can drop its fence-stripping, key-guessing and retry logic. That matters wherever a program consumes the response. When a person reads it, the constraint matters much less.

Where it applies

Use case	What the schema holds	Source
Extraction from unstructured text	Fields from invoices, contracts, clinical notes, product descriptions	AWS lists finance, healthcare, ecommerce and legal examples
Classification and routing	Enums for category, priority and queue	The ticket example below; AWS lists ticket routing under customer service
Tool arguments for agents	The input schema of each tool, with `strict: true`	Bedrock documentation
Separating an answer from commentary	An `answer` field and a `reasoning` field	OpenAI's announcement
Generated interfaces	A typed tree of components	OpenAI's guide
Pipelines that trust model output	The typed record the next stage reads	AWS describes "zero-validation data pipelines" built on schema-compliant output

The tool row connects to how agents work. A tool is a function with a JSON schema, and strict mode applies the same constraint to the arguments the model fills in.

How to use it with Claude Opus 5

Opus 5 on Bedrock accepts the Anthropic Messages API on the bedrock-runtime endpoint. Point the Anthropic SDK at https://bedrock-runtime.<region>.amazonaws.com/anthropic, authenticate with a short-lived Bedrock token, and put the schema in output_config.format. This example classifies a support ticket:

import json
from anthropic import Anthropic
from aws_bedrock_token_generator import provide_token

REGION = "us-west-2"
client = Anthropic(
    base_url=f"https://bedrock-runtime.{REGION}.amazonaws.com/anthropic",
    api_key=provide_token(region=REGION),
)

schema = {
    "type": "object",
    "properties": {
        "category": {"type": "string",
                     "enum": ["billing", "bug", "account_access", "feature_request", "other"]},
        "priority": {"type": "string", "enum": ["low", "medium", "high", "critical"]},
        "needs_human": {"type": "boolean"},
        "order_id": {"anyOf": [{"type": "string"}, {"type": "null"}]},
        "summary": {"type": "string"},
    },
    "required": ["category", "priority", "needs_human", "order_id", "summary"],
    "additionalProperties": False,
}

response = client.messages.create(
    model="global.anthropic.claude-opus-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": f"Classify this support ticket.\n\nTicket: {ticket}"}],
    output_config={"format": {"type": "json_schema", "schema": schema}},
)

if response.stop_reason != "end_turn":
    raise RuntimeError(f"incomplete response: {response.stop_reason}")

text = next(block.text for block in response.content if block.type == "text")
triage = json.loads(text)

Read the text block by type. With default settings and no schema, Opus 5 returned a thinking block ahead of the text in every run I made, so response.content[0].text fails there.

Every object in the schema needs additionalProperties: False, and Bedrock rejects the schema without it. Numeric ranges such as minimum and maximum are outside the supported subset, so check those in your own code.

For tool arguments, add "strict": True to the tool definition. The model's tool call then follows the input schema:

tool = {
    "name": "classify_ticket",
    "description": "Record the classification of a support ticket",
    "strict": True,
    "input_schema": schema,
}

response = client.messages.create(
    model="global.anthropic.claude-opus-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": f"Classify this support ticket using the tool.\n\nTicket: {ticket}"}],
    tools=[tool],
    tool_choice={"type": "tool", "name": "classify_ticket"},
)

args = next(block.input for block in response.content if block.type == "tool_use")

What the response looks like with and without a schema

The input is one support ticket:

Hi, I was charged twice for order #A-48213 this morning and now I can't log in
to see my invoices. This is blocking our month-end close, please fix ASAP!!

Opus 5 thinks by default, which changes the response and the bill. For this comparison I turned thinking off with thinking={"type": "disabled"}, so the schema is the only thing that differs between requests.

Without a schema, I asked Opus 5: "Classify this support ticket and return the result as JSON." Every response opened with a paragraph of prose, put the JSON in a markdown code fence, and usually closed with notes, so json.loads fails on the raw text. Here are two separate runs of that prompt, trimmed:

Looking at this ticket, there are actually two distinct problems bundled together,
plus some urgency signals worth capturing. Here's my classification:

```json
{
  "ticket_id": "A-48213",
  "summary": "Customer reports a duplicate charge on order #A-48213 and is simultaneously unable to log in ...",
  "classification": {
    "primary_category": "Billing",
    "secondary_category": "Account Access",
    "is_multi_issue": true
  },
  "priority": {
    "level": "High",
    "rationale": "Financial impact (duplicate charge) combined with a blocker on a time-sensitive ..."
  },
  ...
}
```

A couple of notes on the judgment calls:

**Why Billing is primary:** The duplicate charge is the root financial harm ...

Looking at this ticket, there are actually two distinct problems bundled together,
plus some urgency signals worth capturing. Here's my classification:

```json
{
  "ticket_id": "A-48213",
  "summary": "Customer reports a duplicate charge on order #A-48213 and is simultaneously unable to log in ...",
  "primary_category": "billing",
  "secondary_issues": [{ "category": "account_access", ... }],
  "sentiment": "frustrated",
  "urgency": "high",
  "priority": "P1",
  ...
}
```

A few notes on the judgment calls:

**Primary category is billing, not account access.** The duplicate charge is the root financial harm ...

Both answers are reasonable. They disagree on where the category lives, whether priority is an object or a string, and what the value says (High with a rationale, or P1). No two of the 20 runs used the same set of fields, three had no priority field at all, and the prompt never named any fields.

Describing the fields in the prompt ("category, one of billing, bug, ...; priority, one of low, medium, high, critical; ...") fixes this. Opus 5 followed it closely, and 19 of 20 responses were raw JSON that json.loads accepted. The twentieth came back in a code fence.

With the schema passed through output_config.format, 20 of 20 responses were the JSON and nothing else:

{"category":"billing","priority":"high","needs_human":true,"order_id":"A-48213","summary":"Customer reports a duplicate charge on order #A-48213 and is also unable to log in to access invoices, blocking their month-end close. Requests urgent resolution (refund of duplicate charge and account access restoration)."}

The fields are the ones I defined, priority is one of the four values I allowed, and order_id is a string or null. Twenty runs is a small sample, but the one fenced response is the failure a parser has to handle, and the schema removes it. A one-in-twenty rate would be fifty parse failures per thousand tickets.

What it costs

A schema moves the bill in two directions. It is added to the prompt, so input grows, and the response is only the JSON, so output shrinks. I priced each request at the published Opus 5 rate of $5 per million input tokens and $25 per million output tokens, using the token counts the API reported. With thinking disabled, 20 runs each:

Request	Input tokens	Output tokens	Cost per 1,000 tickets
No schema, "return the result as JSON"	84	1,081	$27.43
Fields described in the prompt	148	117	$3.67
Schema in the request	473	98	$4.83

Against the bare prompt, the schema is 82% cheaper, because the bare prompt returns about a thousand tokens of prose and notes around the JSON. Against a prompt that describes the fields, the schema is 31% more expensive. It bills 325 more input tokens, about $1.63 per thousand tickets, and saves 19 output tokens, about $0.47. That difference buys the removal of the one fenced response in twenty.

These figures use the thinking-disabled runs from the previous section. Thinking tokens bill as output tokens, so a request that triggers thinking costs more than shown. I did not measure retries avoided.

Where it stops

Truncation breaks the contract. With max_tokens set to 20, the call returned HTTP 200, stop_reason: max_tokens and an empty text block. AWS tells you to check the stop reason for refusals and token limits, and the snippet above does.

A valid response can still lose information. The ticket describes two problems, a duplicate charge and a login failure, and my schema allows one category. Opus 5 returned billing in all 20 runs, and the login failure survived only in the free-text summary. Without a schema, every run recorded the second issue in its own field, such as secondary_issues. The schema made a product decision, one category per ticket, and the model followed it. If the next stage needs both issues, the schema needs a categories array. A schema guarantees the shape of each field and says nothing about whether those fields capture what the next stage needs.

So what

Put a schema on any call where a program reads the answer. The cost is the schema's input tokens and a stop-reason check, and the return is the deletion of fence-stripping, key-guessing and retry code. For tool arguments, add strict: true.

Keep your own validation for ranges and for anything the schema subset cannot express. The schema covers shape. Meaning needs a separate check.

Opus 5.5 Leads a Four-Model OWASP Java Test by 7 to 9 Points. Kimi K3, GLM 5.3, and GPT-6 Sol Tie.

Amit — Sat, 10 Oct 2026 17:46:10 +0000

OWASP Benchmark for Java is the OWASP Foundation's open test suite for scoring vulnerability-detection accuracy. Version 1.2 ships 2,740 deliberately exploitable test cases across eleven weakness classes — SQL injection, command injection, cross-site scripting, path traversal, weak cryptography, and more — each one a fully runnable Java servlet labeled vulnerable or safe against a public answer key. It was built to score static and dynamic security scanners the same way on every run, on two numbers: whether a tool finds the real vulnerabilities, and whether it also wrongly flags the safe code next to them. That second number — false positives — is where three of the four models in this test lose most of their points.

At maximum reasoning effort, Claude Opus 5.5 classified 654 of 660 OWASP Benchmark Java files correctly and missed no vulnerable file. Kimi K3, GPT-6 Sol, and GLM 5.3 scored 607, 598, and 597 — a 1.5-point spread that a paired bootstrap can't call a winner on. They separate clearly on cost and speed instead.

I ran the same 660 files through all four models on Amazon Bedrock with the same settings. The question was what two open-weight models give up against current closed models on a single-call code-review task once each runs at its top reasoning setting, and what each correct answer costs.

TL;DR

Opus 5.5 got 654 of 660 right with zero missed flaws and 5 false alarms, while the other three raised 38 to 41 false alarms each. Its lead over each of them is 7 to 9 points, and the 95% interval stays above zero in all three comparisons.
Kimi K3, GPT-6 Sol, and GLM 5.3 land within 1.5 points of each other, and every paired interval among them includes zero.
GLM 5.3 charges 56% of Kimi K3's input rate and 35% of its output rate, yet it writes 3.4 times as many output tokens, so cost per correct answer comes out at $0.0091 against $0.0094.
Opus 5.5 costs 3.1 to 3.4 times more per correct answer than the other three and averages 36 seconds per call against 7.5 to 10.7.

What the model is asked

Each call is one message with no tools. It names a weakness class, such as command injection or weak hashing, and attaches one servlet file from OWASP Benchmark for Java with line numbers. The model returns one JSON object: vulnerable true or false, an evidence line, and a one-sentence reason.

The answer key is a separate public file that never reaches the model. I score only the true or false verdict. A refusal, a truncated reply, or an unparseable reply counts as a miss and gets reported on its own line.

The 660 files are 330 vulnerable and 330 safe, spread across eleven weakness classes, drawn deterministically from a pinned commit of the benchmark. None of the 660 repeat across my earlier OWASP runs.

Every model ran with the same settings:

the global inference profile, called through Bedrock Runtime
reasoning effort set to max
a 32,768-token output cap, which no call reached
one prompt wording that asks whether the code is vulnerable to the named class
one repeat per file

Claude went through the Messages API, GPT-6 Sol through Responses, and Kimi K3 and GLM 5.3 through Chat Completions. I covered why each family needs its own API in an earlier post on wiring OpenCode to Bedrock. The model cards for GLM 5.3, GPT-6 Sol, and Opus 5.5 list the supported effort levels.

The result

	Opus 5.5	Kimi K3	GPT-6 Sol	GLM 5.3
Correct	654 (99.1%)	607 (92.0%)	598 (90.6%)	597 (90.5%)
Missed flaws	0	11	24	21
False alarms	5	41	38	39
Non-answers	1 refusal	1	0	3
Mean output tokens	1,192	364	707	1,243
Mean seconds per call	36.1	10.7	7.5	10.3
List cost, 660 calls	$19.99	$5.73	$5.96	$5.40
Cost per correct answer	$0.0306	$0.0094	$0.0100	$0.0091

I compared each pair of models on the same 660 files with a paired bootstrap over files. Opus 5.5 beat Kimi K3 by 7.1 points (interval +5.3 to +9.2), GLM 5.3 by 8.6 (+6.5 to +10.9), and GPT-6 Sol by 8.5 (+6.4 to +10.6). Among the other three, the largest gap was 1.5 points, and all three intervals straddled zero.

The paired counts show the shape of the lead. Opus 5.5 was right on 47 files Kimi K3 got wrong, 57 that GLM 5.3 got wrong, and 56 that GPT-6 Sol got wrong. None of the three was right on a file Opus 5.5 got wrong.

Where the gap lives

Weak hashing and weak cryptography carry the largest share. Opus 5.5 scored 70 of 70 on both classes, while the other three scored between 55 and 63 on each. Command injection shows the same direction, with Opus 5.5 at 66 of 70 against 58 to 62.

Ten of the eleven classes put Opus 5.5 at or within one file of a perfect score, and command injection (66 of 70) is the exception. GLM 5.3 matched it at 70 of 70 on weak randomness, and all four models tied at the top on secure cookies and XPath injection. Each class rests on 20 to 70 files, so I treat single-class gaps as leads to follow and not as findings.

The false-alarm column matters in practice. A reviewer who flags 40 safe files out of 330 produces a triage queue, and Opus 5.5 produced 5. The Semgrep team reported that Kimi K3 had lower precision than other models on IDOR detection, 0.684 against 0.84 to 0.91. That was a different task with an agentic harness. In my test Kimi K3's 41 false alarms sit next to GLM 5.3's 39 and GPT-6 Sol's 38, so I see no Kimi-specific precision problem on this task. The outlier is Opus 5.5.

What the open-weight pair buys

Kimi K3 and GLM 5.3 each cost about $5.50 for the whole 660-file run, against $20 for Opus 5.5. That buys roughly 7 to 9 points less accuracy and a 3 to 5 times faster call.

Token counts are exact — I logged every call's usage block directly, and the mean, median, and 95th-percentile figures above come straight from those logs. The price per token checks out against each vendor's own published rate. Anthropic's pricing lists Claude Opus 5.5 at $4 in / $20 out per million tokens on the direct API, and states that US-only inference carries a separate 1.1x surcharge — so the $4/$20 global rate used here is Anthropic's own direct-API price, not an estimate. OpenAI's pricing lists GPT-6 Sol at $2 in / $10 out, and matches the same 1.1x pattern: this repository's own cost tracking found Bedrock's US-Geo rate for GPT-6 Sol at exactly $2.20/$11.00, 1.1x the direct rate. Kimi K3's Bedrock global rate ($3.00 in / $15.00 out) matches Moonshot's own API price exactly, confirmed against the live AWS Price List API. GLM 5.3 is the one real exception: Bedrock's global rate ($1.68 in / $5.28 out) runs 20% above Z.ai's own direct rate of $1.40 in / $4.40 out — a genuine markup, not a reading gap. All four rates in the table are grounded against a vendor or AWS source; none are reconstructed.

GLM 5.3's per-token advantage mostly disappears. Its mean output of 1,243 tokens is 3.4 times Kimi K3's 364. At the rates above, GLM 5.3's output alone costs about $0.0066 per call and Kimi K3's about $0.0055, so GLM 5.3 spends more on output despite the lower rate. Cheaper input recovers most of the difference, and the net gap is 4% per correct answer — small enough that extra reasoning tokens, not price, decide which one is cheaper in practice.

GPT-6 Sol sits with the open-weight pair on both accuracy and cost, and it was the fastest model in the run.

What the effort setting did

Bedrock accepted six effort values on both open-weight models in my probes: none, low, medium, high, xhigh, and max. It rejected minimal, and none produced zero reasoning tokens. Moonshot's own documentation lists only low, high, and max for Kimi K3, so Bedrock exposes a wider surface than the vendor describes.

On three OWASP files, GLM 5.3's reasoning tokens did not rise steadily with the label: high used none on two of them, and low used about 2,700 tokens on the weak-hash file. A sheep-counting puzzle did rise steadily, from 29 tokens at low to 240 at max. I ran only max in the full test, so I can say nothing about cost or accuracy at lower settings. A team picking an effort level for production would want exactly that curve, and this test does not provide it.

What this test cannot show

Memorization. OWASP Benchmark and its answer key have been public for years. Opus 5.5's near-perfect score could reflect recall of the benchmark, and I cannot tell how much. This is my inference, and a post-cutoff set is the test that would settle it.
Scope. One benchmark, Java only, single files, with the weakness class named in the prompt. Whole-repository review and agentic harnesses are different tasks.
One repeat. My earlier OWASP runs showed wrong answers repeating across runs, so extra repeats add little. Gaps under about 4 points between two models are within the noise.
Latency. Approximate, because the runs shared a machine and overlapped in time.
Opus 5.5's refusal. One file returned a refusal with only a thinking block. I counted it as a miss, and Opus 5.5 scores 654 of 660 either way.

Other groups report results for these models on different tasks. Aikido's agentic fresh-CVE test found 25 of 32 for both Kimi K3 and GLM 5.3, which is a tie like mine. Simbian's defensive hunting benchmark put GLM 5.2 slightly ahead of Kimi K3. Z.ai's model card reports a CyberGym lead for GLM 5.3 that I did not test. I found no published OWASP Benchmark result for Kimi K3 or GLM 5.3.

So what

On single-file Java review at maximum effort, I would pick by the failure I can afford. Opus 5.5 gives up no vulnerable files and few false alarms, and charges about three times as much per correct answer. The open-weight pair and GPT-6 Sol are interchangeable on accuracy here, so price per correct answer, latency, and data-handling needs decide between them.

The result also sets up a specific question. Opus 5.5 was never wrong on a file where Kimi K3, GLM 5.3, or GPT-6 Sol was right — every file it missed, they missed too. That asymmetry is the precondition for a model cascade: run a cheap model on every file, and escalate to Opus 5.5 only on the ones it flags as uncertain. FrugalGPT named this pattern in 2023, and the broader cascade literature treats cheap-first, escalate-on-low-confidence as the standard cost play for exactly this kind of accuracy asymmetry. The same literature carries a specific warning that applies here: a 2026 measurement of cascade reliability found that a cheap verifier's blind spot — the fraction of wrong answers it waves through unescalated — grows with the weaker model's capability, and a frontier verifier closes that gap only by escalating on nearly half of all traffic, which erases most of the savings the cascade exists to produce. I have not built that cascade on this benchmark. Whether it lands near the 3x savings a clean cascade promises, or the near-zero savings the blind-spot result warns about, is the open thread — along with whether Opus 5.5's lead holds on code that was never public, which this test also cannot answer.

Most Enterprises Are Chasing AI Autonomy Before Defining the Outcome

Amit — Thu, 24 Sep 2026 18:00:35 +0000

TL;DR

Enterprises are adopting agents faster than they are redesigning the work those agents will perform.
The unit of autonomy is the delegation envelope: one actor, one goal, one allowed action set, one evidence contract, and one recovery boundary.
Capability, delegated authority, and operating maturity are separate scorecards. A system that can perform an action has not automatically earned permission to take it.
The goal is the minimum authority needed to improve a defined outcome, expanded only when the workflow can prove and recover from its work.

IBM found that 64% of 2,000 surveyed CEOs said fear of falling behind drives technology investment before value is clear. Another 61% were adopting AI agents and preparing to scale them, while only 25% of AI initiatives had delivered the expected return. Enterprises are chasing autonomy faster than outcomes.

That is why How autonomous should our AI be? is the wrong opening question. An enterprise does not have one autonomy level. A software-maintenance workflow can complete bounded work while a payment workflow requires approval for every consequential action. Customer service can resolve a routine request while legal review remains advisory.

The useful question is:

What outcome are we trying to achieve, how does the work produce it today, and what authority has each part of that workflow earned?

Start With the Outcome, Not the Agent

“Deploy an agent” is not a business outcome.

Reduce account-opening time. Resolve routine service requests without repeat contact. Cut invoice exceptions. Detect fraud before money moves. These are measurable outcomes. An agent is one possible actor alongside deterministic software, analytical models, policies, systems of record, and human judgment.

Task productivity is not workflow performance. An agent may draft an answer in seconds while the case waits two days for approval, or generate more code while review becomes the bottleneck.

McKinsey’s 2025 global survey found that 62% of respondents’ organizations were experimenting with agents, but only 23% were scaling an agentic system somewhere in the enterprise. In any individual function, no more than 10% reported scaling agents. Adoption, workflow penetration, and business value are different scorecards.

The outcome defines what “done” means, which decisions matter, what the agent may change, which errors are tolerable, when a person intervenes, and how success is measured. Without that sequence, autonomy becomes a technology objective detached from the work.

The Delegation Envelope Is the Real Unit

Authority lives in a delegation envelope: one actor, one goal, one allowed action set, one evidence contract, and one recovery boundary. Change any part and you have a different envelope. This is the unit that can be designed, audited, graduated, and revoked.

A workflow contains multiple envelopes at different levels. The same agent may hold advisory authority for a legal conclusion, bounded authority for assembling its evidence, and no authority to send the result externally. Calling the whole system “L3” hides the consequential boundaries.

Three scorecards must stay separate:

Capability: What can the system accomplish, and with what reliability?
Delegated authority: What effects may it cause without further approval?
Operating maturity: Can the organization observe, constrain, audit, interrupt, and recover that delegation?

“The model can do it” answers only the first question. The levels below describe envelope scope, not corporate progress. There is no universal L1–L5 standard for enterprise agents; this is my design model.

Level	What the envelope permits	Human responsibility
L1 — Advise	Analyze, draft, predict, or recommend. No material external action.	Performs the work and makes the decision
L2 — Per-action approval	One discrete action whose exact occurrence and consequential parameters receive explicit approval	Directs the action and reviews consequential parameters
L3 — Pre-authorized bounded workflow	A multi-step workflow whose class, boundaries, and evidence contract were approved in advance; individual runs are not separately approved	Approves the workflow class and its exceptions—not each step
L4 — Coordinating envelope	Sequences and selects among several already-bounded L2/L3 envelopes toward a defined outcome. Composed authority, not new authority.	Sets constraints, owns material deviations and cross-boundary exceptions
L5 — Continuing objective	Creates and revises its own plans toward a standing objective under policy, budget, and time limits	Sets strategy, prohibited actions, and revocation authority

The L2/L3 distinction is when approval happens: each occurrence at L2, the workflow class at L3. L4 composes bounded envelopes; if the coordinator gains authority none of them had, the boundary has leaked. L5 is a horizon, not a target. The public sources reviewed for this article did not document a standing-objective envelope with self-revised plans under material consequence.

Enterprise Workflows Are Landing at Different Levels

The ranges below are my assessment of public evidence available as of September 2026—not a market standard or benchmark. I use ranges because the evidence does not support point estimates, and I label vendor claims, regulatory boundaries, and thin evidence.

Enterprise workflow	Typical range	Basis, and its limits
Software engineering	L2–L3	Vendor product claim: GitHub documents that its coding agent implements bounded changes and opens pull requests; humans retain merge authority. Documented capability, not measured adoption depth.
IT and employee service	L2–L3	Vendor product claim: ServiceNow's platform announcement describes routine service workflows completing inside existing approvals. Marketing material; no independent outcome data.
Customer service	L2–L3	Routine resolution is the most commonly reported bounded-completion case in the surveys cited above; exceptions stay human-owned. Vendor deflection figures are not comparable across firms; I do not rely on them.
Sales	L1–L2	Weak evidence. No function-specific study cited. Inference from the ≤10% per-function scaling ceiling in McKinsey's survey plus the structural point that pricing and commitments are contractually binding. Hypothesis.
Marketing	L1–L2	Observed adoption: BCG reports agent-led workflows remain a minority; campaigns operate inside approved briefs and budgets.
HR	L1–L2	Observed adoption: employee service and onboarding advance. Regulatory boundary: employment decisions face EU AI Act high-risk obligations—a legal ceiling, not a measurement of practice.
Finance and accounting	L1–L2	Observed adoption: reconciliation and close preparation advance; assurance readiness and material approvals remain limiting (KPMG).
Procurement	L1–L2	Observed adoption: supplier analysis and low-value transactions advance; trust in autonomous decisions is the leading reported barrier (BCG).
Supply chain and logistics	L1–L2	Observed adoption: agents generate and execute narrow plans; cross-network trade-offs typically remain planner-approved (BCG).
Risk, legal, and compliance	L1	Observed adoption: research, review, monitoring, drafting, and evidence collection dominate (Thomson Reuters). Also subject to professional-responsibility limits.
Physical operations	L2–L3 in engineered environments only	Weak evidence. No study cited here covers robotics. Inference from the engineered-environment argument below: bounded robots operate where state and recovery are physically constrained. General-purpose physical agents are earlier. Treat as hypothesis.

Three patterns matter. Engineering leads where evidence and rollback are executable. Regulated functions move more slowly because their actions carry legal or financial consequence. Maturity varies within every function because workflows expose different boundaries.

Why Software Engineering Moved First

Software engineering already had the control system for bounded delegation: machine-readable state, tests, sandboxes, granular permissions, reviewable diffs, independent approval, and rollback.

GitHub's coding-agent documentation—a vendor product description—maps cleanly onto the envelope. The agent evaluates an issue, changes code, runs tests, and opens a pull request. The pull request is the evidence contract; merge authority is the boundary. The agent completes a pre-authorized workflow without inheriting authority over the software outcome.

Other functions need equivalent controls: simulation instead of a branch, deterministic business rules instead of tests, action diffs instead of code diffs, task-scoped credentials instead of deployment access, and compensating transactions instead of rollback.

The ICLR 2026 Measuring Agents in Production study found that across 306 practitioners and 20 case studies, 68% of production agents executed at most ten steps before human intervention, 74% relied primarily on human evaluation, and reliability remained the leading challenge. Production maturity currently comes from bounded authority, not unrestricted action.

A Worked Example: Invoice Exceptions

The outcome is not “process invoices with an agent.” It is “resolve valid exceptions within one business day without an unauthorized posting or payment.”

An L1 envelope classifies the exception and assembles evidence. L2 requests missing documentation under a pre-approved rule. L3 resolves named exception classes below a financial threshold, while payment release remains separately gated. Each step records the source data, rule, and proposed change. Recovery means reopening the exception, reversing a provisional posting, and revoking authority when error thresholds are exceeded.

The envelope earns wider scope only when its goal, action set, evidence contract, authority history, and recovery boundary strengthen together. Widening authority without strengthening those controls is weak governance wearing a level number.

What Executives Are Actually Deciding

Deloitte’s 2026 State of AI research found that 74% of surveyed leaders expected moderate or extensive agent use within two years. Yet only 21% reported mature governance for autonomous agents, 30% were redesigning key processes around AI, and 84% had not redesigned jobs around it.

The gap is concrete: organizations want class-level autonomy before they have evidence contracts that make class-level approval safe. L4 compounds the problem across decision rights, data definitions, permissions, budgets, and accountability. The composition itself needs an owner.

McKinsey found AI high performers were 2.8 times more likely to report fundamental workflow redesign than other respondents—55% versus 20%. That is an association, not proof of causation, but it is a stronger maturity signal than agent count.

The operating sequence is simple: observe the workflow, let the system advise, approve individual actions, pre-authorize proven workflow classes, then coordinate bounded workflows. At each step, collect outcome and exception evidence, monitor drift, and demote or revoke authority when the evidence deteriorates.

The NIST AI Risk Management Framework asks organizations to map an AI system’s intended purpose, context, affected actors, objectives, and legal requirements. NIST’s February 2026 draft on agent identity asks how standards can support identification, authorization, auditing, and non-repudiation. My architectural implication is that agents should receive scoped, auditable delegation rather than silently inherit a person’s full access.

What’s Missing

The market has plenty of agent frameworks. It has fewer mechanisms for operating authority.

Business workflows still lack standard equivalents for tests, action diffs, merge protection, and rollback. Evaluations score answers while production risk lives in tool use, permissions, side effects, and exceptions. Weak human review becomes an approval button without enough context or power to refuse.

The missing layer is not another autonomy label. It is an authority lifecycle that defines the envelope, gathers evidence in simulation and production, graduates scope within explicit thresholds, monitors drift, and revokes authority when performance deteriorates.

So What

Ask which envelope you are designing: whose goal, which actions, what evidence, what recovery, and whether approval happens per action or per workflow class. Some envelopes should remain advisory permanently. Maturity is knowing which can move—and having a mechanism for moving them back.

Here is a bet that can be checked. By September 2028, enterprises publicly reporting durable financial outcomes from agents will be distinguished by workflow-level evidence infrastructure—action diffs, scoped credentials, compensating transactions, and named envelope owners—rather than by the breadth or autonomy level of their deployments. I also expect no credible public case of a sustained L5 envelope operating over material consequence by that date.

I am wrong if enterprises scale agent authority broadly and profitably without those controls, or if a standing-objective agent runs a consequential business objective with self-revised plans and holds up under audit.

The question I have not resolved is how much evidence a workflow should accumulate before it moves up one rung—and who should set that threshold when the benefits and the cost of failure fall on different functions.

Wiring Codex and ChatGPT Desktop to OpenAI Models on Amazon Bedrock Runtime

Amit — Wed, 23 Sep 2026 04:54:39 +0000

Codex CLI and the ChatGPT desktop app can use the same native Amazon Bedrock Runtime configuration. The working pattern has three parts: the built-in Runtime provider, an AWS profile and Region, and a global model profile ID.

I verified this pattern on September 11, 2026 with four OpenAI model profiles: GPT-5.6 Sol, Terra, Luna, and GPT-6 Astra. All four completed headless runs through the native Runtime provider in the desktop app's bundled Codex 0.153.4. Astra also passed with Codex CLI 0.154.0 using the same saved configuration.

TL;DR

Codex CLI and the ChatGPT desktop app read the same local Codex configuration, so one provider definition can serve both surfaces.
The endpoint remains regional, while a global.openai.* model ID selects global cross-Region inference.
The built-in amazon-bedrock-runtime provider uses an AWS profile directly; no static API key, token generator, or local proxy is required.
amazon-bedrock and amazon-bedrock-runtime are different providers. The first selects Mantle in the tested clients; the second selects Bedrock Runtime.

The working architecture

OpenAI documents shared local configuration across Codex surfaces. Amazon Bedrock documents the Runtime endpoint as the recommended endpoint for new applications, including OpenAI-compatible Responses and Chat Completions paths under /openai/v1.

The resulting request path is direct:

Codex CLI or ChatGPT desktop
        |
        | reads ~/.codex/config.toml
        v
Native amazon-bedrock-runtime provider
        |
        | uses AWS profile and Region
        v
bedrock-runtime.<region>.amazonaws.com/openai/v1
        |
        v
global.openai.<model>

There is no gateway in this path. AWS credentials stay in the normal local profile chain, and Codex sends Responses API traffic directly to Bedrock Runtime.

The word global applies to the model profile, not the hostname. The network endpoint still contains a Region such as us-west-2; the global.openai.* model ID tells Bedrock to use its global cross-Region inference profile. The Bedrock Responses API documentation shows the Runtime base URL and global profile naming pattern.

Prerequisites

The setup needs:

A current Codex CLI installation
The ChatGPT desktop app with Codex support
An AWS profile with permission to invoke the selected Bedrock models

Confirm the AWS session before changing Codex:

aws sts get-caller-identity --profile YOUR_AWS_PROFILE

The shared Codex configuration

Add the following provider to ~/.codex/config.toml. Replace YOUR_AWS_PROFILE and select a Bedrock Region where the models are available.

model = "global.openai.gpt-6-astra"
model_provider = "amazon-bedrock-runtime"
model_reasoning_effort = "medium"

[model_providers.amazon-bedrock-runtime.aws]
region = "us-west-2"
profile = "YOUR_AWS_PROFILE"

The Codex configuration sample documents AWS profile and Region settings for built-in Amazon Bedrock providers. This configuration references the selected profile rather than embedding access keys. The profile can still use AWS IAM Identity Center, role assumption, environment credentials, or another supported AWS credential source.

Global OpenAI model profiles

These are the four global profile IDs I verified:

Model	Codex model ID	Verified result
GPT-5.6 Sol	`global.openai.gpt-5.6-sol`	Native Runtime passed
GPT-5.6 Terra	`global.openai.gpt-5.6-terra`	Native Runtime passed
GPT-5.6 Luna	`global.openai.gpt-5.6-luna`	Native Runtime passed
GPT-6 Astra	`global.openai.gpt-6-astra`	Native Runtime passed

AWS introduced Sol, Terra, and Luna on Amazon Bedrock in July 2026 and announced GPT-6 Astra availability in September 2026. Model access still depends on the account, selected Region, and current Bedrock catalog, so discovery in one account is evidence for that environment rather than a universal availability guarantee.

Set Astra as the default in the configuration, or select a model for an individual Codex run:

codex -m global.openai.gpt-5.6-sol
codex -m global.openai.gpt-5.6-terra
codex -m global.openai.gpt-5.6-luna
codex -m global.openai.gpt-6-astra

Verify the Runtime path

A headless run provides a clean test because it removes the interactive interface from the request path:

codex exec \
  --ephemeral \
  --skip-git-repo-check \
  -m global.openai.gpt-5.6-sol \
  "Reply with exactly SOL_RUNTIME_OK"

Repeat the probe with the other model IDs and a distinct expected string. My four probes returned:

SOL_RUNTIME_OK
TERRA_RUNTIME_OK
LUNA_RUNTIME_OK
ASTRA_RUNTIME_OK

This verifies more than model discovery. It confirms that Codex can load the provider, resolve the named AWS profile, reach the Runtime endpoint, select the global profile, and receive a model response.

For the desktop app, fully quit and reopen the application after changing ~/.codex/config.toml. The model picker should then expose the same provider catalog. Existing conversations may retain their original model selection, so use a new task when validating a newly selected model.

The provider name selects the endpoint

The two built-in Bedrock provider names do not select the same endpoint.

Provider	Endpoint selected in the tested clients
`amazon-bedrock`	Bedrock Mantle
`amazon-bedrock-runtime`	Bedrock Runtime

This distinction is visible in an actual call. With Codex 0.153.4, amazon-bedrock sent Astra to a bedrock-mantle URL and received a model-not-found response. The same binary, model ID, AWS profile, and Region succeeded after changing only the provider to amazon-bedrock-runtime.

The native provider passed across both tested Codex versions:

Desktop-bundled Codex 0.153.4 returned SOL_NATIVE_OK, TERRA_NATIVE_OK, LUNA_NATIVE_OK, and ASTRA_NATIVE_OK.
Standalone Codex CLI 0.154.0 returned CLI_NATIVE_RUNTIME_OK from the same saved provider configuration.

Inspect the versions when diagnosing behavior:

codex --version

The desktop app's bundled version is visible in its diagnostic output. Version differences still matter for model catalogs and client features, but they do not require a custom provider for this Runtime path.

Direct Runtime first, gateway when required

Direct Bedrock Runtime is the lower-complexity path when AWS identity, model access, logging, and billing controls meet the requirement. An AWS reference architecture for Codex with LiteLLM positions a gateway as an additional layer for centralized routing, budgets, or policy controls—not as a requirement for basic Codex connectivity.

That distinction matters. A local proxy or hosted gateway adds another credential boundary, another failure point, and another component to operate. Add one when it supplies a control the direct path does not provide.

What remains open

The native Runtime provider passed direct Codex execution in both binaries. The desktop app also restarted with successful configuration and model-catalog reads. I have not yet tested every desktop-only helper action against all four global model profiles; those flows can select models independently from the main conversation.

The configuration boundary is the part to keep under test: one file, two client surfaces, four global model profiles, and one AWS profile. When any client version changes, rerun one headless probe and one fresh desktop task before treating the wiring as stable.

So what: Bedrock model access is only one layer of the setup. The complete contract is endpoint, API format, authentication, model profile, and client configuration—and all five need to work together.

Alternative AI Hardware Is a Systems Problem

Amit — Wed, 23 Sep 2026 04:54:04 +0000

Alternative AI hardware is not one product category.

Huawei Ascend is becoming a complete compute, interconnect, systems, and software platform. FuriosaAI RNGD is a power-conscious inference accelerator that fits into standard servers. Rebellions REBEL starts with a PCIe NPU and scales into multi-card and rack-level systems.

All three reduce dependence on the dominant GPU stack. They do it at different boundaries, with different software commitments and different evidence. This is an architecture comparison; public evidence is not yet consistent enough for a normalized performance ranking.

TL;DR

Huawei competes at the platform boundary: accelerator, scale-up fabric, clusters, compiler, framework, and cloud infrastructure.
FuriosaAI narrows the problem to efficient transformer inference in a familiar PCIe server form factor.
Rebellions combines a data-center NPU with PCIe, CXL-oriented memory expansion, scale-out Ethernet, and rack-level system designs.
The strategic test is not whether an NPU has competitive peak arithmetic. It is whether the surrounding system can run supported models reliably, at the required quality, with enough software and operational evidence.

Three architectures, three ownership boundaries

The phrase “alternative accelerator” hides the most important distinction: who owns the system around the silicon?

Platform	Primary boundary	Physical unit	Software commitment	Strategic objective
Huawei Ascend	Full platform	Chip, server, SuperPod, SuperCluster, and cloud	CANN, MindSpore, framework adapters, cluster software	Build an integrated sovereign compute stack
FuriosaAI RNGD	Efficient inference accelerator	PCIe card, server, and partner system	Furiosa SDK, compiler, runtime, and model support	Add efficient inference capacity to standard data centers
Rebellions REBEL	Scalable inference NPU	PCIe card, multi-card system, rack, and cloud service	RBLN SDK, compiler, runtime, serving integration	Scale transformer inference from card to rack

These systems should not be collapsed into one benchmark row. Huawei is attempting to reproduce much of the platform surrounding a GPU. Furiosa and Rebellions can coexist beside GPU pools because their first deployment boundary remains a server accelerator.

flowchart TB
    APP["Models and applications"]

    subgraph H["Huawei platform boundary"]
        HC["Ascend accelerator"]
        HF["UnifiedBus / system fabric"]
        HS["SuperPod and SuperCluster"]
        HSW["CANN, MindSpore, cluster software"]
        HC --> HF --> HS
        HSW --> HS
    end

    subgraph F["Furiosa deployment boundary"]
        FC["RNGD PCIe cards"]
        FS["Standard inference server"]
        FSW["Furiosa SDK and runtime"]
        FC --> FS
        FSW --> FS
    end

    subgraph R["Rebellions deployment boundary"]
        RC["REBEL PCIe NPUs"]
        RS["Multi-card server or rack"]
        RSW["RBLN SDK and serving stack"]
        RC --> RS
        RSW --> RS
    end

    APP --> HSW
    APP --> FSW
    APP --> RSW

The operator accepts the largest platform commitment with Huawei. The smaller vendors ask for a narrower model and serving commitment, but depend more heavily on host CPUs, merchant networking, server partners, and external orchestration.

Huawei is building the whole stack

Huawei’s architecture has moved far beyond a single Ascend card.

The company’s Atlas 900 A3 SuperPoD combines Ascend compute, high-speed interconnect, storage, management, and liquid-cooled cabinets into a scale-up system. Huawei describes a 384-card configuration with 300 TB of HBM and 122.8 TB/s of scale-up bandwidth.

The published SuperCluster architecture expands that boundary again. Multiple SuperPods connect through the company’s UnifiedBus fabric, with separate scale-up and scale-out layers. Huawei announced Atlas 950 and Atlas 960 SuperPods based on future Ascend generations and described SuperClusters containing hundreds of thousands to more than one million cards.

Those figures are roadmap and vendor architecture claims. They establish direction and intended scale, not measured production efficiency.

The software boundary is equally broad. Huawei’s CANN platform provides compilers, libraries, runtime components, development tools, and framework adapters for Ascend hardware. MindSpore supplies a Huawei-backed framework path, while PyTorch and other frameworks rely on integration layers.

That creates an integrated stack:

Model and framework
  → CANN graph compilation and kernels
  → Ascend runtime and collectives
  → UnifiedBus scale-up fabric
  → SuperPod
  → SuperCluster and cloud operations

The benefit is one architecture across silicon, fabric, server, and software. The cost is a larger migration and validation boundary. Kernel coverage, numerical behavior, framework compatibility, tooling, and distributed execution all need to match the model portfolio.

Huawei is therefore the closest of these three to a sovereign platform strategy. The objective is not merely efficient inference. It is control over the complete compute system and its supply path.

FuriosaAI keeps the facility boundary familiar

RNGD takes a narrower approach.

The second-generation FuriosaAI accelerator is a 180-watt PCIe card with 48 GB of HBM3. The company positions it for large-language and multimodal inference, with two cards fitting inside a conventional server power and cooling envelope.

RNGD’s Tensor Contraction Processor uses a programmable tensor engine rather than exposing a GPU-style collection of fixed matrix units. Furiosa argues that the architecture can map a wider set of tensor shapes while keeping control overhead and data movement low.

The data-center contract remains recognizable:

PCIe attachment;
standard x86 host servers;
air-cooled operation;
HBM on each card;
Kubernetes and container-based deployment;
a vendor compiler and runtime behind supported frameworks.

That matters because adopting a specialist involves two separate changes. The first is physical: power, cooling, network, server qualification, and sparing. The second is logical: model compilation, operator support, quantization, serving, and observability. A 180-watt PCIe card reduces the first change. It does not remove the second.

Furiosa’s software stack includes a compiler, runtime, model tooling, and serving integration. The company publishes supported models and performance material, but the usable boundary remains model-specific. An architecture can execute transformers in principle while still lacking a production-ready path for a particular operator, attention variant, quantization method, or serving engine.

Vendor and partner evidence now includes RNGD volume shipments, a live Samsung SDS NPU-as-a-Service offering, and announced data-center installations. These sources establish commercial activity, but they are not independently normalized production measurements. The key missing figures are sustained utilization, failure rates, model onboarding time, power at the complete server boundary, and independently comparable service cost.

Rebellions scales a PCIe NPU toward the rack

REBEL also starts as a standard data-center accelerator.

The card provides 144 GB of HBM3E and connects through PCIe Gen5. Rebellions describes native support for several low-precision formats and positions the design for large-model inference. A REBEL-Quad system combines four accelerators behind one host, creating a larger local memory and compute domain.

The architecture then extends through merchant data-center interfaces:

PCIe for host attachment;
CXL-oriented memory expansion in system designs;
Ethernet for scale-out;
liquid-cooled rack configurations for higher density;
a Kubernetes-based operational path.

Rebellions’ RBLN software stack provides the compiler, runtime, framework integration, model zoo, and serving tools required to turn checkpoints into executable artifacts. As with Corsair and RNGD, that compiler is part of the product. The hardware cannot be evaluated separately from supported operators, model conversion, quantization, and distributed-serving behavior.

The company has announced cloud and telecom infrastructure work, including a 2026 collaboration with SK Telecom and Arm, along with server-system partnerships. These announcements show an operating ecosystem. They do not yet create a common benchmark boundary across REBEL, RNGD, Ascend, and GPUs.

Rebellions occupies the middle of the three strategies. It does not own a full sovereign software and fabric stack like Huawei, but it aims beyond a single add-in card by defining multi-card and rack-scale inference systems.

Efficiency claims need a service boundary

Each vendor presents performance-per-watt or cost advantages. Those claims can be true within their test boundary and still fail to answer a production question.

The useful unit is not peak TOPS. It is a completed unit of model service:

Measurement	Required boundary
Latency	Prompt length, output length, batch policy, streaming behavior, and percentile
Throughput	Concurrent requests, quality target, scheduler policy, and complete system
Accuracy	Model revision, dataset, quantization, and numerical tolerance
Power	Card, host server, network, cooling, or facility boundary
Cost	Hardware utilization, software work, support, spares, and facility life
Portability	Time and engineering required to qualify another model
Availability	Card, server, rack, compiler, and service failure rates

A PCIe NPU can draw less power than a high-end GPU and still produce a worse service if the model does not compile cleanly, the cards remain underutilized, or requests spill back to a costly compatibility pool.

The reverse is also true. A specialist does not need universal model coverage to create value. It needs enough stable demand for supported models to keep the pool utilized.

Two meanings of sovereignty

These architectures separate two meanings of sovereignty.

One is supply sovereignty: control over silicon, manufacturing access, networking, system design, and software. Huawei is pursuing that broader boundary.

The other is operational choice: adding accelerators from more than one supplier so stable workloads are not tied to one runtime and pricing model. Furiosa and Rebellions contribute to that boundary without replacing the rest of the data-center stack.

What remains unproven

The vendors have demonstrated supported transformer models through published systems, model catalogs, partner deployments, or cloud endpoints. That establishes execution capability for those documented configurations, not broad model coverage or equal production economics.

The gap is whether the surrounding systems can sustain a broad production workload with predictable onboarding time, quality, utilization, and availability. Public material still lacks a consistent comparison of:

the same current model and serving software;
the same quantization and accuracy target;
identical latency percentiles and batch policy;
card, server, rack, and facility power;
compiler and model-porting effort;
system availability and replacement time;
software support across successive model generations.

That missing evidence affects each architecture differently. Huawei must prove the breadth and operability of an integrated platform. Furiosa must prove that a narrow, efficient pool can stay busy across enough models. Rebellions must prove that its card-to-rack expansion preserves the efficiency and simplicity promised at the card boundary.

Alternative AI hardware becomes credible when the complete operating system around the chip is visible. The unresolved question is who will publish enough production evidence to compare that system honestly—without reducing architectural choice to a peak-performance table or treating every alternative as the same kind of accelerator.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

SambaNova Compiles Models Into a Rack

Amit — Wed, 23 Sep 2026 04:53:28 +0000

SambaNova's most important product is not its accelerator chip. It is a compiled model deployment that spans the chip, three memory tiers, a 16-RDU rack, and the software routing requests across them.

That system boundary separates SambaNova from a conventional accelerator card. A model configuration becomes a Processor Executable Format binary, its weights occupy a planned memory hierarchy, and SambaStack deploys the result as a model service rather than treating every request as an arbitrary program arriving at a general-purpose processor.

The thesis: SambaNova compiles models into a rack. The Reconfigurable Dataflow Unit, or RDU, is the execution engine, but the useful commercial unit is the complete inference system around it.

TL;DR

The RDU maps a model's dataflow graph onto programmable compute, memory, and communication resources instead of repeatedly launching isolated kernels.
SN40L combines distributed SRAM, HBM, and attached DDR; the memory system is part of model placement, not merely storage behind the processor.
The mature deployment boundary is SambaRack SN40-16 plus SambaStack on-premises, or the same architecture exposed through SambaCloud.
SambaNova's software compiles specific model, sequence-length, batch-size, and speculative-decoding configurations into deployable binaries and validates whether a deployment fits available RDU memory.
SN50 extends the thesis toward larger scale and disaggregated inference, but its strongest public evidence remains preview benchmarks and vendor roadmap claims rather than broad production measurements.

Dataflow changes what the compiler owns

CUDA's programming model presents a GPU as a device executing kernels. Each kernel performs a bounded operation, while software schedules launches and moves tensors through the memory hierarchy. Fusion can reduce that traffic, but the execution model remains kernel-oriented.

SambaNova makes a different cut. Its current Dataflow Architecture description shows a grid of programmable compute units and SRAM-based programmable memory units connected as a streaming pipeline. While one operator computes, the system can fetch data for the next. Intermediate activations stay near the operators consuming them.

The underlying idea predates the current product naming. SambaNova's architecture material describes an RDU as a tiled array of compute and memory units joined by a programmable communication fabric. The SN40L paper presented at MICRO 2024 adds the current system details: dataflow cores contain compute units, memory units, address-generation units, and an on-chip reconfigurable network.

This does not mean the RDU executes without instructions or memory movement. It means the compiler can plan more of the communication path and keep a longer section of the model graph resident as a spatial pipeline.

flowchart TD
    A[Model checkpoint] --> B[Conversion and compilation]
    B --> C[PEF: compiled model configuration]
    A --> D[Model weights]
    C --> E[ModelProfile: supported runtime shapes]
    D --> F[Model: checkpoint and tokenizer]
    E --> G[Model and profile pair]
    F --> G
    G --> H[Optional ModelBundle]
    G --> I[ModelDeployment]
    H --> I
    I --> J[Memory-fit legalizer]
    J --> K[Inference router and runtime]
    K --> L[Attached DDR: model catalog and cache]
    L --> M[HBM: weight streaming]
    M --> N[Distributed SRAM: active data]
    N --> O[PCU and PMU dataflow pipeline]
    O --> P[OpenAI-compatible API response]

The diagram exposes the adoption trade. A familiar checkpoint enters the system, but it does not remain an arbitrary framework program at runtime. The current SambaStack deployment model defines a PEF as a compiled model binary for an RDU. A ModelProfile groups compatible PEFs and the runtime shapes they support. A Model identifies checkpoints and tokenizers, while a ModelDeployment creates replicas and a routable endpoint. ModelBundle is now optional: it groups several model/profile pairs when they need to be deployed, validated, or shared as one unit.

That compilation boundary is where SambaNova gains control over data movement. It is also where model support, graph coverage, compiler maturity, and artifact lifecycle become commercial constraints.

Three memory tiers make the rack the machine

The RDU's second architectural choice is capacity placement. The current SambaRack SN40-16 datasheet specifies 16 SN40L RDUs with three memory tiers:

Memory tier	SN40-16 published capacity	System role
Distributed on-RDU SRAM	8 GB total	Keeps active operands and intermediate data close to compute
HBM	1 TB total	Streams model weights at high bandwidth
RDU-attached DDR	4 TB total	Holds checkpoints, prompt cache, and a catalog of model configurations

The same rack contains host CPUs, NVMe storage, management components, and 400/200 GbE networking. SambaNova publishes 10 kW typical and 16.4 kW maximum inference power for the air-cooled rack. Those are rack specifications, not measured energy per token.

This hierarchy supports a different capacity strategy from placing every active model entirely in HBM. Large DDR capacity can hold more weights and configurations close to the RDUs. HBM provides the high-bandwidth path, while SRAM feeds the active graph. SambaNova's claim is that this enables fast switching among resident models and reduces repeated transfers from external storage.

The software reveals how tightly memory and deployment are coupled. SambaStack runs a legalizer before deployment to verify that the selected PEFs and checkpoints fit the node's memory constraints. A deployment can include multiple models, sequence-length profiles, batch sizes, and a target/draft pair for speculative decoding. The runtime then selects among compiled configurations according to the request.

The deployable object is therefore not "a model on an RDU." It is a memory-valid deployment distributed across a rack.

The commercial boundary has three forms

SambaNova now exposes the same architectural thesis through different operational boundaries.

Offering	Customer operates	SambaNova supplies	Practical boundary
SambaStack	Data center, network, Kubernetes integration, identity, storage, and service policy	RDU rack, runtime, management software, compiled artifacts, and serving stack	Customer-operated private or sovereign inference
SambaManaged	Facility boundary and application integration	Fully managed inference infrastructure installed in the customer's data center	Dedicated capacity without operating the full stack
SambaCloud	API client, model selection, and application	Infrastructure, model deployment, routing, and service operation	Multi-model inference API

The on-premises path is not a simple appliance with one network cable. SambaNova's current installation guide requires a front-end data network, a private inter-rack network, a management network, customer identity and DNS services, storage, load balancing, and a Kubernetes control plane. SambaStack installs through Helm and exposes deployed models through an OpenAI-compatible API.

At the other end, SambaCloud hides the compiled artifacts and rack operations. Its July 2026 release added prompt caching and continued expanding API compatibility, tool use, structured output, and model support. That surface is easier to adopt, but it offers the curated models and configurations SambaNova has prepared rather than arbitrary code execution.

This is the commercial read: the buyer chooses how much of the system boundary to own. Infrastructure operators can run a private RDU system, let SambaNova operate dedicated capacity in their facility, or consume the architecture as an API without seeing the compiler boundary.

SN40L is deployed; SN50 is the next claim

The generation boundary matters because SambaNova's public material now discusses SN40 and SN50 together.

SN40L is the fourth-generation RDU behind the documented SambaRack and the current on-premises software guides. Independent deployment evidence exists. The Argonne Leadership Computing Facility opened its Metis system to researchers in October 2025 with two 16-RDU SambaRacks. Its current user guide describes a larger six-rack SN40L cluster serving multiple models through OpenAI-compatible endpoints.

Earlier systems also operated alongside conventional supercomputers. Lawrence Livermore National Laboratory integrated SambaNova dataflow hardware into its computing environment for cognitive simulation research. That establishes real installation history, although the older system used an earlier RDU generation and does not prove current LLM-serving economics.

SN50 is SambaNova's fifth-generation inference processor. The company says a 16-chip SambaRack averages 20 kW, offers five times the compute and four times the network bandwidth of SN40, and can connect as many as 256 RDUs across racks. Its February 2026 announcement said customer shipments would begin in the second half of 2026.

The most interesting SN50 direction is not a larger standalone rack. It is disaggregated inference. SambaNova demonstrated NVIDIA GPUs handling compute-heavy prefill while RDUs handled memory-bound decode. A July 2026 demonstration used four H200 GPUs for prefill and one 16-RDU SN50 rack for decode on MiniMax M2.7.

The evidence has advanced beyond first-party performance charts, but it remains early. SambaNova's summary of a SemiAnalysis-run benchmark reports roughly 800 tokens per second at the highest-interactivity point and roughly 400 tokens per second as throughput approached the fastest B200 configuration tested on the same model. The available public summary does not provide the power, utilization, availability, or economic data required for a complete rack comparison.

That changes the competitive framing. SambaNova does not need to replace every GPU to become useful. It can become a specialized decode tier behind a serving layer that sends each phase to different hardware.

Where the architecture fits

Workload	Architectural fit	Evidence boundary
Interactive LLM decode	Strong: sequential generation rewards memory bandwidth, locality, and planned execution	Artificial Analysis independently measures high output speed on several SambaCloud models, but endpoint tests do not reveal rack utilization or energy
Multi-model and agent workflows	Strong in principle: large DDR capacity and deployments keep multiple models and configurations resident	Fast switching and model bundling are documented primarily by SambaNova
Private or sovereign inference	Strong: complete air-cooled racks can run on-premises and expose familiar APIs	Installation requirements still include Kubernetes, networking, identity, storage, and vendor-provided artifacts
Disaggregated prefill and decode	Promising: GPUs retain prefill while RDUs specialize in decode	Public SN40/SN50 demonstrations exist; broad production measurements do not
Custom fine-tuned models	Supported through checkpoint conversion, PEFs, and custom bundles	Support depends on compiler coverage and validated configurations
Large-scale model training	Historically supported by earlier SambaNova systems	Current SN40/SN50 positioning, software, and evidence concentrate on inference
Arbitrary or rapidly changing research graphs	Weaker fit	Compilation and validated artifacts add friction when operators change models or shapes frequently

Artificial Analysis currently reports SambaCloud output rates above 700 tokens per second for some gpt-oss-120b configurations and above 400 tokens per second for MiniMax M2.7. Those measurements establish that the public service can deliver fast single-request generation. They do not establish aggregate rack throughput, sustained concurrency, power efficiency, availability, or total service cost.

That distinction matters. The architecture is optimized around planned model execution, but a production service also contains queues, routers, caches, network paths, replicas, failures, and model-specific quality choices. Tokens per second from one endpoint are evidence about the service experience—not a complete system benchmark.

What's missing

SambaNova publishes enough material to understand the shape of the system. It does not publish enough comparable data to close the economic argument.

There is no current, public evidence set that places SN40 or SN50 beside competing systems with the same model, precision, context distribution, batch policy, concurrency, accuracy target, measured wall power, and availability window. The SN40L MICRO paper is detailed, but its headline comparisons focus on SambaNova's Composition of Experts workload rather than a neutral industry benchmark. The SN50 material is earlier still.

Compiler coverage is the other open boundary. SambaStack documents stable, preview, deprecated, and removed PEF and checkpoint versions. That is healthy operational machinery, but it also shows that model support is a maintained product matrix. A framework checkpoint being available does not mean every shape, feature, quantization, or serving optimization is immediately deployable on an RDU.

So what

SambaNova is best understood as a system for controlling model movement.

The RDU turns a model graph into a spatial dataflow pipeline. The memory hierarchy decides what remains resident and what streams into active execution. The compiler turns framework artifacts into hardware-specific configurations. SambaStack packages those configurations into a rack-level service, while SambaCloud hides the machinery behind an API.

That full boundary is the product. Comparing one RDU with one GPU misses it.

The unresolved question is whether SambaNova can keep compiler and artifact delivery ahead of changing model architectures while proving that phase-specific routing still wins after network, utilization, power, and operational costs are included.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

The GPU Stopped Being the Product

Amit — Wed, 23 Sep 2026 04:52:53 +0000

NVIDIA's most important architectural change is not a faster GPU. It is the expansion of what counts as the computer.

The Tesla P100 arrived as an accelerator inside a server. The A100 made the eight-GPU system a repeatable unit. Hopper joined the CPU and GPU more tightly. Blackwell added a top-end configuration whose scale-up boundary is a liquid-cooled rack. Vera Rubin extends the design across compute racks, networking, power, cooling, and inference orchestration.

At the high end, the product is no longer only a chip. It is a large part of the AI data center assembled as one machine: Tensor Cores change the math, HBM feeds it, NVLink expands the scale-up domain, networking connects racks, and the software stack makes each generation usable.

TL;DR

P100 established the modern foundation with HBM2, NVLink, and the first DGX system; V100 added Tensor Cores.
A100 turned the GPU into elastic infrastructure through TF32, structured sparsity, Multi-Instance GPU, faster NVLink, and a standardized eight-GPU NVSwitch system.
Hopper optimized transformers directly through FP8 and Transformer Engine, while Grace Hopper introduced coherent CPU-GPU coupling.
Blackwell introduced a top-end 72-GPU liquid-cooled rack with 130 TB/s of summed bidirectional NVLink endpoint bandwidth, while B200 also remained available in server configurations.
Vera Rubin doubles NVLink bandwidth again and raises HBM bandwidth to 22 TB/s per GPU. Groq-derived LPX racks add specialized inference paths, while Rubin remains in production ramp rather than broad availability.

Start before A100

The useful starting point is the Tesla P100 and Pascal GP100 architecture in 2016. P100 introduced three pieces that still define NVIDIA's systems:

HBM on the GPU package for high-bandwidth access to model data.
NVLink for GPU-to-GPU communication beyond PCIe.
DGX-1, an integrated eight-GPU server with a tested software stack.

P100 did not have Tensor Cores. Its AI acceleration came from running FP16 arithmetic at twice the rate of FP32 on ordinary CUDA cores. The Tesla V100 Volta architecture made the next decisive move in 2017: 640 first-generation Tensor Cores dedicated to matrix multiplication.

Pascal made the GPU a better parallel processor. Volta began shaping its numerical machinery around neural networks.

flowchart LR
    P["P100 Pascal<br/>2016<br/>HBM2 + NVLink + DGX-1"] --> V["V100 Volta<br/>2017<br/>Tensor Cores"]
    V --> A["A100 Ampere<br/>2020<br/>TF32 + sparsity + MIG"]
    A --> H["H100/H200 Hopper<br/>2022–2024<br/>FP8 + Transformer Engine"]
    H --> B["B200 Blackwell<br/>2024 architecture<br/>dual-die GPU + NVL72"]
    B --> BU["B300 Blackwell Ultra<br/>2025 architecture<br/>288 GB HBM3e"]
    BU --> R["Vera Rubin<br/>2026 production ramp<br/>HBM4 + NVLink 6"]
    R -. roadmap .-> RU["Rubin Ultra<br/>2027"]
    RU -. roadmap .-> F["Feynman<br/>2028"]

The names hide overlapping product lines. H200 is a Hopper memory upgrade, while Blackwell Ultra expands Blackwell's memory and low-precision throughput. Grace Hopper, GB200, and Vera Rubin are platforms spanning CPUs, GPUs, memory, interconnects, and systems.

The generation map

The table uses representative high-end data-center configurations. SXM and rack-scale modules run at different power and bandwidth levels from PCIe cards with the same architecture. Peak Tensor Core figures also change numerical format across generations, so they are not one continuous performance benchmark.

Platform	Architectural step	Representative memory	Scale-up fabric per GPU	Deployment unit	Status on Aug. 30, 2026
P100 Pascal	HBM2, NVLink 1, high-rate FP16	16 GB, 0.72 TB/s	0.16 TB/s bidirectional	Eight-GPU DGX-1 with a direct NVLink mesh	Legacy
V100 Volta	First Tensor Cores, NVLink 2	16–32 GB, 0.9 TB/s	0.30 TB/s bidirectional	DGX-1 mesh; DGX-2 introduced NVSwitch	Legacy
A100 Ampere	TF32, BF16, FP64 Tensor Cores, 2:4 sparsity, MIG	40–80 GB, up to 2.0 TB/s	0.60 TB/s bidirectional	Eight-GPU DGX/HGX NVSwitch baseboard	Mature production
H100 Hopper	FP8, Transformer Engine, Tensor Memory Accelerator	80 GB, 3.35 TB/s	0.90 TB/s bidirectional	Eight-GPU HGX/DGX; external NVLink Switch systems	Mature production
H200 Hopper	Hopper compute with larger, faster HBM3e	141 GB, 4.8 TB/s	0.90 TB/s bidirectional	H100-compatible system generation	Mature production
B200 Blackwell	Two reticle-sized dies act as one GPU; FP4; NVLink 5	180 GB, up to 8 TB/s	1.8 TB/s bidirectional	DGX/HGX server or GB200 NVL72 rack	Production
B300 Blackwell Ultra	288 GB HBM3e, more NVFP4 and attention throughput	288 GB, up to 8 TB/s	1.8 TB/s bidirectional	GB300 NVL72 rack	Production
Vera Rubin	HBM4, sixth-generation Tensor Cores, NVLink 6, Vera CPU	288 GB, 22 TB/s	3.6 TB/s bidirectional	Vera Rubin NVL72 and pod-scale platform	Production ramp; shipments scheduled for fall 2026
Rubin Ultra and Feynman	Larger scale-up domain, then next named architecture	Not fully disclosed	Not fully disclosed	2027 and 2028 roadmap	Roadmap

The status boundary matters. On August 27, AWS received its first Vera CPU server and Rubin GPU for deployment work. NVIDIA still describes production shipments as beginning in fall 2026. Early component delivery is not general availability of complete NVL72 systems.

Three caveats matter:

NVIDIA's NVLink numbers add transmit and receive bandwidth. They are not one-way payload rates.
Sparse throughput assumes the required structured sparsity pattern. It is not dense performance.
FP16, TF32, FP8, FP4, and NVFP4 trade precision, range, storage, and compute differently. A larger headline FLOPS number at a narrower format does not mean every workload runs proportionally faster.

Component 1: Compute changed from general arithmetic to model-aware arithmetic

P100 accelerated FP16 on CUDA cores. V100 added dedicated Tensor Cores that multiplied FP16 matrices and accumulated in FP32, giving neural-network matrix operations their own hardware.

A100's third-generation Tensor Cores added TF32, BF16, IEEE FP64, INT8, INT4, and 2:4 structured sparsity. One matrix engine now covered training, inference, and scientific computing.

The H100 architecture added FP8 and the first Transformer Engine, coordinating lower- and higher-precision execution by transformer layer.

Blackwell extended that direction to FP6 and FP4 with a second-generation Transformer Engine. Rubin's third-generation Transformer Engine adds sixth-generation Tensor Cores and up to 50 petaflops of NVFP4 inference compute according to NVIDIA. That is a vendor peak, not a cross-generation application benchmark.

Generation	Numerical shift	What changed architecturally
Pascal	FP16 on CUDA cores	AI uses the general parallel processor more efficiently
Volta	FP16 Tensor Cores with FP32 accumulation	Matrix multiplication receives dedicated hardware
Ampere	TF32, BF16, FP64 Tensor Cores, INT8/4, 2:4 sparsity	One matrix engine spans training, inference, and HPC
Hopper	FP8 plus Transformer Engine	Precision selection becomes part of model execution
Blackwell	FP6, FP4, microscaling	Low-precision formats become finer-grained and model-aware
Rubin	Expanded NVFP4/NVFP6 and third-generation Transformer Engine	Precision, attention, sparsity, and communication are co-optimized

Component 2: Memory became as important as compute

Neural networks repeatedly move weights, activations, and cache state. More arithmetic units do not help when those units wait for data.

P100's HBM2 placed stacked memory beside the GPU on a silicon interposer. It delivered 16 GB at 720 GB/s. V100 raised bandwidth to 900 GB/s. The original A100 reached 1.555 TB/s with 40 GB, and the later A100 80 GB exceeded 2 TB/s.

H100 increased bandwidth to 3.35 TB/s, but H200 demonstrates why capacity and bandwidth must be separated. H200 retained Hopper compute while moving to 141 GB of HBM3e at 4.8 TB/s. On memory-intensive large-model inference, that change can matter without a new Tensor Core generation.

Blackwell increased the representative B200 package to 180 GB at up to 8 TB/s. Blackwell Ultra reached 288 GB. Rubin keeps 288 GB but replaces HBM3e with HBM4 and raises the vendor-published peak to 22 TB/s.

xychart-beta
    title "Selected vendor-published peak HBM bandwidth"
    x-axis ["P100", "V100", "A100 80GB", "H100", "H200", "B200", "Rubin"]
    y-axis "TB/s" 0 --> 24
    bar [0.72, 0.90, 2.04, 3.35, 4.80, 8.00, 22.00]

The chart keeps one physical quantity on the axis, but it combines product introductions, later memory variants, and Rubin's production-ramp specification. Every value is a published peak rather than achieved application bandwidth.

Rubin's published HBM bandwidth is more than 30 times P100's. Power and cooling rose with it: P100 SXM was rated at 300 watts, A100 SXM at 400 watts, H100 SXM at up to 700 watts, and B200 at up to 1,000 watts per GPU.

Component 3: Packaging and CPU coupling expanded the processor boundary

P100, V100, A100, and H100 were large monolithic GPU dies. Each generation pushed close to the practical manufacturing limit for one piece of silicon.

Blackwell changed the package. Two reticle-limited dies connect through a 10 TB/s chip-to-chip interface and present themselves as one CUDA GPU. NVIDIA could increase transistor count to 208 billion without requiring one manufacturable die of that size.

Rubin continues the dual-reticle approach with 336 billion transistors. NVIDIA calls the inter-die connection NV-HBI. From the programming layer, the goal remains one GPU abstraction. Underneath it, the package has become a tightly integrated multi-die system.

Once a "GPU" contains multiple compute dies, HBM stacks, high-speed die links, NVLink interfaces, and control engines, the package is already a system.

Grace Hopper tightened that relationship. A Grace CPU and Hopper GPU communicate over coherent NVLink-C2C, giving the CPU and GPU a higher-bandwidth path and a coherent memory relationship inside one superchip.

Blackwell's GB200 platform connects one Grace CPU to two Blackwell GPUs. The GB200 NVL72 rack contains 36 Grace CPUs and 72 Blackwell GPUs. Vera Rubin replaces Grace with the Vera CPU while retaining the 36-to-72 relationship.

This creates two different fabrics:

NVLink-C2C connects the CPU and GPU inside a superchip.
NVLink and NVSwitch connect GPUs across the scale-up domain.

PCIe remains the host and peripheral fabric, while NVLink-C2C and NVLink carry the tightly coupled processor paths.

flowchart TB
    APP["Models and applications"]
    ORCH["Inference orchestration<br/>Dynamo, routing, cache coordination"]
    SERVE["Serving and model runtimes<br/>Triton, TensorRT-LLM"]
    LIBS["Accelerated libraries<br/>cuDNN, cuBLAS, NCCL, NVSHMEM"]
    CUDA["CUDA platform<br/>compiler, runtime, graphs, memory APIs"]
    CPU["Grace or Vera CPU<br/>system memory and control"]
    GPU["GPU package<br/>Tensor Cores, cache, HBM"]
    SCALEUP["NVLink + NVSwitch<br/>scale-up domain"]
    NODE["Server or rack I/O domain"]
    SCALEOUT["InfiniBand or Spectrum-X<br/>scale-out fabric"]
    OPS["NGC, GPU Operator, Mission Control<br/>deployment and operations"]

    APP --> ORCH --> SERVE --> LIBS --> CUDA --> GPU
    CPU <-->|"NVLink-C2C"| GPU
    GPU <-->|"NVLink"| SCALEUP
    SCALEUP --> NODE
    NODE <-->|"ConnectX / BlueField"| SCALEOUT
    OPS -. manages .-> SERVE
    OPS -. manages .-> CUDA
    OPS -. manages .-> SCALEOUT

Component 4: NVLink turned multiple GPUs into one scale-up computer

PCIe connects devices to a host. NVLink exists because tightly coupled GPU workloads need a different path.

P100 supported four first-generation NVLinks for 160 GB/s of aggregate bidirectional bandwidth per GPU. DGX-1 arranged eight GPUs in a hybrid cube mesh, so not every pair had a dedicated direct connection.

V100 increased per-GPU NVLink bandwidth to 300 GB/s. DGX-2 then introduced NVSwitch, replacing the fixed mesh with a switched 16-GPU fabric.

A100 doubled per-GPU bandwidth to 600 GB/s and standardized an eight-GPU fully connected NVSwitch baseboard. H100 reached 900 GB/s, while external NVLink Switch systems stretched the scale-up fabric beyond one server.

Blackwell moved the fabric across an entire rack. GB200 and GB300 NVL72 connect 72 GPUs through fifth-generation NVLink at 1.8 TB/s per GPU, producing NVIDIA's rounded 130 TB/s aggregate endpoint figure.

Rubin doubles the per-GPU figure to 3.6 TB/s and the 72-GPU aggregate to 260 TB/s.

xychart-beta
    title "Published aggregate bidirectional NVLink bandwidth per GPU"
    x-axis ["P100", "V100", "A100", "H100", "B200", "Rubin"]
    y-axis "TB/s" 0 --> 4
    line [0.16, 0.30, 0.60, 0.90, 1.80, 3.60]

The directionality deserves emphasis. A100's 600 GB/s means 300 GB/s in each direction. Blackwell's 1.8 TB/s and Rubin's 3.6 TB/s are also bidirectional endpoint totals. The 130 TB/s and 260 TB/s rack numbers multiply those endpoint figures by 72 GPUs. They do not state the fabric's one-way bisection bandwidth.

NVLink generation	Introduction	Per-GPU published bandwidth	Representative scale-up domain
NVLink 1	P100	160 GB/s bidirectional	Eight-GPU direct mesh
NVLink 2	V100	300 GB/s bidirectional	Eight-GPU mesh; 16 GPUs with NVSwitch
NVLink 3	A100	600 GB/s bidirectional	Eight-GPU fully switched baseboard
NVLink 4	H100	900 GB/s bidirectional	Eight-GPU HGX; larger Grace Hopper switch systems
NVLink 5	Blackwell	1.8 TB/s bidirectional	72-GPU NVL72 rack
NVLink 6	Rubin	3.6 TB/s bidirectional	72-GPU NVL72; larger 2027 roadmap domain

The scale-up domain supports frequent, low-latency communication inside one model execution. Scale-out networking joins systems and racks into a larger cluster.

Component 5: Networking and cooling expanded the deployment boundary

NVIDIA's 2020 acquisition of Mellanox brought the network into the platform.

Mellanox added several distinct components:

ConnectX network adapters move data between GPU systems.
Quantum InfiniBand provides a purpose-built scale-out fabric for large compute clusters.
Spectrum-X Ethernet combines Ethernet switches, adapters, congestion control, and software for AI traffic.
BlueField DPUs handle infrastructure work including networking, storage, isolation, and security.

Successive NVIDIA systems used 100 Gb/s EDR, 200 Gb/s HDR, 400 Gb/s NDR or Spectrum-X, and then 800 Gb/s Quantum-X800 and Spectrum-X800. Rubin adds ConnectX-9 while retaining InfiniBand and Ethernet scale-out paths.

NVLink and InfiniBand are not competing names for the same connection:

Fabric	Primary job	Typical boundary
PCIe	Host and peripheral attachment	CPU, GPU, NIC, storage inside a system
NVLink-C2C	Coherent processor coupling	CPU to GPU inside a superchip
NVLink + NVSwitch	High-bandwidth GPU scale-up	GPUs inside a server, rack, or tightly coupled domain
InfiniBand / Spectrum-X	System scale-out	Servers and racks across a cluster

The deployment unit grew with the fabric. The first DGX-1 was a 3U, air-cooled server rated at 3.2 kW. DGX A100 grew to 6U and 6.5 kW, while DGX H100 reached roughly 10.2 kW.

GB200 NVL72 changes the boundary: 72 GPUs, 36 CPUs, nine NVSwitch trays, power shelves, networking, and direct-to-chip liquid cooling form one rack-scale system. NVIDIA documents approximately 120 kW for that rack, while GB300 NVL72 raises the published envelope to about 142 kW.

Those figures do not compare efficiency because the measured boundary grows from one eight-GPU server to an entire 72-GPU rack. They show the architectural transition: electrical delivery, cooling, switching, and compute are now specified together.

Component 6: CUDA expanded into an operating stack for AI infrastructure

CUDA is the longest-running source of continuity. The CUDA programming model, drivers, compilers, and libraries let applications survive multiple hardware generations without starting over.

The stack around CUDA kept expanding:

Layer	Representative components	Role
Programming platform	CUDA	Kernels, memory management, graphs, compilation, runtime
Accelerated libraries	cuBLAS, cuDNN, NCCL	Matrix math, neural-network operations, multi-GPU collectives
Precision layer	Transformer Engine	FP8 and newer precision recipes for transformer models
Model optimization	TensorRT, TensorRT-LLM	Graph optimization, quantization, kernels, KV cache, distributed LLM execution
Serving	Triton Inference Server	Model serving, batching, ensembles, multiple backends
Fleet orchestration	Dynamo and NIXL	Request routing, disaggregated prefill and decode, cache and tensor movement
Packaging and operations	NGC containers, Container Toolkit, GPU Operator, Mission Control	Tested software combinations, deployment, drivers, scheduling, lifecycle management
Resource isolation	MIG and vGPU	Partitioning and sharing accelerator capacity

Volta exposed Tensor Cores through CUDA 9 and accelerated libraries. Ampere added TF32, sparsity, CUDA Graphs, and MIG. Hopper paired FP8 hardware with Transformer Engine. Blackwell paired FP4 with TensorRT-LLM, NeMo, and rack-scale NVLink. Dynamo moved optimization above the model server into routing, KV-cache placement, prefill/decode separation, and worker-to-worker data movement.

CUDA compatibility is valuable, but it is not magic portability. The CUDA compatibility model has minimum driver requirements and version boundaries. Native cubins target specific compute capabilities. TensorRT engines can be tied to library versions and GPU assumptions. MIG profiles differ by architecture. An optimized FP8 or NVFP4 execution recipe may need rebuilding for a different generation.

The application logic may travel. The highest-performance execution artifact often does not.

Groq adds a specialized inference path

The Groq relationship adds one important direction to NVIDIA's architecture: the GPU does not need to execute every phase of inference.

This was not an acquisition of Groq. In December 2025, Groq announced a non-exclusive license of its inference technology to NVIDIA. Several Groq leaders and team members joined NVIDIA, while Groq remained an independent company and continued operating GroqCloud.

NVIDIA used the licensed technology in the Groq 3 LPX rack, which it positions beside Vera Rubin NVL72. One representative pattern separates prefill from decode: Rubin processes the prompt and creates the KV cache, LPX generates output tokens, and NVIDIA Dynamo coordinates routing and data movement.

The pattern matters more than the product pairing. NVIDIA is extending its platform around heterogeneous inference—general-purpose GPUs for broad model execution, specialized processors for latency-sensitive phases, and software that decides where each phase runs.

What measured performance can actually tell us

Peak specifications describe hardware ceilings. Vendor platform claims describe selected workloads and configurations. Neither is a substitute for a controlled benchmark.

The methodologically cleanest comparison in the available results is MLPerf Training v0.7. NVIDIA submitted an eight-GPU V100 DGX-1 and an eight-GPU DGX A100 to the same BERT benchmark under the same release. Applying MLPerf's rule of discarding the fastest and slowest runs and averaging the remaining eight gives roughly 166.2 minutes for V100 and 49.0 minutes for A100—about 3.4 times faster.

The result combines hardware and generation-appropriate software rather than isolating the chip—which matches how users experience the system.

Later comparisons are harder:

NVIDIA reported up to 6.7 times higher H100 performance than its first A100 submission in MLPerf Training v2.1, but "up to" selects the strongest workload and spans different submission vintages.
In MLPerf Inference v4.0, NVIDIA reported about 28% greater H200 performance than H100 at the same 700-watt envelope and up to 45% at 1,000 watts. The equal-power result is the cleaner evidence for HBM3e's contribution.
Blackwell and Blackwell Ultra have measured MLPerf submissions, but accelerator count, numerical format, model, power, and software must match before calculating a ratio.
Rubin had no comparable public MLPerf submission by August 30, 2026. Its performance and cost claims remain NVIDIA measurements until reproducible benchmark systems appear.

There is no honest single line for "GPU performance over time." Workload, precision, sparsity, GPU count, power, and software all move.

What's missing

The stack still lacks a neutral system benchmark that matches how current AI infrastructure is purchased and operated.

MLPerf is the strongest public mechanism available, but the product boundary keeps expanding faster than the benchmark boundary. A useful rack-scale comparison would hold all of these constant:

model and quality target;
prompt and output-length distribution, including concurrency;
time to first token and inter-token latency;
sustained throughput under concurrency;
training or inference precision;
dense versus sparse execution;
rack, network, cooling, and facility power;
software engineering required to reach the result;
failure recovery and degraded-operation behavior;
acquisition price and cost per completed unit of work.

Without that boundary, a vendor can compare one Blackwell rack with many Hopper servers, or FP4 with FP8, or projected utilization with measured throughput. The numbers may all be correct while the comparison remains unhelpful.

Rubin makes the gap larger. NVIDIA now describes benefits from GPU architecture, HBM4, NVLink 6, Vera CPUs, Spectrum-X, power smoothing, liquid cooling, Dynamo, and workload scheduling in one platform claim. That is a reasonable systems argument. It also makes independent reproduction much harder.

So what

NVIDIA's evolution from P100 to Rubin is the story of bottlenecks moving outward.

P100 attacked memory and PCIe. V100 attacked matrix math. A100 attacked precision, utilization, and multi-tenancy. Hopper attacked transformer execution. Blackwell attacked the server boundary. Rubin attacks the rack and pod as operating systems for continuous inference.

That changes how to evaluate the company and its competitors. Comparing accelerator FLOPS is now the narrowest possible cut. The real comparison is the amount of useful model work a complete system produces within its power, latency, reliability, and software constraints.

The GPU did not disappear. It became the center of a much larger machine.

The open question is whether this integration remains one durable platform advantage or creates enough cost, power, and operational complexity for the stack to split again—between general-purpose GPUs, specialized inference processors, open interconnects, and software that can schedule work across all of them.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

There Is No Universal AI Runtime—So Build a Portable Control Plane

Amit — Wed, 23 Sep 2026 04:52:17 +0000

AI accelerators are not interchangeable computers. A CUDA kernel does not become a Trainium executable because both systems run PyTorch, and an inference server supporting two GPU families does not erase their different compilers, libraries, memory models, collectives, or performance envelopes.

The portable layer sits higher.

Applications can call one stable service contract. A model registry can govern several hardware-specific artifacts. A router can place each request on a qualified pool. Observability can compare the resulting service, and policy can decide when moving a workload is economically justified.

That is a portable control plane—not a universal runtime.

TL;DR

Kernels, compiled artifacts, collectives, quantization paths, and performance tuning remain tied to an accelerator stack.
Serving frameworks can present common APIs across several backends, but backend support does not guarantee identical model coverage, behavior, or economics.
The durable architecture keeps NVIDIA, AMD, TPU, Trainium, and specialist accelerators in separate pools behind one service contract.
Model governance, workload qualification, routing policy, service-level telemetry, and cost policy are more portable than execution.
Portability is an operating model: maintain a broad compatibility pool and move stable workloads only after another pool passes correctness, performance, and cost gates.

The runtime boundary is real

CUDA defines a GPU programming model, runtime, compiler toolchain, and libraries for NVIDIA hardware. CUDA C++ device code is compiled by nvcc, which can produce architecture-specific binary images and PTX for later compilation by the CUDA runtime. That mechanism provides compatibility within the NVIDIA platform; it does not produce an executable for an unrelated accelerator.

HIP deliberately resembles CUDA and includes tools that translate many CUDA API calls. The same documentation also states that HIP code must be compiled for a specific AMD GPU architecture and that the resulting binaries contain code for that target. Source translation reduces migration work. It does not turn a CUDA binary into an AMD binary.

The boundary becomes clearer with cloud accelerators. The Neuron compiler transforms framework graphs into a Neuron Executable File Format artifact that the Neuron runtime loads on Neuron devices. PyTorch/XLA lets applications retain familiar PyTorch APIs, but it traces operations into an intermediate graph and sends HLO to XLA for compilation. Changes in graph shape can cause additional compilations, and hardware-specific optimization still happens below the framework.

OpenXLA is a useful portability layer precisely because it separates a versioned operation set from target-specific backends. StableHLO can carry a model graph between framework and compiler, while each backend still performs its own optimization and code generation. The graph can travel farther than the executable.

This produces a simple hierarchy:

Layer	What can be shared	What remains hardware-specific
Application	Request schema, authentication, streaming contract, error model	Model behavior under different precision and implementations
Model source	Framework checkpoint, tokenizer, configuration, evaluation set	Converted, sharded, quantized, or compiled artifacts
Graph representation	ONNX or StableHLO where the model and operators are supported	Unsupported operators, custom extensions, lowering, graph partitioning
Serving	Endpoint shape, admission control, rollout mechanism	Engine build, kernels, cache layout, batching, parallelism
Scheduling	Workload intent, pool labels, quotas, priority	Device discovery, drivers, topology, allocation mechanism
Execution	Nothing universal enough to assume interchangeability	Runtime, compiler, kernels, libraries, collectives, memory placement
Measurement	Correctness, latency, throughput, availability, cost per outcome	Vendor counters, power boundaries, utilization definitions

The mistake is not using abstraction. The mistake is placing the abstraction below the level where the evidence supports it.

One contract, separate hardware pools

The stable unit should be a service, not a device. Each accelerator family operates as a separately qualified pool with its own software image, driver, runtime, serving engine, model artifacts, deployment process, and failure domain.

flowchart TB
    A[Applications and agents] --> B[Stable inference or training contract]
    B --> C[Identity, quotas, admission control]
    C --> D[Model catalog and artifact registry]
    D --> E[Workload qualification and policy router]

    E --> N[NVIDIA pool<br/>CUDA artifacts and kernels]
    E --> M[AMD pool<br/>ROCm and HIP artifacts]
    E --> T[TPU pool<br/>XLA-compiled artifacts]
    E --> R[Trainium pool<br/>Neuron-compiled artifacts]
    E --> S[Specialist pool<br/>Vendor runtime and artifacts]

    N --> O[Normalized service telemetry]
    M --> O
    T --> O
    R --> O
    S --> O

    O --> P[Cost, reliability, and placement policy]
    P --> E

This design does not schedule one container indiscriminately across every device. It selects a pool that already has a tested artifact and a known operating envelope for the requested model, precision, context length, batch profile, and service objective.

The model catalog therefore needs more than a model name. A useful entry binds a logical model version to a set of qualified implementations:

logical model: reasoning-model-v3
evaluation policy: eval-suite-2026-08

qualified implementations:
  - pool: nvidia-hopper
    artifact: fp8-tensor-parallel-8
    status: production
  - pool: amd-mi300
    artifact: fp8-tensor-parallel-8-rocm
    status: canary
  - pool: trainium
    artifact: neuron-compiled-seq4096
    status: batch-only

The logical model gives applications a stable name. Each implementation retains its own artifact identity, software bill of materials, evaluation results, supported limits, and rollback history. Promotion happens per implementation, not once for the abstract model.

Serving frameworks widen the interface, not the binary

Modern serving projects make this architecture easier, but none removes the qualification work.

vLLM supports hardware plugins outside its main repository. SGLang documents support for multiple accelerator families and offers an API compatible with common language-model request patterns. These are valuable common surfaces. Their hardware paths can still differ in kernel availability, quantization support, distributed execution, model coverage, and release maturity.

Triton Inference Server makes the separation visible: every model is associated with a backend, and the backend is the implementation that executes it. Triton can wrap TensorRT, PyTorch, ONNX Runtime, vLLM, or custom logic, but its own documentation warns that not every backend is supported on every platform.

ONNX Runtime follows a similar pattern through Execution Providers. The application uses a consistent API while a provider claims supported nodes or subgraphs and hands them to a hardware-specific library. That is portable application integration with specialized execution underneath. It is not proof that every model will partition, compile, or perform equally on every provider.

KServe can standardize how a model service is declared and operated on Kubernetes. Its serving runtimes still select concrete backends, images, resource requests, and model formats. The operational object becomes portable; the implementation remains qualified for a particular pool.

The practical contract should cover:

logical model and version;
task type and request schema;
streaming and cancellation behavior;
context, input, and output limits;
safety and data-handling policy;
latency and availability objectives;
error categories and retry rules;
response metadata identifying the implementation used.

It should not promise identical tokenization, numerical output, latency, or cost unless those properties were explicitly tested.

The scheduler places workloads; it does not port them

Cluster schedulers already understand heterogeneous resources. Kubernetes device plugins let vendors advertise specialized devices to the kubelet. A cluster can expose resources such as nvidia.com/gpu and amd.com/gpu, then use labels and affinity to place a workload on the intended node type.

Slurm GRES similarly models GPU types as generic resources and lets a job request a specific type and count. Ray defines accelerator resources for NVIDIA GPUs, AMD GPUs, Neuron cores, TPUs, and other devices, then assigns tasks or actors to nodes with the requested resource.

These systems solve allocation. They do not determine whether a model is correct on that accelerator, whether the serving engine has the required kernel, or whether the workload meets its latency target. A portable control plane must make placement eligibility a product of qualification:

eligible pool =
  supported model
  + approved artifact
  + compatible request limits
  + passing evaluation
  + healthy capacity
  + acceptable economics

Only after those gates pass should the router compare eligible pools.

Route on workload shape and evidence

A lowest-cost accelerator rule is too crude. Inference cost changes with context length, prefill-to-decode ratio, output length, concurrency, batch formation, cache reuse, quantization, and the amount of idle capacity held to protect latency.

The routing decision needs a qualification matrix:

Decision input	Why it changes placement
Model and revision	Operator and architecture support change between releases
Precision and quantization	Available kernels and acceptable accuracy loss differ
Context length	Memory consumption, compilation shapes, and cache pressure change
Prefill versus decode	The two phases stress compute and memory differently
Latency objective	A cheap saturated pool can violate the service contract
Batchability	Offline work can use capacity that interactive traffic cannot
Geography and data policy	Some pools may be ineligible before performance is considered
Current queue and warm capacity	A nominally efficient pool can be expensive after delay and cold start
Measured cost per accepted output	Hardware-hour price alone omits utilization and failed work

Routing also occurs at more than one level. Ray Serve’s routing documentation distinguishes model-level ingress routing from replica selection. A cross-hardware control plane needs both: first select the qualified implementation and pool, then let the local serving system select a replica using queue, cache, and locality signals.

The broad compatibility pool plays an important role. New models, unusual operators, rapidly changing context requirements, and workloads without enough evaluation evidence stay there. Specialist pools earn traffic when the workload becomes stable enough to compile, measure, and govern.

Normalize outcomes, preserve native telemetry

Observability is portable only at the service layer. Time to first token, inter-token latency, total response time, requests per second, tokens per second, error rate, evaluation score, queue delay, and cost per accepted output can be defined consistently.

The lower-level evidence remains native. GPU occupancy, NeuronCore utilization, compiler-cache behavior, HBM counters, collective timing, and device power are not one universal metric family. Flattening them into a single “accelerator utilization” number destroys the details needed to diagnose a pool.

The clean design keeps two views:

Normalized service telemetry compares the outcome delivered to the application.
Backend-native telemetry explains why one implementation produced that outcome.

Economic policy should use both. A pool that has a lower device-hour price but compiles frequently, rejects more requests, requires excess standby capacity, or produces lower-quality outputs can have the higher service cost.

Compare service cost, not accelerator price

A portable control plane needs one economic denominator across every pool:

service cost per accepted unit =
  reserved accelerator and host capacity
  + network, storage, and energy
  + amortized model-porting and software work
  + operator and support cost
  + failed, retried, and quality-rejected work
  ------------------------------------------------
  accepted outputs that met the service objective

The denominator can be accepted tokens, images, training steps, or completed jobs, but it must include the quality and latency objective. Utilization belongs inside the calculation because reserved capacity accrues cost while idle. Availability belongs inside it because failed work and standby capacity reduce the accepted output produced by the same reservation.

This illustrative monthly comparison shows why hardware price alone is misleading:

Monthly component	Broad compatibility pool	New specialist pool
Reserved compute, hosts, and fabric	$72,000	$45,000
Energy and facility allocation	$8,000	$4,000
Amortized porting and software work	$6,000	$15,000
Operations, support, and standby capacity	$8,000	$12,000
Failed, retried, or rejected work	$4,000	$6,000
Accepted output meeting the service objective	140 billion tokens	90 billion tokens
Service cost per million accepted tokens	$0.70	$0.91

The numbers are hypothetical; the mechanism is not. The specialist has lower compute and energy cost but loses the comparison because it carries porting work, idle capacity, and failed output across a smaller qualified workload. If model coverage grows, utilization rises, and software work is amortized across more accepted output, the same pool can later become the cheaper path.

The control plane should therefore store both the current service cost and the evidence behind it: measurement interval, reservation model, utilization, failure rate, quality threshold, energy boundary, labor allocation, and software version. Routing on a number without that provenance recreates the same false portability problem at the economic layer.

What is still missing

The software projects above provide pieces of this architecture, not one finished control plane. Schedulers allocate devices. Serving engines execute models. Model registries track artifacts. Gateways route requests. Telemetry systems collect measurements.

The missing layer is a durable qualification record connecting all of them:

which logical model maps to which hardware-specific artifact;
which evaluation approved that artifact;
which request shapes it supports;
which software and driver versions produced the result;
which service and economic thresholds it currently meets;
which policy moved traffic and how to reverse that decision.

Without that record, “multi-accelerator” becomes a collection of independent deployments sharing an endpoint. With it, hardware diversity becomes governable.

So what

Do not wait for a universal AI runtime. The compilers, kernels, collectives, and memory systems are moving in the opposite direction—toward deeper specialization.

Build portability where it holds: the application contract, model identity, evaluation policy, workload router, service telemetry, and economic decision. Keep execution pools separate, qualify every implementation, and route only across proven choices.

The open thread is how much control-plane standardization is possible before the policy itself becomes coupled to the dominant serving engine. If every backend exposes different cache state, queue semantics, compilation behavior, and power evidence, the router may remain portable in name while its best decisions depend on vendor-specific signals.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

The Interconnect Determines How Much of the Chip You Can Use

Amit — Wed, 23 Sep 2026 04:51:41 +0000

An accelerator can finish its arithmetic and still leave the system waiting.

That is the architectural shift behind rack-scale AI. More processors do not automatically produce a faster computer. They produce a larger distributed system whose useful performance depends on moving weights, activations, gradients, expert tokens, and cache state between the right processors at the right time.

At rack and pod scale, data movement can constrain useful accelerator performance more than arithmetic. The chip still matters. But the interconnect increasingly decides how much of the chip the system can use.

TL;DR

Scale-up fabrics connect accelerators into one tightly coupled compute domain; scale-out networks connect servers, racks, and pods. They solve different communication problems.
NVLink and NVSwitch are a mature vertically owned scale-up stack. UALink and Ethernet-based scale-up proposals create open alternatives, but a published specification is not a deployed ecosystem.
InfiniBand and Ethernet/RoCE both support large AI clusters. Their useful performance depends on NIC placement, topology, congestion control, collectives, and operations—not the link rate alone.
NCCL, RCCL, and related libraries turn collective operations into traffic patterns. They are part of the performance architecture, not a thin software wrapper.
Copper remains efficient over short electrical reaches. Pluggable optics dominate longer links. Co-packaged optics and optical I/O move conversion closer to the switch or accelerator as electrical reach and front-panel power become harder constraints.

Scale-up and scale-out are different systems

The cleanest way to understand an AI fabric is to ask where the tightly coupled compute domain ends.

Scale-up connects accelerators that exchange data frequently enough to behave like one larger machine. Latency, bandwidth, memory semantics, ordering, and collective behavior all matter. The fabric may remain inside a server, cross a rack, or eventually span several racks.

Scale-out connects those compute domains into a larger cluster. It carries distributed training collectives, inference traffic, model and checkpoint movement, storage access, and control traffic across a broader topology.

flowchart LR
    subgraph R1["Rack-scale compute domain"]
        A1["Accelerator"]
        A2["Accelerator"]
        A3["Accelerator"]
        SU1["Scale-up switches<br/>NVSwitch, UALink, or another fabric"]
        A1 <--> SU1
        A2 <--> SU1
        A3 <--> SU1
    end

    NIC1["NIC or SuperNIC"]
    SO["Scale-out network<br/>InfiniBand or Ethernet"]
    NIC2["NIC or SuperNIC"]

    subgraph R2["Second compute domain"]
        SU2["Scale-up fabric"]
        B1["Accelerator"]
        B2["Accelerator"]
        SU2 <--> B1
        SU2 <--> B2
    end

    SU1 <--> NIC1
    NIC1 <--> SO
    SO <--> NIC2
    NIC2 <--> SU2

The distinction is architectural, not merely physical. NVIDIA's NVLink carries tightly coupled GPU traffic, while its Quantum InfiniBand and Spectrum-X Ethernet platforms connect systems across the scale-out network. A cable does not become scale-up because it is fast, and Ethernet does not become scale-out-only because that is its conventional role.

That last boundary is now moving. The UALink Consortium defines an open 200 Gb/s-per-lane scale-up specification for accelerator pods. Broadcom's Scale-Up Ethernet framework applies Ethernet technology to the scale-up domain. AMD's UALink over Ethernet, or UALoE, uses that broader direction to connect 72 MI455X GPUs in the Helios rack. The physical and link technologies can converge while the workload contract remains different.

The scale-up contest is also an ownership contest

NVLink's advantage is not only its published bandwidth. NVIDIA owns the GPU interface, NVSwitch silicon, firmware, topology, collective library, rack design, and much of the qualification process.

Blackwell NVLink 5 provides 1.8 TB/s of aggregate bidirectional bandwidth per GPU. Rubin NVLink 6 doubles that published endpoint figure to 3.6 TB/s. In NVL72 systems, NVSwitch extends the tightly coupled domain across 72 GPUs in one rack.

Those figures are endpoint totals, not achieved application throughput or one-way fabric bisection bandwidth. Blackwell systems are deployed, while NVIDIA describes Vera Rubin as ramping into full production. Both belong to an integrated product stack—not an interface specification waiting for implementations.

UALink takes a different path. Its 200G 1.0 specification separates the fabric definition from any single accelerator vendor and supports scale-up domains as large as 1,024 accelerators. That creates room for accelerators, switch silicon, cables, management components, and systems from different suppliers.

The UALink specification family now also separates common, data-link, physical-layer, chiplet, and manageability documents. The open boundary moves integration responsibility outward. Compatibility requires more than matching electrical lanes: memory semantics, error handling, management, firmware, topology discovery, collectives, and system qualification must converge across companies.

Fabric or initiative	Primary boundary	Ownership model	Status by Aug. 30, 2026	What the label does not prove
NVLink + NVSwitch	NVIDIA GPU scale-up	Vertically controlled by NVIDIA	Mature across several generations; NVL72 in production	Application bandwidth, availability, or compatibility with non-NVIDIA accelerators
UALink specifications	Open accelerator scale-up	Consortium specifications and multi-vendor implementations	200G 1.0 plus newer component specifications published; implementation ecosystem emerging	Shipping interoperable systems at NVLink maturity
Broadcom Scale-Up Ethernet	Ethernet-based accelerator scale-up	Merchant switch silicon plus ecosystem	Framework and products published; ecosystem integration under way	That ordinary data-center Ethernet automatically meets scale-up semantics
AMD UALoE	72-GPU Helios scale-up domain	AMD implementation using Ethernet switch silicon	Helios architecture introduced in 2026	Independent interoperability or NVLink-equivalent operational maturity
Ultra Ethernet 1.0.3	AI/HPC scale-out transport	Consortium specification	Current specification published; products and software arriving by vendor	A replacement for scale-up fabrics or an installed multi-vendor fleet

These rows are not benchmark peers. NVLink is a deployed proprietary stack. UALink is an open scale-up specification. Ultra Ethernet is a scale-out transport specification. Scale-Up Ethernet is a family of merchant-Ethernet approaches aimed at a new boundary.

The strategic question is whether an open ecosystem can coordinate those layers quickly enough to behave like one system.

Link speed is not collective performance

Distributed models do not send a uniform stream of independent packets. They execute collective operations.

An all-reduce combines values across workers and returns the result to each worker. An all-gather assembles shards. A reduce-scatter combines and redistributes them. Mixture-of-experts models can produce all-to-all traffic as tokens move toward different experts.

NCCL implements these collectives for NVIDIA GPUs across PCIe, NVLink, InfiniBand, and Ethernet paths. RCCL provides the corresponding collective library in AMD's ROCm stack. Intel maintains oneCCL for distributed communication across its supported compute environments.

The library chooses algorithms and routes based on topology, message size, available transports, and system configuration. A ring can use links efficiently for large transfers. A tree can reduce latency for other message sizes. Hierarchical algorithms can reduce data inside a server or rack before crossing the scale-out network.

flowchart TB
    OP["Model operation<br/>all-reduce, all-gather, all-to-all"]
    LIB["Collective library<br/>NCCL, RCCL, or oneCCL"]
    PLAN["Algorithm and topology choice<br/>ring, tree, hierarchy, channels"]
    LOCAL["Local movement<br/>HBM, PCIe, scale-up fabric"]
    NIC["NIC path<br/>RDMA, GPU memory access, transport offloads"]
    FABRIC["Scale-out fabric<br/>switches, routes, congestion control"]
    RESULT["Useful synchronized work"]

    OP --> LIB --> PLAN
    PLAN --> LOCAL
    PLAN --> NIC --> FABRIC
    LOCAL --> RESULT
    FABRIC --> RESULT

This stack explains why a nominally faster network can deliver less useful performance. A job can lose time through a poor rank-to-NIC mapping, oversubscribed links, uneven paths, head-of-line blocking, retransmission, collective imbalance, or one slow worker.

The accelerator metric that matters is not peak FLOPS. It is the fraction of execution time spent doing useful compute rather than waiting for communication or synchronization.

InfiniBand and Ethernet are systems, not cable choices

InfiniBand entered AI clusters with an integrated remote-memory and lossless-fabric model. NVIDIA's Quantum-2 platform combines switches, ConnectX adapters, RDMA, congestion management, telemetry, and its SHARP in-network-reduction capability. In supported Quantum deployments, the fabric can participate in selected collective operations instead of treating every reduction as endpoint-only traffic.

Ethernet starts with a broader multi-vendor base. Running AI collectives over Ethernet commonly uses RoCE, which carries RDMA over Ethernet. That removes repeated CPU and kernel involvement from the data path, but it does not remove the need to engineer congestion behavior.

Traditional RoCE deployments can combine priority flow control, explicit congestion notification, data-center quantized congestion notification, adaptive routing, and careful buffer configuration. NVIDIA's Spectrum-X packages Ethernet switches, BlueField SuperNICs, telemetry, congestion control, and adaptive routing as one AI-networking platform.

The Ultra Ethernet Consortium's current 1.0.3 specification goes further by defining an AI/HPC transport that includes packet spraying across paths, selective retransmission, congestion management, and collective-communication support. It is an attempt to make large Ethernet fabrics less dependent on every deployment recreating the same behavior through proprietary combinations.

Layer	InfiniBand example	Ethernet example	Operational owner
Link and switching	Quantum switches and InfiniBand links	Ethernet switches and links	Network platform team
Endpoint	ConnectX HCA	NIC or SuperNIC	Server and network teams
Remote-memory transport	InfiniBand RDMA	RoCE or Ultra Ethernet transport	NIC, driver, and fabric stack
Congestion behavior	InfiniBand controls and adaptive routing	PFC/ECN/DCQCN, vendor controls, or UEC mechanisms	Fabric engineering
Collective layer	NCCL/RCCL plus optional in-network operations	NCCL/RCCL/oneCCL plus transport integration	Accelerator platform team
Job placement	Topology-aware scheduler and rank placement	Topology-aware scheduler and rank placement	Cluster platform team

This is why “InfiniBand versus Ethernet” is an incomplete comparison. The real unit is endpoint hardware, switching, transport, congestion control, collectives, topology, and operations.

NICs and DPUs own different work

A NIC terminates the network path. A modern AI NIC can also expose RDMA, access accelerator memory, steer traffic across rails, timestamp packets, and offload transport work.

NVIDIA uses the term SuperNIC for a high-performance network endpoint optimized for accelerator traffic. It positions BlueField-3 SuperNIC around GPU-to-GPU networking, while its DPU configuration adds programmable Arm cores and infrastructure offloads.

A DPU has a broader ownership boundary. It can run networking, storage, security, isolation, and management functions away from the host CPU. AMD's Pensando infrastructure products follow that broader offload model.

Neither device fixes a poor fabric by itself. A faster NIC attached through the wrong PCIe root, mapped to distant GPUs, or connected to an oversubscribed leaf still creates a slow communication path. Hardware topology and software placement have to agree.

Copper reaches a wall before optics does

The physical medium is becoming an architectural choice.

Passive copper is attractive inside a tray or across short rack distances because it avoids optical conversion and can deliver low power and low latency. Active electrical cables extend that reach by adding signal conditioning. As lane rates rise, electrical loss increases and the practical reach shrinks.

Pluggable optical modules move the electrical-to-optical conversion to the switch faceplate. They provide longer reach and replaceable modules, but every port consumes front-panel area and conversion power. Linear pluggable optics remove part of the digital signal processing from the module to reduce power, while placing tighter signal-integrity requirements on the host system.

Co-packaged optics moves the optical engines beside the switch ASIC. Broadcom's co-packaged-optics portfolio and NVIDIA's silicon-photonics switch platforms shorten the high-speed electrical path and increase optical bandwidth density. They also couple optics more closely to switch packaging, cooling, yield, and service procedures.

flowchart TB
    NEED["Required reach, bandwidth density,<br/>power, and service model"]
    TRACE["On-board traces<br/>inside board or package"]
    COPPER["Passive or active copper<br/>tray and short rack links"]
    PLUG["Pluggable optics<br/>conversion at faceplate"]
    CPO["Co-packaged optics<br/>conversion beside switch ASIC"]
    OIO["Optical I/O<br/>conversion beside compute or memory"]

    NEED --> TRACE
    NEED --> COPPER
    NEED --> PLUG
    NEED --> CPO
    NEED --> OIO

Optical I/O pushes the boundary closer to compute. Ayar Labs' TeraPHY packages optical chiplets for chip-to-chip communication. Lightmatter Passage combines optical connectivity with packaging intended to link processors and memory. Celestial AI's Photonic Fabric targets optical scale-up and memory connectivity.

These products do not share one maturity level or deployment model:

Optical boundary	Public maturity by Aug. 30, 2026	Evidence boundary
Pluggable optics	Shipping infrastructure across data-center networks	Mature module ecosystem; power and faceplate density remain constraints
Broadcom CPO and NVIDIA silicon photonics	Commercial portfolios and announced switch platforms	Product availability varies by platform and configuration
Ayar Labs optical I/O	Chiplet products and customer sampling programs	Not yet a broadly interchangeable accelerator interface
Lightmatter Passage and Celestial AI Photonic Fabric	Product programs and partner integrations	Deployment scale and production economics remain vendor-reported

The direction is consistent even though the maturity is not: when electrical reach, connector density, and SerDes power become system limits, designers can move the optical conversion point inward. That choice is driven by distance and packaging boundary, not one universal copper-to-optics migration sequence.

The interconnect changes who owns the computer

NVIDIA's model places the scale-up link, switch, NIC, network, collective library, and rack qualification under one platform owner. That reduces the number of boundaries a system integrator must reconcile.

An open stack distributes those responsibilities. An accelerator vendor can implement UALink. A merchant-silicon supplier can provide scale-up and scale-out switches. A NIC vendor can own RDMA and congestion functions. An OEM can assemble the rack. A collective library can adapt the topology to the framework.

That structure creates supplier choice. It also creates a harder acceptance test. A specification match does not establish that the assembled system sustains collectives under congestion, survives link failures, maps jobs correctly, or exposes one support boundary when performance falls.

The practical evaluation unit is therefore not the accelerator card. It is:

the tightly coupled scale-up domain;
the scale-out fabric joining those domains;
the collective software translating model operations into traffic;
the topology and congestion mechanisms preserving progress;
the physical medium carrying the required bandwidth within the power and service envelope.

What remains unproven

Vendor bandwidth figures are abundant. Comparable useful-work evidence is not.

A credible system comparison needs the same model, precision, collective mix, accelerator count, topology, message sizes, failure policy, and power boundary. It also needs achieved bandwidth, communication-to-compute overlap, tail behavior under congestion, job completion time, and recovery behavior—not only a clean microbenchmark.

Open fabrics face a second proof burden: interoperability. UALink and Ultra Ethernet can publish complete specifications before multiple vendors ship independently qualified systems. That is normal standards work, but it creates a gap between protocol maturity and operational maturity.

So what

The next AI platform decision cannot stop at chip throughput. The fabric determines the size of the useful computer, the software determines how traffic enters it, and optics determines how far that computer can extend before power and signal integrity pull it apart.

The open question is whether the industry can create a genuinely interchangeable scale-up and scale-out ecosystem—or whether every high-performing AI cluster will remain a platform-specific computer whose compatibility ends exactly where its collective software, congestion controls, and optical packaging diverge.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

Groq Makes the Compiler Part of the Processor

Amit — Wed, 23 Sep 2026 04:51:06 +0000

Groq's architectural bet is that an inference processor should execute a plan, not discover one while the model is running.

The Language Processing Unit removes much of the reactive machinery found in conventional processors: hardware caches, dynamic arbitration, speculative execution, and out-of-order scheduling. Its compiler decides where operations run, where data lives, and when each value moves. The processor then follows that schedule across a large on-chip SRAM system.

That changes the unit worth evaluating. One LPU does not hold a modern large language model. The useful machine is the connected system of LPUs, its compiler-generated schedule, and the cloud service around it.

TL;DR

Groq moves instruction scheduling, tensor placement, and much data movement into the compiler, leaving simpler hardware to execute a predetermined program.
SRAM supplies high bandwidth and predictable access time, but limited capacity forces model weights and execution across many connected LPUs.
Deterministic chip execution reduces one source of latency variation; it does not remove API networking, admission control, queueing, or shared-service contention.
Groq is strongest where one user's token rate matters: voice, coding loops, agents, and other interactive inference. That is different from maximizing total batched throughput.
NVIDIA did not acquire Groq. It licensed Groq's inference technology non-exclusively and hired Groq leaders and staff; Groq remains independent and continues to operate GroqCloud.

The processor begins in the compiler

Modern GPUs are also compiler-driven machines. Framework graphs are lowered into kernels, kernels are compiled, and serving engines plan memory and batch work. The distinction is what remains unresolved at runtime.

A GPU retains hardware mechanisms that react to events while a program executes: caches answer memory requests, arbiters resolve contention, and schedulers choose from ready work. Groq's original Tensor Streaming Processor paper describes the opposite cut. It eliminates reactive elements such as caches and arbiters so the compiler can reason about execution timing precisely.

The hardware is functionally sliced. Separate regions perform matrix operations, vector arithmetic, memory access, data movement, and instruction control. Values stream between those regions through a software-controlled interconnect. The compiler maps the model graph onto that physical machine and emits a cycle-level schedule.

Groq describes this as a programmable assembly line: each functional unit receives an instruction, takes data from a named stream, performs its operation, and places the result onto another stream. Hardware does not wait to discover which path wins arbitration because the compiler has already assigned the path and time.

System decision	Conventional GPU-oriented execution	Groq LPU execution
Instruction readiness	Hardware and runtime mechanisms select ready work	Compiler places work into a predetermined schedule
Data locality	Registers, caches, HBM, and runtime-managed movement	Compiler assigns tensors across distributed SRAM
Resource contention	Hardware arbitration resolves competing requests	Compiler attempts to prevent conflicts before execution
Timing model	Performance can vary with cache behavior and contention	Scheduled accelerator execution is predictable cycle by cycle
Optimization boundary	Kernels, runtime, serving engine, and hardware	Model compiler, LPU fabric, and serving system

This is not “software instead of hardware.” The compiler works because the hardware was built to expose timing and data movement directly. Groq makes the compiler part of the processor's effective control plane.

flowchart LR
    GRAPH["Model graph"]
    COMPILER["Groq compiler<br/>partition + place + schedule"]
    ARTIFACT["Compiled execution plan"]

    subgraph SYSTEM["The useful compute boundary"]
        LPU1["LPU<br/>compute + SRAM"]
        LPU2["LPU<br/>compute + SRAM"]
        LPUN["More LPUs<br/>compute + SRAM"]
        FABRIC["RealScale interconnect<br/>scheduled chip-to-chip movement"]

        LPU1 <--> FABRIC
        LPU2 <--> FABRIC
        LPUN <--> FABRIC
    end

    API["GroqCloud or system API"]
    REQUESTS["Inference requests"]

    GRAPH --> COMPILER --> ARTIFACT --> SYSTEM
    REQUESTS --> API --> SYSTEM
    SYSTEM --> API

SRAM makes speed possible—and scale necessary

Groq places the primary working memory on the processor in SRAM. SRAM is faster and more predictable than fetching weights from off-chip DRAM or HBM, but it occupies much more silicon area per stored bit.

The first-generation chip described in the architecture literature contains approximately 220 MB of SRAM. Groq's current eight-accelerator GroqNode lists 1.76 GB of aggregate on-die memory, consistent with eight chips at that capacity. A 70-billion-parameter model requires roughly 35 GB merely to store weights at four bits per parameter, before accounting for scales, activations, code, and key-value cache.

One chip is therefore the wrong capacity boundary. Groq partitions model weights and operations across multiple LPUs, then streams activations through the connected machine. GroqNode exposes 88 chip-to-chip connectors and is positioned for multi-server and multi-rack scaling without a separate external switch in the LPU fabric.

The result resembles a pipeline more than a pool of interchangeable accelerators. Each stage owns part of the model and passes intermediate values to the next stage according to the compiled schedule. Adding LPUs expands both SRAM capacity and execution resources, but it also makes partitioning, communication, and pipeline utilization part of the performance result.

Groq's latest rack specification makes that system boundary explicit. Its inference platform page lists 256 LPUs, 128 GB of on-chip SRAM, and 40 PB/s of aggregate SRAM bandwidth per rack for the Groq 3 generation. Those are vendor specifications, not independently measured application results.

The architectural trade is clear: spend silicon and system complexity on low-latency SRAM and predictable movement, then recover model capacity by joining many processors into one scheduled machine.

Deterministic execution is not deterministic service

Groq uses “deterministic” precisely at the accelerator layer. Given the same compiled program, the LPU schedule specifies when instructions execute and when values move. The hardware does not introduce cache misses or arbitration delays that change the cycle count.

A cloud request crosses a larger system:

client network
  → API gateway
  → admission control and queue
  → model replica selection
  → prompt processing
  → scheduled LPU execution
  → streamed response

Only one part of that chain is the deterministic machine. Request bursts can still create queues. Prompts have different lengths. Tool calls introduce external dependencies. Network paths vary. A shared service can also reserve, batch, or route work differently as load changes.

This distinction matters because latency has several meanings:

Metric	What it measures	What it can hide
Time to first token	Delay before generation begins	Prompt length, queueing, and network transit
Output speed	Tokens delivered per second after generation starts	Queueing before the first token and aggregate capacity
End-to-end response time	User wait for the complete answer	Different answer lengths and reasoning-token behavior
Aggregate throughput	Total work completed by the serving system	Whether any individual user receives tokens quickly
Tail latency	Slow requests at the 95th or 99th percentile	Average results that look healthy while some users wait

Groq's strongest public evidence is output speed. Its GroqCloud model catalog publishes model-specific rates ranging into hundreds of tokens per second, with smaller models reaching about 1,000 tokens per second. These are service specifications and vary by model.

Artificial Analysis independently measures Groq's public endpoints. At the August 30, 2026 cutoff, it reports the fastest tracked Groq configuration near 980 output tokens per second. For the same Llama 3.3 70B model, its provider comparison measures Groq at roughly 298 output tokens per second—more than 20 times one of the slower tracked providers.

That is useful independent evidence for interactive speed. It is not a rack-throughput benchmark, a power measurement, or a total-cost comparison. Artificial Analysis measures hosted APIs; it does not expose how many chips, replicas, queued users, or reserved capacity sit behind each endpoint.

The distinction also explains Groq's workload fit. Fast sequential generation is valuable for conversational voice, coding agents waiting inside a loop, interactive search, and workflows that call a model repeatedly. Offline jobs with large batches care more about total tokens per dollar and total tokens per rack than the token rate of one stream.

SemiAnalysis argues that SRAM-heavy systems trade capacity available for weights and key-value cache against high single-user token rates, while GPUs can win aggregate throughput and cost through batching. That is the right comparison boundary: latency-first and throughput-first inference are different products even when both report “tokens per second.”

GroqCloud is the complete product

The LPU does not arrive as a general accelerator card with a CUDA-like programming surface. GroqCloud presents supported models through an API that is largely compatible with the OpenAI client pattern. The compiler, model artifacts, partitioning, and machine topology remain behind the service boundary.

That lowers adoption work for a supported model: change the endpoint and test behavior. It also moves control to the provider. Model availability, context limits, rate limits, pricing, quantization, and rollout schedules follow the GroqCloud catalog rather than the customer's own accelerator fleet.

The fit is strongest when:

the model and operators are supported by Groq's compiler and service;
low per-user generation latency changes the product experience;
the workload is stable enough to justify a compiled deployment;
API consumption is preferable to owning the hardware and compiler lifecycle.

The fit weakens when:

the workload requires custom or rapidly changing operators;
the model is absent from the supported catalog;
large dynamic key-value caches dominate memory demand;
maximum batched throughput matters more than individual stream speed;
infrastructure ownership or artifact control is mandatory.

GroqCloud therefore sells more than processor time. It packages a compiler-controlled distributed machine behind a familiar inference contract.

NVIDIA licensed the technology; it did not acquire Groq

The NVIDIA relationship is easy to overstate because technology and senior personnel moved together.

On December 24, 2025, Groq announced a non-exclusive inference technology licensing agreement with NVIDIA. Founder Jonathan Ross, president Sunny Madra, and other Groq staff joined NVIDIA. The same announcement says Groq would remain an independent company under CEO Simon Edwards and that GroqCloud would continue operating.

Reuters described the structure as a technology license plus executive hiring that stopped short of a formal acquisition. Reported deal values remain reporting, not disclosed transaction terms from either company.

The distinction now appears in two product paths:

GroqCloud remains Groq's independently operated inference service.
NVIDIA Groq 3 LPX uses licensed Groq technology inside NVIDIA's Vera Rubin platform.

NVIDIA's Groq 3 LPX architecture description specifies a 256-LPU rack with 128 GB of SRAM, 40 PB/s of SRAM bandwidth, and 640 TB/s of rack-scale communication. NVIDIA announced the product in full production in August 2026, with initial cloud availability planned later in the year.

LPX does not establish that every model should run entirely on an LPU rack. NVIDIA presents a heterogeneous design coordinated by Dynamo:

Vera Rubin GPUs can process prefill, attention, and key-value cache.
LPX can execute latency-sensitive feed-forward or mixture-of-experts decode stages.
The two systems exchange intermediate activations according to the selected serving pattern.

NVIDIA calls these patterns prefill-decode disaggregation, attention-FFN disaggregation, and external-drafter speculative decoding. The published numbers—including 3,431 output tokens per second on one 100K-context test—come from NVIDIA-operated LPX systems and vendor-selected configurations, even when Artificial Analysis supplied the test harness.

The evidence floor is narrower than “Groq replaces GPU inference.” LPX shows that a latency-specialized SRAM machine can become one stage inside a larger GPU inference system. GPUs retain the memory-heavy and general-purpose work; LPUs handle selected decode paths where deterministic scheduling and SRAM bandwidth matter most.

What's missing

Groq has credible evidence for high output speed and a coherent explanation for why the architecture produces it. The public record remains thinner on aggregate system economics.

A complete comparison needs the same model, quality target, prompt distribution, context length, concurrency, and service-level objective across systems. It also needs measured rack power, total output throughput, tail latency, compiler coverage, failed-compilation rate, capacity reservation, and cost under sustained load.

No current public MLPerf result establishes a standardized Groq-to-GPU comparison across those boundaries. GroqCloud API benchmarks prove that users can receive tokens quickly. They do not reveal whether the underlying rack delivers the best aggregate tokens per watt or tokens per dollar.

So what

Groq's contribution is larger than a fast inference endpoint. It redraws the processor boundary around the compiler and the connected machine.

SRAM makes each operation fast and predictable. The compiler turns that memory and compute into a timed execution plan. The interconnect lets many small memory domains behave like one model-serving pipeline. GroqCloud turns that system into an API product, while NVIDIA's licensed LPX path places the same architectural idea inside a heterogeneous GPU platform.

The open question is operational: as inference splits into prefill, attention, feed-forward, speculative decoding, and cache management, does the winning system use one broad accelerator—or does the compiler become the place where several specialized processors are assembled into one service?

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

d-Matrix Moves the Math Into Memory

Amit — Wed, 23 Sep 2026 04:50:30 +0000

Inference is becoming a memory-placement problem.

A modern accelerator can perform arithmetic faster than it can repeatedly move model weights into the compute engines. Autoregressive decoding makes the mismatch visible: every generated token requires another pass through the model, often at a batch size small enough that raw matrix throughput is not the limiting resource.

d-Matrix Corsair attacks that boundary by placing digital arithmetic beside a large SRAM tier. The system still needs off-chip capacity memory, PCIe, Ethernet, compilation, quantization, and distributed execution. The architectural bet is narrower and more interesting than “replace the GPU”: keep the latency-critical working set near the arithmetic and use the rest of the memory hierarchy deliberately.

TL;DR

Corsair combines digital in-memory compute with 2 GB of high-bandwidth SRAM per card and up to 256 GB of off-chip capacity memory.
The useful deployment boundary expands from a paired PCIe card to an eight-card server and then a 64-card rack because larger models must be partitioned across multiple SRAM pools.
d-Matrix reduces physical adoption friction by using standard PCIe and Ethernet, but model compilation, compression, partitioning, and numerical validation remain accelerator-specific.
The clearest near-term pattern may be heterogeneous inference: use a specialist for a memory-bound phase while GPUs retain compute-heavy phases.

The bottleneck is the trip to memory

The arithmetic in transformer inference is not uniform.

Prefill processes many prompt tokens in parallel and can keep a large compute engine busy. Decode generates tokens sequentially. At small batch sizes, the accelerator repeatedly reads model weights to produce relatively little new work. The service becomes sensitive to memory bandwidth, placement, and communication latency.

Corsair puts a digital in-memory compute engine beside what d-Matrix calls Performance Memory. One card provides 2 GB of that SRAM at a published 150 TB/s, plus as much as 256 GB of Capacity Memory off chip. A two-card pair doubles those figures and connects through the company’s DMX Bridge.

Those two tiers serve different jobs:

Memory tier	Published card capacity	Architectural job	Main constraint
Performance Memory	2 GB SRAM	Keep latency-sensitive weights and activations next to the compute engines	Capacity
Capacity Memory	Up to 256 GB	Hold larger models and support throughput-oriented execution	Lower bandwidth and greater movement cost
Distributed rack memory	128 GB aggregate SRAM and up to 16.4 TB capacity in the 64-card reference rack	Partition larger models and requests across cards and servers	Fabric latency, partitioning, and software coordination

This is not one flat pool. The compiler and runtime must decide what fits in the fast tier, how the model is divided, and when data moves between cards or servers.

flowchart LR
    MODEL["Model checkpoint"]
    COMP["Aviator compiler<br/>compression and partitioning"]
    CAP["Capacity Memory<br/>larger model storage"]
    PERF["Performance Memory<br/>SRAM beside compute"]
    DIMC["Digital in-memory<br/>compute engines"]
    FABRIC["DMX Bridge, PCIe,<br/>or Ethernet"]
    NEXT["Next card or<br/>inference stage"]

    MODEL --> COMP
    COMP --> CAP
    COMP --> PERF
    CAP --> PERF
    PERF <--> DIMC
    DIMC --> FABRIC
    FABRIC --> NEXT

The diagram captures the trade: less distance between arithmetic and the fast memory tier, followed by more responsibility for placement and distributed execution.

A card is not the complete product

Corsair uses an industry-standard full-height, full-length PCIe Gen5 form factor. That makes it physically familiar, but the deployment unit changes with the model.

The company describes three reference boundaries:

Boundary	Configuration	Published memory boundary	Intended use
Card pair	Two connected Corsair cards	4 GB Performance Memory; up to 512 GB Capacity Memory	Smaller models and low-latency stages
Server	Eight cards	16 GB Performance Memory; up to 2 TB Capacity Memory	Multi-card inference within one host
Rack	Eight servers, 64 cards	128 GB Performance Memory; up to 16.4 TB Capacity Memory	Larger models and higher request volume

The aggregate SRAM determines which model weights can remain in the fastest tier. d-Matrix describes smaller models as an appliance-level fit and larger models as a rack-level problem. That is why the system architecture matters more than the card specification.

Within a node, PCIe and DMX Bridge connect the accelerators. Across nodes, the design uses Ethernet. d-Matrix’s JetStream NIC is intended to let accelerators coordinate transfers without routing every step through the host control path. The company says each JetStream connects four Corsair cards to a leaf switch and extends accelerator-to-accelerator execution across servers.

That is still a distributed system. The architecture reduces some host-mediated handshakes; it does not remove switches, congestion, topology, failure handling, or the need to keep model partitions synchronized.

The software stack carries the memory policy

In-memory compute does not make a PyTorch model automatically executable.

The Aviator software stack contains model templates, compression tools, an MLIR-based compiler, a distributed inference engine, host runtime, device firmware, and operational tooling. It integrates with PyTorch and Triton DSL at the upper boundary, then produces artifacts for Corsair’s spatial architecture and numerical formats.

Software layer	Responsibility
Model Factory	Provides distributed model templates
Compressor	Converts weights into supported block floating-point formats
Compiler	Places and schedules operations for the accelerator
Inference Engine	Coordinates execution across multiple cards
Host Runtime	Manages interaction between server CPUs and cards
Chip Runtime	Launches device work and manages on-chip execution
Operational tools	Monitor cards and analyze workload performance

This is the portability boundary. A framework checkpoint may be shared with another accelerator, but the compiled artifact, quantization choices, partitioning plan, and performance profile are hardware-specific.

The numerical choice matters. Corsair supports microscaling block floating-point formats, including MXINT8 and MXINT4 modes. Lower precision reduces storage and movement, but every model still needs accuracy evaluation. Peak low-precision throughput is not proof that a particular model preserves quality or meets a production latency target.

Standard racks reduce one kind of risk

d-Matrix does not require a proprietary wafer-scale chassis or a new liquid-cooled megawatt rack. Corsair is a PCIe card, and the company positions air-cooled server and rack designs that use Ethernet for scale-out.

That can reduce facility integration work:

standard server mechanics;
familiar PCIe enumeration;
Ethernet-based rack networking;
incremental deployment beside existing accelerator pools;
independent scaling of specialist and GPU capacity.

It does not remove system integration. A 64-card model partition still needs validated servers, network topology, firmware, cooling, power, observability, and replacement procedures. The operator also needs a serving layer that knows which model artifact can run on which pool.

The deployment pattern is therefore less “one new universal accelerator” and more “one specialized pool behind a stable inference contract.”

Heterogeneous inference is the stronger first use

The most useful evidence for Corsair is not a vendor comparison claiming that one rack replaces a GPU fleet. It is a narrower experiment that gives the specialist one phase of the request.

A partner engineering study from Gimlet Labs evaluated Corsair for speculative decoding while leaving prefill and verification on GPUs. The target model was gpt-oss-120b; a 1.6-billion-parameter draft model ran on two Corsair cards. Gimlet reported a 2–10× improvement in interactivity over its GPU-only speculative-decoding configuration at matched energy-efficiency points.

The boundary matters:

the analysis combined measured and modeled data;
it modeled an 8K-input, 1K-output coding workload;
token-acceptance assumptions materially affected the result;
the exact comparison GPU was not disclosed;
the test evaluated one inference phase, not complete replacement of the target-model hardware.

That caveat does not weaken the result. It makes the result useful. A memory-bound draft phase fits SRAM-centric hardware differently from compute-heavy prefill and batched verification. The experiment supports a heterogeneous architecture where each phase runs on the hardware that matches its bottleneck.

flowchart LR
    REQ["Request"]
    PREFILL["Prefill<br/>GPU pool"]
    DRAFT["Draft tokens<br/>Corsair pool"]
    VERIFY["Verify tokens<br/>GPU pool"]
    ACCEPT{"Accepted?"}
    OUTPUT["Stream output"]

    REQ --> PREFILL
    PREFILL --> DRAFT
    DRAFT --> VERIFY
    VERIFY --> ACCEPT
    ACCEPT -->|yes| OUTPUT
    ACCEPT -->|continue| DRAFT

This is disaggregated inference in concrete form. It adds network hops and orchestration, so the specialist must save more time than the handoff consumes.

The evidence is still uneven

d-Matrix publishes large claims for Corsair, including tens of thousands of tokens per second and substantial improvements in latency, power efficiency, and total cost. Its launch announcement labels the performance and cost estimates preliminary and subject to change.

The product page also mixes several boundaries:

card-level peak arithmetic;
server and rack aggregate memory;
model-specific projected performance;
preliminary comparisons against H100;
capacity-mode and performance-mode configurations.

Those numbers are not interchangeable. A useful independent comparison needs the same model, precision, quality threshold, prompt and output lengths, batching policy, service-level objective, power boundary, and complete system cost.

No Corsair result appeared in the MLCommons Inference results available during this August 30, 2026 review. This is a bounded research finding, not proof that a submission cannot exist elsewhere. Public partner and integration announcements establish ecosystem activity, but they do not provide the same evidence as measured production utilization, availability, or cost per accepted token.

The roadmap adds another evidence boundary. d-Matrix says its Pavehawk 3DIMC test silicon is operating in the lab and that a future Raptor architecture will use stacked memory. Those are development milestones and targets, not shipping-product measurements.

What d-Matrix changes in the landscape

d-Matrix is not trying to recreate the entire NVIDIA stack.

It attacks one expensive path: inference workloads where moving weights dominates useful arithmetic. The physical interface remains PCIe and Ethernet. The differentiated layer is the memory-compute complex, followed by the compiler and runtime required to use it.

That makes d-Matrix easier to combine with other accelerators than a closed, facility-scale replacement architecture. It also means the surrounding control plane becomes critical. The model registry needs multiple compiled artifacts. The router needs to understand workload phases. Observability needs to compare latency, throughput, quality, energy, and handoff overhead across dissimilar pools.

The strategic question is not whether every token should run on in-memory compute. It is whether enough stable, memory-bound work can be isolated to keep the specialist busy without making the serving system harder to operate than the savings justify.

If heterogeneous inference becomes the normal design, the unresolved boundary moves upward: who owns the optimizer that decides when a request should cross from a GPU into a specialist—and what evidence proves that decision still wins after network, compilation, capacity, and operational costs are included?

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.

When the Platform Operator Designs the Chip

Amit — Wed, 23 Sep 2026 04:49:55 +0000

When the operator of a cloud, model platform, or captive fleet designs the accelerator, the chip stops being the product. The product becomes a vertically integrated compute system: silicon, memory, interconnect, compiler, scheduler, and service assembled behind one operating boundary.

The thesis: when the platform operator owns those layers together, its infrastructure becomes the computer.

This is a different competitive model from selling an accelerator card. AWS Trainium, Google TPU, Microsoft Maia, Meta MTIA, and OpenAI Jalapeño all pursue it, but they are not at the same maturity or aimed at the same customer. Trainium and TPU are external cloud platforms. Maia is entering Azure infrastructure. MTIA is a captive production system. Jalapeño is working silicon still moving through qualification.

TL;DR

The real unit of platform-designed silicon is a managed system, not a chip: the operator controls compilation, topology, scheduling, capacity, and service delivery.
Trainium exposes the cleanest cloud hierarchy—NeuronCore to chip to UltraServer to UltraCluster—while TPU has the deepest published history of co-designing matrix engines, XLA, optical fabrics, and pod scheduling.
Maia, MTIA, and Jalapeño validate the same vertical-integration thesis at different stages, but their public evidence does not yet support treating them as interchangeable cloud products.
The trade is architectural: tighter co-design can improve utilization and economics, while moving portability upward from device binaries to models, frameworks, APIs, and workload-routing policy.

The platform is the system boundary

A merchant accelerator has to fit into systems that other companies assemble. A platform-designed accelerator starts with a different constraint: it can assume the operator also controls the host, network, rack topology, compiler fleet, workload scheduler, and service API.

That changes where optimization happens. A compiler can target known memory layouts. A scheduler can place a job into a known accelerator topology. The network can expose collectives that match the chip. Capacity planning can treat thousands of devices as one managed pool. The customer requests instances, slices, reservations, pods, or a managed model service rather than buying the underlying board.

flowchart TB
    A["Model and framework<br/>PyTorch, JAX, service API"] --> B["Compiler and graph optimizer<br/>Neuron or XLA-class layer"]
    B --> D["Capacity scheduler<br/>selects an available topology"]
    D --> E["Selected cluster, pod,<br/>or accelerator slice"]
    E --> C["Device runtime and<br/>collective libraries"]
    C --> F["Scale-up server or<br/>accelerator domain"]
    F --> G["Accelerator + HBM +<br/>local interconnect"]

    H["Cloud control plane"] --> B
    H --> D
    H --> E
    H --> F

The diagram is the important part of the argument. The platform operator is not merely substituting one arithmetic engine for another. It is closing the control loop from model graph to physical placement.

Trainium turns EC2 into the machine

Trainium3 makes the hierarchy unusually explicit. One chip contains eight NeuronCores and 144 GB of HBM3e with 4.9 TB/s of memory bandwidth. A trn3.48xlarge instance contains 16 chips. A Trn3 UltraServer joins nine instances—144 chips—through NeuronSwitch-v1 and a 28.8 TB/s NeuronFabric scale-up domain. UltraCluster 3.0 then connects UltraServers through Elastic Fabric Adapter networking.

That hierarchy matters more than any isolated TOPS number:

NeuronCore
  → Trainium3 chip
  → EC2 instance
  → Trn3 UltraServer
  → UltraCluster 3.0
  → managed training or inference service

The software path is equally deliberate. The AWS Neuron SDK connects PyTorch and JAX to Neuron compilers, runtimes, collective communication libraries, profiling, and monitoring. Neuron Kernel Interface gives developers a lower-level route for custom kernels without exposing the hardware as a generic CUDA-compatible device.

Trainium3 also shows why headline generational numbers need careful reading. AWS reports 2.52 PFLOPS of FP8 dense compute and 10.1 PFLOPS at FP4, alongside 1.2 PFLOPS of BF16 dense compute. The largest gains come from lower precision, sparsity, memory bandwidth, and a larger connected system—not an equal multiplier for every model and numerical format. The usable gain depends on whether the compiler maps a workload onto those features without losing model quality or creating excess communication.

The cloud integration reaches beyond instances. Trainium is exposed through EC2 and integrated into managed paths including Amazon SageMaker AI and Amazon EKS. AWS also offers dedicated AI Factories that combine accelerator infrastructure, networking, services, and customer-selected data-center locations.

The maturity signal is customer access: Trainium2 and Trainium3 are purchasable cloud infrastructure rather than lab prototypes. The evidence gap is comparability. AWS publishes specifications, workload claims, and customer adoption, but the public record offers less standardized third-party accelerator benchmarking than the NVIDIA and Google ecosystems. Trainium is a real cloud platform whose strongest evidence is production availability and disclosed usage, not a broad set of independently normalized chip measurements.

TPU makes the pod programmable

Google’s TPU architecture has followed this systems thesis for a decade. The original in-datacenter TPU paper described a domain-specific processor built around matrix multiplication rather than the instruction and cache machinery of a general-purpose CPU or GPU. The architectural through-line is systolic execution: data flows through an array of multiply-accumulate units while software arranges computation and movement around the array.

The more consequential transition arrived at pod scale. TPU v4 combined 4,096 chips with optical circuit switches that could reconfigure the interconnect topology. Optical switching let Google route around unavailable components, change topology for different workloads, and improve system availability without wiring every connection through a fixed electrical fabric.

Ironwood, or TPU7x, extends that model for inference. Each chip provides 192 GiB of HBM3e and 7.38 TB/s of memory bandwidth. A cube contains 64 chips, and a full pod reaches 9,216 chips through the Inter-Chip Interconnect and optical fabric. Google exposes smaller slices through its multislice software, so the physical pod becomes a capacity pool rather than one indivisible machine.

The software boundary is XLA. Cloud TPU system architecture places the user program on a TPU virtual machine while XLA compiles framework graphs into TPU executables. JAX is the native-feeling path because its transformations and array semantics align with XLA, while PyTorch/XLA lets PyTorch programs target the same compiler stack.

The scheduler is part of the architecture. queued resources let users request TPU capacity that Google Cloud provisions when the topology is available. flex-start VMs add a managed queue for workloads that can wait for capacity. Reserved capacity and calendar modes serve workloads that need predictable start times. These are not billing decorations; they decide whether a distributed executable receives the physical shape it was compiled to use.

The next generation makes workload specialization more explicit. Google describes TPU 8 as two designs: TPU 8t for training and TPU 8i for inference. TPU 8t pairs two compute dies with eight HBM3e stacks and introduces an expanded optical scale-up domain. TPU 8i uses four HBM3e stacks and targets inference economics. At this cutoff, TPU 8 remains an early-access system, while Ironwood has the clearer public cloud operating record.

Google also has stronger standardized evidence than the other cloud ASIC programs. Its published MLPerf Training v6.0 results cover TPU v6e, TPU7x, and NVIDIA systems. That does not make every TPU claim independently proven—vendor submissions remain optimized submissions—but it creates a defined benchmark boundary that other stacks often lack.

Five vertically integrated stacks, five maturity levels

Stack	Real system boundary	Compiler and software path	Availability by Aug. 30, 2026	Public evidence boundary
AWS Trainium3	NeuronCore → chip → instance → 144-chip UltraServer → UltraCluster	PyTorch/JAX → Neuron compiler, runtime, collectives, NKI	Customer-accessible EC2 infrastructure	Detailed vendor specifications and production adoption; limited standardized cross-vendor results
Google TPU	TensorCore → chip → cube/slice → optically connected pod	JAX/PyTorch → XLA → TPU runtime and ICI collectives	Ironwood available; TPU 8 in early access	Architecture papers, cloud documentation, and MLPerf submissions; TPU 8 evidence remains early
Microsoft Maia 200	Accelerator → Azure server and network → regional inference fleet	PyTorch models → Maia SDK, compiler, Triton language, firmware and runtime	Deployed in Azure regions; SDK preview	Microsoft specifications and workload claims; external customer and benchmark evidence remains narrow
Meta MTIA 300	Accelerator → eight-chip host → Ethernet scale-up/scale-out fleet	PyTorch → in-house compiler, runtime, PyTorch Distributed, TorchTitan	Production use inside Meta	Concrete production workload and fleet design; not offered as a general cloud service
OpenAI Jalapeño	Accelerator → server → rack → OpenAI inference fleet	PyTorch → Triton-based kernels, compiler, runtime, observability	Engineering samples; deployment targeted for late 2026	Vendor-published silicon measurements; production qualification still underway

The table is not a ranking. Availability, public programmability, and benchmark disclosure answer different questions.

Maia, MTIA, and Jalapeño narrow the target

Maia 200 is an inference accelerator with 216 GB of HBM3e and 7 TB/s of memory bandwidth. Microsoft says it is deployed in its U.S. Central data-center region, with a second region planned, and that its workloads include models from the Microsoft Foundry catalog. The Maia SDK includes a Triton compiler, PyTorch integration, kernel libraries, a low-level programming language, and simulation tools.

That is credible system construction, but the public service boundary is still forming. Maia 200 is not documented as a broadly selectable Azure VM family comparable to established GPU instances or current TPU and Trainium offerings. The evidence establishes deployed hardware and an SDK preview, not an open production ecosystem.

MTIA 300 is farther along as captive infrastructure. Meta reports production use training a recommendation model with more than 150 billion parameters. Each accelerator includes two network interfaces, and an eight-accelerator host uses Ethernet for both scale-up and scale-out communication. The system plugs into PyTorch Distributed and TorchTitan while Meta’s compiler and runtime manage the device-specific path.

MTIA therefore supplies strong evidence for vertical integration and weak evidence for external portability. It is optimized around Meta’s models, data centers, network, and fleet software. That is the point. A captive accelerator does not need to become a universal platform if it can remove cost or capacity pressure from a large, stable workload.

OpenAI Jalapeño takes the same idea into the model-provider layer. OpenAI and Broadcom co-designed an inference accelerator with 128 GB of HBM3e and 3.6 TB/s of bandwidth, then built a PyTorch-oriented compiler, runtime, kernel library, observability layer, server, and rack around it. OpenAI reports first-silicon measurements from engineering samples and plans deployment in its inference fleet by the end of 2026.

Jalapeño is the least mature system in this comparison. The silicon is operating and the software stack exists, but production qualification, manufacturing scale, and fleet behavior remain open. Its significance is strategic rather than proven economic superiority: a model provider is designing the execution substrate around the distribution of its own workloads.

Portability moves above the device

Platform-designed silicon relocates portability rather than eliminating lock-in. Device binaries, kernels, collectives, topology assumptions, and compiled artifacts remain hardware-specific even when several stacks accept PyTorch.

The portable boundary sits at model identity, evaluation, service contracts, and outcome telemetry. There Is No Universal AI Runtime—So Build a Portable Control Plane develops that pattern; the important point here is that vertical integration makes execution less interchangeable while making the complete service easier for one operator to optimize.

What remains unproven

Vertical integration creates a measurement problem. The provider can optimize hardware, software, placement, and pricing together, but outsiders rarely observe every boundary. Peak arithmetic says little about model quality. Chip power says little about facility energy. Tokens per second say little without batch size, latency distribution, context length, model version, and service utilization.

Trainium needs more standardized independent results. TPU 8 needs production evidence beyond early access. Maia needs a clearer customer-facing service boundary. MTIA needs no external ecosystem to succeed, but that makes cross-platform comparison difficult. Jalapeño needs manufacturing and fleet evidence after qualification.

The strongest evaluation unit is therefore the complete service: supported model, required code changes, compilation reliability, useful throughput, latency, accuracy, availability, energy boundary, and total cost over a sustained workload. Any comparison that stops at the chip cuts the system at the wrong layer.

So what

Platform-designed silicon changes the question from “Which chip is fastest?” to “Which operator can turn its workload, compiler, network, scheduler, and capacity into the best service?”

Trainium and TPU show the mature forms of that strategy. Maia and MTIA show how large operators narrow the architecture around workloads they already understand. Jalapeño pushes the boundary one layer higher: the model provider starts designing the machine that serves the model.

The open thread is control. If the cloud becomes the computer, how much of that computer must remain visible—topology, compilation, queueing, power, failure domains, and pricing—for builders to make an informed infrastructure choice rather than accept a black-box service?

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.