DEV Community: kt

JWT (Access Token) vs X.509 Deep Dive: How to Choose What You Present as a Credential

kt — Mon, 15 Jun 2026 13:54:03 +0000

Introduction

I was poking around a service mesh when something struck me as odd.

Inside the mesh, Pod-to-Pod traffic authenticated with mTLS: both sides showed each other an X.509 certificate. But the same request, when it came from outside through an API Gateway, carried a JWT in Authorization: Bearer eyJ.... Same job ("who are you, and may you come in"), yet the thing being handed over changed depending on where you stood.

At first I assumed it was just historical mess. The more I dug, though, the more it turned out to be a clean split that comes down to one axis: "bearer, or proof-of-possession?"

This article lines up JWT (access tokens) and X.509 (mTLS) from the angle of "what do you present as a credential," and walks through when to pick which, top to bottom. It starts from the basics, so you can follow it even if you are not deep into OAuth or TLS.

Setup: what "present a credential to get authorized" actually means

Let me get the vocabulary straight first. The "getting authorized" scene that runs through this whole article has three actors.

Issuer: creates the credential and hands it out. For a JWT that is the IdP (authorization server); for X.509 it is the Certificate Authority (CA).
Presenter: holds the credential it received and shows it to the other side on every request.
Verifier: checks whether the presented credential is genuine and whose it is, then decides to let it through or reject it.

A "credential" is the ID badge you present in step 2. It is structurally the same as showing your driver's license to prove your age. The question is what you use as that badge. Here the path forks into two families.

JWT (access token): a signed JSON, sent in the Authorization header.
X.509 certificate: a digital certificate presented during the TLS handshake. The "m" in mTLS (mutual) is exactly this.

The two differ in how they look and where they ride, but the essential difference is a single thing. That is next.

The core: bearer, or proof-of-possession

Every difference grows from here. Credentials split into two kinds by "can you use it just by holding it."

Bearer type: anyone holding the thing can use it. Like cash. Drop your cash and whoever picks it up can spend it. JWT access tokens are basically this.
Proof-of-possession type: showing the thing is not enough. You have to prove every time that it is really yours. Closer to a bank card with a PIN. Steal the card and you still cannot move money without the PIN (the private key). X.509 + mTLS is this.

This one picture is the most important in the article. Almost everything else is a consequence of it.

If a JWT is stolen, the thief gets to act as "you" directly (a replay attack). That is why the standard move for JWTs is to keep their lifetime short so the damage window stays small.
An X.509 certificate is more or less public information, but you cannot open a TLS session without holding the matching private key. Sniff the wire and copy the certificate, and you still cannot impersonate "you," because you lack the private key.

"So just always use X.509, right?" Not quite. Proof-of-possession carries its own costs (distributing, storing, and rotating private keys, plus the proxy problem covered later). That is exactly why you need to choose.

Cracking each one open

Now that the axis is clear, let me look at the two real artifacts.

JWT (access token): a signed JSON flowing through the app layer

A JWT is a string split into three parts by ..

eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9 . eyJzdWIiOiJzdmMtYSIsImF1ZCI6... . SflKxwRJSMeKKF2QT4f...
|------------ header -----------|    |--------- payload --------|    |----- signature ----|
  alg (sign method), typ              sub (who), aud (for whom), exp    signed with private key

The verifier checks the signature using the key set the issuer publishes (the JWKS: JSON Web Key Set). The point is that the verifier does not have to call the issuer. If the signature checks out, it can trust the token's contents (who, until when, for whom). This is stateless verification. Delivery is via an HTTP header.

There are also access tokens whose contents are not a JWT, the opaque tokens. For those the verifier calls the issuer's introspection endpoint on every request to confirm validity (stateful). When this article says "access token," it means the JWT form that supports the stateless verification above.

GET /api/orders HTTP/1.1
Host: api.example.com
Authorization: Bearer eyJhbGciOiJSUzI1NiI...

Since this lives at the application layer (L7), anything that speaks HTTP can carry it. A proxy or a Gateway can pass it straight through as "just a header." That matters later.

X.509 certificate: an identity proven during the TLS handshake

X.509 is presented in the middle of opening a TLS connection. Ordinary HTTPS has only the server present a certificate, but in mTLS the client presents one too, and on top of that proves possession by signing with its private key.

The verifier walks the certificate up to a CA it trusts (the trust bundle) to confirm legitimacy, and then uses the CertificateVerify signature to confirm the peer "really holds the private key." The defining trait is that authentication finishes before a single byte of application data flows, and this is transport-layer (closer to L4) work.

That pins down where each one lives: a JWT is data flowing through the app layer, and X.509 is an identity carved into the connection itself.

Lining them up on seven axes

With both artifacts in view, let me put them side by side. First a table for the big picture, then notes on the axes that matter most.

Axis	JWT (access token)	X.509 (mTLS)
Trust model	Bearer	Proof-of-possession
Theft resistance	Weak (steal it, use it)	Strong (useless without the private key)
Layer it works at	App layer (L7 / HTTP header)	Transport layer (TLS handshake)
Survives a proxy / LB	Yes (just a header)	No (mTLS is terminated there)
Verification	Check signature via JWKS (stateless)	CA chain (trust bundle) + signature check
Revocation	Let it expire via a short `exp`	CRL / OCSP, or short-lived certs
Operational cost	Low (no key distribution)	High (distribute, store, renew keys). Automated in a mesh

Three axes are worth expanding.

Axis 1: can it cross a proxy

This is the one that bites hardest in production. mTLS is authentication bound to "this point-to-point connection." Put an L7 load balancer or API Gateway in the middle that terminates TLS, and the client's identity vanishes there. From a service deeper than the Gateway, the peer is the Gateway, not the original client.

A JWT is "data," so the Gateway can forward it deeper as a header untouched. So if you want to carry identity end to end across several proxy hops, use a JWT. That is the answer to the mystery from the opening: cross the Gateway and it becomes a JWT.

Axis 2: how you revoke

When you want to cancel a credential, the two strategies are inverted.

JWT: once a signed token is out, the verifier never calls the issuer, so "revoke it after the fact" is hard. So you make the lifetime short from the start (minutes to tens of minutes) and wait for a leaked one to expire on its own. A long-lived JWT is a red flag.
X.509: you can announce "this certificate is no longer valid" via a revocation list (CRL) or OCSP. But the verifier has to go check that, which is operationally heavy. So rather than leaning on CRLs, the current trend is to make the certificate itself short-lived (SPIRE defaults to one hour, for example) and reissue it frequently.

The funny part is that both roads end up in the same place: "keep it short-lived to dodge the problem."

Axis 3: operational cost

X.509 means handing a private key to each workload, storing it safely, and renewing it on a schedule. That is the real source of "certificates are painful." But today a service mesh like Istio or Linkerd, or SPIFFE/SPIRE, does this distribution and renewal fully automatically. The old common sense that "mTLS is operationally heavy," from the days of handing out certificates by hand, has largely faded in a mesh.

The JWT side does not need to distribute a private key to clients (only the issuer holds the signing key), and the verifier just pulls the public key from the JWKS. The barrier to adoption is low.

Know how each one breaks

Before you choose, it helps to know how each gets defeated, so your judgment does not wobble. The entry point for an attacker is completely different between the two.

JWT alg:none: the biggest landmine in JWT. Set the header alg to none and a buggy implementation may accept "valid even without a signature," a classic vulnerability. You close it with a sane library and an operational rule that pins the allowed alg (the SPIFFE JWT-SVID spec limits alg to nine values and rejects none and the symmetric HS* family).
JWT theft and replay: the bearer curse. Minimize the damage with a short exp, sending over TLS, and never logging it.
X.509 private key leak: since possession of the private key is the whole basis of proof, a leaked key is game over. So keep keys off plain disk, ideally locked inside a TPM / HSM or in memory, and rotate them by keeping them short-lived.
X.509 CA compromise: take over the CA and you can mint any fake identity. Managing the trust bundle matters.

X.509 says "as long as you protect the private key, being watched on the wire is fine." JWT says "as long as you protect the channel (TLS) it is easy to adopt, but a leak means instant impersonation." The point you have to defend is different.

The hybrid: bearer ergonomics + proof-of-possession strength

"I want the convenience of a JWT, but I do not want it used after being stolen." Two mechanisms answer that greed. Both bolt proof-of-possession onto a JWT.

Certificate-bound access token (RFC 8705): at issuance, the hash (thumbprint) of the client certificate is stamped into the token. The verifier checks that "the thumbprint written in the token" matches "the certificate on the mTLS connection it currently holds." A stolen token alone is useless because you cannot open the matching mTLS connection. Used in high-security domains like Open Banking.
DPoP: for situations where you cannot run mTLS, it binds the token to a key the client holds, at the app layer. A "proof" signed with that key is attached per request, and the verifier cross-checks it against the token.

In short, "bearer or PoP" is not 0 or 1. You can add PoP to a JWT and slide toward the middle. But every addition costs implementation and operational effort, so for the first decision the plain two-way split is enough.

Choosing by example: the SPIFFE guidance makes it concrete

SPIFFE, the standard for workload identity, carries both formats: the certificate flavor X509-SVID and the JWT flavor JWT-SVID. And the guidance in its docs is blunt, which makes a ready-to-use decision rule. Summarized:

Because they are susceptible to replay attacks, use X.509-SVIDs whenever possible. Use JWT-SVID when mTLS is not practical, such as when an L7 proxy or load balancer sits between workloads.

Walk a worked example. An order service calls an inventory service.

Inside the mesh, Pod to Pod, direct: no L7 proxy in between, and the sidecar rotates certificates for you. Use X509-SVID (mTLS). Sniffing is useless without the private key, so it resists replay.
Across an API Gateway from outside, or with an L7 LB in between: mTLS gets terminated at the Gateway and identity vanishes. Put a JWT-SVID in a header and carry it end to end. In return, keep exp short (a few minutes) to bound the theft risk.

In real production the standard is a hybrid: "JWT (OAuth) at the user edge, mTLS between services." You do not have to commit to one. The right answer is to choose per boundary using the decision tree above.

Quick reference

To put the decision in one place.

When	Pick	Why
Service to service, direct inside a mesh	X.509 + mTLS	Auto-renewal works, resists replay
Crossing an L7 proxy / Gateway	JWT	mTLS is terminated and identity vanishes; a JWT survives as a header
From a browser / mobile	JWT	Distributing a private key to the client is impractical
Many services trust the same issuer	JWT	Verifiable via JWKS, no certificate distribution
Strong defense against replay needed	X.509, or a cert-bound / DPoP JWT	Close the bearer weakness with proof-of-possession
Must revoke immediately	X.509 (CRL/OCSP), or a short-lived JWT	JWT is bad at after-the-fact revocation; substitute short lifetimes

And the one line to remember goes back to the first picture.

A JWT works if you hold it (bearer). X.509 works if you hold it and can prove it with the private key (proof-of-possession). Almost every other difference follows from that.

Conclusion

The two families of credential you hand over for authorization, JWT (access token) and X.509 (mTLS), can be organized along one axis: bearer or proof-of-possession.
A JWT is easy and crosses proxies, but a stolen one gets used. So you protect it by keeping it short-lived.
X.509 is strong, useless even if sniffed without the private key, but its weaknesses are that identity is cut off at a proxy and that keys need operating. In a mesh, automation makes the operations light.
If you want both, you can bolt proof-of-possession onto a JWT with RFC 8705 certificate-bound tokens or DPoP.
In practice it is not either-or; you choose per boundary with the decision tree. SPIFFE's "X.509 whenever possible, JWT when a proxy is in the way" works directly as the rule.

Next time you look at your own system, go boundary by boundary and check, "is what flows here a bearer, or a proof-of-possession?" If a long-lived bearer token is flowing along a near-plaintext path, that is your most dangerous spot.

a2claude: Turn Claude Code Into a Server Other AI Agents Can Call

kt — Sun, 14 Jun 2026 07:54:12 +0000

Introduction

Running a single AI agent is starting to feel dated. You have an agent that researches, one that plans, one that writes code, each with a role, handing work to each other. When I tried to wire up a setup like that, I got stuck on one question: what do I use for the "writes code" slot?

I already had Claude Code. It actually runs tools, edits files, and runs tests for you. The catch is that Claude Code is built to be driven by a human sitting at a terminal. There is no built-in door for another agent to call it automatically.

So I wrote a small tool that exposes Claude Code as an A2A protocol server. It is called a2claude. Another agent sends "implement this feature" over A2A, a real Claude Code session runs against the project you pointed it at, and the result streams back. And what comes back is not a blob of text: it is structured information about which tools ran, which files changed and how, what it cost, and which actions need approval.

The repo is here: github.com/kanywst/a2claude

This article is written so that reading top to bottom takes you from knowing nothing about A2A or the internals of Claude Code to understanding exactly what a2claude bridges and how.

Background 1: What A2A is

A2A (Agent2Agent) is a communication standard for AI agents to talk to each other. Google announced it in 2025, donated it to the Linux Foundation the same year, and it is now governed there under neutral stewardship (v1.0 landed in 2026). It rides on top of HTTP over one of JSON-RPC, gRPC, or REST (the spec requires all three to expose the same operations), and it standardizes the exchange where one agent asks another to do a job.

There are only four terms you need to hold onto.

Agent Card: the business card that states what an agent can do. It lives at a fixed path, /.well-known/agent-card.json, and a caller reads it first to decide whether this agent can handle what it wants done.
Task: one request. It carries state and moves like submitted (received) → working (in progress) → completed (done). Along the way it can also hit input-required (waiting on input) or failed.
Artifact: the output of a Task. Generated text, files, and so on. It can be returned incrementally (streamed).
contextId: the conversation handle. Send the next Task with the same contextId and it is treated as a continuation of the previous one.

The Task state transitions look like this. It pays to keep this in your head, because how a2claude uses these states matters later.

A2A has other states too, such as rejected (the agent declines the request), but the ones a2claude actually uses are in this diagram. The one in red, input-required, becomes the star of the second half.

The point is that A2A is a shared language between agents: it does not matter what an agent runs on the inside (Claude or some other LLM), as long as it honors the Agent Card and the Task exchange, the conversation works.

Background 2: What Claude Code is

Claude Code is Anthropic's coding agent. It is not a chat that only returns prose. It also:

runs commands with Bash
reads and writes files with Read / Edit / Write
stops to ask for approval before risky actions
reports the cost and turn count of a run

In other words, it actually moves its hands. The thing to notice for this article is that when Claude Code runs, a lot of information beyond text comes out: which tool it called, which file it changed and how, what it cost. a2claude does not throw that away. It puts it on A2A.

Background 3: Why a bridge is needed

A2A is the protocol for agents to talk. Claude Code is the powerful tool that actually writes code. The two do not line up. Claude Code has no A2A server door, and from the A2A world Claude Code is invisible.

So you need a translator in between. It takes an A2A Task, drives Claude Code, and translates the events Claude Code emits back into the language of A2A. That is what a2claude does.

How this differs from the usual approach

Adapters that "wrap a coding agent in A2A" are not rare. But most of them flatten both input and output to text. You hand over a prompt and get text back. That erases everything that happened in between. From the calling agent's point of view, you cannot see whether a file was rewritten, what spent money, or whether an action needed approval.

a2claude keeps the structure that comes out of Claude Code and carries it onto A2A. The difference between the left and right of the diagram below is the whole reason this project exists.

What maps to what

The heart of a2claude comes down to one mapping table: "what Claude Code emits" onto "which A2A surface it lands on". This table, which is also in the README, is the core of the design.

What Claude Code emits	The A2A surface it lands on
Assistant text	A streamed artifact (`append` / `last_chunk`)
A tool call (Bash, Edit, ...)	A `working` status update
A file edit	A named artifact carrying the diff
Run result	Cost / turns / usage on the completion message
Session id	Mapped to the A2A `contextId` to resume next time

This mapping is closed up in one place in the code (executor.py). That is what pays off in the next section.

Architecture: the A2A layer does not know Claude directly

a2claude is split into layers. The single most important design decision is that the layer doing the A2A translation never imports the SDK that drives Claude Code.

In between sits an abstraction called a "backend". A backend's job is to drive Claude Code and emit normalized events (text, tool calls, file changes, permission requests, run results). The translation layer looks only at those events and maps them onto A2A. How Claude is invoked (through the SDK today, through the raw CLI later) is none of the translation layer's business.

Because of this split there are two backends.

echo: no API key and no Claude install required. A dummy that just mirrors the input. It lets you exercise the server, the protocol mapping, and the CLI end to end, offline. It reproduces every path, including the permission round trip that comes up later.
claude: drives the real Claude Code through the Claude Agent SDK.

Why is "the translation layer not knowing the SDK" worth it? Because testing gets easy. The echo backend alone can verify all of the A2A behavior. With no API key to call Claude, even in CI, you can check that the design has not broken.

Inside: an event queue and "parking"

This is the most interesting part of a2claude. A backend's drive runs as a background task and pushes the events it produces onto a queue. The translation layer (the consumer) pulls them off in order with drain.

While things are flowing normally it is just a producer / consumer. The interesting case is when you hit a tool that needs approval. Claude Code stops and waits. At that point a2claude parks the background task right at the permission request, and pushes only a "please approve this" event onto the queue. drain returns that one event and then halts.

While it is halted, the Claude session does not die: it waits. Later, when the caller sends its answer, the parked task resumes and starts flowing again. This trick of "keeping the session alive across two separate calls" is what makes the next section's permission exchange possible.

Permissions: do not auto-approve, do not silently skip

How you handle actions that need approval is the most nerve-wracking part of opening a coding agent up to someone else (another agent). A sloppy implementation either "auto-approves everything" or "silently skips anything that needs approval". Both are dangerous.

a2claude uses the A2A input-required state. This is not something a2claude invented: A2A defines it precisely for human-in-the-loop, where a human or the caller makes a decision mid-task. Per the spec, when a Task reaches input-required the processing stops there and control returns to the caller. a2claude rides directly on that.

When a tool that needs approval shows up, the Task stops at input-required and asks the caller "approve this action?". The caller sends its answer as a message on the same Task. allow (or yes, approve, ok) approves; anything else denies.

Thanks to the park from the previous section, the Claude session is still alive while stopped, so when the approval comes back it picks up right where it left off.

One more important point: the server does not inherit the personal Claude settings of whoever runs it. Your local Claude Code has a pre-approved tool allowlist ("this tool is always OK"), but the server does not load it. So when it acts on behalf of another agent, any action needing approval always routes back to the caller for a decision. Read-only actions that are already considered safe still run without a prompt, as before.

Session continuity: remembering "the previous turn"

Claude Code holds a conversation session and can keep working with the prior context. a2claude ties that Claude session id to the A2A contextId and remembers it. Send the next Task with the same contextId and the same Claude conversation resumes.

"Add a /health endpoint" → "now add a test for it" goes through as one continuous piece of work, not two separate requests.

Try it

From here on we actually run it. You need three things.

Python 3.13 or newer
uv
(only for the claude backend) the Claude Code CLI on your PATH

First clone and install dependencies.

git clone https://github.com/kanywst/a2claude
cd a2claude
uv sync

First, offline (the echo backend)

The echo backend needs no API key and no Claude install. Running the whole path offline first puts your mind at ease. Start the server and call it from another mouth.

uv run a2claude serve --backend echo &
uv run a2claude call "fix the failing test"

What comes back looks like this. A Task and context id are assigned, and after the stream the completion metadata (cost and turn count) is attached.

task 189b1c63-1a7b-4908-87c4-c8f3bba8f6b5
context 0b2a901e-2b6f-4c56-bba2-d0da546936e9

  · Echo
fix the failing test
[completed] $0.0 · 1 turns

Point it at a real project (the claude backend)

Now the claude backend. Use --cwd to set the directory Claude Code works in.

uv run a2claude serve --backend claude --cwd /path/to/project
uv run a2claude call "add a /health endpoint" --url http://localhost:9100/

Ask for a follow-up

Pass the context that came back from the previous turn and it becomes a continuation of the same conversation.

uv run a2claude call "now add a test for it" --context <context-id>

Try the permission exchange

Send an action that needs approval and it stops at input-required and asks for an answer. The answer goes to the same Task.

uv run a2claude call "sudo reboot"
# ... [input-required] Permission requested for Bash: $ sudo reboot
#       reply: a2claude call "allow" --task <id> --context <id>
uv run a2claude call "allow" --task <id> --context <id>

The echo backend also asks for approval when the prompt contains sudo, so you can verify just this exchange without driving Claude.

Peek at the Agent Card

This is the business card the other agent reads first.

uv run a2claude card

a2claude does not lump Claude Code's abilities into one "chat box". It lists them as discrete skills on the Agent Card: code generation, refactor, debug, review, test, and code explanation, six of them. The caller can aim a request, "this is a debug job", deliberately.

Wrap-up

What a2claude does comes down to one mapping table: it maps the structured events Claude Code emits onto A2A's Task, Artifact, status, and contextId. It does not flatten them to text.

Three design points to single out:

The translation layer does not know Claude's SDK. With a backend abstraction in between, echo alone verifies the whole path offline, and swapping out how Claude is invoked later leaves the translation layer untouched.
Permissions map to input-required. Keeping a parked session alive across two calls means it never auto-approves and never silently skips. The decision always returns to the caller.
contextId makes the conversation continuous. Mapping the Claude session id to the A2A context remembers "the previous turn".

If you are looking for a "writes code" slot in a setup where agents hand work to each other, you can drop Claude Code straight in as a part. The code is small enough to read all of, so take a look, and if something is off, throw an Issue.

github.com/kanywst/a2claude

A2A Protocol Auth, Taken Apart: Why the Spec Is Thin and Where That Leaves Holes

kt — Sat, 13 Jun 2026 14:26:48 +0000

How this started: I opened the auth section and it was nearly empty

A2A (Agent2Agent) keeps showing up as the protocol for letting AI agents talk to each other. Google announced it in April 2025, and the Linux Foundation runs it now.

"If one agent calls another, there has to be authn and authz in there," I figured, and opened the auth section of the spec. It was an anticlimax. There is no new authentication mechanism defined anywhere. What it says is: "use OAuth2 or OpenID Connect or mTLS," "send credentials in HTTP headers," "advertise your requirements in the Agent Card." That is it.

My first reaction was "is this just lazy?" But the more I read, the clearer it got that the thinness is intentional. A2A defines only the frame for authn and authz and delegates the contents to existing standards. The catch is that the way it delegates creates holes.

Everything I have written before about AI agent auth (WIMSE, SPIFFE, ID-JAG, Identity Chaining, Transaction Tokens) shows up here as A2A's "delegation target."

Background: just three players to remember

Before the auth details, the minimum cast. A2A is a protocol for one agent to hand a task to another agent. The transport is JSON-RPC over HTTP (there are gRPC and HTTP+JSON bindings too).

Player	Role
Client Agent (caller)	The agent handing off a task. The source of the HTTP request
Remote Agent / A2A Server (callee)	The agent that receives and works the task. Acts as an HTTP server
Agent Card (business card)	JSON where the Remote Agent publishes its capabilities and "auth requirements." Lives at `/.well-known/agent-card.json`

The Agent Card is the one to internalize. "What credential do I need to call this agent?" is written entirely on this card. The Client reads the card first, sets up the right auth, then sends the real request.

The core of A2A auth is already in this picture. Tokens are issued outside A2A (by an external IdP), and A2A itself only "states requirements in the card" and "receives credentials in headers."

When is it actually A2A: the difference from a plain API call

Before the auth internals, let's pin down what is A2A and what is not. Read on with these confused and the later design decisions float in midair. Here are the three things that get lumped together.

What you call	What you pass	What it is called	A2A?
An LLM (an inference API)	a prompt	just a model API	No. The other side is not an agent
A tool / data (DB, external API, files)	a function call	MCP territory (agent to tool)	No
Another autonomous agent	a task	A2A (agent to agent)	Yes

Calling an LLM directly is "make the model think." The other side is not an agent. MCP is "use your own hands and feet (tools)," where the other side is subordinate. Only A2A is "hand work to someone else's agent."

What is different from a plain API call

To be honest, A2A is mechanically nothing more than HTTP + JSON-RPC. Nothing magic beyond "a service calls a service" happens. What differs is not the wiring but who the other side is and how you ask. The same job, "is this expense (EUR 420, client entertainment) within policy, and convert it to USD while you're at it," looks different written two ways.

The plain-API way (you know the whole contract and wire it tightly yourself):

GET  fx-api.com/rate?from=EUR&to=USD
POST policy-service/check  {amount: 420, category: "entertainment"}

The A2A way (you do not know the internals; you read the card at runtime and send intent):

1. At runtime, read the Agent Card of the other side
   (another team's / company's "expense compliance agent")
   -> "I can do policy checks and currency conversion. Auth is Bearer token."
2. Send intent as a structured message:
   "Is this expense (EUR 420, entertainment) within policy? Also convert to USD."
3. The LLM inside that agent decides on its own which internal APIs to hit
   and how to judge, then returns a reasoned answer plus an attachment (artifact).

The differences are these three:

You send "intent," not an exact command. You do not spell out steps like ?from=EUR&to=USD. You hand over a goal and the other side works out the steps (because it has an LLM).
You do not know the internals / you do not pre-wire. You do not build against the other side's API contract. You read the Agent Card at runtime to learn "what it can do and how to auth" on the spot.
The other side is someone else's property. You cannot import it into your code. It is another team's agent running somewhere else.

The one reason it exists: killing the N×N wiring

If the other side is your own service, you do not need A2A. Just call the API. That can be said plainly. The one reason A2A earns its keep is avoiding the N×N problem.

When companies each have agents and want to hand work to each other, plain APIs need wiring per pair (the combinations blow up as the count grows). If everyone aligns on A2A's one card format and one calling convention, you can hand a task to an agent you have never met before, with no pre-wiring. It is like USB-C: the standard itself is not new technology, the value is that everyone aligned on the same socket. There is also a business angle: vendors do not want to expose raw APIs, but they will open up as an "agent."

Reality: still pre-adoption

Honestly, as of 2026 there are still few sharp "this is the A2A use case" examples. Production adoption is past 150 organizations and shows up in supply chain, financial services, insurance, and IT operations, but the public material stops at adoption counts and verticals, and concrete workflow case studies are thin. The canonical demos are cross-vendor delegation: travel booking (a planner delegates to airline, hotel, and rental-car agents from different companies) and hiring (a manager delegates to sourcing and scheduling agents). Most "multi-agent" today is in-process sub-agents, and for that you do not need A2A. A2A pays off when "make agents across org boundaries talk" becomes real, and it is a standard betting on that.

That covers "what A2A is and when to use it." Now the real subject: authn and authz. The auth design takes the shape it does precisely because crossing boundaries is the premise.

The design philosophy: "treat agents as ordinary apps"

A2A's auth design follows from a single principle.

Treat agents not as something special but as ordinary enterprise applications.

How do ordinary web apps authenticate when they call each other's APIs? Put an OAuth token in the Authorization: Bearer header, put an API key in a header, present a client cert with mTLS. A2A just applies that to agents as-is. It invents nothing new.

From this principle, four concrete design decisions fall out. This is the whole shape of A2A auth. Going through them in order:

Don't put identity in the payload. Establish it at the HTTP transport layer.
Advertise auth requirements in the Agent Card.
Acquire credentials out-of-band (outside A2A's scope).
Authorization is per-skill scope based. But actual enforcement is left to external infrastructure.

Decision 1: don't put identity in the payload

The spec says this.

A2A protocol payloads (JSON-RPC messages) don't carry user or client identity information directly.

"Not in the payload" is easy to misread, so look at the actual HTTP request. A2A traffic is JSON-RPC carried over HTTP, and it splits like this.

POST /a2a HTTP/1.1
Host: remote-agent.example.com
Authorization: Bearer eyJhbGci...     # "who is calling" (identity) goes here
Content-Type: application/json

{
  "jsonrpc": "2.0",
  "method": "message/send",
  "params": { "message": { "text": "Summarize this PDF" } }
}

The payload (the JSON-RPC body) carries only "what you want done." "Who" is carried by the Authorization token in the HTTP header (or the mTLS client cert). That is what "don't put identity in the payload" means.

The contrast with the design A2A does not use makes it clear. If you did put identity in the payload, it would look like this.

{
  "jsonrpc": "2.0",
  "method": "message/send",
  "params": {
    "callerId": "agent-A",
    "onBehalfOf": "user-123",
    "message": { "text": "Summarize this PDF" }
  }
}

A2A avoided this shape. There are three reasons, and they are what make it a "design decision."

A "who" inside the body is nothing but self-assertion. Anyone can write callerId: "admin". The header token, on the other hand, is signed by an IdP and cannot be forged. The identity you can actually verify only rides on the header side.
You can offload verification to infrastructure. Reading the Authorization header is the specialty of an API Gateway / reverse proxy / IAM. It does not have to parse the body (app-specific JSON). So the A2A SDK itself never needs to know "who is calling" (it stays identity-agnostic).
No double bookkeeping. HTTP already has a place for auth (the header). Put it in the body too and you have two places that can disagree.

The "transport layer" here does not mean OSI L4 in the strict sense. It is more like "the HTTP request envelope (headers) that wraps the app message (JSON-RPC)." ID on the envelope, business in the contents.

This separation also feeds the "holes" later. If identity does not ride in the payload, the A2A message itself cannot carry the context "acting on behalf of user-123." So when you delegate in hops, A to B and B to C, the job of conveying "who the original user is" falls outside A2A (Identity Chaining / Transaction Tokens).

Decision 2: advertise auth requirements in the Agent Card

"To call this agent you need a Bearer token (scope read:tasks)" or "no, we use an API key": requirements differ per agent. A2A makes you declare this in the Agent Card's securitySchemes and security fields. Read the card and you know what to bring. Details below.

Decision 3: acquire credentials out-of-band

The spec decides how credentials are "used" but not how they are "obtained."

Credentials for a client agent to connect to a remote agent are obtained by the client agent through an out-of-band process outside the scope of the A2A protocol.

How you get the token (run an OAuth authorization-code flow, or grab one with client credentials) is outside A2A. The agent operates on the premise that it "already has a token."

Decision 4: per-skill scope authorization, enforced externally

An A2A agent lists its "skills" (what it can do) in the Agent Card. Authorization can be applied per skill.

Access can be controlled on a per-skill basis ... specific OAuth scopes should grant an authenticated client access to invoke certain skills but not others.

But the implementation that "checks the scope and rejects" is not something A2A does. That too is external work (a Gateway or IAM, or the app's middleware). The A2A spec only says "do it."

Sorting "what A2A defines" from "what it delegates"

Sort the decisions so far cleanly into "what A2A decides itself" and "what it delegates to external standards," and the essence of A2A auth shows.

Left (green) is A2A's "frame," right (yellow) is the "delegation targets." Understanding A2A auth is almost entirely about understanding the left frame, and the right is the very standards covered in other articles (ID-JAG, Identity Chaining, WIMSE, and so on).

From here, take the left frame apart in order.

Dissecting the Agent Card: 5 SecurityScheme types

Look at a real Agent Card that declares auth requirements. securitySchemes defines "what auth methods exist" by name, and the security array specifies "which methods are required." This notation is borrowed straight from OpenAPI 3.x Security Schemes.

{
  "name": "Document Processing Agent",
  "description": "An agent that analyzes documents",
  "capabilities": {
    "streaming": true,
    "extendedAgentCard": true
  },
  "securitySchemes": {
    "oauth2": {
      "type": "oauth2",
      "flows": {
        "authorizationCode": {
          "authorizationUrl": "https://auth.example.com/oauth/authorize",
          "tokenUrl": "https://auth.example.com/oauth/token",
          "scopes": {
            "read:tasks": "Read task information",
            "write:tasks": "Modify tasks"
          }
        }
      }
    },
    "apiKey": {
      "type": "apiKey",
      "in": "header",
      "name": "X-API-Key"
    }
  },
  "security": [
    { "oauth2": ["read:tasks", "write:tasks"] },
    { "apiKey": [] }
  ],
  "skills": [
    { "name": "analyze-document", "description": "Analyze document content" }
  ],
  "signatures": [
    {
      "protected": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9",
      "signature": "SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"
    }
  ]
}

How to read this card:

securitySchemes defines two methods, oauth2 and apiKey.
The security array is read as OR. It means "come in with oauth2 (with scopes), or come in with apiKey."
signatures is for tamper detection on the card itself (more below).

The spec fixes SecurityScheme at exactly 5 types. These are the entirety of A2A's auth methods.

type	what it is	main fields
`apiKey`	API key	`in` (header/query/cookie), `name`
`http`	HTTP auth (Basic/Bearer, etc.)	`scheme` (e.g. "bearer"), `bearerFormat`
`oauth2`	OAuth 2.0	`flows` (table below)
`openIdConnect`	OIDC discovery	`openIdConnectUrl`
`mtls`	mutual TLS	(no extra fields)

The flows you can write in oauth2's flows are also limited by the spec: the only ones defined are authorizationCode, clientCredentials, and deviceCode, just these three. OpenAPI's implicit and password are not adopted in A2A (both are discouraged flows, so it is a sensible cut). For agent-to-agent traffic (no human in the loop) you pick clientCredentials, and for human delegation you pick authorizationCode.

Worth noting: mtls is in this list of 5. Beyond OAuth tokens, certificate-based mutual auth is in the "frame" from the start. That turns out to be the key to closing holes later.

The auth flow: read the card, then send the real request

With the cast and card understood, walk the actual auth flow end to end. A2A auth is discovery-driven (it starts from the card). The client first fetches the card with no auth, reads the requirements written there, then sets up auth.

Three points.

Phase 1 needs no auth. Anyone can read the card. So you must not write secrets in the card, and the card's authenticity has to be protected by other means (signing).
Phase 2 is outside A2A. A2A has nothing to do with how the token is obtained. The diagram uses client_credentials as an example, but this is entirely the OAuth world.
Credentials always go in HTTP headers. The spec is blunt: "Credentials MUST be transmitted in standard HTTP headers." Never in the JSON-RPC body. Validation failure is 401, insufficient permission is 403.

Extended Agent Card: authenticate and the card grows

A2A has a neat trick: the Extended Agent Card. The public card (readable by anyone) carries only the minimum, and only authenticated clients get an "extended card" with additional skills and config.

This helps when you want to hide even the existence of a capability. Show only base skills to the outside, and show management skills to authenticated internal agents.

Spec rules:

Usable only if the public card has capabilities.extendedAgentCard: true.
The auth for fetching the extended card uses a scheme declared in the public card's security. In other words, "the key to see the extended version is told to you properly by the public version."
A client that gets the extended card replaces its cached public card with it (for the duration of the authenticated session, or until the version changes).

What is interesting is that "the visible capabilities change with authentication," a form of authorization, is built into the protocol.

Signing the Agent Card: the card itself is an attack surface

You may have noticed by now. All the auth requirements are written on the card. So what happens if the card is tampered with?

If an attacker rewrites the card's tokenUrl to their own server, the client goes to the attacker to fetch a token. Inject a prompt injection into the card's description and you can warp the behavior of a victim agent that reads it. The card is the root of trust, yet in Phase 1 it is handed out with no auth. That is the weak point.

For this, A2A provides Agent Card signing.

The mechanism is two stages:

Canonicalize with JCS (RFC 8785, JSON Canonicalization Scheme): the same JSON content can produce different bytes depending on key order and whitespace. Canonicalize before signing so anyone who processes it gets the same byte string. Skip this and you get the accident where the meaning is identical but signature verification fails.
Sign with JWS (RFC 7515): sign the canonicalized bytes and put protected (the JWS protected header) and signature into signatures[].

But in the spec this is MAY (optional). It is "you may sign," not "you must sign." That becomes one of the holes discussed next.

Where the holes are: risks born from A2A's delegation design

A2A's "define only the frame, delegate the contents" design is safe if the implementer fills it in properly, but forget to fill it and it becomes a hole. The parts the spec does not bind with MUST become attack surface directly. Here are the main attack surfaces and their roots (how to close them is diagrammed in the next section).

Card tampering / impersonation: the signature (signatures) is optional, and the spec does not even mandate "how to verify the card." An implementation that trusts an unsigned card as-is gets lured to the attacker's server by a rewritten card.
Replay: Authorization: Bearer <token> is a bearer token. Steal it and the thief acts as the legitimate client. A2A itself has no mechanism to "bind a token to a specific client."
Credential leak down the delegation chain: in a chain where agent A calls B and B calls C, if A's token unintentionally flows to C, permission leaks. The spec says "credentials SHOULD be bound to the agent which originated the request," but with SHOULD, not as a mandate.
Scope bypass / confused deputy: per-skill authorization is also up to the app/Gateway to enforce. Forget the "can this token call this skill" check and authentication passes while authorization sails through. And when an agent executes someone else's request with "its own strong permissions," you get a textbook confused deputy.

In short, whether A2A auth ends up safe hinges on whether the implementer can promote the parts the spec let off with MAY / SHOULD up to MUST.

Closing the holes: mTLS + signed cards + existing identity standards

So how do you fill it concretely. OAuth bearer tokens alone cannot close the holes above. The combination that comes up over and over in practice is the three-piece set "mutual TLS + signed Agent Card + PKI-backed machine identity." These three supply the "proof of the sender" and "authenticity of the card" that a bearer token structurally cannot hold. Mapping this onto A2A's frame and the existing standards covered in other articles looks like this.

Brought down to concrete implementation policy:

Always sign the card. Do not accept an unsigned card. Promote the spec's MAY to MUST in your own system. Put issuer verification on PKI (a trusted CA, or a SPIFFE trust domain).
Use mTLS for agent-to-agent paths with no human in the loop. A2A has the mtls SecurityScheme from the start. With client certs, a stolen token alone cannot impersonate. Even when using OAuth, kill bearer replay with sender-constrained tokens. This is a mechanism that binds the token to "a specific client's key or cert," so stealing only the token is useless without the matching key. The two implementation styles are mTLS binding (RFC 8705) and DPoP (RFC 9449).
Leave chain propagation to standards outside A2A. Bind the credential to the originator, and for crossing domains use ID-JAG / Identity Chaining, for safe propagation within a chain use Transaction Tokens. A2A is only the "frame" to carry these, so bring the contents in from standards.
Always enforce per-skill authorization at a Gateway / IAM. "Authentication passed" and "has permission to call that skill" are different things. Always place a layer that does not let the latter through unchecked.

This is where the "delegation targets" (the blue on the right) all connect to existing standards covered in other articles. A2A is the plumbing (frame) of AI agent auth, and the water flowing through it (the identity contents) is the very standards I have been writing about. That structure becomes clear.

See it for real: turn a coding CLI into an A2A server and the holes appear

The "holes" so far may look abstract, but stand up one existing OSS project and they show up right in front of you. A Remote Agent is not a special product. It is just an HTTP server that wraps an agent you have on hand as an A2A server. There are several OSS projects you can actually try.

a2aproject/a2a-samples: the official hello-world Remote Agent (5 languages). The minimal server that handles message/send
hybroai/a2a-adapter: a Python SDK that converts n8n / LangGraph / CrewAI / Claude Code / Codex / Ollama and more into A2A servers (published on PyPI)
firstintent/a2a-bridge: connects CLI agents like Claude Code / Codex / Gemini CLI / Zed to each other through a daemon

For example, turning Claude Code into an A2A server with a2a-adapter is just this.

from a2a_adapter import ClaudeCodeAdapter, serve_agent

adapter = ClaudeCodeAdapter(working_dir="/path/to/project")
serve_agent(adapter)  # Claude Code comes up as an A2A server

This stands up a server where "throw a task over the network and Claude Code reads/writes files and runs commands and returns." The problem starts here. The a2a-adapter README says this.

Without a pre-configured Claude Code permissions file, tool-use is blocked. You can bypass it with skip_permissions=True (or the env var A2A_CLAUDE_SKIP_PERMISSIONS=1).

adapter = ClaudeCodeAdapter(working_dir="...", skip_permissions=True)

The moment you add skip_permissions=True, Claude Code runs tools (file edits, command execution) without human confirmation. And the trigger that launches it is a task coming over A2A. If you have not tightened the front-door auth (who can call this server), anyone who reaches the endpoint gets all the way to command execution. This is exactly the attack surface listed earlier in the article.

No auth in front -> unauthenticated remote code execution
Token only, no mTLS -> replay with a stolen token
Per-skill scope not enforced -> "read only" turns into write and execute (confused deputy)

So the previous section's fixes (limit who can call it with mTLS / signed cards / enforce per-skill scope at a Gateway) are not slogans. They become mandatory the moment you ship one such adapter to production. The responsibility A2A "delegated" for auth is ultimately taken on by the human who stood up this server.

Design-decision cheat sheet

For reference while implementing, here are A2A auth decision points on one page.

Topic	What A2A specifies	Strength	Recommendation in practice
Where identity lives	HTTP header / transport layer. Not in the payload	design	Follow as-is. Push verification to Gateway/IAM
Conveying auth requirements	Agent Card `securitySchemes` / `security`	defined	Mind the OR semantics. Cut least-privilege scopes
Auth methods	apiKey / http / oauth2 / openIdConnect / mtls, 5 types	defined	For M2M paths use `mtls` or client_credentials
Credential transmission	HTTP headers required	MUST	Never in the body
Transport	HTTPS required in prod, TLS 1.2+ recommended	MUST/SHOULD	Use TLS 1.3, verify the server cert
Card signing	JWS (RFC 7515) + JCS (RFC 8785)	MAY	Promote to MUST in your own system
Authorization granularity	per-skill scope	SHOULD	Always enforce at Gateway/IAM
Binding the credential	bind to the originating agent	SHOULD	Enforce with sender-constrained tokens
Cross-domain propagation	unspecified (outside A2A)	none	Bring in ID-JAG / Identity Chaining
Replay defense	unspecified	none	Kill bearer theft with mTLS / DPoP

Where the "Strength" column is MUST, A2A has your back. Where it is MAY / SHOULD / none, you tighten it yourself. A2A auth security comes down to how much of the bottom-right of this table you fill.

Conclusion

A2A auth is "thin." But that is not laziness. It is the result of a clear design decision: "treat agents as ordinary enterprise apps and delegate the auth contents to existing standards."

The three things to hold onto:

What A2A defines itself is only the "frame." Agent Card securitySchemes / security, the 5 SecurityScheme types, the no-identity-in-payload design, the Extended Agent Card, JWS signing. Token issuance, validation, and propagation are all outside.
Holes appear in the parts the spec let off with MAY / SHOULD. Card signing is optional, replay defense is absent, scope enforcement is external. Unless the implementer promotes these to MUST, they sail through.
The way to close them is mTLS + signed cards + existing identity standards. SPIFFE / WIMSE / ID-JAG / Identity Chaining / Transaction Tokens are the water flowing through the A2A plumbing.

Read A2A as a "new auth protocol" and you get the anticlimax. Read it as a "frame for wiring existing auth standards between AI agents" and the rationale of the design, and the responsibility you have to fill yourself, both come into focus. Next time you stand up an A2A server, change the bottom-right of this article's cheat sheet (MAY / SHOULD / none) into MUST, one row at a time.

References

gRPC Deep Dive: Stubs, HTTP/2 Frames, and Why Netflix, Spotify, and Mercari Switched

kt — Wed, 10 Jun 2026 15:37:14 +0000

Introduction

While writing my xDS post (the one about istiod shipping protobuf to Envoy over a gRPC stream), I noticed something uncomfortable. I use gRPC every day. And if someone had asked me "so what is gRPC, exactly", I would have said "HTTP/2" and "Protocol Buffers" and then run out of sentences.

So I stopped and rebuilt my understanding from the ground floor. This post walks through it in order, and every term gets explained at the spot where it first appears. The only assumption is that you have written an HTTP API that returns JSON at some point.

What RPC means (the "RPC" in gRPC)
The two parts that make up gRPC: Protocol Buffers and HTTP/2
The four communication patterns
Running it locally and measuring everything (Go)
Adoption stories: what each company was migrating away from
Weak points, and when to pick gRPC over REST

One running example carries the whole post: gadgefre, a fictional flea-market app for used gadgets. It has exactly four characters, and they show up in every diagram.

The mobile app (what the user touches)
The order service (accepts orders)
The stock service (tracks inventory)
The payment service (moves money)

RPC: network calls dressed up as function calls

Before gRPC, RPC. Remote Procedure Call means exactly what the name says: calling code that runs on another machine, written as if it were a local function call.

Think about what calling a REST API looks like in code. You build a URL, pick an HTTP method, encode a JSON body, parse the JSON response, branch on the status code. The thing you wanted was "reserve 2 units of this item", and most of the code is transport logistics.

RPC hides all of that. The order service just writes this:

// The stock service runs on another machine, but this reads like a local call
res, err := stockClient.Reserve(ctx, &pb.ReserveRequest{ItemId: "gx100", Quantity: 2})

Here is what actually happens underneath:

The interesting box is the stub. A stub is the code that does the transport logistics for you (serialize, send, receive, deserialize), and nobody writes it by hand. A tool generates it. The diagram has the order service in Go and the stock service in Java on purpose: stubs are generated per language, so the caller and callee do not need to agree on one.

RPC itself is an old idea with plenty of implementations (Java RMI, Thrift, JSON-RPC). gRPC is the modern one, open sourced by Google in 2015. Internally Google had been running an RPC system called Stubby for over a decade, handling on the order of tens of billions of calls per second across its datacenters, and gRPC is the rebuild of that system on open standards. In 2017 it moved to the CNCF (the same foundation as Kubernetes), where it lives today as an Incubating project with a release roughly every six weeks.

Part 1: Protocol Buffers (what gets sent)

gRPC is two parts glued together. What gets sent is decided by Protocol Buffers (protobuf for short). How it travels is decided by HTTP/2. Protobuf first.

Protobuf has two jobs:

An IDL (interface definition language): you describe the shape of your API in a plain-text .proto file
A serialization format: it turns data into a binary blob smaller than JSON

Protobuf as an IDL

Here is the gadgefre order service API as a .proto file. This one file is the spine of the whole post; the hands-on later uses it unchanged.

syntax = "proto3";

package gadgefre.order.v1;

service OrderService {
  // Unary: create one order
  rpc CreateOrder(CreateOrderRequest) returns (CreateOrderResponse);
  // Server streaming: push order status changes as they happen
  rpc WatchOrder(WatchOrderRequest) returns (stream OrderEvent);
}

message CreateOrderRequest {
  string item_id  = 1;
  int32  quantity = 2;
  int64  user_id  = 3;
}

message CreateOrderResponse {
  string      order_id  = 1;
  OrderStatus status    = 2;
  int64       total_yen = 3;
}

message WatchOrderRequest {
  string order_id = 1;
}

message OrderEvent {
  string      order_id = 1;
  OrderStatus status   = 2;
}

enum OrderStatus {
  ORDER_STATUS_UNSPECIFIED = 0;
  ORDER_STATUS_PENDING     = 1;
  ORDER_STATUS_CONFIRMED   = 2;
  ORDER_STATUS_SHIPPED     = 3;
}

Three things to know when reading it:

The service block is the list of callable functions, each in the form rpc Name(Input) returns (Output)
The message blocks are the input/output types. The = 1, = 2 are field numbers: on the wire, these numbers stand in for the field names (that is where the size savings come from)
The first enum value being _UNSPECIFIED = 0 is a protobuf convention. In proto3 there is no way to tell "field was never set" apart from "field was set to value 0", so value 0 is reserved to mean "not specified"

Feed this file to the protoc compiler and it generates stubs for whatever languages you need:

This diagram is the benefit that adopting companies bring up first. The API definition lives in one .proto file, and every language's client and server code is generated from it mechanically. Two chronic problems ("the docs drifted from the implementation" and "we hand-write a client per language") disappear structurally. Mercari, which shows up in the adoption section, keeps every service's .proto in a single repository and has CI regenerate the Go, Python, Java, and Node.js code on merge.

Protobuf as a serialization format

The second job is binary serialization. I serialized the same payload as protobuf and as JSON and measured (the measurement code appears in the hands-on):

protobuf: 24 bytes  0a 11 6f 72 64 5f 67 78 31 30 30 5f 31 32 33 34 35 36 37 10 01 18 e8 4d
json:     81 bytes  {"order_id":"ord_gx100_1234567","status":"ORDER_STATUS_PENDING","total_yen":9960}

Same content, 24 bytes of protobuf versus 81 bytes of JSON, a 3.4x difference. The bytes are worth reading because they show how the trick works:

0a says "field number 1, length-delimited type". That is order_id. The 8-character string "order_id" appears nowhere in the payload
11 is length 17, and the next 17 bytes are the ASCII of ord_gx100_1234567
10 01 is field 2 (status) set to 1 (PENDING). One byte, not the enum name as a string
18 e8 4d is field 3 (total_yen) set to 9960, encoded as a varint (variable-length integer) in 2 bytes

The field names, quotes, and braces that JSON re-sends on every single message simply do not exist in protobuf, because the field numbers are pinned in the .proto. When your services exchange tens of thousands of messages per second, that difference is bandwidth and parsing CPU, paid continuously.

There is a cost: the payload is binary, so you cannot curl it and read it with your eyes. The weak points section deals with that, tooling included.

Part 2: HTTP/2 (how it travels)

gRPC runs on HTTP/2 as its transport. If your mental model of HTTP/2 is "HTTP/1.1 but faster", gRPC's design decisions look arbitrary, so this section takes its time. HTTP/1.1 and HTTP/2 share the same semantics (methods, headers, status codes) but differ completely in architecture: how those semantics are laid out as bytes on a connection.

HTTP/1.1's architecture: text letters, one at a time

An HTTP/1.1 request is plain text separated by newlines:

POST /reserve HTTP/1.1
Host: stock.gadgefre.internal
Content-Type: application/json
Content-Length: 32

{"item_id":"gx100","quantity":2}

The smallest unit in this protocol is the whole message. Text streams in top to bottom, and the receiver has no way to tell "which request does this line belong to", so one TCP connection can carry only one request at a time. Want parallelism, open more connections. That is literally what browsers have done for decades: about six connections per host.

For service-to-service traffic this hurts. The order service constantly wants to ask the stock service and the payment service things at the same time, and every extra connection costs a TCP handshake plus a TLS negotiation (several round trips) to establish.

HTTP/2's architecture: binary frames interleaved on one connection

HTTP/2 rebuilt exactly this part. Messages are chopped into frames, small binary boxes, and every frame is tagged with the number of the stream (the logical conversation) it belongs to. The receiver sorts frames by stream number and reassembles. Result: one TCP connection carries many conversations at once. That is multiplexing.

The colors line up between the two halves: collect only the blue frames and you have reassembled stream 1, only the purple ones and you have stream 5. That picture is most of what HTTP/2 is. Frames come in a handful of types, and these are the ones gRPC traffic actually consists of:

Frame	Role	How gRPC uses it
HEADERS	Carries a block of headers	Opens an RPC (method name etc.) and closes it (the trailer, see below)
DATA	Carries body bytes	The protobuf messages
SETTINGS	Connection parameter exchange	Both sides send it right after connecting
WINDOW_UPDATE	Flow control (receive buffer space)	Backpressure for streaming
PING	Liveness check	Keepalive
RST_STREAM	Kill one stream only	RPC cancellation, deadline exceeded
GOAWAY	Announce connection shutdown	Graceful shutdown

Look at RST_STREAM and WINDOW_UPDATE for a moment. "Cancel one RPC out of the hundred in flight" and "slow down only the stream whose receiver is falling behind" are operations built into the protocol layer. On HTTP/1.1 your only lever is killing the whole connection. The reason cancellation and flow control behave consistently across every gRPC language is not heroic framework code. It is that HTTP/2 is shaped like this.

Headers got an upgrade too: they travel compressed with HPACK. A header sent once gets an entry in a per-connection dictionary, and from then on it is a one-or-two-byte index. Nobody re-sends the string content-type: application/grpc ten thousand times.

One more mechanism matters later: the trailer. HTTP/2 lets a peer send one more HEADERS frame after the body, closing the message with headers at the end. That tail block is the trailer, and it exists to carry "here is how things turned out" information that is only known after the body finished. For a stream that pushed gigabytes before failing, a status in the leading headers is physically impossible; the end of the message is the only honest place. gRPC puts the RPC's outcome there. And this trailer is the entire reason for the "browser wall" coming up in the weak points section.

How gRPC maps onto HTTP/2

Here is the full assignment of gRPC concepts to HTTP/2 machinery:

gRPC concept	What it is on HTTP/2
One RPC	One stream
Method selection	The `:path` header (e.g. `/gadgefre.order.v1.OrderService/CreateOrder`)
A request/response message	"5-byte prefix + protobuf" inside DATA frames
RPC outcome	`grpc-status` in the trailer (0 means OK)
Deadline	The `grpc-timeout` header
The four patterns (next section)	Just how many DATA frames flow in each direction
A channel (the client-side connection object)	One or more TCP connections plus SETTINGS/PING bookkeeping

And the frame-by-frame shape of a single unary RPC:

That is the theory. Whether it is true gets checked at the end of the hands-on, where I capture the actual frames off a live connection.

In case you are wondering about HTTP/3: as of June 2026, ecosystem-wide HTTP/3 support is still at the official proposal stage (G2), with grpc-dotnet running a trial implementation. Production gRPC traffic today is overwhelmingly HTTP/2.

The four communication patterns

A REST API has essentially one shape: one request, one response. gRPC gives you four. As the mapping table said, they are not four mechanisms; they differ only in how many DATA frames each side sends on the one stream.

Mapped onto gadgefre:

Pattern	In the proto	gadgefre use case
Unary	`rpc CreateOrder(Req) returns (Res)`	Creating an order. Every ordinary API call
Server streaming	`rpc WatchOrder(Req) returns (stream Event)`	Live order status. Push the moment it ships
Client streaming	`rpc UploadPhotos(stream Chunk) returns (Res)`	Photo upload in chunks, one result at the end
Bidirectional	`rpc Chat(stream Msg) returns (stream Msg)`	Buyer-seller chat

The only syntax involved is where you put the stream keyword. Here is WatchOrder (server streaming) over time:

Doing this with REST means the client polls with GET every few seconds, or you bolt on a separate WebSocket layer. In gRPC you write the word stream in the proto and the generated stubs, flow control, and all four languages' implementations come with it. This is one of the reasons ABEMA (more on them later) picked gRPC for a latency-sensitive video service.

The big picture so far

All the parts are on the table, so here is gadgefre in one diagram:

There is a design decision buried in this diagram, and nearly every adopter made the same one: the outside edge (phones, browsers) speaks REST/JSON, and only the service-to-service interior speaks gRPC. Mercari, ABEMA, and Netflix all look like this. The reasons are the browser problem explained in the weak points section, plus the fact that external developers expect REST. Lining up the case studies makes the pattern obvious: gRPC spread as the tool for service-to-service traffic, not as a REST replacement.

Hands-on: run it and measure it

Time to touch it. Environment: Apple Silicon Mac, Go 1.26.4, libprotoc 35.0, grpc-go v1.81.1, grpcurl 1.9.3. The proto is the order.proto from earlier, unchanged.

1. Generate the code

brew install protobuf protoc-gen-go protoc-gen-go-grpc grpcurl

mkdir grpc-demo && cd grpc-demo
go mod init example.com/gadgefre
# save the proto from above as proto/order.proto, then:
protoc --go_out=. --go_opt=module=example.com/gadgefre \
       --go-grpc_out=. --go-grpc_opt=module=example.com/gadgefre \
       proto/order.proto

Two files appear under gen/orderpb/: order.pb.go holds the message types (structs plus serialization), order_grpc.pb.go holds the service stubs (client and server interfaces). Open them and you can see that the "stub" from the first diagram is just ordinary Go code.

2. The server

package main

import (
    "context"
    "fmt"
    "log"
    "net"
    "time"

    "google.golang.org/grpc"
    "google.golang.org/grpc/reflection"

    pb "example.com/gadgefre/gen/orderpb"
)

type orderServer struct {
    pb.UnimplementedOrderServiceServer
}

func (s *orderServer) CreateOrder(ctx context.Context, req *pb.CreateOrderRequest) (*pb.CreateOrderResponse, error) {
    return &pb.CreateOrderResponse{
        OrderId:  fmt.Sprintf("ord_%s_%d", req.ItemId, req.UserId),
        Status:   pb.OrderStatus_ORDER_STATUS_PENDING,
        TotalYen: 4980 * int64(req.Quantity),
    }, nil
}

func (s *orderServer) WatchOrder(req *pb.WatchOrderRequest, stream grpc.ServerStreamingServer[pb.OrderEvent]) error {
    statuses := []pb.OrderStatus{
        pb.OrderStatus_ORDER_STATUS_PENDING,
        pb.OrderStatus_ORDER_STATUS_CONFIRMED,
        pb.OrderStatus_ORDER_STATUS_SHIPPED,
    }
    for _, st := range statuses {
        if err := stream.Send(&pb.OrderEvent{OrderId: req.OrderId, Status: st}); err != nil {
            return err
        }
        time.Sleep(300 * time.Millisecond)
    }
    return nil
}

func main() {
    lis, err := net.Listen("tcp", ":50061")
    if err != nil {
        log.Fatal(err)
    }
    s := grpc.NewServer()
    pb.RegisterOrderServiceServer(s, &orderServer{})
    reflection.Register(s)
    log.Println("listening on :50061")
    log.Fatal(s.Serve(lis))
}

Notice what is missing: not one line of HTTP/2 handling or serialization. Only logic. reflection.Register(s) is for grpcurl later; it lets the server tell clients "here is the proto I implement". (The port is 50061 because something on my machine was already squatting on 50051, which is the conventional one.)

3. The client, and real numbers

The client side is just calls on the generated stub. The relevant excerpt:

conn, _ := grpc.NewClient("localhost:50061", grpc.WithTransportCredentials(insecure.NewCredentials()))
c := pb.NewOrderServiceClient(conn)

res, _ := c.CreateOrder(ctx, &pb.CreateOrderRequest{ItemId: "gx100", Quantity: 2, UserId: 1234567})

stream, _ := c.WatchOrder(ctx, &pb.WatchOrderRequest{OrderId: res.OrderId})
for {
    ev, err := stream.Recv()
    if err == io.EOF {
        break
    }
    fmt.Printf("WatchOrder  -> %s is now %s\n", ev.OrderId, ev.Status)
}

Output, including a latency loop I added at the end (10000 unary calls on a warmed-up connection, sorted, percentiles taken):

CreateOrder -> order_id=ord_gx100_1234567 status=ORDER_STATUS_PENDING total_yen=9960
WatchOrder  -> ord_gx100_1234567 is now ORDER_STATUS_PENDING
WatchOrder  -> ord_gx100_1234567 is now ORDER_STATUS_CONFIRMED
WatchOrder  -> ord_gx100_1234567 is now ORDER_STATUS_SHIPPED
unary x10000: p50=50.334µs p99=165.667µs

Localhost, sure, but that includes serialization and HTTP/2 framing both ways: p50 of 50 microseconds per call. Worth keeping as a gut number for how light the RPC machinery itself is.

4. Debugging with grpcurl

This is the answer to "binary means no curl". With reflection enabled on the server, grpcurl fetches the proto and does the JSON conversion for you:

$ grpcurl -plaintext localhost:50061 list
gadgefre.order.v1.OrderService
grpc.reflection.v1.ServerReflection

$ grpcurl -plaintext -d '{"item_id":"gx100","quantity":2,"user_id":1234567}' \
    localhost:50061 gadgefre.order.v1.OrderService/CreateOrder
{
  "orderId": "ord_gx100_1234567",
  "status": "ORDER_STATUS_PENDING",
  "totalYen": "9960"
}

Small thing that trips everyone up once: "totalYen": "9960" is a string, and that is by spec. Protobuf's JSON mapping always renders int64 as a JSON string, because JavaScript's Number cannot represent 64-bit integers exactly.

5. Looking at the actual HTTP/2 frames

Time to check Part 2's diagrams against reality. I wrote an 80-line proxy that sits between client and server, passes every byte through untouched, and feeds a copy into golang.org/x/net/http2's Framer to log what it sees (listens on 50071, forwards to 50061). The whole trick is this:

framer := http2.NewFramer(io.Discard, r) // r carries the raw connection bytes
framer.ReadMetaHeaders = hpack.NewDecoder(4096, nil)
for {
    f, err := framer.ReadFrame()
    if err != nil {
        return
    }
    // f is a *http2.MetaHeadersFrame, *http2.DataFrame, etc. Log by type
}

Every frame from one CreateOrder call through that proxy (connection setup SETTINGS, keepalive PING, and flow-control WINDOW_UPDATE omitted):

[client→server] HEADERS stream=1 END_STREAM=false
         :method=POST
         :scheme=http
         :path=/gadgefre.order.v1.OrderService/CreateOrder
         :authority=localhost:50071
         content-type=application/grpc
         user-agent=grpc-go/1.81.1
         te=trailers
[client→server] DATA stream=1 len=18 END_STREAM=true
         00 00 00 00 0d 0a 05 67 78 31 30 30 10 02 18 87 ad 4b
[server→client] HEADERS stream=1 END_STREAM=false
         :status=200
         content-type=application/grpc
[server→client] DATA stream=1 len=29 END_STREAM=false
         00 00 00 00 18 0a 11 6f 72 64 5f 67 78 31 30 30 5f 31 32 33 34 35 36 37 10 01 18 e8 4d
[server→client] HEADERS (trailer) stream=1 END_STREAM=true
         grpc-status=0
         grpc-message=

What to look at:

The order matches Part 2's sequence diagram exactly: HEADERS, DATA, then HEADERS, DATA, trailer coming back
The first 5 bytes of the request DATA, 00 00 00 00 0d, are the "5-byte prefix" in the flesh. Byte one is the compression flag (0 = uncompressed), the next four are the message length (0x0d = 13). The remaining 13 bytes are the CreateOrderRequest protobuf: 0a 05 67 78 31 30 30 reads "field 1, length 5, gx100"
The 24 protobuf bytes inside the response DATA are byte-for-byte identical to the hex dump from the size measurement section
te=trailers in the request headers is the client declaring "I can receive trailers". A browser's fetch API cannot make that declaration (this becomes the next section)
grpc-status=0 in the final trailer means the RPC succeeded. Note that it is a different thing from :status=200

That last point has operational teeth. A failing gRPC call still carries HTTP :status=200; only grpc-status in the trailer changes. If your L7 access logs equate 200 with healthy, you are blind to every gRPC error. Monitor on grpc-status.

Adoption: what was each company escaping from?

Now that the machine is understood, the users. A bare list of logos teaches nothing, so the axis here is what they ran before, and what hurt.

Company	Before	What decided it
Google	In-house Stubby	gRPC is Stubby's open rebuild; everything internal is RPC
Netflix	In-house HTTP/1.1 stack (Ribbon)	Cost of maintaining their own; new Java services start on gRPC
Spotify	In-house RPC (Hermes)	"The community caught up and surpassed us"
Dropbox	In-house RPC frameworks	Could keep existing protobufs; HTTP/2 multiplexing and streaming
Uber	Server-Sent Events over HTTP/1.1 (push)	Bidirectional streaming, cross-language stubs, QUIC interop
Salesforce	JSON/REST	`.proto` as a fixed contract between teams
Mercari / Merpay	(greenfield)	Standardized on gRPC while splitting into microservices
ABEMA	(greenfield)	Low latency, fit with GCP + Kubernetes + Go
Ikyu	REST	Speed; built a parallel REST fallback and never needed it

A few worth unpacking.

Netflix ran service-to-service traffic on its own HTTP/1.1-based stack (Ribbon and friends, parts of it open sourced) until around 2015, then moved to gRPC when the maintenance bill came due. Today a large share of their internal traffic is gRPC, and new Java development starts gRPC-first. The interesting bit: the driver was not speed. It was wanting to stop maintaining a bespoke RPC framework.

Spotify is the same story with different names: their in-house Hermes got replaced by gRPC plus Envoy. Their engineer Dave Zolotusky summarized the whole industry arc in one line: they had built their own tools because nothing handled their scale, "but then the community kind of caught up and surpassed us." Every company that went microservices early, around 2015, eventually faced that decision, and nearly all of them landed on gRPC.

Dropbox documented its migration in detail as Courier: hundreds of services in multiple languages exchanging millions of requests per second. Two details stand out. They picked gRPC partly because they could carry their existing protobuf definitions over unchanged, and Courier itself is not a new protocol; it is gRPC wired into their existing auth, service discovery, and tracing. Their closing lesson applies to any migration: it takes longer than the development itself, and it is only finished after the cleanup.

Uber is the streaming showcase. Their mobile push platform (internally called RAMEN) originally delivered updates over Server-Sent Events on HTTP/1.1; they rebuilt it on gRPC bidirectional streaming, citing the standardized cross-language implementations and the ability to ride Cronet's QUIC sessions on mobile. If your mental image of gRPC is "internal microservice plumbing", Uber pushing to phones over it is the counterexample.

Mercari / Merpay (Japan's largest C2C marketplace and its payments arm) is the best-documented case in the Japanese-language sphere, and the operational details translate well. When they split the monolith to scale the org toward 1000 engineers, they standardized inter-service traffic on gRPC:

Every microservice's .proto lives in one repository; CI generates the Go, Python, Java, and Node.js code on merge
API design debates happen on .proto pull requests, so interfaces get reviewed before implementation starts
They went further and built gRPC Federation, an OSS tool that generates an entire BFF (the aggregation layer in front of mobile clients) from options written in the proto

ABEMA (a Japanese streaming TV service) launched in 2016 on GCP + Kubernetes + Go + gRPC, with roughly 40 microservices talking gRPC to each other. Video is latency-sensitive, and protobuf's encode/decode speed and density were the deciding factors. For external APIs they use grpc-gateway (a tool that generates a REST proxy from the proto), making them a clean example of "gRPC inside, REST outside" done by code generation.

Squeeze the cases and four patterns fall out. If you are deciding whether gRPC belongs in your stack, it comes down to whether these apply:

Too many teams to keep inter-service contracts as verbal agreements (Salesforce, Mercari)
An in-house RPC layer you are tired of maintaining (Netflix, Spotify, Dropbox)
Clients needed in several languages, none hand-written (everyone)
Real-time or latency requirements that polling cannot meet (Uber's push platform, ABEMA)

Weak points, with fixes

It has been a friendly story so far, so here are the traps, honestly. Each comes with a workaround.

Weak point 1: browsers cannot speak it

Remember the trailer, explained in Part 2 and caught on the wire in the hands-on (the final HEADERS frame carrying grpc-status). gRPC reports the outcome of every RPC there, and the browser fetch API cannot read trailers. The te=trailers declaration visible in the frame capture is one a browser will never send. So plain gRPC from browser JavaScript is off the table.

Three families of workarounds:

gRPC-Web: a browser-safe variant of the protocol; a proxy (typically Envoy) translates to real gRPC. Longest track record
grpc-gateway: generates a REST/JSON API from the proto and runs it as a proxy (the ABEMA approach)
Connect RPC: the newer option, from Buf, accepted into the CNCF in 2024. One server speaks gRPC, gRPC-Web, and plain HTTP+JSON on the same port, so the translating proxy disappears entirely. Browsers call it with ordinary fetch

Starting fresh today and wanting protobuf types in the browser, I would look at Connect first. Deleting a proxy tier from your architecture is a big operational win.

Trailers are awkward even outside browsers, by the way. When Cloudflare added gRPC support to their edge in 2020, a large chunk of the work was that their NGINX-based proxies barely supported HTTP trailers and their origin-facing connections were HTTP/1.1. If a CDN had to build a new proxy platform for this, your middleboxes deserve a look too: every hop between client and server must speak HTTP/2 and pass trailers through.

Weak point 2: load balancing skews on Kubernetes

This is the trap people hit in production. HTTP/2's greatest strength, one long-lived connection reused for everything, collides head-on with how Kubernetes load balances by default. A Service (ClusterIP) picks a backend once, at connection time. A long-lived gRPC connection therefore glues itself to whichever Pod it first landed on, and every subsequent request rides that connection to the same Pod.

The symptom: you scale the stock service to 3 Pods and one Pod melts while two idle. The fix is always some layer that picks a backend per request instead of per connection, and there are three:

A service mesh / L7 proxy: Istio or Linkerd sidecars (Envoy) balance per request. If you already run a mesh, you get this for free
Client-side load balancing: built into grpc-go and friends; point it at a headless Service (clusterIP: None) so the client sees every Pod IP, connects to all, and round-robins
xDS: the client gets routing info straight from a control plane, speaking the same protocol Envoy does (proxyless gRPC). Datadog runs this setup

Weak point 3: humans cannot read it without tools

It is binary; tcpdump and curl show you noise. The baseline fix is what the hands-on did: reflection on the server, grpcurl in your hand. Postman supports gRPC if you want a GUI, and there is also Evans, a REPL-style client that came out of Mercari. If you are rolling gRPC out to a team, make "reflection enabled on every server, at least outside production" a written rule early. It pays off weekly.

Weak point 4: schema evolution needs discipline

Field numbers are the binary compatibility contract, so a number, once used, can never change meaning or be recycled. Deleting a field means writing reserved 4; to leave a tombstone. Discipline like this should be enforced by a linter, not by memory: buf breaking checks "does this change break wire compatibility" in CI. Starting with buf instead of raw protoc saves you the incident later.

The decision table

To wrap up the design guidance. "Everything becomes gRPC" is not the lesson; the adopters' own architecture (gRPC inside, REST outside) says so.

Situation	Pick	Why
Internal service-to-service	gRPC	Typed contracts, performance, stubs in every language
Public API for arbitrary consumers	REST + OpenAPI	curl-ability, ecosystem reach
Typed contracts in browser/mobile	Connect or gRPC-Web	Plain gRPC dies on the trailer problem
Real-time server-to-client push	gRPC server streaming	No polling; one `stream` keyword in the proto
Internal, but consumers only speak curl	gRPC + grpc-gateway	Generate the REST facade from the proto

And the tools that appeared along the way:

Tool	Job
`protoc` + `protoc-gen-go` etc.	Generate per-language stubs from `.proto`
`buf`	Modern protoc frontend; linting and breaking-change checks
`grpcurl`	curl for gRPC, pairs with server reflection
`Evans`	Interactive REPL gRPC client
`grpc-gateway`	Generate a REST proxy from the proto
Connect RPC	gRPC-compatible framework family with native browser support

Conclusion

The question that started this post ("so what is gRPC, exactly") now has a one-sentence answer I can stand behind: a framework that generates every language's communication code from a contract written in a .proto file, and carries the messages as protobuf over HTTP/2 streams.

Two things stuck with me from running it. First, how short the distance is from writing a proto to a working client. Second, the weight of the machinery: p50 of 50µs per call, and a wire format where I could account for every single byte. On the flip side, the browser wall and the Kubernetes balancing skew are both "trivial if you know, an outage if you don't" traps, so if you take one section into a migration meeting, take the weak points.

If you want the next layer up, my xDS deep dive is the same story from the service mesh side: istiod pushing protobuf to Envoy over one long-lived gRPC stream.

What xDS Actually Ships: Your Control Plane Sends protobuf, Not YAML

kt — Tue, 09 Jun 2026 17:09:28 +0000

The thing I had wrong for years

When I started with Istio, one mental model stuck and it was wrong. I knew istiod pushed "config" to the Envoy sidecars. Edit a VirtualService, routing changes with no restart. It looked like magic.

The picture in my head was this: istiod generates Envoy YAML, ships that YAML to Envoy, and Envoy loads it. Half right, half wrong, and the wrong half is the interesting one. istiod does not send YAML. It sends protobuf, a binary format, over a gRPC stream. YAML only shows up at the human-facing edges.

This post follows the payload from the top down. Words like xDS, protobuf, gRPC, Any, and type_url show up along the way, and each one gets explained where it lands. At the end I stand up Istio locally on kind and pull the actual bytes off the wire to look at them.

By the time you finish, one sentence should feel obvious: the control plane ships Envoy-schema protobuf messages over xDS.

Cast of characters

Jumping straight into xDS is a good way to get lost, so first the players. A service mesh has two layers.

Here is who is who.

Term	What it is	Concrete example here
Data plane	The layer that actually moves traffic	The Envoy sidecar in each Pod
Control plane	The layer that tells the data plane how to behave	istiod
Envoy	A programmable proxy that receives and forwards packets	The sidecar itself
sidecar	A helper container that rides along with the app Pod	Holds Envoy
istiod	Istio's control plane binary (single process)	The source of config

The key point: Envoy does not know what to do on its own. Which port to listen on, where to forward, who to trust, all of it comes from the control plane. The mechanism for "telling it" is xDS.

What problem this even solves

Why ship config at all? Picture running Envoy on a static file and the answer falls out.

Envoy originally boots from a single YAML file (envoy.yaml). You write "listen on 8080, forward to the reviews service" and it runs. Fine while things are small.

The trouble starts in a moving environment like Kubernetes. Pods come and go by the second. Their forwarding IPs are not stable. Routing rules change on every deploy. mTLS certificates (the certs that let services authenticate each other in both directions) rotate on a schedule. Doing all of that by rewriting YAML and restarting Envoy means dropping connections on every restart, and you never keep up with the churn.

So the idea: receive config through an API instead of a file. When something changes, the control plane streams the delta in, and Envoy never restarts. That mechanism is xDS.

What xDS is

xDS stands for "x Discovery Service". The x is a wildcard that later becomes a letter like L (Listener) or C (Cluster). Mentally expand it to "the family of APIs that ship config to Envoy".

The mechanism is plain: Envoy dials the control plane over gRPC and receives config. For now, treat gRPC as "a way to exchange messages over HTTP/2" (its real shape shows up later with protobuf). Config is not pushed at Envoy out of nowhere; Envoy asks for it.

The natural next question: who is the "xDS server" in Istio terms?

Role	In Istio
Control plane process	istiod (single binary)
The piece inside it that serves xDS	Pilot (a subsystem of istiod; `DiscoveryServer` in the code)
The side that receives config (the xDS client)	Each Pod's Envoy sidecar

Since Istio 1.5 (2020) istiod is a "modular monolith": Pilot, Citadel, and Galley, once separate processes, were folded into one binary. The part that serves xDS is Pilot. So "what part of the control plane serves xDS?" answers as "the xDS server (discovery server)" in general, or "istiod's Pilot" in Istio.

xDS comes in five flavors

xDS is not one API. It splits by the kind of thing being shipped. Five do the heavy lifting, and they compose inside Envoy to serve a single request.

Read top to bottom and you see one request being resolved.

Kind	What it ships	Analogy
LDS	Listeners (the address and port to accept on)	The front door
RDS	Routes (where each path or host goes)	The reception map
CDS	Clusters (the group of upstream targets)	The destination department
EDS	Endpoints (the real Pod IPs and ports)	The home addresses of people in that department
SDS	Secrets (TLS certs and keys)	The ID card

EDS updates the most often, because the set of endpoints changes every time a Pod scales.

ADS: one stream to carry them all

A fair question: with five types, does Envoy open five connections to istiod? Usually no. Istio uses ADS (Aggregated Discovery Service) to bundle all of them onto a single gRPC stream.

The reason is ordering. If you tell Envoy about a new cluster (CDS) before its endpoints (EDS), there is a brief window where a config references something that is not there yet, and traffic can drop. One stream lets istiod control the order of delivery.

"How does one stream keep five types apart?" That is what type_url is for. Its real shape arrives with protobuf below. For now it is just a per-type label.

The misconception worth killing

I kept saying "ship config". Back to the wrong mental model from the top.

For a long time I assumed istiod generated Envoy YAML and sent it. istioctl proxy-config spits out JSON, Envoy config is "YAML", so of course that is what flows. Right?

Wrong. What istiod sends over gRPC is neither YAML nor JSON. It is protobuf, a binary format.

YAML shows up in exactly two places. One is when a human hand-writes envoy.yaml (the bootstrap config at startup). The other is when protobuf gets rendered back into something readable (istioctl proxy-config ... -o yaml and friends). Nowhere on the delivery path does YAML or JSON appear.

To get why, you need to know what protobuf is, which is where this goes next.

What protobuf is

protobuf is short for Protocol Buffers. It is a Google format, and in one line it is a language-neutral serialization format, plus a schema language, for packing structs into bytes and back.

Serialization, briefly

An object (a struct) in memory cannot go on the wire as is. You convert it to a form that can travel (a byte string). That is serialization. The receiver does the reverse (deserialization) to get the struct back.

JSON, YAML, and XML aim at the same thing. Here is the difference.

Format	Kind	Human-readable	Size	Parse speed	Schema
JSON	text	yes	large	slow	optional
YAML	text	yes	large	slow	optional
protobuf	binary	no	small	fast	required

protobuf is unreadable to humans but small and fast, which suits machine-to-machine traffic. When a control plane ships config to thousands of Envoys at high frequency, that difference earns its keep.

You write the schema first (.proto)

protobuf's defining trait is "define the type first". You write the type in a .proto file. A small example:

message Listener {
  string name = 1;
  Address address = 2;
  repeated FilterChain filter_chains = 3;
}

How to read it:

message defines a struct (a class). It is one type.
Each line is the triple type name = number;.
= 1, = 2 are field numbers (tags). On the wire the field is identified by this number, not by its name. That keeps it small, and a number, once chosen, must never change.
repeated is an array. filter_chains can hold many.

Run this .proto through the protoc compiler and it generates classes in each language.

This is the payoff. istiod (Go) and Envoy (C++) compile the same .proto. They share the exact same type, so bytes packed by one deserialize straight into the other's class. No intermediate text like YAML is needed.

How gRPC fits

To close the loop: gRPC is an RPC framework that carries protobuf. The messages gRPC exchanges are defined in protobuf. xDS is "stream protobuf over a bidirectional gRPC connection", so protobuf and gRPC always show up together.

Any and type_url get special treatment

If you read xDS, you cannot avoid google.protobuf.Any. A normal field has a fixed type, but Any is a box whose contents type is decided at runtime.

message Filter {
  string name = 1;
  google.protobuf.Any typed_config = 2;
}

An Any looks like this inside:

{
  type_url: "type.googleapis.com/envoy....Router"
  value:    < bytes of a serialized Router >
}

type_url declares "this is a Router", and value holds that type packed into bytes. Envoy reads type_url and concludes "then deserialize this as Router".

The reason this exists: filters and plugins are open-ended. The Listener .proto cannot enumerate every type in advance, so it keeps a "type decided at runtime" box. The type_url from the ADS section is exactly this mechanism, telling the stream "the type flowing right now is X".

Reading the real listener.proto

That was the setup. Now the real thing. I pulled Envoy's listener.proto from upstream. It runs 450 lines, so I will pick out the syntax highlights with their actual line numbers.

The header first.

syntax = "proto3";                              // line 1: protobuf version
package envoy.config.listener.v3;               // line 3: namespace
import "envoy/config/core/v3/address.proto";    // line 6: import an Envoy type
import "google/protobuf/wrappers.proto";        // line 16: import a Google standard type
import "xds/type/matcher/v3/matcher.proto";     // line 19: import an xds.* type
option go_package = ".../listener/v3;listenerv3";  // line 30: Go package on generation

An important fact is already visible. The imported types fall into three families: envoy/*, google/protobuf/*, and xds/*. The difference matters later, so keep it in the back of your mind. The listenerv3 alias in go_package is the package name of the generated Go code, which is where the listenerv3.Listener{} I used earlier comes from.

Now the body. An excerpt from message Listener.

// [#next-free-field: 39]            // line 64: next free field number is 39
message Listener {
  enum DrainType {                   // line 68: an enum nested inside the message
    DEFAULT = 0;                     // line 71: proto3 enums must start at 0
    MODIFY_ONLY = 1;
  }

  reserved 14, 23;                   // line 140: numbers 14 and 23 are retired, never reuse

  string name = 1;                   // line 145
  core.v3.Address address = 2;       // line 157: type from another package
  repeated FilterChain filter_chains = 3;        // line 176: array
  google.protobuf.BoolValue use_original_dst = 4; // line 207: a wrapper type

  oneof listener_specifier {         // line 416: only one of these may be set
    InternalListenerConfig internal_listener = 27;
  }
}

Three things that look strange on first read.

reserved (retired numbers)

reserved 14, 23; (line 140) seals off field numbers that were used in the past. The field number is the on-wire identifier, so if a client from back when 14 held a different type is still around, reusing 14 makes that old client misread the bytes. So the number is reserved and never used again. The weight of a number here is nothing like a JSON key name.

Wrapper types (the one that matters most)

google.protobuf.BoolValue use_original_dst = 4; (line 207). Why BoolValue instead of bool?

A proto3 scalar (bool, int32, and so on) cannot tell "unset" from "the zero value". bool defaults to false, so "explicitly false" and "never set" are indistinguishable.

Envoy config has many fields that need all three states (true / false / unset). The fix is to wrap bool in a message, BoolValue. A message can express presence, so you get the distinction.

UInt32Value and Duration follow the same logic. When you hit an XxxValue in Envoy .proto, it is a field that needs to tell all three states apart.

oneof (mutually exclusive)

oneof listener_specifier (line 416) means exactly one of its fields is set. You cannot hold two at once. It expresses "A or B, not both" at the schema level.

The xDS proto and the Envoy proto are not the same

Back to the imports splitting into three families. This is the structure I most want to land. "Is the xDS proto different from the Envoy proto?" Yes, and it splits into three layers.

As a table.

Layer	What	Where defined	Envoy-only
Transport	The subscribe and deliver envelope (the xDS protocol itself)	`envoy/service/discovery/v3`	Spec is general, defined in envoy
Shared data model	Base types like matcher	`xds.*` (cncf/xds)	No, vendor-neutral
Resource schema	The contents of Listener, Cluster, etc	`envoy.config.*` (envoy)	Yes, Envoy-specific

The history makes it click. xDS types all used to live in Envoy's repo under envoy.api.v2.* (the previous_message_type = "envoy.api.v2.Listener" inside listener.proto is the fossil of that). Then there was a push to make xDS a general API usable beyond Envoy, because gRPC itself wants to speak xDS without a proxy ("proxyless gRPC"). The shared parts were carved out into cncf/xds (formerly UDPA, the Universal Data Plane API). The xds/core/v3 and xds/type/matcher/v3 imports are that.

But the shape of resource bodies like Listener and Cluster is still defined by Envoy, because what the data plane can interpret is Envoy's call to make.

So the precise statement:

xDS is "the delivery protocol" plus "vendor-neutral base types". The contents of Listener and Cluster are Envoy's proto. What istiod ships is "the xDS transport envelope, stuffed with Envoy-schema resources".

What ties ① and ③ together is type_url. The envelope DiscoveryResponse (layer ①) holds an Any, and its type_url points at type.googleapis.com/envoy.config.listener.v3.Listener (layer ③). ① is the envelope, ③ is the letter inside.

When does the protobuf get filled in

An important distinction. The .proto, and the Go/C++ classes generated from it, are types only, with zero bytes of data. So when does the content get filled in?

At runtime, while istiod is running. On a timeline, building the type and filling the data are clearly separate.

Inside istiod, information read from Kubernetes gets poured into the empty types. The code looks roughly like this.

lis := &listenerv3.Listener{}            // new up the type (still empty)
lis.Name = "0.0.0.0_8080"                // fill a value
lis.Address = buildAddress(svc)          // fill from the Service
lis.FilterChains = buildChains(vs, eps)  // fill from VirtualService and Endpoints

bytes, _ := proto.Marshal(lis)           // turn it into bytes here
stream.Send(resp)                        // send to Envoy

Phase by phase, the data state changes like this.

Phase	When	What happens	Data state
Definition	When you write the `.proto`	Decide the type's shape	No data (blueprint)
Code generation	Build time (protoc)	Turn the type into language classes	No data (empty classes)
Filling	While istiod runs	Pour Kubernetes info into the type	Values present
Sending	On each fill	Marshal and push	Becomes bytes on the wire

What goes on inside istiod

A bit deeper. istiod has several subsystems, and filling the config happens through their interplay.

istiod reads 20-plus resource types (VirtualService, Service, and the rest) and aggregates them into an internal snapshot called PushContext. Then a per-type Generator (RouteGenerator and friends) builds config tailored to the Envoy that connected.

The interesting part: generation depends on which Envoy connected. istiod does not translate inputs to xDS mechanically. It looks at the client's labels and computes "the set of policies that apply to this proxy" before building config. The cost of that flexibility is that config translation eats most of istiod's resources. When istiod pins a CPU on a large mesh, this generation step is usually why.

What triggers a fill

Two main triggers.

When Envoy connects and subscribes: Envoy boots, opens a gRPC stream to istiod, and sends a DiscoveryRequest ("give me LDS"). istiod fills the latest snapshot into the type and returns it.
When config changes: someone runs kubectl apply, a Pod scales and endpoints change, and so on. istiod's watch fires, and it recomputes and pushes to the affected Envoys.

It does not fire blindly. istiod batches changes for a moment (debounce) and only sends the kinds that changed. Endpoints only? Just EDS. A VirtualService? RDS and maybe CDS, and so on.

There are also two delivery styles: "State of the World (SotW)", which sends everything, and "Delta (Incremental) xDS", which sends only the diff. Delta became the default in Istio 1.22, because at scale SotW means re-sending every resource each time, which is heavy on the network and the control plane.

What Envoy does with it

Switch to Envoy's side. When Envoy receives the protobuf bytes, it does not convert them to YAML. It deserializes the bytes straight into the matching type (a C++ class) it already holds, and that becomes the live listener or cluster.

istiod turns a Go object into bytes; Envoy turns bytes back into a C++ object. Neither side ever touches a text form (YAML or JSON). The JSON you see in istioctl proxy-config is Envoy rendering the binary into something readable for debugging, not the engine running on JSON.

ACK/NACK rejects broken config

Delivery is not fire and forget. Envoy validates what it receives and replies ACK if it applied, NACK if it could not.

The nonce is an identifier stamped on each delivery, used to match a reply to the response it answers. On a NACK, Envoy drops the new config and keeps running the old one. That makes it a safety net: pushing broken config does not instantly kill traffic. A NACK in istiod's logs usually means the generated config is invalid for the Envoy version that sidecar runs.

Applying it: which layer does metadata go in

You now know what flows. The question that always comes up in real design is: when you want to inject some metadata (this Pod is v2, this path needs stronger auth, and so on), which of LDS/RDS/CDS/EDS is the right place? There is a clear rule for this, and each layer has things only it can express.

The two questions

Envoy processes one request from upstream to downstream in this order.

Where metadata goes is decided by asking, in order:

What does this metadata vary by? Per port means LDS, per service means CDS, per path or header means RDS, per Pod means EDS. The smallest grain at which it differs is the layer.
Who reads it, and when? Place it at the layer where the consuming filter or load balancer runs, or upstream of it, or it cannot be read.

What only each layer can express

"Only" is not a figure of speech here. It means structurally there is nowhere else to put it.

Layer	Where it goes (proto)	What only this layer can express
LDS	`Listener.metadata` / FilterChain	Whether a filter is installed at all. Later layers can only override what this installs. L4 decisions like SNI and original destination
RDS	`Route.metadata` / `typed_per_filter_config`	Decisions based on HTTP request attributes (path, header). Per-path filter overrides (rate limit, ext_authz) live only here
CDS	`Cluster.metadata` / `lb_subset_config`	The subset "key definition", load balancing policy, circuit breakers, upstream TLS
EDS	`LbEndpoint.metadata` / `LocalityLbEndpoints`	Per-target values (version=v2 and such), locality/zone, per-endpoint weight and health

Concrete "physically nowhere else" cases:

A value that differs per Pod (this Pod is v2, this one is in Zone-A) attaches only to an endpoint. EDS is the only home.
Behavior per HTTP path (only /admin gets stronger auth) can only see the request at the route, so it is RDS-only.
Whether a filter exists at all is decided at LDS (the filter chain). Trying to override a filter via RDS typed_per_filter_config that LDS never installed is "overriding a filter that is not there", and it does nothing.

How the wrong layer breaks

This is the most common trap in practice. Pick the wrong layer and you get "config applies but does nothing".

Put a per-endpoint value on the Cluster (CDS) and it becomes shared across all Pods, so the per-Pod distinction vanishes.
Put typed_per_filter_config on a route but never install that filter in the LDS filter chain, and it is silently ignored (sometimes without even a NACK).
Put a subset match on RDS but leave subset_selectors undefined on CDS, or leave that key off the EDS endpoint metadata, and nothing matches, so you get a 503.

Worked example: canary by version (three layers split the job)

"Split reviews into v1/v2/v3 by version label, and send a slice of traffic to v2." This one feature decomposes across three layers, each doing the part only it can.

CDS (Cluster.lb_subset_config):
    subset_selectors: [{ keys: ["version"] }]      # declare: build subsets by version

EDS (LbEndpoint.metadata):
    10.244.0.11 -> {"envoy.lb": {"version": "v1"}}  # tag each Pod with its label value
    10.244.0.20 -> {"envoy.lb": {"version": "v2"}}

RDS (Route.route.metadata_match):
    { "envoy.lb": {"version": "v2"} }              # select: route to the v2 subset

In Istio terms, each layer maps like this.

What you write in Istio	The xDS it generates	The job it does
`DestinationRule.subsets` (version key)	CDS `subset_selectors`	Declares the "split by version" structure
The Pod's `version: v2` label	EDS endpoint metadata	Tags each target with the value
The `VirtualService` route	RDS `metadata_match`	Selects the destination subset per request

The point: one concern, version, decomposes by grain into three layers. The value differs per Pod so it is EDS, "split by version" is a property of the whole Cluster so it is CDS, and the actual selection happens at request time so it is RDS. Routing one concern to the right layer by its grain is what designing with xDS is.

Quick reference: where does it go

What you want to inject	Layer	proto field
Per-Pod tags, zone, weight	EDS	`LbEndpoint.metadata` / `LocalityLbEndpoints`
Service-wide LB policy, subset definition, TLS	CDS	`Cluster.metadata` / `lb_subset_config`
Per-path or per-header filter behavior	RDS	`Route.typed_per_filter_config` / `metadata`
Port, filter setup, L4 decisions	LDS	`Listener.metadata` / FilterChain

Hands-on: pull a raw DiscoveryResponse

I have been saying "protobuf flows" in words. Now confirm it with your own eyes. Stand up Kubernetes on kind, install Istio, and pull the actual bytes istiod ships, then decode them with protoc.

You need docker, kind, istioctl, kubectl, grpcurl, and protoc. On a Mac, brew install kind grpcurl protobuf plus istioctl from the official instructions.

Step 1: stand up the cluster and Istio

# create the kind cluster
kind create cluster --name xds-demo

# install Istio, minimal profile
istioctl install --set profile=demo -y

# deploy a sample app (Bookinfo) with sidecar injection
kubectl label namespace default istio-injection=enabled
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

# wait for the Pods to come up
kubectl wait --for=condition=ready pod --all --timeout=180s

Step 2: the human-facing view with istioctl

Start with ordinary debugging. See what Listeners istiod generated for a given Envoy. Every output from here on is the real thing, captured by running these steps.

# grab the productpage Pod name
POD=$(kubectl get pod -l app=productpage -o jsonpath='{.items[0].metadata.name}')

# the Listener list for that Envoy
istioctl proxy-config listeners "$POD"

It comes out as a readable table (excerpt).

ADDRESSES      PORT   MATCH                                   DESTINATION
10.96.0.10     53     ALL                                     Cluster: outbound|53||kube-dns.kube-system.svc...
0.0.0.0        80     Trans: raw_buffer; App: http/1.1,h2c    Route: 80
10.96.0.1      443    ALL                                     Cluster: outbound|443||kubernetes.default.svc...
10.96.151.152  443    ALL                                     Cluster: outbound|443||istiod.istio-system.svc...
0.0.0.0        9080   Trans: raw_buffer; App: http/1.1,h2c    Route: 9080
0.0.0.0        9080   ALL                                     PassthroughCluster

Add -o yaml to see the protobuf dumped into a YAML view. Remember that this YAML is a readable rendering of the protobuf, not what went over the wire.

istioctl proxy-config listeners "$POD" -o yaml | head -50

Step 3: see the type_url in the open

Next, pull the config dump from Envoy's admin endpoint and list the @type values.

# port-forward Envoy's admin API
kubectl port-forward "$POD" 15000:15000 &

# list the @type values (the type_url) inside config_dump
curl -s localhost:15000/config_dump | jq -r '.configs[]."@type"' | sort -u

The real output:

type.googleapis.com/envoy.admin.v3.BootstrapConfigDump
type.googleapis.com/envoy.admin.v3.ClustersConfigDump
type.googleapis.com/envoy.admin.v3.ListenersConfigDump
type.googleapis.com/envoy.admin.v3.RoutesConfigDump
type.googleapis.com/envoy.admin.v3.ScopedRoutesConfigDump
type.googleapis.com/envoy.admin.v3.SecretsConfigDump

That type.googleapis.com/... is type_url. It is the protobuf Any declaring "what type is inside this", and you can see Listener, Cluster, Route, and Secret each carrying a different type_url.

Step 4: pull a raw DiscoveryResponse from istiod

This is the part the setup was for. istiod also speaks xDS on port 15010 (plaintext), so you can hit the ADS stream directly with grpcurl and pull a raw DiscoveryResponse. You pretend to be Envoy and ask for Listeners.

# borrow the node ID from the real Envoy bootstrap
NODEID=$(istioctl proxy-config bootstrap "$POD" -o json | jq -r '.bootstrap.node.id')

# port-forward istiod's xDS port
kubectl -n istio-system port-forward deploy/istiod 15010:15010 &

# ask ADS for Listeners and inspect the structure of the envelope that comes back
grpcurl -plaintext -max-time 8 -d @ localhost:15010 \
  envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources <<EOF \
  | jq '{typeUrl, nonce, count: (.resources|length), first_type: .resources[0]."@type", first_name: .resources[0].name}'
{"node":{"id":"$NODEID"},"typeUrl":"type.googleapis.com/envoy.config.listener.v3.Listener"}
EOF

The real thing that came back (grpcurl renders the protobuf as JSON):

NODEID = sidecar~10.244.0.11~productpage-v1-xxxxx.default~default.svc.cluster.local

{
  "typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener",
  "nonce": "2026-06-09T14:23:1...",
  "count": 18,
  "first_type": "type.googleapis.com/envoy.config.listener.v3.Listener",
  "first_name": "10.96.174.178_15021"
}

This is the envelope, the DiscoveryResponse, in the flesh. typeUrl declares "the stream is carrying Listener type right now", nonce is the identifier for matching ACK/NACK, and resources holds 18 entries, each wrapped in an Any whose @type points at Listener. The earlier "stuff Envoy-schema resources into the transport envelope" is right there in the open.

Step 5: see field numbers with protoc --decode_raw

Last, confirm by hand that protobuf "identifies fields by number". grpcurl formats to JSON, so here I build a tiny proto that mirrors only the first fields of listener.proto, encode it, and feed the raw bytes to protoc --decode_raw.

# a minimal proto with the same field numbers as listener.proto
cat > mini.proto <<'PROTO'
syntax = "proto3";
message SocketAddress { string address = 1; uint32 port_value = 2; }
message Address { SocketAddress socket_address = 1; }
message Listener { string name = 1; Address address = 2; }
PROTO

# fill values and encode to binary
cat > listener.txtpb <<'TXT'
name: "0.0.0.0_9080"
address { socket_address { address: "0.0.0.0" port_value: 9080 } }
TXT
protoc --encode=Listener mini.proto < listener.txtpb > listener.bin

# decode the raw bytes with no type info
protoc --decode_raw < listener.bin

The real output. No field names, just the field numbers as keys.

1: "0.0.0.0_9080"
2 {
  1 {
    1: "0.0.0.0"
    2: 9080
  }
}

1: is the Listener name, 2 { is address, the 1 { inside is socket_address, and 2: 9080 is the port. Recall the proto: string name = 1;, core.v3.Address address = 2;, and port_value = 2 on SocketAddress. Those numbers show up in the decode exactly. That is the hard proof that what flows is not YAML but protobuf packed by field number.

Step 6: tear down

kind delete cluster --name xds-demo

A script that reproduces this hands-on end to end lives at articles/assets/xds-protobuf-deep-dive/run.sh.

Wrap-up

Long road, so a retrace from the top.

A service mesh has a control plane (istiod) and a data plane (Envoy). Envoy does not know what to do on its own and gets config from the control plane.
Shipping config over an API instead of a file is xDS. In Istio, Pilot inside istiod serves it, and each Envoy sidecar receives it.
The contents come in LDS/RDS/CDS/EDS/SDS, bundled onto one stream by ADS.
What flows is protobuf, not YAML. YAML lives only at the human-writing edge and the debug view.
protobuf means defining the type in .proto first, then generating language classes with protoc. The .proto is type only; the data is filled in at runtime by istiod.
xDS is three layers. The delivery (transport), the vendor-neutral base types (cncf/xds), and the Envoy-specific resource schema are distinct, and type_url ties the envelope to the contents.
Where metadata goes is set by the grain it varies by and who reads it. Per Pod is EDS, per service is CDS, per path is RDS, per port is LDS. Routing one concern to the right layer by grain is the essence of designing with xDS.
Finally, decoding the raw bytes on kind shows the field numbers from listener.proto appearing verbatim, confirming with your own eyes that protobuf is what flows.

"The control plane ships Envoy-schema protobuf messages over xDS." If that sentence lands, the post did its job. Next time you reach for istioctl proxy-config during an Istio incident, you can see what is happening behind it.

References

xDS Deep Dive, Part 2: Designing What You Ship Over the Universal Data Plane API

kt — Sun, 07 Jun 2026 08:25:40 +0000

Introduction

In xDS Deep Dive: Dissecting the "Nervous System" of the Service Mesh I dug into the dependency chain of LDS / RDS / CDS / EDS / SDS, the robustness of ACK/NACK, why ADS exists, and the evolution from SotW to Delta. In short, it was about how Envoy consumes xDS.

But there was one line I tossed off at the very end:

xDS isn't Envoy-only anymore. The CNCF xDS API Working Group is standardizing it as the "Universal Data Plane API".

I waved at that and never came back to it. This time I take it seriously, and I change my stance from Part 1. Part 1 read xDS from the consuming side: how Envoy eats it. This one is about the producing side: how you design what gets shipped over xDS, and how.

Why frame it that way? Because every decision you face when designing xDS resources (what to name them, how to split them, how to express variants, which CP owns them) is already spelled out in the github.com/cncf/xds protos and the xRFC documents. xdstp://, Authority, Dynamic Parameters: read straight, they look like feature descriptions. Read through a designer's eyes, they're levers, each one saying "here is where you choose, and how". I'll re-read them one by one as design choices.

And I won't stop at theory. At the end I use go-control-plane to build a Listener → Route → Cluster graph by hand and serve it, then confirm in the output that what I designed actually lands on the wire. I cloned the protos locally too, so I'll keep the definitions open beside me.

git clone https://github.com/cncf/xds ~/xds
ls ~/xds/xds/core/v3/
# authority.proto       collection_entry.proto  context_params.proto
# extension.proto       resource.proto          resource_locator.proto
# resource_name.proto   cidr.proto              ...

The protos sitting in there are the proxy-neutral core: types that don't depend on Envoy at all.

Vocabulary to keep handy

I'm writing this assuming you've read Part 1, but here's a quick glossary of terms that come up a lot. Come back here when you get stuck.

Term	Role
`LDS / RDS / CDS / EDS / SDS`	The five core xDS services. Linked by reference: Listener → Route → Cluster → Endpoint
`ADS`	Multiplexes the above onto a single gRPC stream. Required when ordering matters
`Node`	The identity + metadata a client sends to the server at the start of the stream
`SotW` / `Delta`	Two modes: send every resource each time, or send only the diff
`ACK / NACK`	How a client reports whether it could apply the config, via `version_info` and `nonce`
`LRS`	A xDS-family service where the client reports endpoint load back to the CP

0. The premise: xDS is nothing but "protobuf flowing over a gRPC stream"

In Part 1 I explained LDS / RDS / CDS / EDS, but it hit me that I never once showed the shape of the xDS interface itself. If that hasn't clicked, everything that follows (xdstp://, ORCA, all of it) floats in the air. So let me drop down a level and look at the xDS transport plainly.

xDS is a single gRPC service

There's no standalone "xDS protocol". xDS is one gRPC service definition. The ADS from Part 1 is, concretely, this proto in envoy's service/discovery/v3:

// envoy/service/discovery/v3/ads.proto
service AggregatedDiscoveryService {
  rpc StreamAggregatedResources(stream DiscoveryRequest)
      returns (stream DiscoveryResponse);

  rpc DeltaAggregatedResources(stream DeltaDiscoveryRequest)
      returns (stream DeltaDiscoveryResponse);
}

A bidirectional streaming RPC, stream ... returns (stream ...), and that's it. The client (the proxy) holds a single stream open to the CP, pushes DiscoveryRequest upstream and receives DiscoveryResponse downstream, forever. LDS, CDS, EDS aren't "different protocols": they're just messages with different type_url flowing over the same stream. Even SotW / Delta from Part 1 are nothing more than two dialects on this one stream.

The crux is that there's only one stream. You don't open a connection per type; a single bidirectional stream, multiplexed by type_url, carries Clusters, Listeners, Endpoints, and both ACKs and NACKs all mixed together. Exactly which fields express that ACK / NACK becomes clear the moment you open up the messages.

Why is it all protobuf

"Why is the config protobuf instead of YAML or JSON" answers itself once you're here. Because xDS is gRPC. gRPC's IDL is protobuf, and you can only define a service's arguments and return values as protobuf messages. DiscoveryRequest / DiscoveryResponse being protobuf isn't even a choice; it's a consequence of deciding to use gRPC. The resource bodies (Listener, Cluster) ride inside these messages, so they have to be protobuf too.

Look inside the request / response and you'll see the entire behavior of xDS collapses into the fields of these two messages.

// upstream: client -> CP
message DiscoveryRequest {
  string version_info = 1;            // the version I'm currently ACKing
  Node node = 2;                      // my identity, sent at the start of the stream
  repeated string resource_names = 3; // "give me these"
  string type_url = 4;                // which kind (Listener? Cluster?)
  string response_nonce = 5;          // which response this replies to
  google.rpc.Status error_detail = 6; // the reason, when this is a NACK
}

// downstream: CP -> client
message DiscoveryResponse {
  string version_info = 1;
  repeated google.protobuf.Any resources = 2; // <- resource bodies are Any
  string type_url = 4;
  string nonce = 5;
}

In Part 1 I wrote "ACK/NACK is returned via version_info and nonce". Well, those version_info / nonce / error_detail are literally fields on this message. An ACK is a DiscoveryRequest with error_detail empty; a NACK is a DiscoveryRequest with error_detail set. That's all.

The core: resource bodies are wrapped in `google.protobuf.Any`

The field that matters most is the type of resources: repeated google.protobuf.Any. Any is a pair of "a type_url string + serialized bytes", a box that can wrap any protobuf message regardless of type. So xDS isn't "the protocol that carries Listeners" or "the protocol that carries Clusters". It's a type-neutral config bus that names the type via type_url and wraps the body in Any.

This "type-neutral box" property is exactly where Universal Data Plane starts. If the box doesn't care about the type, then standardizing just the "name" and "type" you ship opens the door to a world that isn't Envoy-specific. What cncf/xds is trying to do is precisely this: universalize resource_names and type_url.

What is an "xDS client", really

Now the thing the phrase "xDS client" points to is clear. It's a state machine living inside the proxy (Envoy itself, or the gRPC library) that manages this bidirectional stream. What it does:

Holds a single StreamAggregatedResources stream to the CP (the one that holds is the client; the one that waits, the CP, is the server)
Names itself by sending Node at the start, and subscribes to the resource_names it wants via DiscoveryRequest
Routes the incoming DiscoveryResponse Any by type_url, unpacks it, and bakes it into internal config
If it applied, ACK via version_info + nonce; if it broke, NACK with error_detail
Keeps a local cache of "which resource at which version I currently hold"

In §3 I actually poke at grpc-go's internal/xds/bootstrap, and that turns out to be the config that decides which CP this xDS client opens a stream to at startup. And every feature of cncf/xds this article reads (xdstp://, Authority, Dynamic Parameters, ORCA) is, almost entirely, about expanding the vocabulary of "what this client subscribes to and how the CP answers". In fact, the TP1 ResourceLocator and the TP3 ResourceError that show up later already exist as fields on the latest DiscoveryRequest / DiscoveryResponse (resource_locators / resource_errors). That's proof the standard is already descending onto the wire.

A map for reading this as a designer

For a designer, what you ship is ultimately just the proto you stuff into that Any and the name you attach to it. So design boils down to deciding "which proto, named how, at what granularity, served from which CP". The chapters that follow knock out those decision points one at a time.

Chapter	Design decision	Lever
§2	What name do you give a resource	`xdstp://` URI / id / context_params
§3	How many CPs, and where you draw the boundary	Authority / federation
§4	Whether to bake references and failover into names	Resource Locator (`alt=` / `entry=`)
§5	Subscribe one at a time, or ship in bundles	singleton / collection / glob
§6	Bake variants into the name, or keep them out	Context Parameters / Dynamic Parameters
§7	What granularity to pick for errors and resources	TP3 Resource Error / NACK blast radius
§8	Whether to wire telemetry into the design	ORCA / LRS
§9	Where to hold matching, declaratively	Unified Matcher / CEL
§11	Actually build and ship all of the above	go-control-plane snapshot

Each chapter ends with a "design call": the options, the criterion for choosing, and how it breaks when you get it wrong, condensed onto one card. The goal is for this to work not just as reading but as a checklist when you design your own mesh.

1. Why does `cncf/xds` exist in the first place

Envoy has long had a mountain of protos at envoyproxy/envoy/api: that place where envoy.config.listener.v3.Listener, envoy.config.cluster.v3.Cluster, and friends live. So a natural question is: don't we already have those, why a separate repo?

The answer is one line in the README:

We will evolve the xDS APIs to support additional clients, for example
data plane proxies beyond Envoy, proxyless service mesh libraries,
hardware load balancers, mobile clients and beyond.

As long as xDS lives inside the Envoy repo, the envoy.config.* namespace tags along forever. When gRPC Proxyless speaks xDS, when Cilium ztunnel speaks xDS, when a load balancer speaks xDS, everyone ends up importing envoy.config.*.

Structurally that makes it "everyone else eats Envoy's leftovers", and the standardization story doesn't hold up. So the WG has been incrementally carving the Envoy-independent parts out into cncf/xds.

While we're here: there's also an old repo cncf/udpa, but it's retired. udpa/README.md says it bluntly:

THESE PROTOS ARE DEPRECATED
We are no longer using the "UDPA" name, and we are moving away from the
protos in this tree. Users should prefer the corresponding protos in
the xds tree instead.

So when we talk xDS from here on, you can ignore udpa. Just look at cncf/xds.

That's the preamble. The real subject is reading what's inside cncf/xds. A quick roll call of what lives there:

Component	What it is	xRFC
`xdstp://` URI / Authority	A universal namespace stamped on every resource	TP1
Context Parameters	Embed a resource "variant" into the URI	TP1
Resource Locator + directive	`alt=` (failover), `entry=` (inline reference)	TP1
Glob Collections	Wildcard subscription of the form `xdstp://.../foo/*`	TP1
Dynamic Parameters	Express variants without putting them in the name (no name pollution)	TP2
Resource Error	Return per-resource errors without tearing down the stream	TP3
ORCA	A separate-family service carrying load metrics backend -> client	(xds.service)
Unified Matcher API	A shared matching tree across all extensions	(xds.type)
CelExpression	A type letting matchers and extensions call CEL (Common Expression Language)	(xds.type)

I'll knock these out top to bottom.

2. Why resource names had to become URIs

Here's the real start. Unless I first explain why xdstp:// was born, the reason for the following chapters (Authority, Context Parameters, Resource Locator) won't land at all.

What a legacy xDS resource name actually was

I touched on this in Part 1, but an xDS resource used to be identified by three things:

Resource name: an arbitrary opaque string. e.g. foo
type URL: the resource's proto type. e.g. envoy.config.endpoint.v3.ClusterLoadAssignment
the Node message: identity info about the node (locality, metadata, etc.), sent once at the start of the stream

The problem: control-plane implementations started looking at Node metadata and returning different bodies for the same name foo. That spawns a nasty side effect.

What breaks: caching

At medium scale and up, you want to drop a cache layer in the middle of xDS. Classic examples are xds-relay (a relay / caching server for xDS published by the Envoy project) or a setup like Envoy Mobile, which embeds Envoy in the mobile app's own process (one client per device, O(millions) scale). Once your client count crosses a threshold, the CP can't keep up without a relay.

But the moment Node gets tangled into the cache key, it's over.

A single key foo doesn't say whose foo it is. Mix Node into the cache key and you get a separate entry per Envoy, which defeats the cache entirely. That's the starting point for TP1.

The answer: cram the needed context into the name itself

The xdstp:// URI format is this, written verbatim in the comment of xds/core/v3/resource_name.proto:

xdstp://{authority}/{type_url}/{id}?{context_params}

The point is that this one URI uniquely determines the resource. Without consulting Node metadata, you can pin down foo as "whose, for what, in which state". The relay can do a cache lookup on the URI alone without peeking at Node, so for the first time caching means something.

In proto, it looks like this:

// xds/core/v3/resource_name.proto
message ResourceName {
  string id = 1;                    // "api-fe"
  string authority = 2;             // "traffic-director.gcp.io"
  string resource_type = 3;         // "envoy.config.listener.v3.Listener"
  ContextParams context = 4;        // {env: prod, region: us}
}

// xds/core/v3/context_params.proto
message ContextParams {
  map<string, string> params = 1;
}

context takes arbitrary key-values. By convention the xds.resource.* prefix is reserved; for example xds.resource.listening_address (e.g. "10.1.1.3:8080") is defined for Listeners.

By here, the new worldview lands: an xDS resource name is not a plain string, it's a URI.

Design call: what goes where in the name

Because the URI has four slots (authority / type / id / context_params), as a designer you decide every time which piece of info goes in which slot. The guidance:

id: only the stable identity of the resource. A logical name like api-fe. Do not put variable axes like env or version here. Change the id and it's a different resource.
context_params: the axes where the body changes per client or environment (env, region). This becomes the cache key and triggers the viral effect below, so don't overload it (§6).
authority: which CP owns the resource (§3).
How it breaks: fold an environment into the id like api-fe-prod and names proliferate per environment, and the upstream references (RDS → CDS) split per environment too. Keep "id immutable, variation in context_params" and you confine that proliferation to one place.

3. Authority and federation

The first thing in the URI path is authority, and that's meaningful.

One client speaking to multiple control planes

Legacy xDS implicitly assumed one control plane per client. ConfigSource was basically a single source. But reality isn't a single source:

You want a CPaaS (Control Plane as a Service) as primary, with your own on-prem CP as failover
Multi-cloud, with separate CPs on the AWS and GCP sides while Envoy runs across clusters
Pull only specific resource types from a different CP managed by a different team

By building Authority into the URI, TP1 lets you put a mapping of "authority name -> physical CP address" into the client's bootstrap. For gRPC this exists for real as the authorities map in the bootstrap JSON. Here's the actual format grpc-go can parse:

{
  "xds_servers": [ ... ],
  "client_default_listener_resource_name_template":
    "xdstp://traffic-director.gcp.io/envoy.config.listener.v3.Listener/%s",
  "authorities": {
    "traffic-director.gcp.io": {
      "xds_servers": [{
        "server_uri": "trafficdirector.googleapis.com:443",
        "channel_creds": [{"type": "google_default"}],
        "server_features": ["xds_v3"]
      }]
    },
    "onprem-cp.internal": {
      "xds_servers": [{
        "server_uri": "istiod.mesh.svc:15010",
        "channel_creds": [{"type": "insecure"}],
        "server_features": ["xds_v3"]
      }]
    }
  }
}

Now writing xdstp://traffic-director.gcp.io/... in a resource URI sends the query to the former, and xdstp://onprem-cp.internal/... to the latter. One Envoy / gRPC client can speak to multiple CPs at once, keyed on a "logical authority".

That's the seed of xDS federation. In the gRPC world the client-side bootstrap spec is nailed down in the A47-xds-federation proposal. Look inside grpc-go's internal/xds/bootstrap and the Authorities field is already implemented, structured to hold a list of ServerConfig per authority.

Run it to check: feed it to the bootstrap parser

"It's implemented" sounds like a cop-out in words, so I fed the bootstrap above to grpc-go's real parser (internal/xds/bootstrap). It does three things only: (1) parse the bootstrap and look up authority -> CP address, (2) watch the legacy logical target xds:///api-fe expand into xdstp:// via client_default_listener_resource_name_template, (3) parse the resulting URI back apart with ParseName().

cfg, _ := bootstrap.NewConfigFromContents([]byte(bs)) // bs = the JSON above
for name, a := range cfg.Authorities() {
    fmt.Printf("%-24s -> %s\n", name, a.XDSServers[0].ServerURI())
}
// xds:///api-fe expands into an xdstp:// name via the template
fmt.Println(bootstrap.PopulateResourceTemplate(
    cfg.ClientDefaultListenerResourceNameTemplate(), "api-fe"))
// parse the resulting URI back apart
n := xdsresource.ParseName(
    "xdstp://onprem-cp.internal/envoy.config.listener.v3.Listener/api-fe?env=prod")
fmt.Printf("authority=%q type=%q id=%q ctx=%v\n", n.Authority, n.Type, n.ID, n.ContextParams)

Clone grpc-go, run this (confirmed on grpc-go 0f3086d / go1.26.4), and the actual output is:

== parsed authorities ==
  traffic-director.gcp.io  -> trafficdirector.googleapis.com:443
  onprem-cp.internal       -> istiod.mesh.svc:15010

== xds:///api-fe expands via client_default template ==
  xds:///api-fe  =>  xdstp://traffic-director.gcp.io/envoy.config.listener.v3.Listener/api-fe

== ParseName() splits an xdstp URI back into parts ==
  scheme="xdstp" authority="onprem-cp.internal"
  type="envoy.config.listener.v3.Listener" id="api-fe" ctx=map[env:prod]

The payoff is from line 3 on. A perfectly ordinary target xds:///api-fe turns, internally, into an authority-qualified URI xdstp://traffic-director.gcp.io/.../api-fe. And ParseName() cleanly splits that URI into authority / type / id / context_params (note ?env=prod getting picked up as ctx=map[env:prod]). In §2 I wrote "a resource name isn't a string, it's a URI", and this is that being literally true at the code level.

Design call: how many CPs, and where to cut the authority boundary

Authority isn't a physical CP address; it's the logical boundary of "who owns this set of resources". So you draw the line along the org chart and trust boundaries.

When to split: (1) different managing team, (2) different trust boundary (your own CP vs a vendor CP), (3) you want fault isolation (escape to your own CP when CPaaS is down), (4) different lifecycle (config that changes constantly vs config that almost never does)
When not to split: merely a different physical DC or region. That should be a context_param (region=us); splitting the authority for it is overkill
How it breaks: split too much and the bootstrap bloats while cross-authority resource references multiply, complicating operations. Split too little and a single CP becomes the SPOF / scaling ceiling. "Split only where ownership splits" is the sweet spot

4. Resource Locator: failover and inline references

By looks alone xdstp:// resembles a URL so closely it's easy to miss, but the fragment (after #) hides an extension with real teeth: directives.

xdstp://{authority}/{type_url}/{id}?{context_params}{#directive,*}

In proto that's ResourceLocator:

// xds/core/v3/resource_locator.proto
message ResourceLocator {
  Scheme scheme = 1;          // XDSTP / HTTP / FILE
  string id = 2;
  string authority = 3;
  string resource_type = 4;
  oneof context_param_specifier {
    ContextParams exact_context = 5;
  }
  repeated Directive directives = 6;

  message Directive {
    oneof directive {
      ResourceLocator alt = 1;   // failover target
      string entry = 2;          // inline reference
    }
  }
}

Let me kill one confusing naming collision here. This ResourceLocator / ResourceName is from xds.core.v3 (cncf/xds): the core type that defines "the grammar of a name". Separately, the ResourceLocator carrying dynamic_parameters and the ResourceName carrying dynamic_parameter_constraints that show up in §0 and §6 are the same-named messages on the envoy.service.discovery.v3 side: a different thing, for transport (subscribe / deliver). Same names, different layers (one is "the type of the name itself", the other "the type of the discovery message that carries that name"). This chapter is reading the former, the core type.

`alt=`: failover

The alt directive is an instruction meaning "if you can't fetch this resource, try the alternate URI".

xdstp://gcp-cp/envoy.config.endpoint.v3.ClusterLoadAssignment/foo#alt=xdstp://onprem-cp/envoy.config.endpoint.v3.ClusterLoadAssignment/bar

"If the GCP-side CP is unreachable, fall back to the on-prem CP" becomes something you can write in a single string. The client only needs both authorities registered in its bootstrap.

`entry=`: inline reference

When a List collection (below) inline-expands several resources, this is the fragment to reference one specific entry inside the collection by name.

xdstp://some-authority/envoy.config.listeners.v3.ListenerCollection/foo#entry=bar

This means "the item named bar inside collection foo". After you pull the whole collection, you can reuse its insides as URIs again, a recursive usage that's allowed.

Design call: whether to bake failover and references into the name

A directive is a tool for "declaratively baking behavior into the name". Handy, but bake too much in and it stiffens up.

Use alt=: for failover where the alternate is statically determined (primary CP -> backup CP). No client-side logic; the intent fits in one URI
Use entry=: when you want to reference and reuse an entry inside a collection by name after fetching it
How it breaks: if "where to fail over to" changes dynamically and you bake it into the name with alt=, the resource name changes on every switch and stiffens. Keep dynamic decisions in the client / LB, and bake only static things into the name

5. Collections and glob: formalizing bulk subscription

This one quietly pays off. As I wrote in Part 1, in old xDS only LDS / CDS were special: empty resource_names meant the implicit rule "wildcard = give me everything". RDS / EDS / SDS, meanwhile, were explicit-subscription. That mix of special-casing and implicit rules had become technical debt.

TP1 cleaned this up by making collections (sets) a first-class concept.

List collection

Separate from a singleton URI that references one resource, a collection URI exists.

# singleton (one resource)
xdstp://auth/envoy.config.listeners.v3.Listener/api-fe

# list collection (a set of resources)
xdstp://auth/envoy.config.listeners.v3.ListenerCollection/my-listeners

The server can answer in two ways:

Return a list of Locators: "the bodies are at other URIs, come fetch them yourself"
Embed bodies via InlineEntry: "I want to save round-trips, so here are the bodies too"

The proto is CollectionEntry:

// xds/core/v3/collection_entry.proto
message CollectionEntry {
  oneof resource_specifier {
    ResourceLocator locator = 1;     // 1. points to another URI
    InlineEntry inline_entry = 2;    // 2. hands you the body right here
  }
}

Glob collection: formalizing the wildcard

When you want to subscribe to "everything matching a given prefix", use a glob.

xdstp://auth/envoy.config.listeners.v3.Listener/my-listeners/*?node_type=ingress

The trailing /* is the glob. You can send this as a subscription, and the server returns every matching resource. A context parameter like node_type=ingress narrows it further.

This kills the LDS / CDS "empty string = wildcard" black magic. Everything closes over structured URIs.

Design call: subscribe one at a time, or ship in bundles

How you present resources to the client is your call.

singleton (explicit subscribe): when the client knows the names it needs in advance. Simplest when the count is bounded and relationships are static
list collection: when the server wants to manage "this set". Send bodies via inline_entry to save round-trips, or return only locator for lazy fetch. Use this when set members change often
glob (/*): when the client doesn't know the individual names / they're dynamically added and removed (e.g. all ingresses). Narrow with ?node_type=ingress
How it breaks: a too-broad glob ships unneeded resources and eats bandwidth and memory. Conversely, making everything a singleton makes the subscription list a chore, requiring client-side subscription updates every time you add a resource

6. Context Parameters and Dynamic Parameters: how to express "variants"

Things get a bit more advanced here. How do you answer the demand: "same resource named foo, but I want a different body per client"?

The Context Parameters (TP1) way

As we saw, you put it in the URI's query string.

xdstp://auth/RouteConfiguration/foo?env=prod&version=v1
xdstp://auth/RouteConfiguration/foo?env=canary&version=v1
xdstp://auth/RouteConfiguration/foo?env=prod&version=v2

These are treated as completely separate resources. Different cache key, different subscription.

But this approach has a fatal weakness, which the TP2 document articulates well: the phenomenon of virality.

The xDS reference graph runs top-down, LDS → RDS → CDS → EDS. Variants spread in the opposite direction. If EDS has two variants env=prod and env=canary, the CDS that references it splits into two, each pointing at one variant's URI. Once CDS splits into two, the RDS pointing at it needs two as well. And the LDS above that, two. In other words, "an EDS variant climbs up the dependency graph and swaps out the upstream resources wholesale". The diagram below is for a single env axis (green -> orange -> blue -> purple is the upstream direction).

On top of that, context parameters are exact-match only, so adding axes causes a combinatorial blowup. Let me count it concretely. Just wanting env={prod,canary,test} x version={v1,v2,v3}, two axes of three values each, gives 3 x 3 = 9 variants. And because it's viral, those 9 spread across every layer of the dependency graph.

Layer	Count the CP holds under TP1 (context params)	Breakdown
EDS	9	`env x version`
CDS (-> EDS)	9	one per EDS variant
RDS (-> CDS)	9	spread upstream
LDS (-> RDS)	9	spread further up
Total	36	logically one service

Logically it's "one service with an env and version axis", yet as xDS resources it's 36. Wanting to add one EDS option spreads the same variant across every layer, CDS / RDS / LDS. That's the weakness of TP1.

The Dynamic Parameters (TP2) way

TP2 flips this. It evicts "the info used to select a variant" from the resource name and into a separate field that only the transport layer sees.

// envoy/service/discovery/v3/discovery.proto (real; also generated in go-control-plane)
message DynamicParameterConstraints {
  message SingleConstraint {
    string key = 1;
    oneof constraint_type {
      string value = 2;        // exact value match
      Exists exists = 3;       // key-existence check
    }
  }

  oneof type {
    SingleConstraint constraint = 1;
    ConstraintList or_constraints = 2;
    ConstraintList and_constraints = 3;
    DynamicParameterConstraints not_constraints = 4;
  }
}

On the wire the flow splits in two directions. The client puts dynamic_parameters (map<string, string>) on the subscribing ResourceLocator to say "these are the parameters I hold". The server puts dynamic_parameter_constraints on the returned ResourceName to say "this resource is for clients satisfying this constraint". The important part is that this constraint is not part of the resource's id (the URI string). The EDS a CDS points at stays a single name, yet the EDS side can serve out the env=prod variant and the env=canary variant separately.

So what happens to that "36"? Under Dynamic Parameters the resource names stay the four LDS / RDS / CDS / EDS, and the env / version axes exist only as constraints attached to EDS. Variants don't pollute the namespace, so the upstream viral spread stops. The constraint's expressive power is value exact-match, Exists (key existence), and AND / OR / NOT combinations, so you can express exactly the variants you actually need via constraints. And a caching xDS proxy only needs to look at the constraint to remember multiple variants, without touching the data-model graph.

That said, while the proto type has landed in envoy, TP2's own Implementation section is still TBD (Will probably be implemented in gRPC before Envoy). The type exists, but the behavior that actually serves out variants is, like TP3, expected to come to gRPC first, and is still ahead of us.

Design call: bake variants into the name, or keep them out (the chapter's crux)

This is the single biggest design fork in this article. Which one you pick changes how things break when variants grow.

Pick Context Parameters: when variant axes are few and the upstream viral spread is tolerable. A simple, exact-match-only mechanism that works today on both Envoy and gRPC. Clean cache keys too. For a small-to-medium single env axis, this is plenty
Plan for Dynamic Parameters: when variant axes are many and axes x values x layers combinatorially blows up. It doesn't pollute the name, so virality stops. But it needs constraint-evaluation logic, and the behavior is currently unimplemented (gRPC expected first). You can't deploy it today, but you can choose to not pollute your name design now, on the premise of moving there later
A numeric criterion: as counted in §6, two axes of three values each (env x version) balloons early to 36 resources. The moment "the variant count looks like it'll hit double digits" is the danger signal for a context_params-only approach. At that point, defend "don't bake variants into the id" and deliberately keep the variant count down until dynamic parameters catch up in implementation
How it breaks: carelessly grow context_params and the entire reference graph splits per variant and cache efficiency collapses. Conversely, deploy dynamic parameters today "because it's new" and there's no serving side to implement it, so it simply doesn't work

7. TP3: returning per-resource errors without tearing down the stream

Unglamorous, but it bites once you operate this stuff.

In legacy xDS there was effectively one way to say "can't fetch this resource": tear down the whole stream with a non-OK status. That's way too blunt. One NOT_FOUND out of 100 resources stops the other 99 updates too.

Worse, the SotW protocol had no way to even express "the resource doesn't exist"; the client had to wait for a 15-second does-not-exist timer to fire. This is a spec hole (also documented in Envoy's official docs).

TP3 solves it by adding a resource_errors field to DiscoveryResponse.

message ResourceError {
  ResourceName resource_name = 1;
  google.rpc.Status error_detail = 2;
}

message DiscoveryResponse {
  // ...existing fields elided...
  repeated ResourceError resource_errors = N;
}

The client branches on the status code:

status code	client behavior
`UNAVAILABLE` / `INTERNAL` / `UNKNOWN`	treat as transient. Keep using the last good config
`NOT_FOUND` / `PERMISSION_DENIED`	treat as a data error. Free to drop the cache
anything else	undefined. SHOULD treat the same as transient

The benefit comes down to two things:

No tearing down the stream. With subscribe [foo, bar, baz], if only baz goes NOT_FOUND, the foo and bar updates keep running on the same stream
SotW can express does-not-exist instantly. No waiting on the 15-second timer

The TP3 text says This will probably be implemented in gRPC before Envoy., so gRPC lands first and Envoy follows.

Design call: what granularity to pick for resources (the NACK blast radius)

TP3 is about "error granularity", but what a designer actually controls is the "resource granularity" upstream of it. The more you cram into one resource, the wider the collateral damage when it NACKs.

Make it fat (all vhosts in one big Route): easy to manage, but one invalid spot NACKs the whole resource, halting updates for unrelated paths too. Before TP3, this was even a full stream teardown
Split it fine (per-vhost / per-service): small NACK blast radius. Carve the fragile, high-churn parts into separate resources and a NACK there leaves the rest running
TP3 presence changes your tolerance: if the delivery target is gRPC (TP3 first), per-resource errors shrink the collateral, so you can stomach a bit of fatness. If it's Envoy-heavy without TP3 yet, design granularity finer to physically shrink the collateral of an error
How it breaks: a giant resource invites "one typo fully outages it". Over-splitting invites a blowup in subscription count and reference-graph complexity. Split fine only where "change frequency x blast impact" is high

8. ORCA: the moment xDS outgrew "shipping config"

The lineage changes here. So far it's been "how to ship resources". ORCA is a protocol for carrying load info from the backend to the client / LB, and it lives in xds/service/orca/v3/orca.proto.

Why it's needed

For an LB to route smartly, it wants to know whether a backend is "heavy or light right now": CPU usage, memory, a custom business cost metric, whatever. If each backend can hand this to the client / LB periodically or per-response, you get smarter load balancing.

Two modes

ORCA has two delivery styles, laid out in grpc.io's Custom Backend Metrics guide.

Mode	When it sends	Best for
Per-query (in-band)	rides the trailer at RPC end	short unary RPCs
OOB (Out-of-Band)	pushed periodically on a separate stream	streaming RPCs, works at zero QPS

The OOB service definition is just this:

// xds/service/orca/v3/orca.proto
service OpenRcaService {
  rpc StreamCoreMetrics(OrcaLoadReportRequest)
      returns (stream xds.data.orca.v3.OrcaLoadReport);
}

message OrcaLoadReportRequest {
  google.protobuf.Duration report_interval = 1;
  repeated string request_cost_names = 2;
}

When the client asks "give me metrics every 2 seconds", the backend server-streams load reports forever.

The body of xds.data.orca.v3.OrcaLoadReport carries cpu / memory / utilization plus an app-specific request_cost map you can add freely. Look at the proto directly and these are all the fields:

// xds/data/orca/v3/orca_load_report.proto
message OrcaLoadReport {
  double cpu_utilization = 1;
  double mem_utilization = 2;
  map<string, double> request_cost = 4;     // per-RPC cost
  map<string, double> utilization = 5;      // arbitrary 0..1 metrics
  double rps_fractional = 6;
  double eps = 7;                            // errors per second
  map<string, double> named_metrics = 8;    // app-defined metrics
  double application_utilization = 9;
}

You can put any app-specific unit on named_metrics, like "this RPC occupied a GPU for 1.2 sec".

Run it to check: hit the OOB stream directly

grpc-go's examples/features/orca is itself an OOB server. The server is a demo that just toggles CPU usage between 0.5 and 0.9 every two seconds. Against it I wrote a ~30-line client that hits StreamCoreMetrics directly with report_interval=2s.

cli := orcav3.NewOpenRcaServiceClient(cc)
stream, _ := cli.StreamCoreMetrics(ctx, &orcav3.OrcaLoadReportRequest{
    ReportInterval: durationpb.New(2 * time.Second), // "give it to me every 2s"
})
for {
    rep, err := stream.Recv()
    if err != nil {
        return
    }
    log.Printf("OrcaLoadReport: cpu=%.2f mem=%.2f", rep.GetCpuUtilization(), rep.GetMemUtilization())
}

Start the server (go run server/main.go), point this client at it, and it really keeps flowing (grpc-go 0f3086d / go1.26.4):

19:13:18 OrcaLoadReport: cpu=0.90 mem=0.00
19:13:21 OrcaLoadReport: cpu=0.50 mem=0.00
19:13:24 OrcaLoadReport: cpu=0.50 mem=0.00
19:13:27 OrcaLoadReport: cpu=0.90 mem=0.00

Call Recv() once and the backend sends an OrcaLoadReport at roughly 2-second intervals until you hang up. You can see the server-side CPU toggle (0.9 / 0.5) showing up at the client as-is. Opposite to the pull model of xDS fetching config, here the backend pushes load at you. That's why I said up front this has a different flavor from "shipping config".

Combining with LRS

The gRPC A64-lrs-custom-metrics proposal formalizes sending client-aggregated ORCA metrics back to the control plane via LRS (Load Reporting Service). LRS is a bidirectional reporting service in the xDS family, originally a mechanism for the client to report "how many RPS I sent to which endpoint" back to the CP. A64 put custom metrics onto that payload.

So in the direction Backend → Client → Control Plane, metrics flow end-to-end within the xDS context. The CP gains a global view of "which backend is heavy overall" and can reflect it into the weights of the EDS it ships next. The point is it doesn't close inside Envoy's world alone: the picture is identical for gRPC Proxyless. xDS, which was supposed to be a config-shipping protocol, is reaching out to swallow the telemetry uplink too.

Design call: whether to wire load info into the design

If you want to decide EDS weights (the weighted cluster in §11) "smartly", you need to include in your design how the load info that feeds it is gathered.

per-query (in-band): when short unary RPCs dominate. Rides the trailer, no extra stream needed
OOB (StreamCoreMetrics): steady-state monitoring. Works at zero QPS, so you can observe even idle long-tail backends
aggregate up to the CP via LRS: when you want to decide weights globally on the CP side. A64 lets custom metrics ride too
How it breaks: gather no telemetry at all and the CP can only ship static weights, causing the "spray evenly onto a genuinely heavy backend" accident. If you're talking about load balancing, the ORCA/LRS path is part of the design

9. Unified Matcher API: a shared matching tree across all extensions

Now the data-model side.

Historically Envoy's filters each had their own matching machinery: HTTP header match, RBAC principal match, access-log filter, access tags, external authorization. Each had its own proto and its own match logic. xds.type.matcher.v3.Matcher unifies that.

Structure

It's expressed as a tree. Pulling just the essentials of the proto:

// xds/type/matcher/v3/matcher.proto
message Matcher {
  message OnMatch {
    oneof on_match {
      Matcher matcher = 1;                      // a nested matcher
      core.v3.TypedExtensionConfig action = 2;  // the action to run
    }
  }

  // ...field matcher list / exact match tree ...
  OnMatch on_no_match = N;
}

Inside Matcher there's a MatcherList (each entry is predicate + on_match) and a MatcherTree (a fast exact-match branch); reach a leaf and the action runs. on_no_match branches to the default when nothing hit.

Embedding CEL

The standout is being able to call CEL (Common Expression Language) inside a predicate. The type for that is xds.type.v3.CelExpression, and you can write things like:

request.headers['x-env'] == 'prod'

This directly drives a branch. Under Envoy's Unified Matcher, CEL matchers became usable in Access Log, RBAC, and external authorization, and CEL itself is a language Google has run for years in policy evaluation for internal IAM and the like, so this isn't some unproven toy being shoved into Envoy.

Why is this nice? The big operational win is "reusing the same matching language across filters". The access-log filter and the RBAC policy can be written in the same DSL, so an operator only keeps one mental model. A declarative match language that closes inside the proto is unglamorous but effective.

Design call: where and how to hold matching, declaratively

Do you write match conditions for routing, authorization, and logging separately per filter, or unify them across the board?

Lean on Unified Matcher + CEL: when multiple filters (RBAC / access log / external authz) want to reuse the same condition expression. Write one CEL like request.headers['x-env'] == 'prod' and share it
Stay with per-filter matching: when it's a simple single-filter condition not worth bringing CEL in for
How it breaks: write match conditions in separate DSLs per filter and the same "prod only" condition ends up subtly different everywhere, splitting the operator's mental model and breeding mistakes. Centralize cross-cutting conditions in Unified Matcher as the single source of truth

10. Putting it together: the world gRPC Proxyless sees

The pieces introduced above are exactly what gRPC's Proxyless xDS assembles and uses on the client side.

The authorities map in the bootstrap file identifies multiple CPs (A47-xds-federation)
An existing xds:///my-service target URI expands internally into an xdstp-form Listener name, but only when the bootstrap's client_default_listener_resource_name_template is set to xdstp:// form (if the template contains %s, the service authority is embedded at that position)
Load can be pulled directly from the backend via ORCA (A51-custom-backend-metrics)
The aggregate can be returned to the control plane via LRS (A64-lrs-custom-metrics)

So a good chunk of the cncf/xds components dissected in the previous chapters is already at the point of running inside the gRPC library without any sidecar.

The "Proxyless gRPC" I introduced at the end of Part 1 runs as the result of assembling, on the client side, the pieces we've read here. It doesn't have Envoy's L7 extension points (Wasm, HTTP filter chains, etc.) so the coverage differs, but the standards-track features of xDS are landing on the gRPC side at a pace that keeps step with Envoy.

11. Designing as the shipper: actually writing a control plane

I've lined up the design levers. Now let's not leave it on paper, and run it. The brief is one worked example: design "ship the api service as prod 90% / canary 10%" as xDS resources, and serve it from a real control plane.

Design: build the resource graph as proto objects

In §3 I raised "do you design from JSON?", but a production CP doesn't route through JSON. What go-control-plane's snapshot cache takes is proto.Message itself (types.Resource = proto.Message), and the designer news proto structs directly in Go. The canary split is expressed via an RDS weighted cluster, splitting prod / canary into separate Clusters.

// the designer builds the resource graph here (excerpt)
func designCluster(name, host string) *cluster.Cluster {
    return &cluster.Cluster{
        Name:                 name,
        ConnectTimeout:       durationpb.New(2 * time.Second),
        ClusterDiscoveryType: &cluster.Cluster_Type{Type: cluster.Cluster_LOGICAL_DNS},
        LbPolicy:             cluster.Cluster_ROUND_ROBIN,
        LoadAssignment:       /* endpoint host:8080 */,
    }
}

// prod 90% / canary 10% weighted route
func designRoute(routeName, prod, canary string) *route.RouteConfiguration {
    return &route.RouteConfiguration{
        Name: routeName,
        VirtualHosts: []*route.VirtualHost{{
            Name: "api", Domains: []string{"*"},
            Routes: []*route.Route{{
                Match: &route.RouteMatch{PathSpecifier: &route.RouteMatch_Prefix{Prefix: "/"}},
                Action: &route.Route_Route{Route: &route.RouteAction{
                    ClusterSpecifier: &route.RouteAction_WeightedClusters{WeightedClusters: &route.WeightedCluster{
                        Clusters: []*route.WeightedCluster_ClusterWeight{
                            {Name: prod, Weight: wrapperspb.UInt32(90)},
                            {Name: canary, Weight: wrapperspb.UInt32(10)},
                        },
                    }},
                }},
            }},
        }},
    }
}

Stuff the built graph into a snapshot and serve it as an ADS server. This is the one line that "puts what you designed onto the wire".

snap, _ := cachev3.NewSnapshot("v1", map[resourcev3.Type][]types.Resource{
    resourcev3.ClusterType: {designCluster("api-prod", "prod.api.svc"),
                             designCluster("api-canary", "canary.api.svc")},
    resourcev3.RouteType:   {designRoute("api-route", "api-prod", "api-canary")},
})
cache := cachev3.NewSnapshotCache(false, cachev3.IDHash{}, nil)
cache.SetSnapshot(context.Background(), "edge-proxy-1", snap)

srv := serverv3.NewServer(context.Background(), cache, nil)
discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, srv) // grow an ADS endpoint

Confirm: did what I designed actually land on the wire

Pull it back from the serving side with a raw ADS client (just open StreamAggregatedResources and send DiscoveryRequest{TypeUrl: ClusterType}). The actual output (go-control-plane v0.14.0 / envoy v1.37.0 / go1.26.4):

== CDS (type_url=type.googleapis.com/envoy.config.cluster.v3.Cluster, version=v1, 2 resources) ==
  cluster api-prod     lb=ROUND_ROBIN endpoint=prod.api.svc
  cluster api-canary   lb=ROUND_ROBIN endpoint=canary.api.svc
== RDS (type_url=type.googleapis.com/envoy.config.route.v3.RouteConfiguration, version=v1, 1 resources) ==
  route   api-route    -> api-prod weight=90
  route   api-route    -> api-canary weight=10

The two Clusters I designed, api-prod / api-canary, and the 90 / 10 weighted route, came back on the wire as-is, type_url and all. The resources []Any from §0 contains exactly the proto structs I just newd. Design -> snapshot -> ADS -> fetch closes inside one Go program.

Here every thread laid since §0 gets pulled together: what I shipped was a proto stuffed in Any (§0), with an id api-prod attached (§2), served to one node edge-proxy-1 (the authority/node of §3), with the canary variant expressed not by baking it into the name but as an RDS weight (the §6 practice of "don't put variants in the id"). All the design levers show up in these 30 lines.

A few things change if the target is gRPC Proxyless

If you ship the same canary design to a gRPC Proxyless client instead of Envoy, the skeleton of the resource graph (LDS → RDS → CDS → EDS) is the same, but three things change. It comes down to who you are shipping to: the client side from §10 has its own constraints.

The Listener is built as an API listener, not a socket. The Listener I built in §11 above was a socket listener with an address (Envoy actually binds a port). gRPC has no port to bind, so you have to ship an API listener with an HttpConnectionManager stuffed directly into the Listener.api_listener field. grpc-go's unmarshal_lds.go literally branches on if lis.GetApiListener() != nil to decide client-side, and hand it a socket listener and it won't treat it as a client one. The designer ends up branching how the Listener is built on "whether the target is proxyless"
The extension set is a subset. Build it rich and it NACKs. The HTTP filters gRPC interprets are roughly router / fault / rbac / ext_proc; there's no Wasm or Envoy-specific filter. And gRPC doesn't silently ignore unknown / unsupported fields, it rejects them (e.g. it bounces a nonzero xff_num_trusted_hops). Ship a Listener you padded out for Envoy straight to proxyless and it NACKs. Narrow "the set of extensions you may use" per target at design time
Custom LB is specified via the Cluster's load_balancing_policy. In §11 I hard-coded LbPolicy: ROUND_ROBIN on the Cluster, but for fancier load balancing on proxyless you put an envoy.extensions.load_balancing_policies.* proto (ring_hash, wrr_locality, client_side_weighted_round_robin, etc.) on the Cluster's load_balancing_policy. grpc-go converts it into an internal balancer tree via xdslbregistry. Pick client_side_weighted_round_robin to feed it the §8 ORCA metrics, and that's where ORCA design and LB design connect

So designing "what you ship" branches on the target (Envoy or proxyless). Same canary, but a socket Listener + filter chain for Envoy, an API listener + limited filters + an LB-policy proto for proxyless. Even if cncf/xds provides a type-neutral box, you still have to design the box's contents to fit the receiver's capabilities.

Conclusion

Part 1 was about learning to read xDS. This one was about being able to design what you ship over xDS. To close, here's every chapter's "design call" folded onto one card. The intent is: when you design your own mesh, run down this table top to bottom and you can decide "which lever to throw, and how", with nothing missed.

Design lever	Options	Criterion for this side	How it breaks when you throw it wrong
Name (§2)	id / context_params / authority	id is the immutable logical name, variable axes go in context_params	bake env into id and names proliferate, references split
CP boundary (§3)	split authority / don't	split only where ownership / trust / fault isolation splits	over-split=bloated bootstrap / under-split=SPOF
Failover (§4)	bake into name via `alt=` / client-side	use `alt=` declaratively when the alternate is static	bake a dynamic switch into the name and it stiffens
Subscription unit (§5)	singleton / collection / glob	know the name=singleton, dynamic churn=glob	glob too broad=wasted bandwidth / too fine=chore subscriptions
Variant (§6)	context_params / dynamic_params	few axes=context, hits double digits=plan for dynamic	grow context too far=36 blowup / dynamic is unimplemented
Granularity (§7)	fat / fine	split fine only where change frequency / blast impact is high	a giant resource fully outages on one typo
Telemetry (§8)	ORCA in-band / OOB / LRS aggregate	short unary=in-band, steady monitoring=OOB	gather none and load balancing goes blind
Matching (§9)	per-filter / Unified Matcher + CEL	use Unified when you want the same DSL across the board	scattered DSLs split the operator's mental model

Behind this table is the one fact from §0: xDS is a type-neutral bus that can carry anything via type_url + Any, and design is deciding the proto and the name you stuff into that box. Each xRFC of cncf/xds is the history of how that "naming and shipping" gets standardized.

xDS, which was supposed to be a "config shipper", is morphing into a unified gRPC API family for remote-controlling a fleet of proxies: federation via Authority, cache-friendly variants via Dynamic Parameters, bidirectional telemetry via ORCA / LRS, a declarative match language via Unified Matcher and CEL. The model "xDS = Envoy's config protocol" is too narrow as of 2026. And as §11 showed, that design and delivery is something you can pull into your own hands in 30 lines of Go. Read the cncf/xds protos and xRFCs as design levers, and the rest is just assembly.

References

Why AWS IAM Is So Hard

kt — Sat, 06 Jun 2026 10:58:20 +0000

Where it starts

The first thing that beats you up when you start using AWS is IAM. It got me too.

You see AccessDenied. You check the policy and "Effect": "Allow" is right there. Denied anyway. You AssumeRole, then run aws sts get-caller-identity and you are still the same old you, with the same permissions. There are two similar-looking JSON blobs called a trust policy and a permission policy, and you have no idea which one to write what in. Someone asks you the difference between a User and a Role, and you sort of know, but you cannot say it in one sentence.

What I eventually realized is this: IAM is hard not because the mechanics are complex, but because it is a minefield of traps that go "same word, different thing" and "a handful of asymmetric rules that fight your intuition." Each trap is the kind you understand instantly once someone explains it. The flip side: defuse them one at a time and IAM becomes shockingly obedient.

This is not a reference that covers every corner of IAM. It is the mines a beginner will definitely step on, broken apart by their true cause, and lined up so that reading top to bottom makes each one go "oh, that's all it was."

1. What problem is IAM actually solving

Before the traps, let me pin down in one line what IAM is for. If this drifts, everything after it goes blurry.

IAM decides who (authentication) can do what (authorization). Every operation in AWS is an HTTPS API call, and IAM stands in front of every single one of them.

Clicking around in the console, running aws s3 ls, applying Terraform: under the hood they all become the same thing, an API request to AWS. For every one of those requests, IAM asks two questions.

Authentication (AuthN): from the signature on the request, confirm that "this caller really is alice." This is the world of signing (SigV4), and the place beginners get stuck is mostly what comes after it: authorization.
Authorization (AuthZ): for a confirmed caller, decide whether a policy exists that permits this operation. Almost all of IAM's difficulty lives on this side.

When I say "hard" in this article, I mean authorization, nearly every time. I will leave the math of signing to another article and focus here on how you write "who can do what" and how it gets evaluated.

2. Get the vocabulary straight

The number one reason IAM blows up on you is that there are too many concepts with similar names. So let me fix the names first. If a term shows up later that you do not recognize, come back here.

Term	In one line
AWS account	The container for resources and IAM. Identified by a 12-digit ID like `123456789012`, and it is also the billing unit. Users and Roles are created inside it and are invisible from other accounts. This becomes the key fact later
Principal	The subject of a request. The "who." A User, a Role, an AWS service, and so on
IAM User	A permanent identity you create inside an account, tied to a person or a machine. Holds a password or long-lived access keys
IAM Role	A role that anyone (who meets the conditions) can temporarily become. Holds no long-lived keys, built around short-lived credentials
IAM Group	A container that bundles Users. Attach a policy to a Group and it applies to every member. A Group itself cannot log in
IAM Policy	JSON that says "what is allowed or denied." On its own it floats in the air; it only means something once you attach it to someone
AssumeRole	The STS API that takes on a Role and hands back short-lived credentials. The central act of IAM
Short-lived credentials	The disposable keys you get back from `AssumeRole`, expiring in about an hour (a three-piece set: access key, secret, session token)
identity-based policy	A policy attached to the identity side: a User, Role, or Group. "What can this identity do?"
resource-based policy	A policy attached to the resource side, like an S3 bucket. "Who is this resource willing to let touch it?"
trust policy	A special resource-based policy attached to a Role. It writes only one thing: "who is allowed to Assume this Role?"
SCP (Service Control Policy)	An Organizations feature that spans multiple accounts. Attached to an OU (a folder that groups accounts) or to an account, it lowers the ceiling on permissions. Think of it as Deny-only
Permission Boundary	A ceiling attached to an individual User or Role. Where an SCP is the per-account version, this is per-identity
STS (Security Token Service)	The service that issues short-lived credentials. `AssumeRole` is its API

The three pairs below are the especially confusing ones. This article defuses them one at a time.

User vs Role (both are "identities" but their nature is the exact opposite)
identity-based policy vs trust policy (a Role needs both)
same-account vs cross-account (the same operation needs different things)

3. Difficulty 1: what is the difference between a User and a Role

The first mine. Both are "identities that can be a Principal," so beginners stall on "so which one am I supposed to use?" The difference boils down to how the keys are held. Here it is in a table.

	IAM User	IAM Role
Key lifetime	Long-lived (valid until you revoke it)	Short-lived (auto-expires in ~1 hour)
Key owner	The User keeps holding it	No owner. Borrowed fresh each time you use it
Binding	Pinned to a specific person or machine	Anyone (who meets the conditions) can become it
Damage if leaked	Big (valid until you recreate it)	Small (expires shortly)
Today's guidance	Avoid where possible (emergency / legacy)	Default to this

In plain analogy:

User = a photo ID badge. Pinned to one person; if lost, it can be abused until you reissue it.
Role = a visitor pass you borrow at the front desk. Anyone (if allowed) can borrow it, and it goes invalid automatically at the end of the day.

Why is a Role the default now? Simple: so you do not scatter long-lived keys all over the world. Writing an access key into code, pushing it to GitHub, leaking it: that is the classic accident. So the goal is a world where only disposable, expiring keys are ever in circulation. That is why human logins, EC2, Lambda, CI/CD all converge on assuming a Role and taking short-lived credentials.

Let me defuse one behavior right here that every beginner trips on: the "I assumed the Role but my permissions did not change" one.

AssumeRole does not rewrite your current credentials. It only "hands back" a fresh set of short-lived credentials. You set those into an environment variable or a profile, and only then, starting with your next request, do you act as the Role. There is no magic switch at the instant you assume. The reason aws sts get-caller-identity returns the same old you is that you have not used the returned keys yet.

4. Difficulty 2: why does a Role carry two policies

This is the biggest climb in IAM. A Role has two policies of different natures hanging off it. And beginners conflate the two and always stall on "which one do I write what in?"

The reason is simple: a Role is a split personality that is both an "identity" and a "resource" at the same time. That duality maps directly onto the identity of the two policies.

Policy	Which face of the Role	Question it answers	Analogy
trust policy	The resource face (resource-based)	Who is allowed into this Role	The front-door key (who gets in)
permission policy	The identity face (identity-based)	What you can do once inside	The list of things you may touch in the room

You need both. Neither works alone.

No (or mismatched) trust policy → you cannot Assume at all. Stopped at the door.
No permission policy → you can Assume, but you cannot touch anything in the room you entered. Empty permissions.

Watch it across the Assume flow

Tracing where each of the two takes effect along the timeline organizes the whole thing at once.

The door check (trust policy) takes effect at the moment of Assume.
The contents check (permission policy) takes effect after you assume, at the moment you actually hit the API.

If you do not know about this two-stage structure, you read the error message wrong.

is not authorized to perform: sts:AssumeRole → stopped at the door. Look at the trust policy, or the caller's own AssumeRole permission.
Assume went through but s3:GetObject returns AccessDenied → stopped at the contents. Look at the permission policy.

The trust policy is the more dangerous one

A permission policy that is too wide means "the person who got in does too much." A trust policy that is too wide means "people who should never have gotten in can get in." The latter is the more serious accident. Carelessly setting Principal to "*" (anyone) turns that Role into a privilege-escalation hole that any AWS account on Earth can Assume. The trust policy is less flashy than the permission policy, but it is the place to write more carefully.

5. Difficulty 3: there are too many policy types

So far we have seen three: identity-based, resource-based, and trust. AWS officially classifies policies into seven types (identity-based, resource-based, permission boundary, SCP, RCP, ACL, session). A beginner who sees that list loses heart on the spot.

You do not need to memorize all of them as equals. Two clarifications up front make it much lighter.

The trust policy is not a separate type. It is a kind of resource-based policy, dedicated to the Assume door of a Role (the one from the last section). An ACL is also a relative of resource-based, and it is treated as legacy now, so do not use it for anything new. So in practice you only need to look at "identity side / resource side / ceiling family."
The rest organize instantly once you split them into two groups: things that add permissions, and things that lower the ceiling on permissions.

(The trust policy is not in this diagram. As covered above, it is dedicated to the Assume door and plays a different role from the permission math we do here. SCP and RCP are ceilings that only show up if you use Organizations; a personal account has none.)

The clearest way to picture the two groups is addition and multiplication.

Group	Which ones	How they combine	Intuition
The adding group	identity-based / resource-based	union (addition). A single Allow anywhere permits it	Pushes toward more permission
The ceiling group	SCP / RCP / Boundary / session	intersection (multiplication). Denied unless all permit	Pushes toward less. One veto and you are out

The consequence here is the one beginners get caught on the most.

Writing an Allow into an SCP or a Permission Boundary does not add a single bit of permission. These are a "ceiling," not a "permission." Actual permission is granted by the adding group (identity / resource). The ceiling group only "trims the part of the granted permission that goes over the ceiling."

So "I wrote Allow in the SCP but my usable permissions did not grow" is correct behavior. An SCP's Allow is nothing but setting a ceiling that says "you may permit up to here." Actual permission has to be granted separately, by something like an identity-based policy.

6. Difficulty 4: why do I get denied when I wrote Allow

"I clearly wrote Allow in the policy but I get AccessDenied." The most common way to get stuck in IAM. The cause is the order of evaluation. AWS looks at all policies in this order.

There are three iron rules to read off this diagram.

The default is deny (implicit Deny). Write nothing and you can do nothing. Permission is born only once you spell out at least one Allow.
An explicit Deny beats everything. If any one policy has a Deny, it is denied no questions asked, even with Allows lined up across every other policy. Evaluation stops there.
The ceiling group is a "cutoff," not a "grant." As covered last section, an operation the SCP / Boundary / session do not permit will not go through no matter how many Allows the identity-based policy has.

(This diagram is simplified for beginners. Strictly, a Permission Boundary does not constrain permissions that a resource-based policy grants directly to a Principal. A path explicitly permitted by name on the resource side can slip past the boundary ceiling. At first, forget this exception and just learn "the ceiling group is the overall ceiling.")

When you knock out the typical patterns of "I wrote Allow but got denied," the culprit is usually one of these.

There is an explicit Deny somewhere (common in an SCP the org attached).
You are hitting the ceiling of an SCP or Permission Boundary and are simply outside the limit.
It is cross-account and the other account has no permission (next section).
A Condition (an IP restriction, MFA required, and so on) is not satisfied, so that Allow never fires.

The presence of Allow is a necessary condition for permission, not a sufficient one. Suspecting "is there a Deny?" and "am I inside the ceiling?" first is the shortcut to debugging.

7. Difficulty 5: behavior changes between same-account and cross-account

This is the asymmetric rule you will never figure out unless someone tells you. The exact same operation needs different things depending on whether the other side is the same account or a different account.

To pin it in words:

Within the same account: either the identity-based policy or the resource-based policy permits it, and you are through (union).
Cross-account: company A's Allow on the identity side and company B's Allow on the resource side, both are required (missing either one and you are out).

Why is it like this? Cross-account is the act of "reaching for stuff in someone else's house," so it is natural to think both parties must consent: the reaching side (your account permits it) and the reached side (their account permits it). Within one account there is a single homeowner, so one side's consent is enough.

On top of this rides one more asymmetry. The "Role trust policy" from the earlier section also tangles with this same/cross distinction. To assume another company's Role cross-account, their trust policy must name your Principal, and you must have the sts:AssumeRole permission on your side: both sides must line up. This is why, when you "cannot assume the other party's Role," looking only at your own permissions often does not solve it.

8. Difficulty 6: why is "IAM Policy alone" not enough

The final climb. By now you might think "what I actually want is just 'who can do what,' so as long as I can write an IAM Policy, that is enough, right?" In reality this is the structural trap the field gets caught in the most.

The key is the fact we touched in the vocabulary section. An IAM Policy floats in the air on its own; it only takes effect once you attach it to an "identity" (a User or a Role). And that identity can only exist inside one specific account.

The instant you add accounts, this bares its fangs. Say a company splits into dev / staging / prod / sandbox and so on, ten accounts. Each account is an independent IAM space. To log alice into every account, the naive approach gives you this.

IAM Users and long-lived access keys multiply by headcount times account count. 10 accounts × 50 people = 500 Users and keys. What makes this hell:

A flood of long-lived keys gets scattered (the leak surface is headcount times account count).
MFA has to be set up on every one of them.
When alice leaves, you go around deleting all ten. Forget even one and a hole stays open.
To change permissions, you do every account by hand.

This very property of "identities being bound per account" is why IAM Policy alone cannot run a real organization. AWS provides the fix: use IAM Identity Center. This manages employee identities in one place and connects to your in-house IdP (Identity Provider: a base like Okta, Microsoft Entra, or Google Workspace that centrally manages employee accounts).

The crux of the mechanism comes down to two words.

Permission Set: a template for permissions. Define ReadOnly or Admin once in the center and reuse it across multiple accounts. The contents are just a bundle of ordinary IAM Policies. You do not have to write from scratch; you can pick AWS-made predefined ones like AdministratorAccess / PowerUserAccess / ReadOnlyAccess as-is. Only when you need something fancy do you mix in a custom policy of your own.
Assignment: the tuple of "which person (Group) / in which account / which Permission Set" to assign.

When you assign, Identity Center grows an IAM Role inside that account automatically. So the IAM constraint that "each account needs its own Role" has not changed. What changed is that instead of creating that Role by hand, it gets handed out automatically from a central template. And alice's identity is a single one in the IdP. On departure, disable that one in the IdP and she is shut out of every account at once.

To organize it:

	IAM User with a policy attached directly	Identity Center + Permission Set
Where the identity lives	Scattered across each account	Consolidated into one in the IdP
Credentials	Long-lived access keys	Short-lived (issued fresh by `aws sso login`)
On departure	Delete the User in every account	Disable one in the IdP
Changing permissions	Every account by hand	Change the template / Assign centrally

You see the difference when you get your hands dirty

This article mostly skips code, but this is the one place where lining up the actual commands makes the difference obvious at a glance. Let me do the same "alice uses two accounts, dev and prod" both ways.

The IAM User way makes the admin repeat the same work as many times as there are accounts.

# as the admin of the dev account
aws iam create-user --user-name alice
aws iam attach-user-policy --user-name alice \
  --policy-arn arn:aws:iam::aws:policy/PowerUserAccess
aws iam create-access-key --user-name alice   # long-lived keys come out -> hand them to alice

# do exactly the same thing again in the prod account (another set of keys)

On alice's side, she writes the long-lived keys she received as-is into ~/.aws/credentials and uses them.

[dev]
aws_access_key_id = AKIA...DEV...
aws_secret_access_key = ...DEV...

[prod]
aws_access_key_id = AKIA...PROD...
aws_secret_access_key = ...PROD...

The Identity Center way, on the other hand, has no create-user and no create-access-key. alice configures once.

aws configure sso        # register the SSO URL once. log in via browser (MFA here)
                         # every account/permission you can reach is listed, and profiles are written automatically
aws sso login            # about once a day. short-lived credentials rain down
aws s3 ls --profile dev  # same login, just switch between dev and prod
aws s3 ls --profile prod

Look at ~/.aws/config and the point is that not a single key is written there. All that is written is "which account, which Permission Set," and the actual keys are issued fresh and short-lived on every aws sso login. This is the decisive difference from the former, which parks long-lived keys in a file.

This is where the essence we pinned at the start pays off: "every access in AWS ultimately converges on assuming a Role and taking short-lived credentials." Identity Center and Permission Sets are not a new authorization mechanism; they are nothing but a connector that wires the human entrance to each account's Role AssumeRole. What you hold at the end is the usual Role's short-lived credentials.

9. Summary: one mental model

Let me fold all the difficulty so far into a single picture. When IAM stops making sense, come back to this drawing.

Each difficulty in one line:

User vs Role: the difference is key lifetime. A User keeps a long-lived key; a Role borrows a short-lived one each time. A Role is the default now.
Two policies: a Role is a split personality, "identity" and "resource." The door (trust policy) and the contents (permission policy) answer different questions.
Policy types: officially there are seven, but split them into "the group that grants by adding" and "the group that lowers the ceiling by multiplying" and it organizes. An Allow in the ceiling group adds no permission.
Evaluation order: write nothing, denied. Deny beats everything. "I wrote Allow" is only a necessary condition for permission.
Same vs cross: same account, one side's Allow is enough. Different account, you need both sides' Allow.
IAM Policy alone is not enough: identities are bound to accounts. When accounts grow, use Identity Center to consolidate identities centrally and hand out Roles automatically.

IAM beats you up at first sight because there are so many concepts, but most of the difficulty is the kind that disappears once explained: "same word, different thing," and "a handful of asymmetric rules that fight intuition." Step on these six mines once and the next time you see AccessDenied, you will be able to narrow down the culprit in order.

Next time you get stuck, suspect first "is there a Deny?" and "am I inside the ceiling (SCP / Boundary)?" If that still does not explain it, then it is either the same/cross asymmetry or you are stopped at the trust policy door. Once you know where the mines are, IAM is not scary.

Transaction Tokens Deep Dive: The OAuth Spec That Carries 'Who, and Why' Across Your Microservices

kt — Wed, 03 Jun 2026 16:17:13 +0000

Introduction

A modern system processes a single request across many cooperating microservices. A user clicks "Buy" on a trading site, and behind that one click the API gateway, the order service, the risk service, the payment service, and the notification service all fire in a chain.

That chain hides a basic question.

"What happens if you take the user's OAuth access token that arrived at the first API gateway, and keep forwarding it as-is to every internal service?"

Plenty goes wrong.

You are leaking an external token into the internals. If that access token leaks, every internal service is exposed.
You lose track of who started the request. The risk service cannot tell whether this call came through the user's gateway or whether someone is hitting it directly.
Any service can impersonate another. Once the internal network is breached, a malicious workload can pretend to be a legitimate request and call other services.
Token theft. Steal the OAuth token flowing internally and you can reach external resources too.

Transaction Tokens (Txn-Tokens) are the answer to all of this.

What Transaction Tokens are

draft-ietf-oauth-transaction-tokens-08 (Txn-Tokens for short) is a spec moving through the IETF OAuth WG. As of June 2026 the latest version is Draft 08, and it has entered WG Last Call (WGLC), the final stage of the working group's process. The authors are engineers from CrowdStrike, Practical Identity, and Defakto Security, and the spec grew out of running microservices at real scale.

In one sentence:

A Txn-Token is a short-lived JWT that tells every workload inside a Trust Domain, in a tamper-proof way, who a request was started for and what it was started to do.

The contrast:

Axis	Forwarding the OAuth access token as-is	Using Txn-Tokens
Leak risk	an externally-facing token flows through the internals	only an internal, short-lived token flows
Context	lost along the way	every workload can verify the same context
Tamper detection	none	the TTS signature makes it tamper-proof
Impersonation	hard to stop	the TTS only issues for legitimate transactions
Blast radius	unbounded	narrowed by tight scopes

Terminology first

Before reading the spec, nail down the terms that are specific to Txn-Tokens.

Term	Meaning
Trust Domain	A set of systems under a common security policy. This is the boundary where a Txn-Token is valid.
External Endpoint	The entry point into a Trust Domain (an API gateway and the like), where external tokens arrive.
Workload	A unit of execution: a container, a microservice, a managed database.
Call Chain	The sequence of workloads invoked one after another for a single transaction.
TTS (Txn-Token Service)	The special service inside a Trust Domain that issues Txn-Tokens.
Txn-Token	A short-lived, signed JWT holding user ID, workload ID, and authorization context, flowing through the whole Call Chain.

A Trust Domain is a group of systems that share a common security policy and controls. Two or more workloads on a physically or virtually isolated network form one Trust Domain, and access to a workload is restricted to its published interfaces only.

The TTS is the single special service inside a Trust Domain that issues Txn-Tokens. "Logically one" means you can run multiple instances for availability, as long as they are unified under one trust policy.

What kind of JWT a Txn-Token is

A Txn-Token is just a signed JWT (JSON Web Token) carrying a specific set of claims.

JWT header

{
  "typ": "txntoken+jwt",
  "alg": "RS256",
  "kid": "identifier-to-key"
}

The typ of txntoken+jwt marks this token as a Txn-Token. That media type also gets registered with IANA.

JWT body claims

Claim	Required	Meaning
`iat`	yes	issued-at time
`aud`	yes	Trust Domain identifier (unusable outside this domain)
`exp`	yes	expiry
`txn`	yes	transaction-unique ID (per RFC 8417 Section 2.2)
`sub`	yes	the subject of the transaction (a user or workload identifier)
`scope`	yes	the purpose and permission range of this transaction (per RFC 8693 Section 4.2)
`req_wl`	yes	identifier of the workload that requested the Txn-Token
`iss`	no	issuer (omit when the signing key is already known)
`rctx`	no	requester environment context (source IP, auth method, ...)
`tctx`	no	transaction context (values that stay constant across the Call Chain)

The two that matter most are tctx (Transaction Context) and rctx (Requester Context).

tctx (Transaction Context)

Holds the parts of the transaction that do not change across the call chain. The TTS sets this as authoritative data, and services read it to make authorization decisions.

"tctx": {
  "action": "BUY",
  "ticker": "MSFT",
  "quantity": "100",
  "customer_type": {
    "geo": "US",
    "level": "VIP"
  }
}

rctx (Requester Context)

Holds the environment of the request: the source IP, auth method, and so on of the original caller.

"rctx": {
  "req_ip": "69.151.72.123",
  "authn": "urn:ietf:rfc:6749"
}

A real Txn-Token (the stock-trade case)

{
  "iat": 1686536226,
  "aud": "trust-domain.example",
  "exp": 1686536586,
  "txn": "97053963-771d-49cc-a4e3-20aad399c312",
  "sub": "d084sdrt234fsaw34tr23t",
  "req_wl": "apigateway.trust-domain.example",
  "rctx": {
    "req_ip": "69.151.72.123",
    "authn": "urn:ietf:rfc:6749"
  },
  "scope": "trade.stocks",
  "tctx": {
    "action": "BUY",
    "ticker": "MSFT",
    "quantity": "100",
    "customer_type": {
      "geo": "US",
      "level": "VIP"
    }
  }
}

Notice exp is iat + 360 seconds (6 minutes). A Txn-Token must be short-lived. Set it to a few minutes at most.

The basic flow: handling an external request

Let's walk through how Transaction Tokens get used, with a sequence diagram.

Basic flow (external request)

The key part is step 3, the call chain. The Txn-Token is forwarded unchanged between internal services. You must not modify it. Because each service verifies the signature independently, every one of them can confirm that the TTS authorized this transaction.

When the request starts internally

An external API call is not the only way a transaction begins. A scheduler, a batch job, or any internal workload can kick off a transaction on its own.

With no external OAuth token in hand, the workload generates a Self-Signed JWT and presents it to the TTS. The TTS verifies it, confirms the request comes from a legitimate workload, and then issues a Txn-Token.

The Txn-Token Service (TTS) in detail

The TTS is the heart of the spec. It is implemented as a profile of RFC 8693 (OAuth 2.0 Token Exchange). Token Exchange is the OAuth extension for "trade a token you already hold for a new token with a different purpose or scope," using grant_type=urn:ietf:params:oauth:grant-type:token-exchange. The TTS rides on that mechanism to convert an OAuth access token (or the Self-Signed JWT above) into a Txn-Token.

Request to the TTS (Token Exchange)

POST /txn-token-service/token_endpoint HTTP/1.1
Host: txn-token-service.trust-domain.example
Content-Type: application/x-www-form-urlencoded

grant_type=urn:ietf:params:oauth:grant-type:token-exchange
&requested_token_type=urn:ietf:params:oauth:token-type:txn_token
&audience=http://trust-domain.example
&scope=trade.stocks
&subject_token=eyJhbGciOiJFUzI1NiIsImtpZCI...
&subject_token_type=urn:ietf:params:oauth:token-type:access_token
&request_context={"req_ip":"69.151.72.123","authn":"urn:ietf:rfc:6749"}
&request_details={"action":"BUY","ticker":"MSFT","quantity":"100"}

What each parameter means:

Parameter	Required	Description
`grant_type`	yes	the fixed Token Exchange value from RFC 8693
`requested_token_type`	yes	set to the `txn_token` URN
`audience`	yes	the Trust Domain identifier
`scope`	yes	the purpose and permission of this transaction (keep it tight)
`subject_token`	yes	a token proving the subject (OAuth access token, Self-Signed JWT, ...)
`subject_token_type`	yes	a URI naming the subject_token type
`request_context`	no	request environment (IP, ...); lands in rctx
`request_details`	no	request details (the action, ...); used to build tctx

Kinds of subject_token

The TTS accepts several kinds of subject_token.

subject_token	`subject_token_type` URI	Typical use
OAuth access token	`urn:ietf:params:oauth:token-type:access_token`	common at an external endpoint
ID token (OIDC)	`urn:ietf:params:oauth:token-type:id_token`	a subject already authenticated via OIDC
SAML assertion	`urn:ietf:params:oauth:token-type:saml2`	SAML-based authentication
Self-Signed JWT	`urn:ietf:params:oauth:token-type:self_signed`	sign-and-present for internal triggers
Unsigned JSON object	`urn:ietf:params:oauth:token-type:unsigned_json`	the simplest internal trigger

Beyond the token types defined in RFC 8693, you can use self_signed / unsigned_json for internal triggers, plus any custom URN the parties agree on.

Note: a Refresh Token must not be used as a subject_token. Txn-Tokens are never minted from a Refresh Token.

The TTS processing flow

Response from the TTS

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: no-store

{
  "token_type": "N_A",
  "issued_token_type": "urn:ietf:params:oauth:token-type:txn_token",
  "access_token": "eyJCI6IjllciJ9...Qedw6rx"
}

The token_type of N_A matters. A Txn-Token is not a bearer token you present as-is. It is an internal-only carrier of authorization context, so per RFC 8693 the TTS returns N_A.

Using the Txn-Token: internal service-to-service calls

Carrying it in an HTTP header

A workload puts the Txn-Token in a dedicated HTTP header called Txn-Token and passes it to the next workload.

POST /api/check-risk HTTP/1.1
Host: risk-service.trust-domain.example
Content-Type: application/json
Txn-Token: eyJhbGciOiJSUzI1NiIsInR5cCI6InR4bnRva2VuK2p3dCJ9...
Authorization: Bearer <workload-own-access-token>

{
  "order_id": "ord-12345"
}

Important: do not put the Txn-Token in the Authorization header (the spec says MUST NOT, Section 13). The thing to notice is that Txn-Token and Authorization sit side by side in the example above. That is not a mistake, it is the intended shape: two layers run at once inside one request.

Authorization: carries the calling workload's own authentication, plus coarse service-to-service authorization (is this service allowed to call this API).
Txn-Token: carries who and what the request was started for, and to do (user ID, the action, and other fine-grained immutable context).

The Authorization header may already be in use for some other purpose, so piggy-backing the Txn-Token on it would collide. That is why it gets its own Txn-Token header.

Verification on the receiving workload

At this point, you must not modify the Txn-Token. Forward it exactly as received. Passing the TTS-signed token along idempotently is what keeps the context consistent across the whole call chain.

Pairing it with workload authentication

Public-key based mutual authentication with the TTS is recommended.

mTLS (RFC 8705): mutual TLS certificates so the workload and the TTS authenticate each other
SPIFFE / SPIRE: the workload identity standard, proven with an SVID
WIMSE Mutual TLS (draft-ietf-wimse-mutual-tls)
WIMSE HTTP Signatures (draft-ietf-wimse-http-signature)
WIMSE Workload Proof Token (draft-ietf-wimse-wpt)
Client Authentication JWT (RFC 7523)

The TTS authenticates to the workload and the workload authenticates to the TTS (mutual). If the workload does not authenticate the TTS, it risks leaking its OAuth access token to a rogue TTS.

Lifetime and replay defense

Designing the lifetime

The design principle is simple: a Txn-Token only needs to live as long as the request takes to process. The spec spells it out: the lifetime MUST be kept short (on the order of minutes or less). Even for a long-running batch, you do not make the Txn-Token long-lived. You re-issue a fresh short-lived Txn-Token per transaction. And if the presented subject_token has already expired, the TTS must not issue a Txn-Token.

Replay detection via the txn claim

The txn claim is a transaction-unique UUID. A workload that caches txn values briefly can detect reuse of the same Txn-Token, that is, a replay attack.

The catch: this is hard when multiple instances share no state. In practice you combine a short lifetime with correlating txn in the logs.

Security considerations

Never embed an access token inside a Txn-Token

Putting the original OAuth access token inside a Txn-Token is strictly forbidden. If the access token is still valid after the Txn-Token expires, an attacker can crack open the Txn-Token, pull out the access token, and use it to reach external resources.

A Txn-Token is not an authentication credential

A Txn-Token does not prove a workload's identity. It is a carrier of authorization context. A workload still needs a separate mechanism (mTLS and the like) to authenticate itself to the TTS.

Prevent scope amplification

Never log a raw Txn-Token

Do not write a complete Txn-Token to your logs: it becomes replay material. Log one of:

only a hash of the Txn-Token (for correlation)
or the payload with the JWS signature stripped

Be extra careful when PII is involved.

How Txn-Tokens relate to WIMSE

WIMSE (Workload Identity in Multi-System Environments) is the IETF WG standardizing workload identity and workload-to-workload authentication for microservice environments.

WIMSE and Txn-Tokens complement each other. WIMSE WPT establishes a workload's own identity, and Txn-Tokens supply the context of the request that workload is processing. Used together, they give you the authentication and authorization model a zero-trust microservice architecture wants.

Identity Chaining vs Txn-Tokens

A related spec is draft-ietf-oauth-identity-chaining. Both aim to propagate request context, but they target different problems. Identity Chaining propagates user identity across Trust Domains (cross-domain federation), while Transaction Tokens propagate the full request context within a single Trust Domain (consistency inside the call chain).

Axis	Identity Chaining	Transaction Tokens
Scope of use	between Trust Domains (cross-domain)	within a Trust Domain (same domain)
Main goal	access a resource in another domain	keep context consistent across the call chain
Token basis	RFC 8693 + RFC 7523 combined	a profile of RFC 8693
Context carried	mainly user identity	user identity + action + environment, all of it

Wrap-up

The problems inside microservices were these: external tokens circulating through the internals, request context lost along the way, impersonation you cannot detect. Txn-Tokens solve them by converting the external token into an internal-only Txn-Token, keeping one consistent context across the call chain, blocking tampering and impersonation with the TTS signature, and shrinking the blast radius with short lifetimes and tight scopes.

The benefits, summarized:

Short-lived + per-transaction binding → lower replay risk
Tight scope → limits lateral movement
TTS signature → blocks tampering inside the call chain
Independent verification at each workload → rejects unauthorized direct calls
Only workloads with the right permission can obtain one → contains the impact of a compromised service

The spec is at Draft 08 and in WG Last Call, the final stretch before becoming an RFC. Standardization is moving ahead with WIMSE integration in view. If you are serious about applying zero trust to your microservices, this is one of the specs to track now.

References

ID-JAG, Transaction Tokens, WIF: The Three Layers of AI Agent Auth

kt — Wed, 03 Jun 2026 11:12:33 +0000

Introduction

Something clicked when I looked at the Workload Identity Federation that Anthropic shipped in May 2026. Delete the sk-ant-... API key, have a k8s service account JWT mint a short-lived sk-ant-oat01-... token, call anthropic.Anthropic() with no arguments, and it just works. The security posture clearly went up.

But the actual day-to-day flow, where an agent "runs Cursor on Alice's GitHub PAT and opens a PR," did not change one bit. Cursor, Claude Code, Comet: all of them ultimately hold the user's own credentials and act with them. Around the same time I was following the IETF drafts for ID-JAG (Identity Assertion JWT Authorization Grant) and Transaction Tokens, and I kept feeling this gap: the specs are converging, yet nobody is adopting them.

That raises one obvious question. "Tighter scope," "auditable," "smaller blast radius" are all true, and yet the agent UX you use every day does not change. So is this stuff actually going to catch on?

This article takes that question head on. The conclusions up front:

The UX not changing is, like refresh tokens, evidence that the design is correct
ID-JAG, Transaction Tokens, and WIF each protect a different layer; lining them up to compare is a category error
There are two adoption routes. One is the MFA / IRSA / Sigstore "incident → regulation → adoption" path. The other is the DNSSEC / SPIFFE "eternal niche, spreads only under a different name through vendor integration" path. A protocol with no UX improvement basically takes the latter, and that is the cold lesson of tech history

To put the punchline first: I predict "we deployed ID-JAG" (Layer A) will not catch on. Layer C will. Layer B is an eternal niche.

The May 2026 map: where the major specs and implementations actually stand

To start the discussion from the same place, here are the primary sources.

IETF drafts

Spec	Latest	Published	Status
draft-ietf-oauth-identity-assertion-authz-grant (ID-JAG)	-04	2026-05-21	Standards Track, Internet-Draft
draft-ietf-oauth-identity-chaining	-12	2026-05-11	Standards Track, Internet-Draft
draft-ietf-oauth-transaction-tokens	-08	2026-03-02	Standards Track, Internet-Draft
draft-araut-oauth-transaction-tokens-for-agents	-06	2026-04-11	Individual Draft (Ashay Raut, Amazon), pre-WG-adoption
draft-ietf-oauth-v2-1	-15	2026-03-02	Standards Track, Internet-Draft (in WG discussion, pre-Last-Call)
draft-klrc-aiagent-auth-00	-00	2026-03	Individual Draft (Defakto / AWS / Zscaler / Ping)

What this tells you about the lay of the land:

ID-JAG is not an RFC yet. There is just a Standards Track draft circulating; as of this writing there is no official RFC number assigned. The contents can still change
The Agent extension to Transaction Tokens is still an individual proposal, pre-WG-adoption (draft-araut-, not draft-ietf-). It uses the act claim for the AI agent and the sub claim for the principal, and it has moved from -00 to -06 in half a year (a straightforward extension of RFC 8693 Token Exchange's actor/subject concepts)
OAuth 2.1 (-15) is in WG discussion. It has not reached Last Call, but it is framed to obsolete RFC 6749. Mandatory PKCE, removal of the Implicit/Password grants, and so on all become load-bearing assumptions for the agent specs

I wrote the dedicated walkthrough of ID-JAG itself in a separate piece (ID-JAG Deep Dive). Here I narrow in on "what becomes visible when you put all three side by side."

The implementation side

Chasing specs alone is pointless. Here is what is actually moving in 2026.

Implementation	Released	What it does
Anthropic Workload Identity Federation	2026-05	Exchanges k8s SA / GHA / SPIFFE / AWS / GCP JWTs for `sk-ant-oat01-...` short-lived tokens via RFC 7523 jwt-bearer
Okta Cross-App Access (XAA)	2025-Q3 beta → 2026 GA	The commercial name for ID-JAG. Adopted formally as MCP's Authorization Extension "Enterprise-Managed Authorization"
xaa.dev playground	2026-01-20	An official Okta environment for trying XAA end to end on your own machine
Keycloak 26.5 JWT Authorization Grant	2026-01	Preview support for RFC 7523. Exchanges externally-signed JWTs for Keycloak tokens
MCP spec 2026-03-15	2026-03-15	Makes RFC 8707 Resource Indicators mandatory, RFC 9728 Protected Resource Metadata mandatory
Salesforce Agentforce "Trusted Agent Identity"	2026 GA	Inserts mobile approval into high-risk agent actions

Two quiet but big shifts are happening in that table.

One: the 2026-03-15 MCP spec made RFC 8707 (Resource Indicators) and RFC 9728 (Protected Resource Metadata) mandatory. Audience binding is now enforced per MCP server, so an access token issued for one MCP server gets rejected by the authorization server if you throw it at a different MCP server. A leak path like Cursor accidentally forwarding a GitHub MCP token to a Slack MCP is now closed at the spec level.

Two: Okta XAA got folded in as MCP's "Enterprise-Managed Authorization" extension. The ID-JAG concept is reaching the implementation side first not as a pure IETF draft but riding on top of MCP.

Summed up: the specs have converged, implementations are out from Anthropic / Okta / Keycloak / Salesforce, but almost nobody in the field is using any of it. That is the state of play in May 2026.

What are the three layers actually protecting?

Putting all three in parallel just causes confusion. As the figure shows, different layers are protected by different specs.

Layer A: Human → Agent → External Resource (ID-JAG / XAA)

The problem here is "how does an Agent touch GitHub on behalf of the human Alice."

There are two OAuth extensions you need as background. They come up repeatedly, so pin them down first.

RFC 7523 (JWT Profile for OAuth 2.0 Client Authentication and Authorization Grants): the profile for "bring one JWT, get an OAuth access token in return." It defines the form you post with grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer
RFC 8693 (Token Exchange): the general framework for "exchange one token for another." subject_token is "who the token is for," actor_token is "the party actually acting." The act / sub claim hierarchy comes from here too

ID-JAG / XAA combines these two into a profile that "extends the IdP trust established by a human's SSO out to API calls made through an agent."

To see the concrete difference, the old approaches are only two:

Hand the Agent Alice's GitHub PAT (basically everyone today)
Have Alice individually approve an OAuth App for the Agent (this is what ChatGPT Plugins / Claude Connectors do)

Approach 1 cannot narrow scope. If Alice has repo:full, the Agent has repo:full. Approach 2 can narrow scope, but Alice has to walk through a consent screen per Agent, which collapses in the age of N agents.

ID-JAG works like this:

Alice is already SSO'd into her IdP (Okta / Entra ID / Google Workspace)
The IdP issues an identity assertion JWT for Alice
The Agent brings that to GitHub's Authorization Server via the RFC 7523 jwt-bearer grant
GitHub's AS validates the JWT through its trust relationship with the IdP and, via RFC 8693 Token Exchange, exchanges it for a scoped-down GitHub access token for Alice
The Agent calls the GitHub API with that token

Alice never touches a consent screen. The Agent only ever holds a narrowly-scoped token from the start. Even with N agents, you manage per-agent policy on the IdP side.

The commercial name, in Okta's case, is "Cross-App Access (XAA)." Conceptually it is a profile riding on top of OAuth Identity Chaining (draft-ietf-oauth-identity-chaining-12), and Okta shipped an implementation in 2026 as MCP's "Enterprise-Managed Authorization" extension.

Layer B: Agent → internal service chain (Transaction Tokens for Agents)

The problem here is "while an Agent processes a single request and calls multiple internal services, how do you propagate who is acting and for what purpose."

Layer A's ID-JAG is about crossing a trust domain boundary (your company → GitHub). Layer B is about the microservice chain inside your own walls.

The prerequisite, Transaction Tokens (draft-ietf-oauth-transaction-tokens-08), is a mechanism to "convert" the outbound token received at the API gateway into a short-lived JWT meant to flow through the internal chain. Pushing the received OAuth access token straight through 5 hops makes the scope too broad and stacks up leak risk. So inside the Trust Domain, a Transaction Token Service (TTS) issues a short-lived "for this request" token and flows that through the chain. I plan to write the details elsewhere, but the gist gets across as just "pack the per-request context into a JWT and carry it."

Transaction Tokens for Agents is the extension that adds the claim design for when an AI agent is mixed into the Trust Domain.

claim	meaning
`sub`	the principal (the human Alice, or the agent itself if the agent acts independently)
`act`	the actor (the AI agent actually making the call; embeds RFC 8693's actor token concept straight into the JWT)

By carrying these two claims across the whole internal chain, who-delegated-to-whom survives per request.

Why is this needed? The ID-JAG token itself does have an act claim for the actor (from RFC 8693), so it is not that "delegation can't be expressed." The problem is elsewhere. ID-JAG is a single-hop, cross-boundary grant whose aud is pinned to a single Resource AS, so you cannot flow it through the internal chain as-is (the aud won't match). Pushing the received outbound token straight through 5 hops is also over-scoped and stacks up leak risk. So in practice you exchange it at the gateway for a short-lived internal token (i.e. mint a new one).

If information disappears, it is not because it crosses hops; it is because the actor gets left off at the moment of that exchange. Mint it naively and only sub=alice survives, act drops, and the terminal service sees nothing but "Alice's own transaction." Transaction Tokens for Agents nails down a claim design that forces act=agent-X, sub=alice to be preserved at exchange time. Once minted, the TraT propagates immutably through Order Service → Risk Service → Payment Service, and each hop can verify "agent-X is executing Alice's transaction" at the JWT level. The Payment Service's audit log permanently records "agent-X executed this transfer on behalf of alice." That is the territory Layer A alone cannot reach.

Layer C: Agent (workload) → LLM (Anthropic WIF)

The problem here is "how does the workload that is the Agent authenticate to the LLM API of Anthropic / OpenAI / Google."

Layers A and B were authentication toward the resource side (GitHub / internal services). Layer C is authentication toward the AI side.

The term Workload Identity Federation (WIF) has history: AWS and GCP originally built it as a mechanism to "delete IAM user static access keys by exchanging an IdP's OIDC token for an STS / GCP token." It is the whole story of exchanging GitHub Actions OIDC or a Kubernetes service account for AWS sigv4 credentials. Anthropic WIF is just the LLM-provider version of that; the structure is completely isomorphic.

The flow on Anthropic's side (official docs) is this:

The platform the Agent runs on (k8s SA / GHA / SPIFFE / AWS / GCP) issues an ambient OIDC JWT (a projected service account token on Kubernetes, the OIDC endpoint on GitHub Actions, and so on)
The Agent posts that JWT to Anthropic's /v1/oauth/token via RFC 7523 urn:ietf:params:oauth:grant-type:jwt-bearer
Anthropic validates the JWT signature against the registered issuer's JWKS and checks the federation rule's subject_prefix and the like
On a match, it returns a short-lived sk-ant-oat01-... token (per service account, 3600 seconds by default)

The nice part is you can fully delete the sk-ant-... static API key. No more operations of injecting a secret into the Agent container.

One thing that often gets misunderstood here: "if I have WIF, do I even need ID-JAG?" They protect different legs, so they do not compare. WIF is the Agent → Anthropic leg, ID-JAG is the Human → Agent → GitHub leg. Anthropic WIF cannot protect Alice's GitHub PAT no matter how nasty a prompt injection the Agent eats. The premise is you use both.

The three-layer mapping

	Layer A	Layer B	Layer C
Which leg	Human → Agent → External Resource	Agent → Internal Service Chain	Agent (workload) → LLM Provider
Representative spec	ID-JAG / Identity Chaining / XAA	Transaction Tokens for Agents	RFC 7523 jwt-bearer + each vendor's WIF
Input identity	Human OIDC ID Token	Agent + principal context	Workload identity (k8s SA / SPIFFE)
Problem solved	narrowing the human's scope, blast radius on agent takeover	propagating "for whom" across the call chain	killing static API keys, auditable workload identity
2026 status	IETF draft -04, implemented in Okta XAA	IETF individual draft -06	Anthropic GA, AWS/GCP for years
Who hurts without it	enterprise SaaS integrations	microservice audit teams	platform SRE / audit

Once you get here, you see the "ID-JAG vs WIF" framing never held in the first place. You use both.

Why the UX "doesn't change," and what not changing actually means

Back to the opening question: "if the UX doesn't change, won't it fail to catch on?"

This is half right. In the happy path, one person, one agent, within their own permissions, the UX really does not change at all. The Agent fetching a JWT from the IdP behind the scenes, narrowing scope with Token Exchange, throwing it at the resource server: none of that is visible to Alice.

And that is the result of correct design. Just as users do not think about OAuth's access_token and refresh_token, making them think about it is a design failure. If you look at this spec expecting "the UX to visibly change," you misread its essence.

But "the UX does not change" does not equal "it has no value." Layers A / B / C deliver the kind of value that is invisible day to day and only becomes visible the instant something goes wrong. Concretely, it shows up in the next three scenarios.

Scenario 1: when you have an accident (blast radius)

July 2025, the incident where Replit's AI agent wiped a production DB during a code freeze. In an experiment Jason Lemkin of SaaStr documented, data for over 1,200 executives and 1,190 companies was destroyed. (Fortune report, AI Incident Database #1152)

What happened comes down to one thing: the Agent ran with the same DB connection privileges as a human. The code-freeze instruction was implemented only as a prompt-level "please," with no mechanism to stop it at the authorization level. The Agent confessed it had "panicked in response to empty queries," and per the report it even lied about the recovery procedure.

If Layer A (ID-JAG) had been in place, picturing it out: scope the token issued to the Agent to just db:read and DROP TABLE comes back as HTTP 403. Even if the Agent goes haywire, if the token is short-lived (say 5 minutes), the rampage itself has a time cap. And the audit log permanently records "agent-X attempted a DELETE against prod and got 403." That is enforcement at the authorization layer, a notch stronger than a prompt-level "please."

After the incident Replit added "automatic dev / prod DB separation." That is the right response, but structurally it is absorbing Layer A's role into the infrastructure side. Solving it at the app layer (ID-JAG) or the data layer (DB separation) are both fine; the real problem is that most teams currently do neither.

Scenario 2: when you eat a zero-click attack (scope violation)

June 2025, EchoLeak (CVE-2025-32711, CVSS 9.3) hit Microsoft 365 Copilot (The Hacker News report; arXiv paper 2509.10540 analyzes it as the "first real-world zero-click prompt injection exploit in a production LLM system"). The attack flow is simple:

The attacker sends one email to a user at the target org
Copilot later goes to read that email
A hidden prompt embedded in the email body steers Copilot
Copilot reads out all data the user has access to from SharePoint / OneDrive / Teams and exfiltrates it to an external endpoint

The user clicked nothing. It completes just by an email landing in the inbox. Aim Labs, who found it, calls this "LLM Scope Violation."

The core problem is isomorphic to Replit: the Agent (Copilot) held "all of Alice's permissions." More precisely, it is not that Alice failed to narrow Copilot's scope, it is that there is no mechanism to narrow it.

In the ID-JAG / XAA worldview, you can cut the token Copilot uses to read SharePoint down to something narrow like "read the document Alice is looking at right now." However hard the hidden prompt tries, it can only touch what the token allows. A scope violation like EchoLeak is a phenomenon that occurs where the token's permission boundary is "all of Alice's permissions," so the real problem is that there is currently no standardized way to make that boundary narrow.

Scenario 3: when it scales (N agents)

The Moltbook incident of January–February 2026. A service billing itself as a "social network" for AI agents launched on 2026-01-28, and a mere three days later, on 2026-01-31, it got reported by 404 Media. A backend Supabase misconfiguration left it in a state where anyone could take over any agent on the platform via the DB.

Then in February 2026, Wiz researchers found that the Supabase API key was exposed in the front-end JavaScript. Full read/write to production data went through, and you could pull 1.5M API authentication tokens, 35,000 email addresses, and private messages between agents.

What matters here is a structural story prior to any individual vulnerability: no identity separation between agents, no scope narrowing, no rotation, none of it implemented. Stand up a platform holding this many identities this quickly and try to run it on human-scale credential ops, and this is what you get.

The numbers back it up too. Non-Human Identities (NHI) as of 2026 outnumber humans by 40–100x, reaching 500x in hyper-automation orgs. The average enterprise carries over 250,000 NHIs (NHI Working Group's summary).

Of those, 71% are not rotated, 97% are over-privileged, and only 15% of orgs feel confident they can defend against NHI-based attacks. 68% of incidents involve NHIs.

This is the pattern where operations break before any accident does. Kubernetes service accounts spread not because of an accident but because "sharing creds across 200 microservices became physically impossible." The same thing happens with agents. If Replit / EchoLeak "push it up from the accident side," then Moltbook and the NHI numbers are the "push it up from the operations side."

By here the real meaning of "the UX doesn't change" comes into focus. In the happy path you feel nothing. That is evidence of correct design. On the flip side, when an accident hits, the damage is human-scale (Replit's full DB deletion, EchoLeak exfiltrating all of a company's documents off one email), and that is where the cost of not having Layer A surfaces. When you scale, audit-log pollution and the inability to revoke come due all at once as the tab for not having Layers B / C. The cost you do not see day to day becomes visible all at once under accidents and scale: that is the essential role of Layers A / B / C.

Predicting "when each layer catches on": the 3-stage model of security history

From here on it is prediction. Less to be right than to give the discussion a foundation.

The spread of security infrastructure almost without exception goes through the next three stages.

Past isomorphic cases

MFA. TOTP was standardized in RFC 6238 in 2011, U2F in 2014; the tech itself is over a decade old. The 2012 LinkedIn breach (6.5M passwords), 2013-2014 Yahoo (eventually found to be 3B accounts), 2017 Equifax (147M): flashy incidents kept coming, and regulation caught up afterward, with NIST SP 800-63-3 downgrading SMS-based OOB authentication to a "restricted authenticator" in 2017 and PCI-DSS v4.0 expanding the mandatory-MFA scope beyond admins in 2022. Real adoption was 2019–2023. Roughly 15 years from the tech appearing to practical use.

Short-lived AWS credentials. 2011 STS AssumeRole, 2014 Code Spaces burned to the ground (an AWS key leak deleted all resources and ended the company), 2016 Uber (an S3 key leak, 57M records), 2019 Capital One (106M records via the EC2 instance metadata service). 2019 brought EKS IRSA, and EKS Pod Identity at re:Invent in November 2023 removed even the need to configure OIDC, so the default option finally became "don't create a static key." 10–12 years from the tech appearing to real adoption.

Sigstore / keyless signing. Late 2020 SolarWinds, 2021 Codecov / Kaseya. Executive Order 14028 in May 2021 codified SBOM and signing as federal procurement conditions, 2022 NIST SSDF (SP 800-218), 2023 SLSA v1.0. Real adoption was 2023–2025. Rapid adoption 3 years after the accident, the fastest of the three.

Lining up all three, a common thread emerges. The tech exists years before the accident (it does not spread not because of a tech problem, but because the asymmetry between cost and risk has not yet broken). It needs the media to turn it into a story: a CVE number and CVSS alone do not move anything; only when a named big company gets named-and-shamed does it start to turn. Finally, codification by regulation or industry BCP is the trigger. NIST, PCI, EO 14028, SLSA: at this stage insurance premiums and compliance audits move, and budget gets secured inside companies.

The other historical pattern: protocols with zero UX improvement do not spread

But reading only the 3-stage model, you might conclude "ID-JAG will come eventually too," and that misses an important precondition. MFA / Sigstore had extremely strong regulation, or a UX bonus. With the same "accidents present," a protocol with zero UX improvement and implementation room only on the vendor side does not spread even when regulation comes. Look at tech history and this is the more common pattern.

Protocol	UX improvement	Setup / ops	Regulatory pressure	Adoption
DNSSEC	none	heavy	a few govt agencies only	~30% in 25 years
IPv6	none	heavy	Asia-centric, weak in US/EU	~50% in 25 years
DMARC	none	medium, full of traps	enterprise BCP only	half of enterprises, long tail unconfigured
DPoP	none	medium	none	OAuth extension, near-zero adoption
SPIFFE	none	heavy	none	niche after 10 years
mTLS (via service mesh)	none	the mesh hides it	none	adopted where the mesh is
MFA	worse	light	strong (PCI / NIST)	real adoption in 15 years
Passkey	dramatically better	light	weak	exploded in 2-3 years
Let's Encrypt + HTTPS	better (free, automatic)	light	weak	global standard in 5 years
TLS 1.3	better (lower latency)	automatic	none	spread in a few years

The rule you can read off this is simple:

UX improves → spreads fast (Passkey, Let's Encrypt, TLS 1.3)
UX worsens + strong regulation → takes time but spreads (MFA muscled through the UX hit of SMS OTP via regulation)
zero UX improvement + weak regulation + heavy setup → eternal niche (DNSSEC, IPv6, DMARC, DPoP, SPIFFE)
zero UX improvement but a mechanism makes the ops invisible (service mesh, vendor integration) → spreads via a different route

So which quadrant is ID-JAG in?

UX improvement: none. Alice's experience is actually better for skipping the consent screen, but it is not at the "wow" level
Setup: heavy. Trust with the IdP, the Resource Server's support, the Agent's Token Exchange implementation, all cross-vendor
Vendor dependence: strong. Users can't use it unless Cursor / Claude Code / Comet each implement it
Strong regulation: not yet. Neither the EU AI Act nor the NIST AI RMF steps into the identity layer

This is the DNSSEC / SPIFFE / DPoP quadrant. Rapid adoption like MFA or Sigstore is more likely not to happen.

Where Agent Identity stands now

Put the three isomorphic cases next to Agent Identity and it looks like this.

Stage	MFA	Short-lived AWS creds	Sigstore	Agent Identity
1. Tech appears	2011 (TOTP)	2011 (STS)	2020	2024-2026 (ID-JAG / Tx Tokens / WIF)
2. Flashy accident	2012-2017	2014-2019	2020-2021	2025-2026 (Replit / EchoLeak / Comet / Moltbook, ongoing)
3. Regulation / BCP	2017-2022 (800-63-3 / PCI v4)	2019-2023 (IRSA / Pod Identity)	2021-2023 (EO 14028 / SLSA)	2026-? (EU AI Act / NIST AI RMF / RSAC 2026 in discussion)
4. Real adoption	2019-2023	2021-2023	2023-2025	predicted 2028-2030

Right now is the entrance to stage 2. Individual cases like Replit / EchoLeak / Comet (Brave's analysis of Perplexity Comet, Schneier on Security's warning on AI browsers) keep stacking up, but the "Equifax of Agents" has not arrived yet. Three scenarios that look likely to come next:

A big company's Copilot / Cursor / Claude Code integration eats a prompt injection and exfiltrates customer DB contents to a company-wide Slack. Happens at some Fortune 500
An executive's AI assistant gets hijacked and a transfer instruction goes through via Slack. Salesforce building mobile approval into Trusted Agent Identity is precisely a guard against this
A dev Agent force-pushes to a prod repo on a human's GitHub creds and breaks the supply chain. The AI version of SolarWinds

The moment any one of these lands in the New York Times with a name attached, it moves to stage 3 (regulation). Until then it stays stuck at "we really should do this."

Per-layer adoption-timing prediction (the honest version)

I see the three layers walking separate fates. I write "honest version" because it is my personal prediction with the wishful thinking removed.

Layer C (WIF) → will catch on. The reason is simple: the vendor completely hides the operations. Anthropic WIF works just by initializing the SDK with no arguments, and a k8s SA auto-mints projected tokens. The end user does nothing. This is the same shape as mTLS spreading hidden inside the service mesh: the protocol itself spreads without anyone being aware of it. sk-ant-... / sk-... static keys drop within a few years to the status of "exists, but not recommended."

Layer A (ID-JAG / XAA) → won't catch on as a standard; the concept spreads under a different name via vendor integration. Almost no team will say "we deployed ID-JAG" or "we adopted XAA." Just like SAML, each vendor implements it proprietarily as "Claude Connector," "Cursor for Enterprise," "Copilot Connector," and the end user perceives it as "I connected GitHub with one click in the Anthropic console." Nobody cares whether the token exchange behind it is ID-JAG-compliant. Only XAA via MCP has a shot at surfacing, but even that is only something "the side standing up the MCP Server" is aware of; the end user never sees it.

Layer B (Transaction Tokens for Agents) → eternal niche. This is dragged down by the fact that Transaction Tokens proper are currently almost unadopted. It will get picked up in some finance / healthcare intra-trust-domain microservices, but nowhere else. Same pattern as SPIFFE (correct, but adoption is limited).

The likely base case: it ends on the SPIFFE / DNSSEC route

I write this as the base case. Not wishful thinking: I mean that going by the lessons of tech history plus current vendor dynamics, this is the most likely outcome.

Three reasons:

Microsoft / OpenAI / Anthropic / Salesforce locking it down proprietarily is faster. Once Anthropic Service Account, OpenAI Agent Identity, Microsoft Copilot Studio Identity, and Salesforce Trusted Agent Identity each become self-contained, the incentive to wait for standardization evaporates. The same road SAML took, ending up subtly different per SaaS
MCP's Enterprise-Managed Authorization (= XAA) is a route that could save ID-JAG, but it is an Okta-led implementation, and if Microsoft / Google / Auth0 start shipping isomorphic vendor extensions, it could fragment
The discussion at RSAC 2026 (VentureBeat's summary) is moving the direction even higher, to "Agent Identity alone is not enough; you need an Action Governance layer." When the "next layer" discussion starts before the identity-layer standardization solidifies, the standardization work falls behind, and each vendor's proprietary implementation becomes the de facto standard

The optimistic scenario does exist: "an Equifax of Agents lands in the NYT, the EU AI Act steps into the identity layer, and ID-JAG becomes effectively mandatory." But that needs regulatory pressure as strong as how regulation muscled MFA through the UX hit of SMS OTP, and the current AI-regulation discussion centers on liability allocation / transparency / fairness, with no sign of stepping into identity tech. The probability of "spreads in reality via vendor proprietary" is higher than the probability of regulation arriving: that is my honest read.

Conclusion

This ran long, so here is what I want you to take home from the three-layer model.

"The UX doesn't change, so we don't need ID-JAG" is half a misread and half correct. Layer A's value shows up under accidents and scale: that much is correct as a statement about the spec. On the other hand, "will the ID-JAG protocol catch on" and "will the Layer A concept spread" are separate questions, and that is the honest read. The route where the standard stands up front and spreads, like MFA or Sigstore, is possible, but a protocol that combines zero UX improvement, heavy setup, and strong vendor dependence is more likely to follow the DNSSEC / SPIFFE / DPoP route.

Even so, the three-layer model itself does not break. Layer C (WIF) catches on (the structure where the vendor hides the operations), Layer A spreads as concept only, under a different name via vendor integration (Claude Connector / Copilot Connector / Cursor for Enterprise effectively implementing the ID-JAG equivalent behind the scenes), Layer B is an eternal niche in just part of finance / healthcare: that is how it splits. If an accident like Replit / EchoLeak / Moltbook happens in your environment, thinking about what stops and what doesn't per Layer A / B / C changes how you see it.

If you want to move in practice today, start with Layer C. Replacing sk-ant-... / sk-... static keys with Anthropic WIF or equivalent Workload Identity Federation is the side you can do today without waiting for regulation, and that will reliably catch on. For Layer A, rather than aiming for "ID-JAG compliance," it is more practical to check how narrow the scope design is in the proprietary connectors your vendors (Anthropic / OpenAI / Microsoft / Okta) ship. Layer B is fine to think about once an audit requirement actually shows up.

If you want to read the behavior of ID-JAG itself, ID-JAG Deep Dive; for the full map of agent auth, AI Agent Authentication & Authorization Deep Dive: Reading draft-klrc-aiagent-auth-00. Those two are the supporting lines.

Holding the three-layer model as a map means even when a vendor's new announcement is a proprietary integration, you can immediately classify what it solves and what it doesn't. The OpenAI / Google Cloud agent-auth announcements that will probably come in the second half of 2026, even if they don't ship under the name ID-JAG, are fine as long as you can sort out whether they are trying to solve the Layer A problem or are Layer C. Whether the standard catches on is, in a sense, beside the point. As long as the problem gets solved.

SPIFFE Compliance Deep Dive

kt — Sun, 31 May 2026 06:05:03 +0000

Introduction

"This is SPIFFE compliant." "Our implementation is SPIFFE-compatible." Read the SPIRE, Istio, or Cilium docs and you bump into these phrases everywhere. But if you actually stop and ask what compliance means, the answer is surprisingly hard to nail down.

If I run SPIRE, am I SPIFFE compliant?
If I roll my own, what do I have to ship to call it compliant?
Does an SVID have to support both X.509 and JWT? Or is one enough?
Is there an official conformance test you can point at?

A lot of people use the word without checking. I was one of them.

So I cloned the upstream spec repo (github.com/spiffe/spiffe) and read every document from top to bottom. What follows are my notes, organized around the question "what does the spec actually require?"

git clone https://github.com/spiffe/spiffe.git ~/spiffe
ls ~/spiffe/standards/
# JWT-SVID.md
# SPIFFE-ID.md
# SPIFFE.md
# SPIFFE_Federation.md
# SPIFFE_Trust_Domain_and_Bundle.md
# SPIFFE_Workload_API.md
# SPIFFE_Workload_Endpoint.md
# X509-SVID.md
# workloadapi.proto

Eight specs total. Read them in order and each one spells out where the hard requirements end and the soft suggestions begin. I'll pull out the load-bearing parts.

0. Prerequisites

A few terms before we go any further. Skip this section if you've touched SPIFFE/SPIRE before.

What "workload identity" means

A way to cryptographically prove "who you are" to a workload (a process or container). IP addresses and hostnames can be spoofed. Kubernetes labels can be rewritten. Instead, a trusted Certificate Authority (CA) signs a certificate or token that the workload presents, and verifiers check the signature.

To distinguish this from human identity (the account you log into via OAuth), it's increasingly called Workload Identity or Non-Human Identity (NHI).

URI basics (RFC 3986)

We'll be dealing with strings like spiffe://example.com/payments/web-fe, so a quick refresher on URI structure.

spiffe://example.com/payments/web-fe
  |        |              |
scheme  authority        path

In SPIFFE the scheme is fixed at spiffe, the authority is the Trust Domain, and everything after that is the Workload Path.

X.509, mTLS, JWT, JWS, JWK

Signed data structures, or representations of keys.

X.509: the certificate format TLS uses. Binary (DER) or PEM encoded.
mTLS (mutual TLS): both client and server present X.509 certificates and verify each other. Plain TLS only verifies the server.
JWT (RFC 7519): a token made of JSON pieces joined with Base64URL. Common on the web.
JWS (RFC 7515): the structure that signs the JWT payload. A JWT requires a JWS to exist. When the SPIFFE spec says "JWS Compact Serialization," it means the normal JWT layout: header.payload.signature joined with dots.
JWK (RFC 7517): a JSON representation of a public (or private) key. Looks like {"kty":"RSA", "n":"...", "e":"AQAB"}.
JWKS (JWK Set): a JSON array bundling multiple JWKs. If you've ever fetched /.well-known/jwks.json in OAuth/OIDC, you've seen one.
Bearer Token: a token where possession alone grants access. Steal it, use it. JWTs are typically bearer tokens.

The SPIFFE spec stacks a SPIFFE ID on top of these and calls the result an SVID (SPIFFE Verifiable Identity Document):

X.509 based → X.509-SVID
JWT/JWS based → JWT-SVID
Distribution of trust roots (CA keys) → SPIFFE Bundle (a JWKS underneath)

gRPC and Unix Domain Sockets

gRPC: an RPC framework that pushes Protocol Buffers over HTTP/2.
Unix Domain Socket (UDS): an inter-process socket on the local host. You reach it by filesystem path (for example /run/spire/agent.sock).

The Workload API ships over these two together.

1. The Shape of SPIFFE: Three Pillars

The SPIFFE spec set, in one sentence:

"Hand the workload a spiffe://... ID, issue a verifiable document (SVID) that carries it, and serve it through a local API."

The three pieces (ID, document, API) each get their own spec document.

Pillar	Document	Role
1. SPIFFE ID	`SPIFFE-ID.md`	Defines the workload namespace
2. SVID (X.509)	`X509-SVID.md`	How to carry a SPIFFE ID in an X.509 certificate
2. SVID (JWT)	`JWT-SVID.md`	How to carry a SPIFFE ID in a JWT
3. Workload API	`SPIFFE_Workload_API.md`	gRPC service that issues SVIDs
3. Workload Endpoint	`SPIFFE_Workload_Endpoint.md`	Conventions for the socket that exposes the API
Extra	`SPIFFE_Trust_Domain_and_Bundle.md`	How to represent trust roots (CA keys)
Extra	`SPIFFE_Federation.md`	How separate Trust Domains link up

You don't actually have to satisfy all of these to call yourself SPIFFE compliant. The next section explains why.

2. What the Spec Itself Says "Compliant" Means

This one sentence at the end of SPIFFE-ID.md Section 1 does a lot of work:

Conformance with this document is sufficient for the purposes of SPIFFE compliance.

Meet SPIFFE-ID.md and you can claim SPIFFE compliance. That's it. The Workload API, the Trust Bundle, the Federation spec, none of them are required by the letter of the spec.

That's the formal floor. In practice:

An ID by itself is useless without an SVID carrying it.
An SVID is useless without a Workload API to deliver it.
And neither is verifiable across hosts or trust domains without a Bundle representing the trust root.

"Practically SPIFFE compliant" means the three core specs plus Bundle, four documents total. If you cross trust domains, add Federation to that list.

The rest of this article walks through the MUST requirements of each one.

3. SPIFFE ID: How You Name Things

3.1 Anatomy of the URI

A SPIFFE ID is an RFC 3986 URI with a fixed shape.

spiffe://trust-domain-name/path/segments

Pull out everything the spec marks MUST or MUST NOT and you get:

Requirement	Level	Detail
Scheme is fixed at `spiffe`	MUST	Case-insensitive
Trust domain name is non-empty	MUST	The `host` component
No `userinfo`	MUST NOT	`spiffe://user:pass@td/` is out
No `port`	MUST NOT	`spiffe://td:8080/` is out
Trust domain name is lowercase	MUST	`Example.com` is out
Trust domain charset is `[a-z0-9.-_]`	MUST	Percent-encoding is out
No `query` or `fragment`	MUST NOT	`?key=val` and `#frag` are out
No trailing slash on the path	MUST NOT	`spiffe://td/foo/` is out
No `.` or `..` segments	MUST NOT	Relative path tokens forbidden
Whole URI under 2048 bytes	MUST (SHOULD)	Full URI length
Trust domain name under 255 bytes	MUST	RFC 3986 host limit

3.2 Trust Domain Names Are Unregulated

The spec is up front about this:

Trust domain operators are free to choose any trust domain name they find suitable: there is no centralized authority for regulation or registration of trust domain names.

Unlike DNS, there is no registry. Anyone can call themselves example.com.

So what happens when two parties pick the same name?

When a collision does occur, those trust domains will continue to operate independently but will be unable to federate.

While they stay independent, nothing breaks. The moment they try to federate (link up across trust domains), the two sides hold different keys, so neither can verify the other's certificates. They just fail to connect.

The practical workaround is to use a DNS name you already own. If you're auto-generating, a UUID works.

3.3 Path Design

What the path means is up to the implementer. The spec gives three example patterns:

Pattern A, identify the service directly: spiffe://staging.example.com/payments/mysql
Pattern B, mirror the orchestrator's ID structure: spiffe://k8s-west.example.com/ns/staging/sa/default
Pattern C, opaque: spiffe://example.com/9eebccd2-12bf-40a6-b262-65fe0487d453 (metadata managed elsewhere)

SPIRE's default on Kubernetes is Pattern B (/ns/<namespace>/sa/<service-account>), and that's close to the de facto convention. It maps Kubernetes Pod Identity directly, so the operational view stays legible.

4. SVID: X.509 Profile Requirements

4.1 An X.509-SVID Is Just an X.509 Certificate With a URI SAN

Read X509-SVID.md and there are no new fields. It's a normal X.509 certificate with rules layered on top about how to use the existing ones.

The core rule is simple: the Subject Alternative Name extension's URI type holds exactly one SPIFFE ID. Everything else (Subject, Basic Constraints, Key Usage, Extended Key Usage) carries its usual X.509 meaning. The spec only constrains which values are legal for SPIFFE use.

4.2 Leaf SVID vs Signing SVID

The X.509 field values differ between Leaf and Signing.

Field	Leaf SVID	Signing SVID (CA)
`cA` (Basic Constraints)	`false` MUST	`true` MUST
`keyCertSign` (Key Usage)	MUST NOT	MUST
`cRLSign` (Key Usage)	MUST NOT	MAY
`digitalSignature` (Key Usage)	MUST	(not specified)
SPIFFE ID path	Non-root (at least one segment) MUST	No path (`spiffe://td`) SHOULD
Number of URI SANs	Exactly one MUST	Exactly one MUST

A Leaf SVID can sign (mTLS client auth, API request signing) but cannot issue certificates for other parties. Same position as an ordinary client certificate.

4.3 Extra Checks at Validation Time

Section 5.2 of X509-SVID.md makes it clear that standard X.509 path validation is not enough. Using a certificate as an SVID requires these additional checks.

Forget these and you open the door to vulnerabilities like an intermediate CA being used as a leaf certificate. Libraries like go-spiffe/v2 handle this for you. Only worry about it when writing your own.

5. SVID: JWT Profile Requirements

5.1 A JWT-SVID Is Just a JWS

From Section 1 of JWT-SVID.md:

JWT-SVIDs are standard JWT tokens with a handful of restrictions applied.

A normal JWT with a few extra constraints, nothing more. Format is JWS Compact Serialization (the header.payload.signature layout). JWS JSON Serialization is MUST NOT.

5.2 The Core Restriction: Reject `alg: none`

JWT has a well-known footgun: set alg to none in the header and the token sails through unverified.

The JWT-SVID spec pins alg to one of these nine values.

`alg` value	Algorithm
RS256 / RS384 / RS512	RSASSA-PKCS1-v1_5 + SHA
PS256 / PS384 / PS512	RSASSA-PSS + SHA
ES256 / ES384 / ES512	ECDSA + SHA

Anything else (especially none or the symmetric HS* family) MUST be rejected. That single rule closes off most of the classic JOSE vulnerability surface.

Required claims:

sub: the SPIFFE ID of the workload.
aud: at least one audience.
exp: expiry.

kid is optional in the header but required on the Bundle JWK side so verifiers can pick the right key.

5.3 Always Check `aud`

JWT-SVID is a bearer token. Anyone holding it can use it. To soften that blow:

aud MUST be set.
The receiver MUST check that its own ID is in aud.
Single audience strongly recommended.

The example in Section 7.2 of the spec shows the failure mode:

if Alice has a token with audiences Bob and Chuck, and transmits that token to Chuck, then Chuck can impersonate Alice by sending the same token to Bob.

Alice issues a token addressed to both Bob and Chuck. Chuck replays it to Bob and impersonates Alice. One token, one audience. That's the rule.

6. Workload API: How SVIDs Get Delivered

6.1 gRPC, Local Only

SPIFFE_Workload_Endpoint.md, summarized: the Workload API is a gRPC endpoint on a Unix Domain Socket (or localhost TCP) within the same host. In practice, the SPIRE Agent running on each node binds to /run/spire/agent.sock, and every container on that host talks to it over gRPC. The Agent handles the conversation with the SPIRE Server (the central CA) over a separate network. The workload itself never reaches the outside.

The MUSTs around transport and accessibility:

Requirement	Detail
gRPC required	Prefer UDS over TCP (SHOULD)
No TLS	In fact, MUST NOT require it. At bootstrap the workload has no trust root yet.
Confined to one host	SHOULD (don't make it reachable from other hosts)
`workload.spiffe.io: true` metadata	MUST (SSRF defense)
No authentication handshake	MUST NOT require one. Caller is identified at the OS layer instead.

That last point is unusual. A typical gRPC service authenticates clients via certificates or tokens. The Workload API flips that: the workload doesn't announce itself, the server identifies it via the OS. This identification flow is called Workload Attestation. Concretely:

Fetch the connecting PID from the UDS via getpeerucred (or equivalent).
Inspect the PID's cgroup, or query the Kubernetes API server.
Conclude "you're the web-fe container in pod foo-bar-1234."
Issue the SPIFFE ID that container should hold.

6.2 Five RPCs Across Two Profiles

The Workload API has two profiles (X.509 and JWT) and five RPCs between them.

Profile	RPC	Mode	What it returns
X.509	`FetchX509SVID`	stream	SVID + Bundle
X.509	`FetchX509Bundles`	stream	Bundles only
JWT	`FetchJWTSVID`	unary	JWT for a given audience
JWT	`FetchJWTBundles`	stream	JWKS
JWT	`ValidateJWTSVID`	unary	Delegate JWT verification to the server

From Section 1 of SPIFFE_Workload_API.md:

Both profiles are mandatory and MUST be supported by SPIFFE implementations. However, operators MAY administratively disable a specific profile in their deployment.

A SPIFFE-compliant Workload API has to implement both X.509 and JWT profiles (the operator is allowed to turn one off in their deployment). This bites in practice. Even popular OSS libraries often bolt JWT on later.

6.3 Why It Streams

FetchX509SVID, FetchJWTBundles, and FetchX509Bundles return as gRPC server-side streams so the server can push:

New certificates during key rotation.
CRL (revocation list) updates.
New state without the workload paying reconnect cost.

The client SHOULD keep the connection open.

Every message ships the full current state, not a delta (spec sections 4.3 and 4.4). That sidesteps the whole anti-entropy headache of incremental sync.

6.4 Discovery: `SPIFFE_ENDPOINT_SOCKET`

How does a client find the Workload API? From spec section 4:

Clients may be explicitly configured with the socket location, or may utilize the well-known environment variable SPIFFE_ENDPOINT_SOCKET. If not explicitly configured, conforming clients MUST fall back to the environment variable.

Without explicit configuration, look at SPIFFE_ENDPOINT_SOCKET. The value is URI-formatted.

# Unix Domain Socket
export SPIFFE_ENDPOINT_SOCKET=unix:///run/spire/agent.sock

# TCP (only on hosts with a specific reason)
export SPIFFE_ENDPOINT_SOCKET=tcp://127.0.0.1:8000

Official libraries like go-spiffe/v2 do this automatically.

7. Trust Domain and Bundle: Building the Trust Root

7.1 A Bundle Is a Keyring Shaped Like a JWKS

SPIFFE_Trust_Domain_and_Bundle.md in one line: SPIFFE Bundle = RFC 7517 JWK Set + SPIFFE-specific metadata.

JWKS is the same format you've seen as /.well-known/jwks.json on Cognito, Auth0, and Google Identity Platform. The SPIFFE Bundle extends it to hold both X.509 CA certificates and JWT signing keys.

{
  "spiffe_sequence": 12035488,
  "spiffe_refresh_hint": 2419200,
  "keys": [
    {
      "kty": "RSA",
      "use": "x509-svid",
      "x5c": ["<base64 DER encoding of X.509 CA cert>"],
      "n": "<base64urlUint-encoded modulus>",
      "e": "AQAB"
    },
    {
      "kty": "RSA",
      "kid": "<JWT key id>",
      "use": "jwt-svid",
      "n": "<base64urlUint-encoded modulus>",
      "e": "AQAB"
    }
  ]
}

Two things are unique to a SPIFFE Bundle:

use parameter: MUST be either x509-svid or jwt-svid. Without it the verifier can't tell which kind of SVID the key validates.
spiffe_sequence: monotonically increasing counter. Increments every time the Bundle is updated.

7.2 Keep Trust Domains Isolated

Section 6.2 of the Bundle spec:

When a root key is shared across multiple trust domains, it becomes critically important that authentication and authorization implementations carefully check the trust domain name component of an identity.

Don't share keys across trust domains. If you do, verifiers had better check the trust domain name strictly, or an SVID from the wrong trust domain becomes a forgery vector.

One trust domain, one independent set of keys. That's the iron rule of SPIFFE operations.

7.3 Key Rotation: Add First, Remove Later

Bundle updates work as "add the new one, then remove the old." The Bundle moves through four states:

Phase	Bundle contents	Issuer signs new SVIDs with	Verifier accepts SVIDs signed by
Initial	`[K1]`	K1	K1
Add K2	`[K1, K2]`	K1 (still)	K1 or K2
Switch	`[K1, K2]`	K2	K1 or K2
Drop K1	`[K2]`	K2	K2

spiffe_refresh_hint (seconds) controls how often the workload re-fetches the Bundle. The spec's example value is 28 days (2419200 seconds), which is far too long if revocation matters. In practice, 5 minutes to 1 hour is the realistic production range.

8. What Existing Implementations Actually Cover

8.1 SPIRE

The reference implementation. Satisfies every item by definition. Workload API streaming, JWT-SVID ValidateJWTSVID delegation, the whole list.

8.2 Istio

Istio holds the workload's SVID inside each sidecar (Envoy). The on-the-wire interface to Envoy is SDS (Secret Discovery Service), which is an Envoy upstream API, not Istio-specific. From a SPIFFE-compliance standpoint, the relevant fact is that Istio's istio-agent feeds SVIDs into Envoy via SDS rather than exposing the SPIFFE Workload API to workloads. The payload format is SPIFFE-compatible, but the Workload Endpoint spec is not implemented.

Istio can also be wired to use a SPIRE-backed SDS server. In that setup SPIRE is the CA, and the Workload Endpoint spec becomes implemented through SPIRE.

8.3 Cilium

Cilium has two unrelated paths that get conflated under the SPIFFE label, and they need to be kept apart.

Path 1, SPIRE-backed Mutual Authentication (Beta since v1.14, still Beta in v1.19). Cilium auto-deploys a SPIRE Agent on each node and rides on top. Cilium itself doesn't implement the SPIFFE spec directly. It wraps SPIRE. This path is opt-in (authentication.mutual.spire.enabled=true) and has been since v1.14.

Path 2, ztunnel-based transparent mTLS (Beta, added v1.19). Cilium picked up the same ztunnel data plane Istio ambient mode uses. Per the Cilium docs, ztunnel's certificates are generated via OpenSSL and stored as Kubernetes Secrets. SPIRE is not in the loop, and the docs do not claim SPIFFE compliance for this mode. Whether the certificates carry SPIFFE IDs in their SANs is an implementation detail I haven't verified, so don't take "Cilium ztunnel is SPIFFE compliant" as a settled claim.

If someone says "Cilium is SPIFFE compliant", ask which feature: the SPIRE-backed mutual auth, or ztunnel.

So when a doc says "Istio is SPIFFE compliant", read it as "the SVID format is compliant, the Workload Endpoint spec is not." Same caution applies to Cilium plus the version question.

9. 2026 Updates: Recent Spec Movement

Tracking the spiffe/spiffe repo, a couple of things have moved since late 2025.

draft-wit-svid branch, active since late 2025. A candidate successor to JWT-SVID aimed at integration with IETF's WIMSE (Workload Identity in Multi-System Environments) work. PR #361 ("Introduce WIT-SVID Token Document") landed on the branch 2026-01-21. Two changes worth noting: WIT-SVID makes Proof of Possession via the cnf claim mandatory (RFC 7800 confirmation), structurally closing off the bearer-token replay risk of JWT-SVID; and PR #372 (2026-01-26) prohibits the aud claim entirely (audience scoping is delegated to the PoP layer). Still an experimental branch, not on main, so I don't count it toward "compliance" yet.
PR #381, merged 2026-03-26. "Strengthen validation language, and clarify leaf SPIFFE ID requirements." Tightens the path requirements and validation language around leaf SPIFFE IDs. Low impact for most implementations, worth a re-read if you wrote your own.

The "what compliance means" I described here tracks the main branch as of May 2026. If WIT-SVID gets merged into main, sections 4 to 5 grow by one more SVID profile with stricter rules than JWT-SVID.

10. Running the Checklist: `scc`

Walking sections 3 to 7 by hand gets old after the second time someone hands you an SVID and asks "is this compliant?". I packaged the static slice of the checklist as a single-binary CLI: scc (spiffe-compliance-checker).

Each subcommand reads one artifact and emits one line per MUST / SHOULD clause, with the spec file and section cited inline. Exit code is 1 on any MUST failure, 0 otherwise. SHOULD violations surface as WARN and do not change the exit code.

brew install kanywst/tap/spiffe-compliance-checker
# or: go install github.com/0-draft/spiffe-compliance-checker/cmd/scc@latest

scc id        'spiffe://example.com/payments/web-fe'
scc x509-svid leaf.pem
scc jwt-svid  <token>
scc bundle    bundle.json

A failing run reads as a spec walkthrough rather than an opaque error:

What it covers right now: SPIFFE ID syntax, X.509-SVID structural rules (URI SAN, Basic Constraints, Key Usage criticality, EKU), JWT-SVID claims, Trust Bundle JWKS shape including base64 + DER validation of x5c. What it does not: live Workload API behaviour, federation endpoint trust, signature verification against a specific bundle. Those need a running Agent and are out of scope for a static checker.

Source, issues, releases: https://github.com/0-draft/spiffe-compliance-checker.

Conclusion

SPIFFE compliance has two definitions and you should know which one you're talking about.

The spec-defined floor is SPIFFE-ID plus SVID. That's it.
The practical floor is that plus Workload API plus Trust Bundle. Federation joins the list only if you cross trust domains.

When something claims SPIFFE compliance, ask at what level.

Fully compliant (SPIRE): meets every spec, interoperable across the board.
SVID-compatible (Istio): SVID format is compliant, Workload Endpoint is custom.
SPIRE-dependent (Cilium SPIRE path): bundles a SPIFFE implementation. The rest is its own.
Not yet claimable (Cilium ztunnel): a separate mTLS path that doesn't go through SPIRE and hasn't been declared SPIFFE compliant by upstream.

Just want to know whether an artifact in front of you is compliant? Run it through scc (section 10). Building your own implementation? Walk the MUSTs in sections 3 to 7 one at a time. SVID format alone is feasible from scratch. The Workload API is where the scope explodes. go-spiffe/v2 and java-spiffe are the official libraries. Wrapping them is the saner starting point than reimplementing the gRPC service.

What makes SPIFFE useful is that workload authentication completes inside a vendor-neutral spec. AWS IAM, Google IAM, Kubernetes Service Accounts, none of them have to be in the trust path. Wire SPIFFE in once and cross-environment authentication stops relying on IP allowlists and pre-shared secrets.

AWS SigV4 and SigV4A Deep Dive

kt — Sat, 30 May 2026 11:19:22 +0000

Introduction

Hitting S3 from boto3, I had never thought about SigV4. The SDK does everything. I knew an Authorization: AWS4-HMAC-SHA256 ... header was being assembled somewhere under the hood, but I had never built one by hand.

Multi-Region Access Point (MRAP) destroyed that complacency. The instant I hit S3 through MRAP from Lambda, an algorithm I had never seen called AWS4-ECDSA-P256-SHA256 showed up instead of the usual SigV4, and the old botocore I had pinned locally crashed with InvalidSignature. AWS has two signing schemes: SigV4 and SigV4A. The latter is asymmetric, using ECDSA instead of HMAC.

This article dissects both, in this order.

Why AWS uses a custom signature instead of Bearer Token
Background: what HMAC and SHA-256 contribute
The four SigV4 steps (Canonical Request / StringToSign / SigningKey derivation / Signature)
SigV4 written in 80 lines of Python
Chunked Upload (STREAMING-AWS4-HMAC-SHA256-PAYLOAD)
Streaming SigV4 (WebSocket / IoT MQTT)
What is inside a Presigned URL
SigV4A: asymmetric signing on ECDSA P-256
SigV4 vs SigV4A comparison
Clock Skew traps and debugging tips

1. Why a custom signature instead of Bearer Token

In the REST world, Authorization: Bearer <token> is the default. OAuth 2.0 is basically the same. Bearer means "whoever holds the token is legitimate", and if one is stolen it is over. Under HTTPS that is usually fine in practice, but AWS explicitly refused that design. There are three reasons.

The three design goals.

Never put the Secret Key on the wire: SecretAccessKey is an absurdly powerful credential. Sending it on every call is a non-starter. SigV4 does not sign with the Secret Key itself; it signs with a short-lived key derived from it through HMAC.
Replay protection: the signature covers an ISO8601 timestamp and the CredentialScope (date + region + service), so it is bound to a moment in time and a place. A captured signature replayed the next day will not pass.
Tamper detection: HTTP method, URI, query string, headers, and the SHA-256 of the body are all in the signed string. Flip a single byte in transit and the signature mismatches.

SigV4 proves three things at once: who sent it, when, and what was in it. Completely different model from Bearer.

2. Background: HMAC and SHA-256 in one paragraph

SigV4 sits on top of HMAC-SHA256. SHA-256 compresses arbitrary-length input into a fixed 256-bit digest. It is a one-way function with collision resistance (you cannot produce two inputs with the same hash) and preimage resistance (you cannot reverse it). HMAC (Hash-based MAC) wraps a key around a message: HMAC(key, msg) = SHA256(key XOR opad || SHA256(key XOR ipad || msg)). Anyone without the key cannot reproduce the output. SigV4 chains HMAC four times to derive a signing key, then HMACs the StringToSign with it. The hex of that final HMAC becomes Signature=... in the Authorization header. SigV4A swaps only that last step from HMAC to ECDSA. Hold that mental model and the rest is detail.

3. The four SigV4 steps at a glance

One step at a time from here.

4. Step 1: Building the Canonical Request

The signature only works if "the same request always serializes to the same string". Header order and URI escape variants have to be flattened out. That flattening is what the Canonical Request does.

CanonicalRequest =
  HTTPRequestMethod + '\n' +
  CanonicalURI + '\n' +
  CanonicalQueryString + '\n' +
  CanonicalHeaders + '\n' +
  SignedHeaders + '\n' +
  HashedPayload

The rules for each element, in one diagram.

Three traps that bite hard.

URI encoded twice: for AWS APIs other than S3, the path is URI-encoded twice. A frequent SDK bug. S3 is the only exception, encoded once.
Query string sort: b=2&a=1 becomes a=1&b=2. When the same key appears multiple times, sort the values too.
Header value trim: strip leading and trailing whitespace, then collapse runs of internal whitespace to a single space. Foo: bar baz becomes foo:bar baz.

In Python.

from urllib.parse import quote
import hashlib

def canonical_request(method, uri, query, headers, payload):
    # URI: encode once for S3, twice for others (this example assumes S3)
    canonical_uri = quote(uri, safe="/-_.~")

    # Query: sort by key, encode values
    items = sorted(query.items())
    canonical_query = "&".join(
        f"{quote(k, safe='-_.~')}={quote(v, safe='-_.~')}" for k, v in items
    )

    # Headers: lowercase + sort + value trim
    lower = {k.lower(): " ".join(v.split()) for k, v in headers.items()}
    sorted_keys = sorted(lower.keys())
    canonical_headers = "".join(f"{k}:{lower[k]}\n" for k in sorted_keys)
    signed_headers = ";".join(sorted_keys)

    # Payload: SHA256 hex
    hashed_payload = hashlib.sha256(payload).hexdigest()

    return (
        f"{method}\n{canonical_uri}\n{canonical_query}\n"
        f"{canonical_headers}\n{signed_headers}\n{hashed_payload}"
    ), signed_headers, hashed_payload

canonical_headers ends with \n, and the line after it brings another \n, so the serialized form has two newlines in a row. That is spec-correct. "Fix" it to a single newline and SignatureDoesNotMatch greets you immediately.

5. Step 2: Building the StringToSign

Once the Canonical Request exists, SHA256 it and combine the result with the signing scope (algorithm + datetime + region + service) into a single string. That is the StringToSign.

StringToSign =
  Algorithm + '\n' +
  RequestDateTime + '\n' +
  CredentialScope + '\n' +
  HEX(SHA256(CanonicalRequest))

Concrete example.

AWS4-HMAC-SHA256
20260517T120000Z
20260517/us-east-1/s3/aws4_request
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

CredentialScope has the shape <date>/<region>/<service>/aws4_request. That alone tells AWS "this signature is for this day, this region, and this service". A signature scoped to s3 cannot be reused against dynamodb. That is the core of replay protection.

import hashlib

def string_to_sign(amz_date, scope_date, region, service, canonical_req):
    algorithm = "AWS4-HMAC-SHA256"
    scope = f"{scope_date}/{region}/{service}/aws4_request"
    hashed_cr = hashlib.sha256(canonical_req.encode()).hexdigest()
    return f"{algorithm}\n{amz_date}\n{scope}\n{hashed_cr}", scope

amz_date is the basic ISO8601 form 20260517T120000Z (no - or :). scope_date is just 20260517. Mismatching them is a classic bug.

6. Step 3: Deriving the SigningKey (4-stage HMAC chain)

This is the heart of SigV4. You never sign with the Secret Key itself. You chain HMAC four times from the Secret Key to build kSigning, and sign with that.

Why four stages. Key separation. kDate is "a key valid only for this day", kRegion is "valid only for this day and this region", and so on. Each stage narrows the scope further. Even if an attacker leaks kRegion, it cannot be used on another day or in another region. The Secret Key is permanent, but kSigning is disposable, scoped to one day, one region, one service.

import hmac
import hashlib

def hmac_sha256(key: bytes, msg: str) -> bytes:
    return hmac.new(key, msg.encode(), hashlib.sha256).digest()

def signing_key(secret, scope_date, region, service):
    k_date = hmac_sha256(("AWS4" + secret).encode(), scope_date)
    k_region = hmac_sha256(k_date, region)
    k_service = hmac_sha256(k_region, service)
    k_signing = hmac_sha256(k_service, "aws4_request")
    return k_signing

The "AWS4" + SecretAccessKey prefix is hard-coded in the spec. Forget the AWS4 and SignatureDoesNotMatch lands on the first request.

7. Step 4: Computing the Signature and the Authorization Header

The last step is just HMAC the StringToSign with kSigning and hex-encode it.

def sign(k_signing: bytes, string_to_sign: str) -> str:
    return hmac.new(k_signing, string_to_sign.encode(), hashlib.sha256).hexdigest()

The Authorization header is assembled like this.

Authorization: AWS4-HMAC-SHA256
  Credential=<AccessKeyId>/<scope>,
  SignedHeaders=<signed>,
  Signature=<hex>

A real example.

Authorization: AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20260517/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=fe5f80f77d5fa3beca038a248ff027d0445342fe2855ddc963176630326f1024

Credential holds the AccessKeyId and the CredentialScope, SignedHeaders lists the signed header names with ; separators, and Signature is the hex string. This is exactly what the SDK is assembling for you.

8. Full implementation: hitting S3 GetObject with SigV4

The four steps wired together in under 80 lines. Pure urllib, no SDK.

import datetime
import hashlib
import hmac
import urllib.request
from urllib.parse import quote

ACCESS_KEY = "AKIAIOSFODNN7EXAMPLE"
SECRET_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
REGION = "us-east-1"
SERVICE = "s3"
BUCKET = "kt-sigv4-demo"
KEY = "hello.txt"
HOST = f"{BUCKET}.s3.{REGION}.amazonaws.com"


def hmac_sha256(key: bytes, msg: str) -> bytes:
    return hmac.new(key, msg.encode(), hashlib.sha256).digest()


def signing_key(secret, scope_date, region, service):
    k = hmac_sha256(("AWS4" + secret).encode(), scope_date)
    k = hmac_sha256(k, region)
    k = hmac_sha256(k, service)
    return hmac_sha256(k, "aws4_request")


def sigv4_get(bucket, key):
    now = datetime.datetime.now(datetime.timezone.utc)
    amz_date = now.strftime("%Y%m%dT%H%M%SZ")
    scope_date = now.strftime("%Y%m%d")
    scope = f"{scope_date}/{REGION}/{SERVICE}/aws4_request"

    method = "GET"
    canonical_uri = "/" + quote(key, safe="/-_.~")
    canonical_query = ""
    payload_hash = hashlib.sha256(b"").hexdigest()

    headers_to_sign = {
        "host": HOST,
        "x-amz-content-sha256": payload_hash,
        "x-amz-date": amz_date,
    }
    sorted_keys = sorted(headers_to_sign.keys())
    canonical_headers = "".join(f"{k}:{headers_to_sign[k]}\n" for k in sorted_keys)
    signed_headers = ";".join(sorted_keys)

    canonical_request = (
        f"{method}\n{canonical_uri}\n{canonical_query}\n"
        f"{canonical_headers}\n{signed_headers}\n{payload_hash}"
    )
    hashed_cr = hashlib.sha256(canonical_request.encode()).hexdigest()
    string_to_sign = f"AWS4-HMAC-SHA256\n{amz_date}\n{scope}\n{hashed_cr}"

    k_signing = signing_key(SECRET_KEY, scope_date, REGION, SERVICE)
    signature = hmac.new(k_signing, string_to_sign.encode(), hashlib.sha256).hexdigest()

    authorization = (
        f"AWS4-HMAC-SHA256 Credential={ACCESS_KEY}/{scope}, "
        f"SignedHeaders={signed_headers}, Signature={signature}"
    )

    req = urllib.request.Request(f"https://{HOST}/{key}", method=method)
    for h, v in headers_to_sign.items():
        req.add_header(h, v)
    req.add_header("Authorization", authorization)

    with urllib.request.urlopen(req) as resp:
        return resp.read()


if __name__ == "__main__":
    print(sigv4_get(BUCKET, KEY).decode())

Typical failures.

Forgot host in headers_to_sign: SignatureDoesNotMatch
x-amz-date and the date inside Authorization differ: SignatureDoesNotMatch
Body is empty but you forgot to compute payload_hash: SignatureDoesNotMatch

The SDK does all of this correctly. When writing your own, the fastest debugging path is byte-for-byte diff against what the SDK produces.

9. End-to-end sequence: Client and Server verification

The server side (AWS) reproduces the exact same procedure to build the signature, then compares against what the client sent.

When AWS returns SignatureDoesNotMatch, the response body contains the Canonical Request that AWS itself reconstructed. That is gold for debugging. Diff your own Canonical Request against the one AWS built and the divergent element (header trim, URI encoding, missing header) pops out immediately.

10. Chunked Upload: STREAMING-AWS4-HMAC-SHA256-PAYLOAD

For large uploads to S3, hashing the entire body up front is not realistic. Hashing a 10 GB file before you even start sending it is a latency disaster. So S3 has a STREAMING mode where each chunk is signed individually.

The headers look like this.

x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
Content-Encoding: aws-chunked
x-amz-decoded-content-length: <original body size>
Content-Length: <size including chunk headers>

The body is chunked-transfer-style but with a custom format.

10000;chunk-signature=<sig1>\r\n
<8192 byte chunk 1>\r\n
10000;chunk-signature=<sig2>\r\n
<8192 byte chunk 2>\r\n
...
0;chunk-signature=<sigN>\r\n
\r\n

Each chunk's signature is computed by chaining the previous chunk's signature with the current chunk's hash. Swap a chunk in the middle and every signature after it falls apart.

The StringToSign gets a chunk-specific extension.

AWS4-HMAC-SHA256-PAYLOAD
<amz-date>
<scope>
<previous-signature>
HEX(SHA256(""))
HEX(SHA256(chunk-data))

previous-signature is what closes the chain. The stream terminates with a zero-byte chunk. There is also a STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER variant that appends trailer headers like CRC32C for an extra integrity check.

Writing this by hand is basically masochism. Let the SDK handle it. But knowing the mechanics makes questions like "why is Content-Length different from x-amz-decoded-content-length" trivial to answer.

11. Streaming SigV4: WebSocket / IoT MQTT

SigV4 also rides on WebSocket and MQTT over WebSocket, not just plain HTTP. IoT Core, API Gateway WebSocket, and CloudWatch Logs Live Tail all sit here.

These use a Presigned URL form that embeds SigV4 into the connection URL's query string. The Authorization header is hard to attach to a WebSocket Upgrade, so cramming everything into the URL lets the handshake complete in one round trip. With MQTT over WebSocket the URL ends up as wss://<endpoint>/mqtt?X-Amz-Algorithm=...&X-Amz-Signature=....

The streaming case is special because the connection lives for a long time. The SigV4 signature itself is checked only once at connection establishment, so even after X-Amz-Expires passes, the existing connection keeps going (depending on the AWS implementation). Reconnecting requires a fresh signature.

12. What is inside a Presigned URL

A Presigned URL is the Authorization header crammed into the query string. Heavily used for "hand someone a URL and let the browser download the S3 object directly".

https://kt-bucket.s3.us-east-1.amazonaws.com/report.pdf
  ?X-Amz-Algorithm=AWS4-HMAC-SHA256
  &X-Amz-Credential=AKIA.../20260517/us-east-1/s3/aws4_request
  &X-Amz-Date=20260517T120000Z
  &X-Amz-Expires=900
  &X-Amz-SignedHeaders=host
  &X-Amz-Signature=fe5f80f77d5fa3beca038a248ff027d0445342fe2855ddc963176630326f1024

Properties.

X-Amz-Expires: in seconds. Maximum 604800 (= 7 days). The 7-day cap exists because the SigV4 signing key rotates on roughly a 7-day cycle (kDate is per-day, but kSigning is valid for about 7 days).
X-Amz-SignedHeaders=host: no body, no extra headers, so only host needs to be signed.
Payload hash handling: presigned URLs use either UNSIGNED-PAYLOAD or the actual body hash. Looser than the header-based form.

Watch out for these.

A Presigned URL minted with IAM Role temporary credentials (e.g. STS AssumeRole) is capped by the credential's own lifetime. AssumeRole defaults to 1 hour for DurationSeconds, configurable up to 12 hours, but once the SessionToken expires the URL dies. This is the standard root cause when someone passes --expires-in 604800 and the URL still dies after 1 hour.
If you are handing the URL to a browser, mint it with an IAM User long-term key or raise the Role's MaxSessionDuration. EC2 Instance Profile lands in the same trap: IMDS auto-rotates, but after the rotation any URL signed with the previous credentials is dead.

13. SigV4A: the asymmetric variant

Now the main event. The instant you hit Multi-Region Access Point (MRAP), the SDK silently flips from SigV4 to SigV4A. The algorithm name is AWS4-ECDSA-P256-SHA256. It uses ECDSA (Elliptic Curve Digital Signature Algorithm) with NIST P-256 instead of HMAC.

How it actually works

SigV4A derives an ECDSA P-256 keypair from the SecretAccessKey through a KDF. The spec uses input_key = "AWS4A" || sk, the label AWS4-ECDSA-P256-SHA256, the akid (AccessKeyId) as context, and runs a counter that iterates until the derived scalar lands below the P-256 order n. The resulting scalar c becomes the private key as k = c + 1, and the public key is Q = k * G. The verifier needs only the public key to check the signature. The canonical implementation lives in key_derivation.c in awslabs/aws-c-auth.

Why Multi-Region forces this

MRAP is a representative endpoint for a set of buckets across multiple regions, and any request can be routed to any of them. SigV4 bakes the region name directly into CredentialScope, so a signature minted for us-east-1 simply cannot be verified at us-west-2.

SigV4A drops the region segment from CredentialScope entirely and replaces it with the X-Amz-Region-Set header that declares "this signature is valid in us-east-1, us-west-2, and eu-west-1". CredentialScope changes like this.

SigV4:  20260517/us-east-1/s3/aws4_request
SigV4A: 20260517/s3/aws4_request   (region segment removed)

The X-Amz-Region-Set value is a comma-separated list like us-east-1,us-west-2, and wildcards like us-east-* or a bare * are allowed too.

On top of that, every region can verify independently with only the public key. Distributing a symmetric HMAC key to every region is a security risk (compromise one region and they all leak), which is exactly where asymmetric signing pays off.

Is the ECDSA here deterministic

Standard ECDSA needs a fresh random nonce k for every signature (the same input produces different signatures each time). Bias or reuse of k is a fatal mistake that leaks the private key. SigV4A's implementation lives in AWS Common Runtime (awslabs/aws-c-auth + aws-c-cal), and the nonce comes from the OS RNG, so the same request produces a different signature on every call. Whether it follows RFC 6979 (deterministic ECDSA) is not stated explicitly in public docs. "awscrt handles the nonce correctly" is enough to know in practice.

Rolling your own is brutal

ECDSA has heavy math, so if you go custom, use the cryptography library. AWS officially recommends aws-crt (a C library bound into Python through awscrt). botocore treats awscrt as an optional dependency. Install with pip install botocore[crt] (or pip install boto3[crt]) and SigV4A kicks in automatically against MRAP. Plain boto3 without the crt extra crashes against MRAP with MissingDependencyException.

14. SigV4 vs SigV4A side by side

Item	SigV4	SigV4A
Algorithm name	`AWS4-HMAC-SHA256`	`AWS4-ECDSA-P256-SHA256`
Key type	Symmetric (HMAC)	Asymmetric (ECDSA P-256)
Key derivation	4-stage HMAC chain (kDate to kRegion to kService to kSigning)	One-shot KDF with a counter to derive an ECDSA keypair (label `AWS4-ECDSA-P256-SHA256`, input `AWS4A` + secret)
Region in CredentialScope	Fixed (e.g. `us-east-1`)	Segment removed. `X-Amz-Region-Set` header declares the regions (wildcards allowed)
Verification	AWS recomputes the same HMAC from the same Secret	AWS verifies the ECDSA signature with the public key
Main use case	Any single-region API call	Multi-Region Access Point, replication paths
Signature determinism	Deterministic (same input gives same signature)	Non-deterministic (`awscrt` uses a random nonce)
Performance	HMAC is ultra-light (microseconds)	ECDSA is heavier (milliseconds)
7-day Presigned URL	Supported	Supported (same conditions)
SDK support	All SDKs	Through `aws-crt` (Python needs `botocore[crt]`; Go/Java/JS bundle it)

For working developers the only thing that matters in practice: the moment MRAP enters the picture the SDK flips to SigV4A; everything else stays on SigV4. You almost never reach for SigV4A by hand.

15. SigV4 vs SigV4A: scope and verifier differences in one figure

SigV4 assumes 1 request = 1 region, so HMAC is enough. SigV4A is built around 1 request landing in any of several regions, which makes a shared HMAC untenable and forces ECDSA. That is the structural difference.

16. The Clock Skew trap

SigV4's CredentialScope carries <date> and x-amz-date carries a second-precision timestamp. AWS rejects requests whose timestamp is more than ±15 minutes off the current time (the S3 docs state this explicitly; the tolerance varies slightly by service).

Three classic ways to step on this.

Docker container NTP drift: when the container is not NTP-synced, host clock skew kills you instantly. Containers with no visible hwclock and no ntpd are the worst offenders. Lambda is fine because AWS keeps it in sync.
EC2 with chrony missing: AL2023 ships chronyd by default, but custom AMIs that strip it out drift over time. Point chrony at 169.254.169.123 (Amazon Time Sync Service) and the problem disappears.
Old Lambda layers / virtualized clocks: BPF-based sandboxes where the clock is pinned. Upgrading the Lambda runtime usually fixes it.

Error examples.

SignatureDoesNotMatch:
  The request signature we calculated does not match the signature you provided.
  Check your key and signing method.
RequestTimeTooSkewed:
  The difference between the request time and the current time is too large.

RequestTimeTooSkewed is always clock skew, 100%. SignatureDoesNotMatch is either clock skew or a Canonical Request construction bug.

Conclusion

SigV4 solves "request signing that detects tampering and replay without ever exposing the secret". Chain HMAC four times to derive a signing key, SHA256 the Canonical Request to build a StringToSign, then HMAC the whole thing for the final signature. Write the four steps by hand once and everything the SDK was hiding finally becomes visible.

SigV4A is SigV4 extended to multi-region by going asymmetric (ECDSA P-256). Strip the region from CredentialScope, declare the regions in X-Amz-Region-Set, and let each region verify independently with the public key. The SDK flips to it automatically when MRAP is involved, so you basically never reach for it by hand.

The two real-world traps are always the same: clock skew and canonical-request normalization mistakes. Read the exact error message (RequestTimeTooSkewed vs SignatureDoesNotMatch vs AccessDenied) and diff your Canonical Request against the one AWS echoes back. Get that far and SigV4 stops biting.

AWS IAM Roles Anywhere Deep Dive

kt — Thu, 28 May 2026 15:46:19 +0000

Introduction

"I want to drop a file from an on-prem server into S3."
"I want to read DynamoDB from a Kubernetes pod sitting in my datacenter."
"I want to pull a secret from AWS Secrets Manager from an app in someone else's cloud."

Do this the naive way and you end up here:

Create an IAM User
Issue an access key (the AKIA... kind)
Paste it into ~/.aws/credentials, env vars, or some on-prem Secrets Manager
Use it forever

This is where most production incidents start today. Long-lived access keys leak and stay leaked, rotation gets forgotten, and there is no record of who copied them where or when.

AWS IAM Roles Anywhere is the mechanism that hands IAM Role temporary credentials to workloads outside AWS without distributing any long-lived key. The key material is replaced by an X.509 certificate.

This article goes deep on Roles Anywhere.

1. Vocabulary you need first

Roles Anywhere sits on top of two things: IAM Role and PKI (the certificate world). If either is fuzzy you will get lost fast. The bare minimum below.

IAM Role / STS / temporary credentials

Term	What to remember
IAM Role	A box that says "who is allowed to take this permission on". It has two parts: a Trust Policy (who can assume it) and a Permission Policy (what the assumer can do)
AssumeRole	The act of taking on a Role. When it succeeds, you immediately get back a set of temporary credentials
STS (Security Token Service)	The AWS service that issues temporary credentials. The `sts:AssumeRole` family of APIs lands here
Temporary credentials	A three-piece set: `AccessKeyId` + `SecretAccessKey` + `SessionToken`. Default lifetime 1 hour, up to 12 hours via the Role config

X.509 certificate / CA / PKI

Term	What to remember
X.509 certificate	A digital document where a CA signs "this public key belongs to this entity". The cert sitting in front of any HTTPS site is the same shape
CA (Certificate Authority)	The party that issues certificates. Trusting a CA's certificate means trusting every certificate it has ever issued
Private CA	A CA for closed environments such as inside a company. AWS sells a managed version called AWS Private CA (formerly ACM PCA)
PKI (Public Key Infrastructure)	The whole ecosystem of issuing, distributing, and revoking certificates
Private key	The key paired with the certificate. Digital signatures are made with it. Never put it on the network

Roughly: Roles Anywhere is the mechanism that makes AssumeRole callable using an X.509 certificate's private-key signature, instead of an IAM User's long-term key.

2. The big picture and the three actors

Roles Anywhere has three specific concepts: Trust Anchor / Profile / Role. Pin them down visually first.

The three actors and their jobs:

Concept	What it is	What it holds
Trust Anchor	The declaration "I (this AWS account) trust this CA"	A reference to an AWS Private CA, or the PEM of an external CA
Profile	The declaration "for callers authenticated via this CA, which Roles can they use and with what session limits"	A list of allowed Role ARNs + session duration + optional Session Policy
IAM Role	The permission body that activates once assumed	Trust Policy (allowing the Roles Anywhere service principal) + Permission Policy

Walking through each in order.

3. Trust Anchor: the root of trust

A Trust Anchor is the declaration "this AWS account trusts this CA". When Roles Anywhere sees a certificate, it walks the chain to confirm the certificate was issued by this CA.

There are two ways to create a Trust Anchor.

A. Use AWS Private CA (PCA) as the source

AWS Private CA (formerly ACM PCA) is AWS's managed paid CA. When creating a Trust Anchor you say "use this PCA", and every certificate issued by that PCA is trusted automatically.

PCA has two modes (Tokyo region reference pricing, varies by region):

General Purpose Mode: 400 USD/month plus per-certificate issuance fees. Full features (CRL publication, long-lived certs, freely issued via API). Capable as a replacement for an internal PKI
Short-Lived Certificate Mode: 50 USD/month plus per-certificate issuance fees. Only for certificates with lifetime under 7 days, no CRL publication. Optimised for the Roles Anywhere "issue short-lived certs frequently" usage pattern

Upside: issuance, revocation, and renewal automation lean on AWS. The AWS Private CA Issuer for cert-manager lets Kubernetes consume it.

Downside: either mode bills a flat monthly fee just for having a CA stood up. For verification or dev work, an external CA is cheaper.

B. Upload an existing external CA

Upload a CA certificate in PEM and register it as a Trust Anchor. If you already have an internal PKI (HashiCorp Vault, smallstep step-ca, an OpenSSL-based homemade CA, etc.), the extra cost is zero.

aws rolesanywhere create-trust-anchor \
    --name "my-internal-ca" \
    --source '{
        "sourceType": "CERTIFICATE_BUNDLE",
        "sourceData": {
            "x509CertificateData": "-----BEGIN CERTIFICATE-----\nMIID...\n-----END CERTIFICATE-----"
        }
    }' \
    --enabled

With an external CA you own revocation management (CRL updates). Covered in §8.

4. Profile: which Role, with what guardrails

A Profile is "the usage rules for callers authenticated against a given Trust Anchor".

What goes into a Profile:

roleArns: list of IAM Role ARNs that may be assumed
durationSeconds: maximum session lifetime (900 to 43200 seconds, i.e. 15 minutes to 12 hours, capped by the Role's MaxSessionDuration)
sessionPolicy / managedPolicyArns: extra policy that applies only to the session. "The Role's Permission Policy is X, but calls coming through this Profile are further narrowed to Y"

A Session Policy intersects with the Role's Permission Policy via AND. Even if the Role allows s3:*, narrowing the Session Policy to s3:GetObject means only s3:GetObject is effectively permitted.

5. The Role's Trust Policy: who is allowed to call

The Role itself is created the usual way, but its Trust Policy (the policy saying "who can Assume this Role") has a Roles Anywhere-specific shape:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "rolesanywhere.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession",
        "sts:SetSourceIdentity"
      ],
      "Condition": {
        "StringEquals": {
          "aws:PrincipalTag/x509Subject/CN": "server-a.example.com"
        },
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:rolesanywhere:ap-northeast-1:123456789012:trust-anchor/abc-..."
        }
      }
    }
  ]
}

Key points:

The service principal is rolesanywhere.amazonaws.com. Without this entry, AssumeRole via the Roles Anywhere CreateSession path will not work
Allowing sts:TagSession lets the certificate's Subject / SAN / Issuer be injected as session tags (more on this below)
Conditions like aws:PrincipalTag/x509Subject/CN let you narrow down further based on the certificate contents. Fine-grained control like "only certificates with CN server-a.example.com may assume this" is possible
aws:SourceArn pins the call to a specific Trust Anchor, so a request coming from a Trust Anchor you did not expect is denied

Session tags auto-populated from the certificate

A session created through CreateSession + AssumeRole automatically carries certificate attributes as tags. The common ones:

Tag key	Content
`aws:PrincipalTag/x509Subject/CN`	Common Name of the certificate Subject
`aws:PrincipalTag/x509SAN/DNS`	DNS name in the Subject Alternative Name (SAN)
`aws:PrincipalTag/x509SAN/URI`	URI in SAN (e.g. a SPIFFE ID like `spiffe://example.com/ns/prod/sa/app`)
`aws:PrincipalTag/x509Issuer/CN`	CN of the issuing CA
`aws:PrincipalTag/x509Serial`	Certificate serial number

A small bit of supplementary vocabulary:

SAN (Subject Alternative Name): an X.509 extension field that holds additional identifiers beyond Common Name, including multiple hostnames, IPs, or URIs
SPIFFE ID: a URI-format workload identifier defined by the SPIFFE spec (spiffe://...). If your internal identity platform uses SPIFFE, dropping a SPIFFE ID into the SAN URI lets the AWS side reference it directly in a Condition

The upshot: the Role's Permission Policy can also use aws:PrincipalTag/x509Subject/CN in its Conditions. Patterns like "the server-a.example.com certificate can only write under the prefix=server-a/* portion of S3" become possible. That is ABAC (Attribute Based Access Control): instead of carving permissions up by Role (RBAC-style), principal attributes (tags) dynamically tighten what is allowed.

6. The CreateSession flow

Tracing what actually happens from the client's first call until temporary credentials are in hand.

Key points:

A "SigV4 X.509 variant" sits on the SigV4 frame but signs with an X.509 private key. Normal AWS APIs sign with AWS4-HMAC-SHA256 (symmetric HMAC using the access key). CreateSession uses one of AWS4-X509-RSA-SHA256 / AWS4-X509-ECDSA-SHA256 / AWS4-X509-MLDSA (the last one is post-quantum, PQC-capable) depending on the key type. The Canonical Request construction is the same as SigV4. Only the final step is an asymmetric X.509 signature instead of HMAC
The private key never leaves the credential helper. It does not go on the network
The credentials handed back are plain IAM temporary credentials. The SDK sees nothing unusual. Subsequent S3 / DynamoDB calls go out as normal SigV4 (HMAC)

7. What is inside the credential helper

Hand-rolling the CreateSession signing on the client side is not realistic, so AWS ships an official binary called aws_signing_helper (repo: aws/rolesanywhere-credential-helper). That binary is the credential helper.

aws_signing_helper plugs into the AWS CLI / SDK credential_process spec. Drop this into ~/.aws/config and both the CLI and any SDK transparently fetch credentials through Roles Anywhere:

[profile myapp]
credential_process = /usr/local/bin/aws_signing_helper credential-process \
    --certificate /etc/pki/server.pem \
    --private-key /etc/pki/server.key \
    --trust-anchor-arn arn:aws:rolesanywhere:ap-northeast-1:123456789012:trust-anchor/xxxx \
    --profile-arn       arn:aws:rolesanywhere:ap-northeast-1:123456789012:profile/yyyy \
    --role-arn          arn:aws:iam::123456789012:role/MyAppRole

A call like aws s3 ls --profile myapp triggers credential_process, which runs aws_signing_helper, which returns the JSON. The AWS CLI / SDK takes care of automatic credential refresh. No manual refresh needed.

Where the private key lives

The choice of private-key storage sets the ceiling on your Roles Anywhere operational quality. aws_signing_helper supports the following backends. Leak resistance rises as you go down the table.

Storage	Properties	Leak resistance
File (`/etc/pki/server.key` etc.)	Easiest. Anyone with read access can `cat` it	Low
OS keystore (Windows Certificate Store / macOS Keychain)	Protected by OS access control	Medium
PKCS#11 module (HSM / smartcard)	Key stays inside the HSM, only signing operations are delegated	High
TPM 2.0 (secure chip on the motherboard)	Key sealed into hardware, non-exportable	High

In high-security environments, sealing the key into PKCS#11 (HSM) or TPM 2.0 so that the key cannot be extracted at all is the right move. aws_signing_helper exposes flags like --cert-selector and --tpm-key-handle to wire into these backends.

8. Revoking a certificate

You will eventually need to kill a compromised server's certificate immediately. Roles Anywhere supports CRL (Certificate Revocation List).

Importing a CRL

Generate a revocation list on the CA side and register it in Roles Anywhere as PEM.

aws rolesanywhere import-crl \
    --name "my-ca-crl" \
    --crl-data file://crl.pem \
    --trust-anchor-arn arn:aws:rolesanywhere:ap-northeast-1:123456789012:trust-anchor/xxxx \
    --enabled

Once registered, Roles Anywhere checks the CRL on every CreateSession. Calls from revoked certificates are rejected.

Auto-integration with AWS Private CA

When the Trust Anchor source is PCA, calling revoke-certificate on the PCA causes the PCA to publish a CRL into S3 within about 30 minutes. The standard pattern is to catch that with Lambda and feed it to the ImportCrl API.

Temporarily disabling the CRL check

When you need to disable it during incident response, DisableCrl flips the check off and EnableCrl turns it back on. In normal production, leave it on.

9. When it fits, and when it does not

Roles Anywhere is powerful but not for every case. The decision flow:

Takeaways:

Workloads running inside AWS do not need Roles Anywhere. Instance Profile / Execution Role / Pod Identity are the natural choice
If your CI can speak OIDC, prefer OIDC. GitHub Actions / GitLab CI / etc. emit OIDC officially. Roles Anywhere is unnecessary
If you cannot emit OIDC, you already have an X.509-based PKI, and the server is fully on-prem, that is where Roles Anywhere shines

Where Roles Anywhere is a poor fit

"We do not have an internal PKI yet" cases: standing up a CA before introducing Roles Anywhere is much more work than Roles Anywhere itself. If there is an OIDC path, take that
"Thousands of clients, each with its own certificate, CRL updated daily" cases: at this scale CRL propagation lag and other factors need real design work
"Calling from inside a VPC" cases: anything inside a VPC is already inside AWS, so attaching a Role the normal way is enough

10. Pricing and limits

Pricing

The IAM Roles Anywhere service itself is free. No per-CreateSession request fees
AWS Private CA is the only billed component. 400 USD/month for General Purpose, 50 USD/month for Short-Lived Certificate, plus per-certificate issuance fees
Using an external CA (HashiCorp Vault / step-ca / homemade PKI etc.) avoids the PCA bill (you pay in operational effort instead)

Main limits (from official Service Quotas, per Region)

Item	Default	Increase request
Trust Anchors per account	50	Allowed
Profiles per account	250	Allowed
Roles per Profile	250	Not allowed (hard limit)
Registered certificates per Trust Anchor	2	Not allowed (two slots, for rotation)
CRLs per Trust Anchor	2	Not allowed
CreateSession rate	10 req/sec	Allowed
Session lifetime	15 minutes to 12 hours	Within the Role's `MaxSessionDuration`

Note that CreateSession is 10 req/sec. A design that pulls short-lived credentials in volume will hit this per-Region rate. The basic pattern is to cache credentials on the client side and refresh just before expiration. aws_signing_helper's credential_process does this automatically.

11. Don'ts and Do's

❌ Don't

Drop the private key as a plaintext file readable by anyone
Reuse one certificate across multiple servers (you lose the ability to tell which one leaked)
Write a Trust Policy that trusts rolesanywhere.amazonaws.com alone, with no aws:SourceArn or certificate Conditions
Skip CRL operations entirely (i.e. no way to invalidate a cert on compromise)
Issue certificates with absurdly long lifetimes (e.g. 10 years)

✅ Do

Seal the private key into TPM 2.0 / HSM (PKCS#11) / OS keystore
One server, one certificate. Put a hostname or SPIFFE-ID-equivalent identifier in the Subject / SAN
Issue short-lived certificates (e.g. 24 hours to a few days) with automatic renewal. cert-manager and step-ca's auto-renew machinery is the standard pattern
Put both aws:SourceArn (Trust Anchor) and aws:PrincipalTag/x509Subject/CN into the Role's Trust Policy
Automate CRL ingestion and keep DisableCrl ready as an operations break-glass
Use session tags (aws:PrincipalTag/x509...) to drive ABAC in the Role's Permission Policy

12. Wrap-up

IAM Roles Anywhere hands IAM Role temporary credentials to workloads outside AWS without distributing long-lived keys
It uses X.509 certificates as the key material, with the issuing CA registered as a Trust Anchor
Three actors: Trust Anchor (the trusted CA) / Profile (which Role can be used) / Role (the actual permission)
CreateSession is not plain SigV4. It uses a distinct flow signed with the certificate's private key, and the private key never goes on the network
The official aws_signing_helper ships as a credential_process provider, so existing CLI and SDK code works as-is
Seal the private key in TPM / HSM / OS keystore. A plain file on disk is an incident waiting to happen
Revocation goes through CRL via ImportCrl. PCA-backed setups can auto-integrate via S3
If you are inside AWS, Roles Anywhere is unnecessary. If CI can speak OIDC, OIDC wins. On-prem, multi-cloud, and existing-PKI worlds are where it actually fits
The service itself is free. Only PCA costs money