Every pipeline I've seen in the last three years had security steps in it. Scanners, linters, sometimes a signing job someone added after a conference talk. Almost none of them had the other half: a cluster that refuses to run what the pipeline didn't bless. The scan report lands in an artifact folder nobody opens. The signature sits in the registry, verified by no one. All of it is advisory.
There's a one-line test for this:
kubectl run test --image=nginx:latest -n production
A real image, from a registry you never chose, signed by nobody. If that pod runs, your supply chain security is a set of suggestions.
I wanted something I could point to in a design discussion instead of hand-waving, so I built a reference implementation: k8s-secure-supply-chain. It runs on a laptop with kind, converges through Argo CD, and ends with make verify proving three things: an image that CI built, scanned, signed and attested is running, the same cluster just refused an unsigned one at admission, and it refused an image from a registry it never approved.
The tool list (Syft, Trivy, Cosign, Kyverno, Argo CD) is the least interesting part. Two engineers can pick identical tools and build very different platforms. What shaped this one is four decisions, each of which cost me something, plus three mistakes I only found once it was already working. The mistakes are near the end, and they're what I'd most want a reviewer to read. The repo documents the decisions as ADRs; this is the readable version.
First, the map: the whole path once, from an empty laptop and a git push to a running pod.
What actually happens between a push and a running pod
There are two halves that never talk to each other directly. One runs on GitHub every time I push. The other runs on my machine, and I set it up once.
ON GITHUB, every push ON MY MACHINE, once
───────────────────── ───────────────────
lint ─────────── yamllint, shellcheck, make up
terraform fmt Terraform: local registry
policy-tests ─── kyverno test, offline + 2-node kind cluster
source-scan ──── gitleaks + Semgrep make bootstrap
gate-test ────── vulnerable image must Argo CD + one root app
be refused by Trivy
│ │
▼ all green, main only ▼
secure-build Argo CD pulls from git:
build ─► SBOM ─► Trivy gate ─► push Kyverno (wave 0)
─► sign digest ─► attest SBOM policies (wave 1)
│ demo app (wave 2)
▼ │
ghcr.io: image, signature, ◄───── pulls ────────┤
SBOM attestation (+ Rekor log entry) ▼
Kyverno admission:
approved registry?
right identity? SBOM attached?
tag pinned? limits set?
─► run, or refuse
The registry is the only place the two halves meet, and that's deliberate. The cluster never trusts the pipeline. It trusts the evidence the pipeline left behind, and it checks that evidence itself.
Before anything is built, CI runs lint, offline Kyverno policy tests, and gitleaks plus Semgrep over the full git history. One more job exists because of something slightly embarrassing. The demo image is built on a minimal Chainguard base with zero known vulnerabilities, so since the first build the Trivy gate had only ever been seen passing, which looks exactly like a gate that's switched off. Now CI builds a deliberately vulnerable image from a frozen old Python base (133 fixable HIGH and CRITICAL findings), never publishes it, and requires the gate to reject it. The test reads the gate's settings straight out of the real build workflow, so if someone loosens the gate, the test loosens with it and goes red.
The build itself, which only runs on main and only after all of that passes. The order is the part worth reading:
- Build the image on the runner. Not pushed yet.
- Syft reads it and writes a CycloneDX SBOM, the list of every package inside.
- Trivy scans it and fails the job on fixable HIGH or CRITICAL findings.
- Only now is it pushed to ghcr.io, tagged with the commit SHA and with
main. Same image, same digest. - Cosign signs the digest, keyless, using the runner's OIDC token.
- Cosign attaches the SBOM to that digest as a signed attestation.
Scanning before pushing means the registry only ever holds images that passed the gate, so nobody can pull a failed build by accident. Signing the digest instead of the tag matters because a tag is a pointer anyone with push rights can move. And the whole job holds zero stored secrets: GITHUB_TOKEN for the registry push, a short-lived OIDC token for signing, both minted per run and gone when it ends.
The cluster, and what Terraform is for. Terraform only ever runs on my laptop: make up to create, make down to destroy. It never runs in the pipeline; CI only checks that the Terraform code is formatted. It creates two things: a plain registry:2 container on localhost:5001 for pushing local builds, and a two-node kind cluster (a control plane and a worker) pinned to Kubernetes 1.30, with containerd configured to use that registry. On my WSL2 machine that took about four minutes.
The one fiddly part is ordering. The registry has to sit on the Docker network kind creates, and that network only exists once kind has made its first cluster. So Terraform starts the registry disconnected, creates the cluster, and then connects the registry. Declaring the network up front works on any machine that has run kind before and fails on a clean one, which is the kind of bug you only find on someone else's laptop.
Terraform stops at the cluster boundary on purpose. It builds the cluster, and nothing inside it. Kyverno, the policies and the app are all Argo CD's job, pulled from git. Two tools with one clear line between them is easier to reason about than one tool that half-owns the cluster.
Delivery, handled by Argo CD. make bootstrap installs it and applies exactly one manifest by hand, a root application pointing at the repo. After that, nothing Argo manages gets changed by hand. Argo pulls in everything else, and sync waves force the order: Kyverno first, then the policies (read straight from the same catalog folder anyone can copy from), then the app. That ordering isn't cosmetic. Admission control only judges a Pod when it's created, so an app that lands before its policies are enforcing walks straight past the gate, and stays running until something recreates it. At best a background scan flags it in a report afterwards. I hit a version of this for real: a Deployment created before the policies existed couldn't be updated afterwards until I deleted it and let Argo recreate it through the gate.
The pull model matters for a second reason. With push-based deploys, CI holds cluster credentials, so owning CI means owning the cluster. Here CI never gets cluster credentials at all. The cluster reaches out to git, and git never reaches in.
Admission, the only step that can actually say no. When a Pod is created, Kyverno fetches the image's signature and checks that the certificate was issued to this repo's build workflow on main, confirmed against the Rekor log. It fetches the SBOM attestation and checks it really contains a CycloneDX document, not just an empty envelope with the right label. It rewrites the tag to the digest it just verified, so what runs is byte for byte what was checked. In front of all that sits a registry allowlist: in the workload namespace, an image from anywhere other than ghcr.io/yodim/ is refused before signatures even come into it. Then the two plain hygiene policies: no :latest, and CPU and memory limits set. All five policies fail closed: if Kyverno itself is down, nothing new gets admitted. Fail any check and the Pod object is never created. Every policy also checks init containers, and the ones that apply check the ephemeral containers kubectl debug attaches too, so debugging a pod can't smuggle an unverified image in.
That's the whole machine. The next four sections are why each piece looks the way it does.
kind, not a real cloud cluster
This one stung a bit. I have an Azure certification and I spend my workdays on real clusters. A managed cluster with workload identity would look more like production, and more impressive in a README.
I went with kind anyway, because a reference implementation you can't reproduce is just a blog post with extra steps. kind means anyone can run the entire platform for free, on any machine with Docker, in about the time it takes to make coffee. The moment I require a cloud account, I lose most of the people who would actually try it.
What I gave up: cloud IAM, a managed registry, realistic ingress. What I kept: every control that matters here behaves identically on kind and on AKS. SBOMs, signatures, attestations and admission policy don't care where the kubelet lives. The realism I dropped was mostly cosmetic. There's an AKS overlay on the roadmap, but as an addition. The free path stays.
Kyverno, not OPA Gatekeeper
Gatekeeper is the incumbent, and Rego is the more powerful language. It's also a language your whole team has to learn, and that's a tax on everyone who might want to lift a policy out of this repo into their own cluster.
Kyverno policies are Kubernetes YAML. Anyone who can review a manifest can review one in a pull request. And the deciding factor for this project: image verification is native. verifyImages handles Cosign signatures and in-toto attestations out of the box, keyless included, no external data providers bolted on. This is the entire trust pin:
attestors:
- entries:
- keyless:
subject: "https://github.com/yodim/k8s-secure-supply-chain/.github/workflows/secure-build.yml@refs/heads/main"
issuer: "https://token.actions.githubusercontent.com"
One repo, one workflow file, one branch. Nothing else produces images that run in the workload namespace. That sentence was false for a while after I first wrote it, which is the third item in the section below.
The honest counterargument: Rego skills transfer to Terraform checks, API authorization, CI gates. Kyverno's pattern language runs out of road when the logic gets complicated. If your team already writes Rego, ignore me and stay where you are. Most platform teams I've met don't.
Keyless signing, and the dependency I accepted for it
The classic Cosign setup generates a key pair and puts the private key in CI secrets. Congratulations, you now own a crown jewel. You get to rotate it, protect it from fork pull requests, and explain in an audit who has access to it. Worse, the signature only proves possession of a key. Whoever holds it is you.
Keyless flips the model. The CI job presents an OIDC token, Fulcio issues a certificate that lives a few minutes and is bound to one specific workflow identity, the signature goes into the Rekor transparency log, and there is no key to steal. At admission, verification pins that exact identity, so a signature produced by a fork, or by me on my laptop, doesn't verify. "Built by this workflow, on this branch, in this repo" is a much stronger claim than "someone had the key", and nothing needs rotating. Ever.
The price: a hard dependency on public Sigstore infrastructure. If Fulcio or Rekor is down, builds can't sign, and matched images fail closed at admission. I chose failing closed deliberately, integrity over availability, and for a reference platform that's the right default. For a payment system or a hospital cluster I'd sit with that trade-off a lot longer, and whichever side loses, I'd write it down. Also worth knowing: your repo identity lands in a public log. Regulated environments will want private Sigstore or plain keys, and the policy README documents both variants.
An SBOM is not a scan
This is the decision I'd defend the hardest, mostly because the industry keeps merging the two into one pipeline step since one tool can do both.
They're different artifacts with different lifespans. An SBOM is inventory: what is inside this image. It's true forever. A scan verdict is an assessment: is anything inside it currently known to be vulnerable. It starts rotting the moment it's produced, because tomorrow's CVE feed changes the answer.
So the pipeline treats them differently. Syft generates a CycloneDX SBOM that gets attached to the image as a signed attestation and travels with it for the rest of its life. Trivy produces a verdict that gates this one build and is then thrown away. Kyverno closes the loop by requiring the SBOM attestation at admission, which quietly turns inventory from a compliance checkbox into a runtime precondition. Nothing of mine runs unless it can prove what's in it.
When the next Log4Shell arrives, this is the difference between running a query against attested SBOMs and starting an archaeology dig through everything currently deployed. Ask anyone who was near Java services in December 2021 which of those they'd rather do.
One sub-decision people will disagree with: the Trivy gate ignores vulnerabilities that have no available fix. Failing builds over things nobody can action teaches teams to ignore the gate, and a gate people ignore is worse than no gate. Some compliance regimes see it differently. It's one flag, flip it if yours does.
The part I got wrong
Everything above is the version I could write before building it. Then I built it, and the same failure the article opens with showed up inside my own work, three times. Every time it looked fine.
A policy field that enforced nothing. Both image-verification policies carried failureAction: Enforce, sitting right next to the signature configuration, reading exactly like the thing that makes the policy block. It does nothing on the Kyverno version this repo pins (1.12, installed from Helm chart 3.2.6). The field isn't in the CRD schema for that version, so the API server quietly prunes it on the way in. Enforcement was really coming from validationFailureAction at the top of the spec. I found this by accident, chasing an unrelated Argo CD diff.
The reason it matters more than a stray line of dead YAML: failureAction is real in Kyverno 1.13+, where it replaces the deprecated top-level setting. So the trap is live. Someone tidying up, or copying these policies onto a newer cluster, could reasonably delete validationFailureAction on the grounds that the per-rule field already covers it. On 1.12 that silently drops both policies to Audit. The cluster keeps admitting images, the policies still list as Ready, and kubectl get clusterpolicies looks entirely healthy. You would find out when something unsigned ran.
A test that proved the wrong thing. The negative control, the one I called the whole point of the platform, ran this:
kubectl run test --image=ghcr.io/yodim/unsigned-test:1.0.0
Kyverno rejected it. Test passed. Except that image was never published, so the rejection was:
image tag not found: MANIFEST_UNKNOWN: manifest unknown
It failed because the registry had nothing to return. That result is identical with every verification rule deleted from the cluster. I had written a test that proves a registry can return 404, and put its output in an article as evidence of admission control.
The fix was to publish a deliberately unsigned image, FROM scratch with a text file in it, and then assert on why admission refused, not merely that it did. Now the rejection reads no signatures found, and verify.sh fails loudly if it ever sees a resolution error again, because that would mean the test quietly went back to proving nothing.
A control that only checked what it was told to check. The signature policy matches ghcr.io/yodim/*, which is correct, since that's the only place signed images come from. But an image that doesn't match the pattern isn't failed. It's never examined. So docker.io/library/nginx:1.27, with a pinned tag and resource limits, walked straight into the workload namespace, unsigned, while both verification policies reported healthy. Every test I had passed, because every test used my own registry.
I had confused two different guarantees. "Our images are signed" needs signature verification. "Only signed images run" also needs an allowlist: a fifth policy that refuses any image from outside ghcr.io/yodim/. Its pattern is identical to the verification policy's on purpose, so anything the allowlist lets in is guaranteed to be checked. The trailing slash matters more than it looks: without it, ghcr.io/yodim-evil/app, from an account anyone can register, would pass.
Proving the fix had its own twist. To show the new check fails without the policy, I deleted the policy and ran make verify. It passed. Argo CD had restored the policy from git within a second, so the test was measuring GitOps reflexes, not the control. Pausing sync on the policies app wasn't enough either, because the root app put the child's sync settings back. Only with both paused did the deletion stick. Then the new check went red while the other two stayed green, and that combination is what makes it evidence: a healthy, enforcing cluster with exactly one control missing. If you test controls in a self-healing cluster, expect your red test to fight your own automation.
One limit, stated plainly: the allowlist covers the workload namespace only. kube-system, Argo CD and Kyverno pull from upstream registries, and locking them down means mirroring every image the platform itself uses. That's real work, and it's next on the list, along with flipping the scope so new namespaces are covered by default instead of by remembering to add them.
I'm including all three because they're the same bug as the one in the opening paragraph, just one level in. A scan nobody reads is theater. A signature nobody verifies is theater. A policy field that's pruned before it reaches the API, a green test that would pass with the controls switched off, and a policy that reports healthy while ignoring everything it wasn't told about, are also theater, and they're considerably harder to spot, because everything renders as working. The lesson I took from building this: for every control, know the failure you would see if it were silently off. If that looks the same as success, you don't have a control yet, you have a decoration.
Which is why the thing worth checking in your own cluster isn't whether the pipeline is green. It's whether you can make it go red on purpose.
What make verify proves
Everything above compresses into the output of make verify:
==> Positive control: signed demo-app should be Running
PASS signed + attested image admitted and running
==> Negative control: unsigned image should be rejected at admission
PASS unsigned image rejected by signature verification
==> Negative control: image from an unapproved registry should be rejected
PASS image from an unapproved registry rejected by the registry allowlist
The unsigned-image line used to read "rejected by Kyverno admission", which was true and almost meaningless. Naming the reason is the difference between a test and a screenshot.
If you remember one thing, make it the two negative controls. Everything that happens before admission is preparation. The rejection is the platform.
Take the pieces
The repo is structured as a catalog, not a monolith: each policy, the Terraform module and the two reusable GitHub Actions workflows (secure build, source scan) are self-contained, documented, tested where offline testing is possible, and Apache-2.0. You can lift the signature policy into your cluster without adopting anything else, or wire the whole secure build into your repo with a single uses: line.
One thing to know before you run it. If you just clone it, make verify checks my published images against my signing identity and should go green. If you fork it to build your own images, they won't verify at first, because the trust pin above is an identity, not a general "is this signed" check. You point the policies at your own workflow, let your CI build and sign the demo image once, and then it passes. That ordering is inherent to pinning identity rather than a rough edge, and the local-testing doc walks through it.
If you do take a piece and it breaks, or the trade-off I picked is wrong for your context, open an issue and tell me why. For a reference project, "I tried to reuse this and here's where it hurt" is the most valuable feedback there is.
Mehdi Batrone, Systems & DevSecOps engineer.
The repo is at github.com/yodim/k8s-secure-supply-chain, and more of my work is at batrone.com.
Top comments (0)