Sign and Verify Container Images with Cosign and an Admission Policy That Actually Blocks

Signing a container image is a ten-minute task. Making your cluster refuse anything unsigned is the part most teams never finish. The result is a supply chain security story that looks great in a slide deck and provides exactly zero protection at runtime: the signatures exist in the registry, nobody checks them, and a kubectl run with an arbitrary image from Docker Hub still lands in production.

This article builds the full chain end to end: keyless OIDC signing in CI, a CycloneDX SBOM attestation attached to the image, and a Kyverno verifyImages policy that fails closed on an OpenShift or Kubernetes cluster. Every step is verifiable from a terminal.

The problem: signatures nobody verifies

A Cosign signature is just another OCI artifact pushed next to your image (historically a sha256-<digest>.sig tag, now also discoverable through the OCI referrers API). Pushing it changes nothing about how the kubelet pulls images. The kubelet resolves the tag, pulls whatever the registry hands back, and starts the container.

That leaves three concrete attack paths wide open:

1. Registry compromise or tag mutation. app:v1.4.2 is a mutable pointer. Anyone with push access — a leaked CI token, a compromised bot account — can repoint it at a different digest.
2. Typosquatting and copy-paste deploys. A developer deploys ghrc.io/acme/api (note the typo) or a random docker.io/someuser/debug-tools:latest image to "quickly check something" in a namespace that happens to be production-adjacent.
3. Provenance gaps. You have no machine-checkable proof that a given digest came out of your pipeline, from your repository, on your protected branch.

Admission control is the only place where you can turn "we sign images" into an invariant. And it has to be fail-closed: a policy in Audit mode is a logging feature, not a control.

The chain we are building

• CI builds the image and pushes it by digest.
• Cosign signs the digest keyless: a short-lived certificate from Fulcio bound to the GitHub Actions OIDC identity, logged in Rekor. No private key to store or rotate.
• Syft generates a CycloneDX SBOM; Cosign wraps it in an in-toto attestation signed with the same identity.
• Kyverno intercepts every Pod creation, verifies signature and attestation against the expected identity and issuer, and rewrites the tag to the verified digest.

Step 1 — Keyless signing in CI

Why keyless

Key-based signing means a long-lived private key in a CI secret. Whoever can read that secret can sign anything, forever, and you will not know. Keyless signing replaces the key with a workflow identity: Cosign requests an OIDC token from the CI provider, Fulcio issues a certificate valid for ~10 minutes with that identity embedded in a SAN extension, the signature and certificate are recorded in the Rekor transparency log, and the key material is discarded.

Verification then becomes a statement about who signed, not about which key file you trust: "this image was signed by the workflow .github/workflows/release.yml in acme/payment-api on refs/heads/main".

The workflow

name: build-sign-attest

on:
  push:
    branches: [main]

permissions:
  contents: read
  packages: write
  id-token: write          # mandatory: without it, keyless signing fails

jobs:
  build:
    runs-on: ubuntu-24.04
    env:
      IMAGE: ghcr.io/acme/payment-api
    steps:
      - uses: actions/checkout@v4

      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - id: build
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ${{ env.IMAGE }}:${{ github.sha }}

      - uses: sigstore/cosign-installer@v3.7.0
      - uses: anchore/sbom-action/download-syft@v0

      - name: Sign and attest by digest
        env:
          DIGEST: ${{ steps.build.outputs.digest }}
        run: |
          set -euo pipefail
          REF="${IMAGE}@${DIGEST}"

          cosign sign --yes "$REF"

          syft "$REF" -o cyclonedx-json=sbom.cdx.json
          cosign attest --yes --type cyclonedx \
            --predicate sbom.cdx.json "$REF"

          cosign verify \
            --certificate-identity-regexp "^https://github\.com/acme/payment-api/\.github/workflows/.+@refs/heads/main$" \
            --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
            "$REF" > /dev/null
          echo "signed: $REF"

Three details matter here:

• Always sign the digest, never the tag. cosign sign image:v1 resolves the tag first anyway, but scripting by digest removes the race window between push and sign.
• id-token: write is the permission that enables the OIDC token. Missing it is the number one cause of error getting signer: getting key from Fulcio.
• Verify in CI too. It costs two seconds and catches a broken signing step before the cluster does.

Inspect what landed in the registry

cosign tree ghcr.io/acme/payment-api@sha256:3f1e...c9

# └── 💾 Attestations for an image tag: ghcr.io/acme/payment-api:sha256-3f1e...c9.att
#    └── 🍒 sha256:a71d...
# └── 🔐 Signatures for an image tag: ghcr.io/acme/payment-api:sha256-3f1e...c9.sig
#    └── 🍒 sha256:0c44...

And check the SBOM attestation independently:

cosign verify-attestation --type cyclonedx \
  --certificate-identity-regexp "^https://github\.com/acme/payment-api/" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  ghcr.io/acme/payment-api@sha256:3f1e...c9 \
  | jq -r '.payload | @base64d | fromjson | .predicate.components | length'

Step 2 — Attestation vs. attached SBOM

cosign attach sbom is deprecated and was never signed by default. An attestation is an in-toto statement — {subject: [digest], predicateType: "https://cyclonedx.org/bom", predicate: {...}} — wrapped in a DSSE envelope and signed. Only attestations are worth enforcing on, because only they bind the SBOM to the exact image digest with a verifiable identity.

Two predicate types you will meet:

Tool flag	predicateType
--type cyclonedx	https://cyclonedx.org/bom
--type spdxjson	https://spdx.dev/Document
--type slsaprovenance1	https://slsa.dev/provenance/v1

Pick one and be consistent — Kyverno matches on the predicate type string.

Step 3 — Enforcing at admission with Kyverno

Install

helm repo add kyverno https://kyverno.github.io/kyverno
helm repo update
helm install kyverno kyverno/kyverno \
  --namespace kyverno --create-namespace \
  --set admissionController.replicas=3 \
  --set admissionController.container.extraArgs."imagePullSecrets"=ghcr-pull

oc -n kyverno get pods

On OpenShift, Kyverno runs fine under the restricted-v2 SCC. If your images live in a private registry, Kyverno needs credentials to fetch signatures — create the secret in the Kyverno namespace:

oc -n kyverno create secret docker-registry ghcr-pull \
  --docker-server=ghcr.io \
  --docker-username=acme-bot \
  --docker-password="$GHCR_TOKEN"

Before enforcing anything, extend the resource filters so Kyverno never evaluates cluster operator workloads:

oc -n kyverno edit configmap kyverno
# add to resourceFilters:
#   [*/*,openshift-*,*][*/*,kube-system,*][*/*,kyverno,*]

This is not optional on OpenShift. A fail-closed webhook that also intercepts openshift-etcd or openshift-apiserver pods will brick a node reboot.

The policy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-signed-images
  annotations:
    policies.kyverno.io/severity: critical
spec:
  validationFailureAction: Enforce   # Kyverno >=1.13: use rule-level failureAction
  webhookTimeoutSeconds: 30
  failurePolicy: Fail                # fail closed
  background: false
  rules:
    - name: verify-cosign-keyless
      match:
        any:
          - resources:
              kinds:
                - Pod
              namespaceSelector:
                matchLabels:
                  image-policy: enforced
      verifyImages:
        - imageReferences:
            - "ghcr.io/acme/*"
          required: true
          mutateDigest: true        # rewrite tag -> verified digest
          verifyDigest: true
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/acme/*/.github/workflows/*@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev
          attestations:
            - type: https://cyclonedx.org/bom
              attestors:
                - count: 1
                  entries:
                    - keyless:
                        subject: "https://github.com/acme/*/.github/workflows/*@refs/heads/main"
                        issuer: "https://token.actions.githubusercontent.com"
                        rekor:
                          url: https://rekor.sigstore.dev
              conditions:
                - all:
                    - key: "{{ components[?type=='library'] | length(@) }}"
                      operator: GreaterThan
                      value: 0

Note: on Kyverno 1.11 and later the attestation field is type; older releases used predicateType. The conditions block runs JMESPath against the predicate body — here it simply asserts the SBOM is not empty, but this is where you would reject images containing a banned package or an SBOM older than N days.

Close the bypass: restrict allowed registries

verifyImages only evaluates images matching imageReferences. Deploy docker.io/library/nginx and the rule never fires. You need a companion rule:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  failurePolicy: Fail
  background: true
  rules:
    - name: only-trusted-registries
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaceSelector:
                matchLabels:
                  image-policy: enforced
      validate:
        message: "Images must come from ghcr.io/acme/ or the internal OpenShift registry."
        pattern:
          spec:
            =(initContainers):
              - image: "ghcr.io/acme/* | image-registry.openshift-image-registry.svc:5000/*"
            containers:
              - image: "ghcr.io/acme/* | image-registry.openshift-image-registry.svc:5000/*"

Without this second policy, the first one is theater.

Test it

oc create namespace prod-payments
oc label namespace prod-payments image-policy=enforced

# 1. unsigned third-party image -> blocked by the registry policy
oc -n prod-payments run nginx --image=docker.io/library/nginx:1.27
# Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
#   resource Pod/prod-payments/nginx was blocked due to the following policies
#   restrict-image-registries: only-trusted-registries: 'Images must come from ...'

# 2. image from the right registry but never signed -> blocked by verifyImages
oc -n prod-payments run legacy --image=ghcr.io/acme/legacy-tool:0.1
# ... image verification failed for ghcr.io/acme/legacy-tool:0.1:
#     .attestors[0].entries[0].keyless: no matching signatures

# 3. properly signed image -> admitted, and mutated to a digest
oc -n prod-payments run api --image=ghcr.io/acme/payment-api:9f2c1ab
oc -n prod-payments get pod api -o jsonpath='{.spec.containers[0].image}{"\n"}'
# ghcr.io/acme/payment-api:9f2c1ab@sha256:3f1e...c9

That last line is the whole point: what the kubelet pulls is a digest Kyverno verified, not a mutable tag that could have moved since.

You can also test policies in CI without a cluster:

kyverno apply policies/verify-signed-images.yaml \
  --resource manifests/deployment.yaml --cluster=false

Failure modes to plan for before you flip to Enforce

• Rekor or Fulcio outage. With failurePolicy: Fail and the public Sigstore instances, an upstream outage blocks every pod creation in enforced namespaces. Mitigate with Kyverno's image verification cache (enabled by default, TTL configurable) and by running enforcement only on namespaces that tolerate it. Air-gapped clusters should run their own Fulcio/Rekor and point rekor.url and ctlog at them after cosign initialize --mirror <tuf-mirror> --root <root.json>.
• Registry rate limits and auth. Every unique digest triggers registry calls from the Kyverno admission controller. Missing credentials surface as failed to fetch image descriptor: UNAUTHORIZED, which with fail-closed means an outage.
• Webhook timeout. 30 seconds is generous; if verification regularly approaches it, your Kyverno pods are undersized or the registry is slow. Scale admissionController.replicas to 3 and set resource requests.
• Image deletion / GC. If a signature is garbage-collected by a registry retention policy while the image remains, running pods are fine but any rescheduling fails.

Security best practices

• Pin by digest in Git, not just at admission. mutateDigest protects the cluster, but your GitOps repo should still reference digests so rollbacks are deterministic.
• Make the identity regex strict. subject: "https://github.com/acme/*" accepts any workflow in any repo of the org, including a PR workflow a contributor just added. Pin the repo, the workflow path and @refs/heads/main or @refs/tags/v*.
• Never use --certificate-identity-regexp '.*' or Kyverno's subject: "*". That verifies only that somebody signed, which is worthless.
• Keep CI OIDC permissions minimal. id-token: write at job level, not workflow level, and never on workflows triggered by pull_request_target.
• Enforce namespace by namespace. Roll out with a namespaceSelector label, start in Audit, read the PolicyReport objects (oc get polr -A), then flip to Enforce.
• Use PolicyException resources with an expiry process rather than deleting the policy when something breaks at 2 a.m.
• Add defense in depth at the node level. OpenShift 4.17+ supports ClusterImagePolicy/ImagePolicy CRs that configure sigstore verification directly in CRI-O. Admission control protects the API server path; CRI-O policy protects every pull, including static pods.
• Monitor the policy itself. Alert on kyverno_admission_requests_total dropping to zero for a policy, and on Deployments of the Kyverno webhook configuration being deleted. A policy that silently stopped matching is indistinguishable from no policy.
• Treat the SBOM attestation as data, not decoration. Once it is enforced to exist, wire the conditions block to actually reject known-bad components, or feed the attestations into a VEX workflow.

Conclusion

The gap between signing and enforcing is where most supply chain programmes die. Closing it requires four things, and all four are load-bearing:

1. Keyless signing by digest in CI, with id-token: write and a self-verification step so a broken signer fails the build, not the deploy.
2. A signed SBOM attestation, not an attached blob — in-toto + DSSE is what binds the SBOM to the digest and to your workflow identity.
3. A Kyverno verifyImages policy with failurePolicy: Fail, required: true and mutateDigest: true, scoped by namespace label so you can roll it out incrementally.
4. A registry allow-list rule, because verifyImages ignores every image that does not match imageReferences — without it, the bypass is a single docker.io/ prefix.

Verify the whole chain the way an attacker would: try to deploy an unsigned image into an enforced namespace and confirm the API server says no. If oc run nginx still succeeds, nothing upstream of that matters.