devtocash

Posted on Sep 19 Originally published at devtocash.com

Cosign Keyless Signing in GitHub Actions: Sign Images, Attest SBOMs, and Enforce It with Kyverno

#security #devops #kubernetes #githubactions

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

The short version

Cosign keyless signing means your GitHub Actions job proves its identity with its OIDC token, gets a ten-minute certificate from Sigstore's Fulcio CA, signs the image digest, and throws the private key away. There is no signing key to store, rotate, or leak. What you verify later is not "was this signed by key X" but "was this signed by this workflow file, on this branch, in this repo."

The whole loop is three moves: add id-token: write and a cosign sign step to the build job, attach an SBOM as a signed attestation, and make the cluster refuse anything that fails verifyImages. Most guides stop at the first move. The parts that actually bite in production are the identity string you verify against, the registry cleanup rule that silently deletes your signatures, and what happens to pod scheduling when the verifier can't reach the registry. This guide covers all of it.

Step 1: sign by digest in the build job

# .github/workflows/build-sign.yml
name: build-sign
on:
  push:
    branches: [main]

permissions:
  contents: read
  packages: write
  id-token: write        # without this, keyless signing has no identity

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: sigstore/cosign-installer@v3
        with:
          cosign-release: "v2.5.3"   # pin it; see the format note below

      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Lowercase image name
        run: echo "IMAGE=ghcr.io/${GITHUB_REPOSITORY,,}" >> "$GITHUB_ENV"

      - id: build
        uses: docker/build-push-action@v6
        with:
          push: true
          tags: ${{ env.IMAGE }}:${{ github.sha }}

      - name: Sign the digest
        env:
          DIGEST: ${{ steps.build.outputs.digest }}
        run: cosign sign --yes "${IMAGE}@${DIGEST}"

Three details in there are deliberate.

Sign the digest, never the tag. A tag is a mutable pointer. Between your push and your sign step, anything with registry write access can move it, and you would be signing someone else's bytes. build-push-action hands you the digest it just pushed; use it. Cosign prints a warning when you sign by tag for exactly this reason.

Lowercase the reference yourself. GITHUB_REPOSITORY preserves case, so a repo named Acme/Payments-API produces an invalid OCI reference. The Docker actions quietly lowercase for you; your hand-built cosign sign argument does not, and the error message ("could not parse reference") doesn't mention case.

Pin the cosign release. Cosign 3 changed the default way signatures are stored (a new bundle format attached as OCI referrers, rather than the sha256-....sig tag scheme of 2.x). Both are fine, but your admission controller has to understand whatever your pipeline writes. Pin the signer, test verification in the cluster against that exact version, and upgrade both sides on purpose rather than when an installer default moves.

If your build also runs on pull_request, gate the signing step with if: github.event_name != 'pull_request'. Fork PRs don't receive an OIDC token anyway, and you do not want same-repo PR branches producing signatures at all — more on that next.

Step 2: know the identity you are verifying

Verification takes two required inputs: the OIDC issuer and the certificate identity.

cosign verify \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  --certificate-identity "https://github.com/acme/payments-api/.github/workflows/build-sign.yml@refs/heads/main" \
  ghcr.io/acme/payments-api@sha256:4f0c...e91a | jq '.[0].optional'

That identity string is the workflow file path plus the git ref it ran from. This is where most policies are written too loosely. A pattern like https://github.com/acme/* accepts a signature from any workflow on any branch in any repo in the org. Anyone who can push a branch can add a workflow that builds an arbitrary image, signs it, and passes your admission policy. The signature is genuine; the policy just doesn't say anything useful.

Pin at least the workflow filename and the ref:

What you deploy from	Identity to require
`main` only	`.../.github/workflows/build-sign.yml@refs/heads/main`
Release tags	`.../.github/workflows/build-sign.yml@refs/tags/v*`
PR builds	Never. A PR run's ref is `refs/pull/N/merge` — don't match it

Then protect that ref: branch protection on main, and CODEOWNERS on .github/workflows/. The signature's trust is exactly as strong as the review gate on the file it names.

The reusable-workflow gotcha. If signing happens inside a reusable workflow (say acme/ci-templates/.github/workflows/build.yml), the certificate identity is the reusable workflow's path and ref — not the calling repo's. Teams discover this when every verification fails after centralizing CI. It is actually the better design: one blessed pipeline is the only thing in the org that can produce an admissible signature. To still distinguish callers, check the certificate extensions:

cosign verify \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  --certificate-identity "https://github.com/acme/ci-templates/.github/workflows/build.yml@refs/tags/v2" \
  --certificate-github-workflow-repository "acme/payments-api" \
  --certificate-github-workflow-ref "refs/heads/main" \
  ghcr.io/acme/payments-api@sha256:4f0c...e91a

Step 3: attach the SBOM as an attestation

Older tutorials (including the snippet in our Kubernetes security best practices checklist) use cosign attach sbom. That command is deprecated, and for a good reason: an attached SBOM is an unsigned blob sitting next to the image. Anyone with push access can replace it. An attestation wraps the SBOM in a signed in-toto statement bound to the image digest, carrying the same workflow identity as the signature.

Add to the job, after the sign step:

      - uses: anchore/sbom-action/download-syft@v0

      - name: Generate and attest SBOM
        env:
          DIGEST: ${{ steps.build.outputs.digest }}
        run: |
          syft "${IMAGE}@${DIGEST}" -o spdx-json=sbom.spdx.json
          cosign attest --yes --type spdxjson \
            --predicate sbom.spdx.json "${IMAGE}@${DIGEST}"

And prove it round-trips — this is the command you will want during the next "are we running the affected version of libfoo" fire drill:

cosign verify-attestation --type spdxjson \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  --certificate-identity "https://github.com/acme/payments-api/.github/workflows/build-sign.yml@refs/heads/main" \
  ghcr.io/acme/payments-api@sha256:4f0c...e91a \
  | jq -r '.payload' | base64 -d \
  | jq -r '.predicate.packages[] | select(.name=="openssl") | .versionInfo'

The same mechanism carries scan results. Trivy can emit a cosign-shaped predicate (trivy image --format cosign-vuln --output vuln.json), which you attest with --type vuln. That lets admission require "scanned within the last 7 days" rather than trusting that CI ran a scanner at some point. Feed the scan itself to something like the CVE triage agent so the findings become a ranked fix list instead of a wall of red.

cosign tree shows everything hanging off a digest, which is the fastest sanity check that both artifacts landed:

cosign tree ghcr.io/acme/payments-api@sha256:4f0c...e91a

Step 4: enforce at admission with Kyverno

A signature nobody checks is decoration. If Kyverno isn't installed yet, the Kyverno policy-as-code guide covers the HA install; this policy is the hardened version of the short verifyImages example there.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Audit      # flip to Enforce after a clean deploy cycle
  webhookTimeoutSeconds: 30
  background: false
  rules:
    - name: require-ci-signature
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["payments", "checkout"]
      verifyImages:
        - imageReferences:
            - "ghcr.io/acme/*"
          mutateDigest: true
          verifyDigest: true
          required: true
          attestors:
            - entries:
                - keyless:
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/acme/*/.github/workflows/build-sign.yml@refs/heads/main"
                    rekor:
                      url: https://rekor.sigstore.dev
          attestations:
            - type: https://spdx.dev/Document
              attestors:
                - entries:
                    - keyless:
                        issuer: "https://token.actions.githubusercontent.com"
                        subject: "https://github.com/acme/*/.github/workflows/build-sign.yml@refs/heads/main"

What each non-obvious field buys you:

mutateDigest: true rewrites image: ...:v1.4.2 to the verified digest at admission. What was verified is what runs, even if the tag moves a minute later.
The wildcard sits in the repo position only. Workflow filename and ref stay pinned, so the policy scales across services without reopening the "any workflow, any branch" hole.
The attestations block requires a signed SPDX SBOM from the same identity. No SBOM, no admission. Add a second entry with type https://cosign.sigstore.dev/attestation/vuln/v1 and a time_since condition on metadata.scanFinishedOn when you're ready to enforce scan freshness.
Namespaced match is the rollout lever. Start with two namespaces, not the cluster. Never match kube-system or the kyverno namespace itself.

For private registries, Kyverno must be able to pull signatures: reference a pull secret with imageRegistryCredentials.secrets on the verifyImages entry, or grant the admission controller's ServiceAccount registry access via IRSA/workload identity.

Test without creating anything — server-side dry run still goes through admission webhooks:

kubectl run sigtest -n payments --dry-run=server \
  --image=ghcr.io/acme/payments-api:unsigned-test
# Enforce mode: "image verification failed ... no matching signatures"

kubectl get policyreport -n payments \
  -o jsonpath='{range .items[*].results[?(@.result=="fail")]}{.policy}{"  "}{.resources[0].name}{"\n"}{end}'

Stay in Audit through at least one full deploy cycle of every service in the namespace, including CronJobs that fire weekly. With background: false, audit only sees what gets admitted during the window. The usual surprises are third-party sidecars pulled through your registry mirror and a forgotten job still built by an old pipeline.

The failure modes nobody mentions

Registry cleanup deletes your signatures. With the 2.x tag scheme, signatures and attestations live in the same repo as sha256-DIGEST.sig and sha256-DIGEST.att tags. A common ECR lifecycle setup is "keep the last 30 tags prefixed v" plus a catch-all "expire everything else after 14 days." The .sig tags don't start with v. On day 15 the image is still there, its signature is gone, and the next node scale-up can't admit the pod. Add a higher-priority rule with tagPatternList: ["sha256-*"] whose retention matches your release images. With referrer-based storage, the equivalent trap is any "delete untagged manifests" job — referrers are untagged by design.

Admission now depends on the registry and the verifier. Every uncached verification pulls signature manifests. Kyverno caches results (--imageVerifyCacheTTLDuration, 60 minutes by default), and digest mutation makes cache hits likely, but a registry outage plus failurePolicy: Fail means new pods don't schedule — during exactly the kind of incident where you are trying to scale out. Decide this on purpose: keep Fail for production namespaces, run the admission controller with three replicas, and have the break-glass path written down before you need it:

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: hotfix-payments-2026-09-19
  namespace: kyverno
spec:
  exceptions:
    - policyName: verify-image-signatures
      ruleNames: ["require-ci-signature", "autogen-require-ci-signature"]
  match:
    any:
      - resources:
          kinds: ["Pod", "Deployment"]
          namespaces: ["payments"]
          names: ["payments-api-hotfix*"]

Note the autogen- rule name: Kyverno generates a Deployment-level rule from your Pod rule, and an exception that lists only the original name still blocks the Deployment. Restrict who can create PolicyException objects with RBAC, and alert on every one created — an exception is a signed-off bypass, so treat it like one. If AI agents deploy into this cluster, the same rule applies to them; their write path should be pull requests through the signed pipeline, as described in GitOps for AI agents, never a ServiceAccount that can create exceptions.

Keyless signing is public. Every signature writes an entry to the public Rekor transparency log containing your workflow identity — org name, repo name, workflow path. For most teams that's acceptable; if repo names are sensitive, sign with a KMS key instead (cosign sign --key awskms:///alias/image-signing) and verify with the keys attestor type. You trade the leak for a key you must now protect and rotate.

A signature is provenance, not safety. It proves which pipeline built the bytes. It says nothing about whether those bytes contain a vulnerable dependency or whether the commit was malicious. Signing closes the "someone pushed straight to the registry" path; it needs scanning, branch protection, and — for the GitOps half — the manifest-side controls in Argo CD 3.5 supply chain security to be a supply-chain story rather than a checkbox.

Rollout checklist

Add id-token: write, pinned cosign, and sign-by-digest to one service's workflow. Verify locally with the full identity string.
Add the SBOM attestation; confirm with cosign tree and verify-attestation.
Fix registry retention so .sig and .att artifacts live as long as the images they cover.
Deploy the Kyverno policy in Audit, scoped to one or two namespaces. Read the policy reports for a full deploy cycle.
Write and test the PolicyException break-glass, with RBAC and an alert on creation.
Flip to Enforce per namespace. Expand the namespace list only after each one has been quiet for a week.

Steps 1 and 2 are an afternoon. Steps 3 through 5 are the ones that decide whether the first person to notice your signing policy is an attacker or your own on-call engineer at 3 a.m.

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.

DEV Community