We run an auth service, so the question "is the container that just started actually the container we built?" is not academic. Our answer was signatures: sign every image in CI, verify before deploy, and then, as a second layer, put a webhook in the cluster so that even a hand-typed kubectl could not run something unsigned. That went in on a Tuesday afternoon. By evening our auth Deployment on the dev cluster had accumulated 2,242 ReplicaSets, was minting a new one roughly every three seconds, reported Available=False, and served nothing.
The obvious hypothesis is that the new webhook was rejecting our images. It was not. Its logs, for every request, said allowed: true. It admitted everything we sent it, all evening, while the Deployment it was admitting fell apart.
The layer we were adding
The pipeline already signed and verified. Every image gets a keyless signature at build time, tied to the GitHub Actions OIDC identity, and the deploy job runs a verification against that identity before anything reaches the cluster. That gate is fail-closed and it is the real control.
The cluster-side layer is defence in depth: sigstore's policy-controller, running as an admission webhook, holding two policies. One says images matching our own registry path must carry a keyless signature from our workflow identity. The other is a catch-all that lets everything else pass, because "no policy matched" means deny, and without the catch-all the cluster loses its Vault agents, its CSI drivers, and every sidecar it did not build. Install, label the namespace, done. The label is the switch: no label, no enforcement.
The first outage, which was boring
Turning it on with failurePolicy: Fail and the chart's default ten second timeout took dev down almost immediately, in the way everyone expects an admission webhook to take you down. A cold controller has to reach Fulcio and Rekor to verify a signature it has not seen before. Cold, that work does not finish in ten seconds. Fail-closed plus a blown deadline means pod creation is refused, which means the rollout cannot place pods, which means the service has no replicas.
That failure mode is well documented and the fix is the documented one: raise the webhook timeout to thirty seconds and set failurePolicy: Ignore. Ignore sounds like giving up, and in a single-layer design it would be. In ours the pipeline gate is the fail-closed primary, and the cluster layer exists to catch what never went through the pipeline at all. A webhook that can never take the cluster down is worth more to us than a webhook that catches the last one percent, because the last one percent has already been caught upstream.
We shipped that at 17:40. It fixed the first outage completely. It also created the second one, and this is the part worth reading.
Mutating, not just validating
Everyone thinks of an admission webhook as a bouncer: it inspects the object and returns yes or no. Policy-controller is not only that. It is a mutating webhook, and what it mutates is the thing it just verified. When it admits a pod spec that references an image by tag, it rewrites that reference to include the digest it resolved and checked. myregistry.io/authagonal-auth:abc123 goes in. myregistry.io/authagonal-auth:abc123@sha256:... comes out.
That is a genuinely good idea. A tag is a mutable pointer, so verifying a tag and then letting the kubelet resolve it again later leaves a window where the two could differ. Pinning the digest at admission time closes it. The object that runs is the object that was verified.
Now put that next to how a Deployment works. The Deployment controller hashes your pod template, and that hash is what identifies the ReplicaSet that owns the pods. It hashes the template you declared. The webhook rewrites the template on the ReplicaSet it admits. So if your Deployment's template says :abc123, its own child ReplicaSet says :abc123@sha256:..., and the two no longer agree.
The controller reconciles, hashes the template, looks for a ReplicaSet with that hash, and finds one whose template is different. There is exactly one thing that means in Kubernetes: a hash collision, two different templates landing on the same hash. The controller handles collisions the way it is supposed to. It increments collisionCount, which perturbs the hash, and creates a new ReplicaSet. Three seconds later it reconciles again, and the new ReplicaSet has been mutated too. The loop does not converge, because the disagreement it is trying to resolve is being recreated by the webhook every time it tries.
Two thousand two hundred and forty-two ReplicaSets is what an infinite loop looks like when you catch it in the evening rather than the morning.
Why only one Deployment broke
Four workloads run in that namespace. One spiralled. Three were completely fine, which is the detail that kept us looking in the wrong place, because a systemic misconfiguration should not be selective.
failurePolicy: Ignore is why. When the webhook is slow or cold and a request times out, Ignore means the object is admitted unmutated. Whether a given apply came back with a digest baked in or with the bare tag it went in with depended on whether the webhook answered in time. The three healthy Deployments had been applied while it was warm, so their templates already carried digests, and the webhook's rewrite was a no-op that matched what was already there. The one that spiralled had been applied while the webhook was cold, kept its bare tag in the template, and then had every child ReplicaSet mutated out from under it.
So the trigger was a race, decided per apply, that any of the four could have lost on any deploy. Ignore did not cause the bug. It turned a deterministic bug into an intermittent one and hid it behind three healthy workloads.
The fix is a fixed point
The instinct is to stop the webhook from mutating. Better instinct: give it nothing to mutate. If the pod template we submit is already exactly what the webhook would produce, the rewrite changes nothing, the hash stays stable, and the loop cannot start.
So the deploy job now resolves the digest itself before applying. For each image it asks the registry what the tag currently points at, then writes the fully pinned reference into the overlay:
digest=$(az acr repository show --name "$acr" \
--image "authagonal-${img}:${tag}" --query digest -o tsv)
kustomize edit set image "${ref}=${ref}:${tag}@${digest}"
What ships is :tag@sha256:..., tag and digest together. The tag stays for human readability, the digest is what actually resolves. The webhook verifies it, finds nothing to rewrite, and returns the object unchanged. collisionCount has been flat ever since.
There is a small trap in the tail of this. Because CI is what writes the digest, applying the overlay by hand from a laptop sends the base manifests' placeholder tag instead, and the policy now correctly refuses it with "must be an image digest." The cluster is telling you the truth: that thing you just typed was never verified.
A mutating webhook is a second writer, not a checkpoint
A mutating admission webhook is not a checkpoint. It is a writer inside a control loop that is already comparing what you asked for against what exists. Two writers with different opinions about the same field is not a policy question, it is a distributed systems question, and the one thing that makes it safe is idempotence: the state you declare has to be a fixed point of the mutation, so that applying the mutation to it produces it again.
That reframing generalises past sigstore. Anything that rewrites your specs at admission time, whether it injects sidecars, adds default resource limits, or normalises image references, is in the same position, and the same question applies. If you ran your own manifest through this thing, would you get your own manifest back? If not, something is going to keep noticing the difference. It will be patient, and it will be much faster than you.
And the smaller lesson, the one we actually had to unlearn on the night: allowed: true does not mean the webhook is not the problem. We spent an hour treating those logs as an alibi. The webhook was telling the truth the whole time. We were asking it the wrong question, because we had only ever thought of it as something that says no.
If you would rather your identity provider came with its supply chain already argued out, Authagonal is a hosted auth service whose images are signed in CI, verified before deploy, and pinned by digest at the point of admission, which is a sentence we can write because we already lost an evening earning it.
Top comments (0)