ImagePullBackOff on an image nobody had touched since March

Our WebRTC signaling server was unavailable for 2 days, 17 hours and 55 minutes, from 2026-11-08 13:46 to 2026-11-11 07:41 UTC. The last commit to its code was on 2026-03-23 and the Deployment had pointed at the same image tag since then. The outage started with a disk filling up on the node. It lasted because of what a registry cron job had done to that image on 2026-03-24.

This is a two-node k3s cluster that I run alone for a few side projects. Times are UTC.

What it looked like

By the morning of 2026-11-11 the Deployment's ReplicaSet had three pods. The original, created in March, had ended on 2026-11-08 at 13:46:42 with exit code 0 and phase Succeeded. A replacement created one second later had never run and ended with ContainerStatusUnknown, exit code 137. A second replacement, created at 16:10:52, was Pending with this event repeating every five minutes:

Failed to pull image "registry.example.com/dam-signaling:prod-<sha>":
  failed to pull and unpack image ...:
  .../manifests/sha256:211614369c9d... : not found

The audit tool I mention below counted 1,565 of these events in 24 hours on that node. On the registry side, the server log had 1,068 requests for that manifest digest since 2026-11-09 23:58, every one answered 404 manifest unknown.

The usual causes of manifest unknown did not fit. The tag existed in the registry's filesystem. The same node had pulled other images from the same registry that afternoon. The Deployment had not changed since March.

Finding it

I found the outage on 2026-11-11 at 05:08, from a read-only audit of the cluster (kubegap) that I had run for another reason. It listed the Deployment with zero available replicas and the pull failures next to it. The investigation then went through the disk, the registry, and the node cache, in that order.

The disk

The node's root filesystem is 106.7 GiB. On 2026-11-08 free space went from 19.4 GiB at 12:45 to 2.6 GiB at 13:48. The node reported DiskPressure from 13:47 to 14:10 and again from 16:10 to 16:20.

node_filesystem_avail_bytes{mountpoint="/"}      GiB, one-minute samples
12:45  19.4
12:58   8.1
13:07  11.8
13:38  11.4
13:48   2.6     DiskPressure 13:47–14:10
13:56  14.4
14:05  27.3
14:18  13.3

I caused this. That afternoon I pushed 25 deploy commits for a new service, and one of them shipped a 4.5 GiB image with the service's ML models inside. The Deployment picked it up at 13:37 and the pod started at 13:40. The kubelet's evictionHard is imagefs.available<5% and nodefs.available<5%; on k3s with a single root filesystem those are the same disk. When a node crosses that threshold the kubelet first reclaims node-level resources, which for an image filesystem means deleting every image no container is using, and then evicts pods if it is still under. The jump from 2.6 GiB to 27.3 GiB free between 13:48 and 14:05 is consistent with that. By 16:15 the node's eviction counters read 23 for imagefs and 27 for nodefs, and 16 pods across the cluster carried the Evicted reason.

The signaling pod is not among those 16. It ended at 13:46:42, one minute before the DiskPressure condition was recorded, with exit code 0. That is not what an eviction looks like, and I could not read the kubelet journal on that node that day, so I do not know what stopped it. The sequence I can show is: the container stopped inside the disk-pressure window; the first replacement died during the second window; the second replacement needed an image that was no longer in the node's store.

The models-in-image problem got the obvious fix the same evening: models on a volume, 1.6 GiB image.

The registry

The pull was failing on a tag that had been pulled without trouble in March. The registry is docker/distribution 2.8.3 with filesystem storage, so I looked at the files:

repositories/dam-signaling/_manifests/tags/prod-<sha>/current/link
  -> sha256:447fa22a...      OCI image index, 2 children

children of the index
  sha256:211614369c9d...     linux/amd64 image manifest   (what containerd asked for)
  sha256:203e8548...         annotation vnd.docker.reference.type: attestation-manifest

repositories/dam-signaling/_manifests/revisions/sha256/
  447fa22a...                present
  211614369c9d...            missing
  203e8548...                missing
  (directory mtime 2026-03-24 04:00:01)

blobs referenced by the image manifest: 10
  present     4   (base layers shared with other images)
  missing     6   (config + 5 layers)

The tag pointed at an image index. The index was there. Both manifests the index pointed at had been unlinked from the repository, and 6 of the 10 blobs behind the platform manifest were gone. The revisions directory had last been written at 04:00:01 on 2026-03-24, the night after the push.

04:00 is when the registry's garbage-collection CronJob runs. From mid-March until 2026-06-02 that job ran registry garbage-collect --delete-untagged=true. I had turned the flag on to reclaim disk. I turned it off after another image broke, and did not go back to check what else the flag had removed.

What happens, in three parts:

• Since buildx 0.10 (January 2023; docker/build-push-action v4 picked it up), a push attaches a provenance attestation by default. Even for one platform the result is an OCI image index with two children, the platform manifest and the attestation manifest. The tag points at the index. The children have no tag of their own.
• distribution 2.x's garbage collector with --delete-untagged enumerates manifests, keeps those a tag resolves to, and deletes the rest. It does not descend into an index. It therefore kept the index, deleted both children, and then deleted every blob that only the children referenced.
• Afterwards the tag list is intact and the tag resolves to the index. A client that follows the index to the platform manifest gets manifest unknown.

This was reported as distribution/distribution#3004 in 2019 and again as #3178 in 2020. The fix, #4285, was merged in April 2024 and is only in the 3.x releases. 2.8.3, the last 2.x release, predates it.

The node cache

containerd on the node had the image, and with the default imagePullPolicy: IfNotPresent for a non-latest tag nothing asks the registry for an image that is already present. The pod restarted five times between March and November, most recently on 2026-10-28, each time from the local store. The first client to ask the registry for the platform manifest after March was the replacement pod on 2026-11-08, after the cached copy had been removed.

Knowing what to look for, I checked every image tag referenced by a running workload against the registry filesystem:

images referenced by workloads                            40
complete                                                  34
broken (index kept, children unlinked, blobs missing)      6

All six had been pushed in March 2026 while the flag was on: four on the 14th between 19:02 and 19:14, two on the 23rd. Every image pushed after May was either a single manifest or an intact index. Five of the six broken ones were still running from the node's cache, with no pullable copy anywhere.

The fix

Rebuild from the same commit with two lines added to the workflow:

- uses: docker/build-push-action@v6
  with:
    provenance: false
    platforms: linux/amd64

With no attestation and one platform, buildx pushes a single application/vnd.oci.image.manifest.v1+json, and there is no index child for this collector to orphan. The new tag resolved to that manifest with every referenced blob present. The pod was ready at 07:41:37 on 2026-11-11.

Then the other five. Four of those repositories had CI that pushed to GitHub's container registry and edited manifest paths that no longer existed, so whatever had put their images into our registry in March, it was not the workflow in the repo. Three of the four built successfully and failed at the deploy step before I noticed. I pointed the four workflows at the real registry and the real manifest paths, rebuilt, and by 09:20 all six workloads were running from complete single-manifest images.

Timeline

When (UTC)	What
2026-03-14 18:19	Registry GC CronJob created; --delete-untagged=true is in a commit dated the next day. Four of the six images that would break are pushed between 19:02 and 19:14
2026-03-23 13:59	Signaling image pushed as an OCI index with an attestation child; pod starts
2026-03-24 04:00	GC unlinks both children and deletes 6 of 10 blobs. Pod keeps running from the node's cache
2026-06-02	Flag removed from the live CronJob after a different image breaks. Existing damage not surveyed
2026-11-08 13:37	4.5 GiB image for another service deployed; pod starts 13:40
2026-11-08 13:46:42	Signaling pod ends, exit 0. First replacement created
2026-11-08 13:47	DiskPressure. Image reclaim and evictions begin
2026-11-08 14:30	KubeDeploymentReplicasMismatch for the Deployment starts firing
2026-11-08 16:10	Second DiskPressure window. First replacement fails (137); second replacement created, ImagePullBackOff
2026-11-11 05:08	Cluster audit lists the unavailable Deployment and 1,565 pull failures
2026-11-11 07:41	Rebuilt image running. Outage 2 d 17 h 55 min
2026-11-11 09:20	Other five images rebuilt and rolled out

What I am changing

The alert rule existed and fired. KubeDeploymentReplicasMismatch went pending at 14:00 and firing at 14:30 on 2026-11-08 and stayed firing until the fix. Alertmanager's route sends everything above info to a Slack channel and repeats every four hours. I have not gone back through the channel history, and I did not act on it. One week earlier, on 2026-11-01, that channel already had nine alerts firing at the same time, including a TURN server that the audit dates as crash-looping since September. The rule was there. The channel was not being read.

Done:

• provenance: false and platforms: linux/amd64 in the six workflows that push to this registry.
• Models out of the image and onto a volume.
• Four CI pipelines reconnected to the manifests they are meant to update.

Not done yet:

• The remaining workflows in the organisation, or upgrade the registry to distribution 3.x, which handles index children in GC.
• A daily job that walks every image tag referenced by a workload: resolve the tag, fetch the index, fetch each child, HEAD each blob. On a broken image the tag list and the index both look normal, so the check has to go down to the blobs.
• Fix or silence the nine standing alerts. A page, not a Slack line, for available == 0 lasting more than ten minutes on anything with a public hostname.
• ephemeral-storage requests on the one workload allowed to be large, and an alert on node disk under 15 %.
• The registry's manifests under the same GitOps reconciliation as everything else. They are in git, but nothing applies them: the live CronJob was patched by hand on 2 June, two days after the commit that removed the flag.

If you run, or have ever run, docker/distribution 2.x with --delete-untagged, check every multi-manifest image pushed in that window by asking for the platform manifest, not the tag. A running pod only proves that the node has the image.

Public post-incident reports where pods could not come back because the registry could not serve their image are collected under the image pull failures symptom in Incident Library.

---

Originally published as a kubegap guide on benchlet.com.