DEV Community: Onkar Raghunath Shelke

Deleting the corrupted chunk file made it worse

Onkar Raghunath Shelke — Sat, 26 Sep 2026 02:26:35 +0000

A Prometheus pod went into CrashLoopBackOff and stayed there. Nine restarts, the same fatal error every time, the first one logged at 10:06 UTC:

level=ERROR source=main.go:1409 msg="Fatal error"
  err="opening storage failed:
       /prometheus/chunks_head/000225: invalid magic number 0"

The pod ran with a PVC-backed TSDB. Something had killed the process while it was mid-write, and one file in chunks_head/ had been created but never got its header written. Zero bytes where the magic number should be.

The fix looks obvious. One file is bad, so delete the bad file. That's what makes this incident worth writing up, because deleting it makes things worse:

level=ERROR source=main.go:1409 msg="Fatal error"
  err="opening storage failed: found unsequential head chunk files
       /prometheus/chunks_head/000224 (index: 224) and
       /prometheus/chunks_head/000226 (index: 226)"

Now the sequence has a hole in it and Prometheus won't open the directory at all. You've traded one broken file for a broken directory.

What's in those files

chunks_head/ holds memory-mapped chunk files, numbered in sequence. They're the on-disk side of Prometheus's in-memory head block: the couple of hours of samples that haven't been compacted into a persistent block yet. Each file grows to at most 128 MiB, and when the next write won't fit, Prometheus cuts a new file with the next number in the sequence. That's why the numbers climb to 000225 on a busy instance rather than sitting at single digits.

Each file opens with an 8-byte header: a 4-byte magic number, a 1-byte version, then 3 bytes of padding. From tsdb/chunks/head_chunks.go:

// MagicHeadChunks is 4 bytes at the beginning of a head chunk file.
MagicHeadChunks = 0x0130BC91

Prometheus preallocates the file before it writes into it, and writes pass through a 4 MiB buffer on the way to disk. So there's a real window in which the file exists and its header doesn't. Kill the process inside that window and you get a file full of zeros. Zero isn't 0x0130BC91, so validation rejects it, and the error prints the value it found in hex. That's the invalid magic number 0. Unless the zeroed file happens to be the newest one, which turns out to matter a lot. More on that below.

Two checks run over that directory at startup, and they are exactly the two failures above. One function walks the files, rejects a sequence with a gap in it, then rejects any file whose magic doesn't match. Both are fatal. The gap check runs first, which is why deleting the bad file swaps one error for the other instead of getting you halfway.

You don't need Prometheus to apply those rules yourself. Forty lines of Python will do it:

#!/usr/bin/env python3
import os, struct, sys

MAGIC = 0x0130BC91  # MagicHeadChunks
HEADER = 8          # 4 magic + 1 version + 3 padding

d = sys.argv[1] if len(sys.argv) > 1 else "chunks_head"
files = sorted(f for f in os.listdir(d) if f.isdigit())

bad, gaps, prev = [], [], None
for f in files:
    path = os.path.join(d, f)
    size = os.path.getsize(path)
    if size < HEADER:
        bad.append((f, f"short file ({size} bytes)"))
    else:
        with open(path, "rb") as fh:
            magic = struct.unpack(">I", fh.read(4))[0]
        if magic != MAGIC:
            bad.append((f, f"invalid magic number {magic:x}"))
    seq = int(f)
    if prev is not None and seq != prev + 1:
        gaps.append((prev, seq))
    prev = seq

print(f"{len(files)} files, {files[0]}..{files[-1]}")
for f, why in bad:
    print(f"  CORRUPT  {f}: {why}")
for a, b in gaps:
    print(f"  GAP      {a:06d} -> {b:06d}: Prometheus will refuse to open this")

Point it at a directory with a zeroed file in the middle and it reports what Prometheus reports:

3 files, 000224..000226
  CORRUPT  000225: invalid magic number 0

Delete the offending file, run it again, and the second failure is waiting:

2 files, 000224..000226
  GAP      000224 -> 000226: Prometheus will refuse to open this

Run this against a copy of the directory before you touch the real one. It tells you whether you're dealing with one bad file or several, and that decides how much you're about to throw away.

How the process died

The incident record doesn't say what killed it. The usual candidates all leave the same file on disk: an OOM kill, which shows up as exit code 137 and is easy to confirm from the pod's last state; a SIGKILL after the termination grace period expired during a rollout or a node drain; a node that lost power or was force-deleted; or kubelet evicting the pod under disk or memory pressure. Any of those can land inside the window between a chunk file being created and its header reaching disk.

Worth checking anyway, because the class of death tells you whether it'll happen again next week. If it was an OOM kill, the memory limit is the real bug and the corrupt chunk file is a symptom. If it was a grace-period timeout, Prometheus needed longer to shut down cleanly than it was given, and raising terminationGracePeriodSeconds is cheaper than another one of these.

Except the obvious story doesn't quite survive contact with the source, and this is the part of the incident I still can't close. When a process is killed abruptly, the file it was in the middle of cutting is by definition the newest file in the directory. Prometheus handles that case on its own, as the next section explains, and nobody ever sees an error. For this to reach a human at all, 000225 needed at least one file after it, and the second error confirms there was one: it names 000226.

So something zeroed a file that wasn't the newest, and a process dying mid-write doesn't explain that. The candidate I'd look at first is the storage layer underneath the PVC losing a writeback that Prometheus believed had landed. That's a hypothesis, not a finding, and I don't have the evidence to settle it. But it does change what you check after clearing the directory. If a not-newest file was zeroed, the interesting question isn't how the pod died. It's whether that volume acknowledges writes it hasn't actually committed.

Why Prometheus didn't fix this itself

Prometheus repairs a lot of its own storage damage, which is why this crashloop is surprising once you know that. It's worth laying out what it does and doesn't handle, because the pattern is not obvious from the outside.

Start with the one that's most relevant here, because it's the reason this incident is odd rather than routine. Before any validation runs, openMMapFiles calls repairLastChunkFile. That function looks at the highest-numbered file only. If it's too short to hold a magic number, or its magic reads as zero, it deletes the file and carries on without logging anything alarming:

// We either don't have enough bytes for the magic number or the magic number is 0.
if size < MagicChunksSize || binary.BigEndian.Uint32(buf) == 0 {
    // Corrupt file, hence remove it.
    if err := os.RemoveAll(files[lastFile]); err != nil {
        return files, fmt.Errorf("delete corrupted, empty head chunk file during last file repair: %w", err)
    }
    delete(files, lastFile)
}

That's the exact failure this article is about, self-healed, as long as the zeroed file is the newest one. Which is the common case, and which is why most people never see this error at all.

Write-ahead log corruption is repaired too. When the head hits a corrupt WAL record during init, Prometheus increments prometheus_tsdb_wal_corruptions_total and calls Repair, which truncates the log at the last good record and carries on.

Corruption found while reading memory-mapped chunk contents is also repaired. Head.Init() loads those chunks, and if the load fails it calls removeCorruptedMmappedChunks, deletes the bad files, and replays the WAL to rebuild what they held. There's a metric for that too, prometheus_tsdb_mmap_chunk_corruptions_total.

Neither path ran here, and the reason is a call-ordering detail. The magic-number and sequence checks don't live in Head.Init(). They run earlier, in openMMapFiles, which is called from NewChunkDiskMapper while storage is still being opened. When they fail, tsdb.Open returns an error and the process exits. That's why the log says opening storage failed rather than anything about chunks. The repair logic sits behind a door that never got opened.

There's a wrinkle worth knowing if you go reading that code. The comment above the magic check says the error is deliberately passed up so the Head's repair mechanism can deal with it. In practice the failure here happened while storage was being opened, which is upstream of that mechanism, so the process died instead. Whatever the intent, the observed behaviour is a crashloop.

So the rule of thumb is four-tiered. A zeroed newest chunk file: deleted quietly before you notice. Bad WAL records: self-healed. Bad chunk contents: self-healed. A chunk file set that fails validation at open, which means a gap, a bad magic number anywhere but the last file, or an unrecognised format version: fatal, and yours to fix by hand. A missing header on a file that isn't last is the fourth kind, and that's this incident.

One thing not to reach for: there is no promtool tsdb repair. The tsdb subcommands are analyze, bench, create-blocks-from, dump, dump-openmetrics and list, and that's true both in current Prometheus and in 3.4.0. promtool tsdb list and analyze are genuinely useful here for confirming your persistent blocks are fine, but nothing in promtool repairs a head chunk.

Recovering it

The order matters, mostly because the operator will fight you.

Pause the Prometheus CR first. Skip this and the operator recreates the pod while you're working on the volume, and you get to do the whole thing twice:

kubectl -n monitoring patch prometheus prometheus \
  --type merge -p '{"spec":{"paused":true}}'
kubectl -n monitoring scale statefulset prometheus-prometheus --replicas=0
kubectl -n monitoring delete pod prometheus-prometheus-0 \
  --force --grace-period=0 --ignore-not-found

Pausing the individual CR beats scaling the prometheus-operator deployment to zero. It isolates the change to one instance instead of freezing reconciliation for every Prometheus in the cluster, which matters if you're running per-tenant instances and only one of them is broken.

Mount the PVC somewhere else. Use a Prometheus image rather than something like busybox, so promtool is there when you want to check the blocks:

apiVersion: v1
kind: Pod
metadata:
  name: prom-fix
  namespace: monitoring
spec:
  containers:
    - name: fix
      image: quay.io/prometheus/prometheus:v3.4.0
      command: ["sleep", "3600"]
      volumeMounts:
        - { name: data, mountPath: /prometheus }
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: prometheus-prometheus-db-prometheus-prometheus-0

Find out where the data actually is. This one costs people ten minutes. If the StatefulSet mounts the volume with a subPath, the TSDB isn't at the root of the PVC. Here the pod used subPath: prometheus-db, so inside the debug pod the data sat at /prometheus/prometheus-db/, with chunks_head/ under that. Check before deleting anything:

kubectl -n monitoring exec prom-fix -- ls -la /prometheus /prometheus/prometheus-db

Clear the whole directory, not the one file. This is the part the first attempt got wrong. Removing 000225 on its own is what produced the sequence gap. Remove all of them and Prometheus rebuilds the head from the WAL at startup. My rule now: clear the directory, never a single file.

kubectl -n monitoring exec prom-fix -- sh -c 'rm /prometheus/prometheus-db/chunks_head/*'

Put it back:

kubectl -n monitoring delete pod prom-fix
kubectl -n monitoring scale statefulset prometheus-prometheus --replicas=1
kubectl -n monitoring patch prometheus prometheus \
  --type merge -p '{"spec":{"paused":false}}'

What it costs

Less than it looks like, which is the good news buried in this.

The memory-mapped chunk files are not the system of record. The WAL is. Anything in chunks_head/ that hasn't been compacted yet is also in the write-ahead log, so clearing the directory and letting the WAL replay reconstructs the head. Your persistent blocks, the ULID-named directories holding the bulk of the history, are never touched by any of this. They were intact here and replayed cleanly.

What you're risking is the window the WAL itself covers, and only if the WAL is also damaged. The head becomes compactable once it spans one and a half times the chunk range, and the chunk range defaults to two hours, so somewhere between two and three hours of samples can be sitting in the head at any moment. That's the rough size of the exposure. In this incident nothing was lost from the head at all. The visible cost was about 30 minutes with no scraping while the pod was down and being fixed, which shows up as a gap in the graphs rather than as missing history.

That trade is worth stating plainly, because it changes how fast you can move. Clearing chunks_head/ is close to free when the WAL is healthy. Confirm the wal/ directory exists and isn't empty, then delete and stop agonising.

I wouldn't delete a PVC for this, however tempting it looks at 3am. It's faster to type and it throws away every block on that volume, which is the one part of the storage that was never in danger. If you find yourself reaching for that, it's usually because the subPath confusion above has convinced you the data is gone. It isn't.

The part that actually needed fixing

Nobody was paged for this. The pod sat in CrashLoopBackOff and somebody noticed during a routine check.

That's the real finding. A Prometheus that is down cannot alert on itself being down, and any rule evaluated by the broken instance is worth nothing while it's broken. The check has to come from outside: a second Prometheus scraping this one, a blackbox probe against its /-/healthy endpoint, or an Alertmanager watchdog wired to a dead man's switch that pages when the heartbeat stops rather than when it arrives.

The cheapest version, if you run kube-state-metrics and have another Prometheus that can see the namespace:

kube_pod_container_status_waiting_reason{
  namespace="monitoring", container="prometheus", reason="CrashLoopBackOff"
} == 1

Give it for: 5m. That metric is a gauge, so compare it against a value and leave it at that. If you'd rather alert on restarts, kube_pod_container_status_restarts_total is a counter and needs increase() over a window, not a bare comparison.

Two more things worth doing while it's fresh. Write the recovery down, because the sequence-gap trap is exactly what gets rediscovered at 3am by someone who reads the first error and reaches straight for rm. And check how the process died, since a corrupt chunk file caused by a memory limit will be back on its own schedule.

The short version

invalid magic number 0 means a chunk file was created but its header never reached disk.
Deleting that one file gets you found unsequential head chunk files instead. Clear the whole chunks_head/ directory.
Prometheus already deletes a zeroed newest chunk file by itself, in repairLastChunkFile. It self-repairs bad WAL records and bad chunk contents too. A chunk file set that fails validation at open is fatal, and that's this one.
Which raises the question I couldn't answer: a not-newest file was zeroed, so suspect the volume, not the process.
There is no promtool tsdb repair.
Pause the Prometheus CR before touching the volume, and check for a subPath before you delete.
The WAL is what saves you. Persistent blocks are never involved, so don't delete the PVC.
Alert on Prometheus's own crashloop from outside Prometheus, or you'll find it by accident.

Kubelet watches inodes. Just not until it's an emergency.

Onkar Raghunath Shelke — Thu, 17 Sep 2026 05:34:56 +0000

A worker node pages with NodeFilesystemFilesFillingUp. It's on course to run out of inodes. You check the disk first, because that's what everyone checks first, and it reads 83% used: 103G used, 22G available. Then you check inodes: 67% used, 11M used, 5.3M free.

The disk is the fuller of the two, and the inode table is the one that paged. That's not a contradiction. NodeFilesystemFilesFillingUp is a trajectory alert built on predict_linear, so it fires on the slope, not on the level. Nothing was wrong with 67%. What was wrong was how fast it was getting there, and the fact that the only mechanism in kubelet that would have intervened is the one that evicts pods.

Two notes on what follows. I no longer have access to that cluster, so I can't paste the original df, du or tune2fs captures. Where a number comes from the incident record I say so, and where I've reconstructed something from arithmetic I show the arithmetic. The ext4 demonstrations further down you can run yourself in about two seconds.

If you want to compare your own node before reading on:

df -h / && df -i /

What kubelet actually watches

Kubelet's defence against images filling a node is image garbage collection. Once the image filesystem passes imageGCHighThresholdPercent (85 by default), it deletes unused images, oldest first, until usage drops to imageGCLowThresholdPercent (80).

Both of those are percentages of bytes. Image GC never looks at file counts.

Kubelet isn't blind to inodes, though, and it's worth being exact here, because the detail changes the conclusion. On Linux the default hard eviction set is five signals (pkg/kubelet/eviction/defaults_linux.go, quoted here at tag v1.37.0):

var DefaultEvictionHard = map[string]string{
"memory.available":   "100Mi",
"nodefs.available":   "10%",
"nodefs.inodesFree":  "5%",
"imagefs.available":  "15%",
"imagefs.inodesFree": "5%",
}

Both filesystems get an inode threshold, and both sit at 5% free. That's a Linux-only file. The two next to it are shorter: defaults_windows.go and defaults_others.go each carry three signals, memory.available, nodefs.available and imagefs.available, with no inode thresholds at all. On Linux the coverage is there. Check it against whatever version you're running, because these files do change.

So kubelet will act on inodes. The question is when. 5% free means 95% used, and that's the hard eviction floor: the point where kubelet reclaims what it can and starts evicting pods to save the node. Image garbage collection, the mechanism that runs quietly in the background before anything is on fire, triggers at 85% and stops at 80%, and both of those are bytes.

Count the mechanisms watching each resource. Bytes get two: background image GC at 85%, and hard eviction at 10% free on nodefs or 15% free on imagefs. Inodes get one, and it's the eviction floor. There is no early trigger on inodes, so the first thing that happens when you run low is also the most disruptive thing that can happen.

Why small files hit a wall the disk doesn't

Quick question before the numbers. Fill a filesystem with nothing but one-byte files. What percentage of the disk is used by the time you run out of inodes? Take a guess.

ext4 decides how many inodes a filesystem gets at creation time, and that number never changes afterwards. It comes from inode_ratio: one inode per N bytes of capacity. The stock /etc/mke2fs.conf uses 16384. And any file that isn't empty takes at least one 4 KiB block, no matter how small it is.

You can see the ratio on a throwaway image file. No root needed:

$ truncate -s 1G img && mkfs.ext4 -q -F -i 16384 img
$ tune2fs -l img | grep -E '^(Inode count|Block count|Block size):'
Inode count:              65536
Block count:              262144
Block size:               4096

262,144 blocks × 4,096 bytes ÷ 16,384 is 65,536 inodes. If every file uses one block, all 65,536 inodes are gone after 256 MiB. That's 25% of the disk. If you guessed higher, most people do.

Watch it fail

You don't have to take the arithmetic on trust. mkfs.ext4 -d builds a filesystem straight from a directory, so you can pack small files into an image without mounting anything and without root:

mkdir files
for i in $(seq 1 4000); do printf x > files/f$i; done
truncate -s 64M img
mkfs.ext4 -q -F -i 16384 -d files img
dumpe2fs -h img 2>/dev/null | grep -E '^(Inode count|Free inodes|Block count|Free blocks):'

Inode count:              4096
Block count:              16384
Free blocks:              11040
Free inodes:              84

97.9% of inodes used, 32.6% of blocks. The files themselves account for only 24.4 of those 32.6 points: 4,000 files at one 4 KiB block each is 15.6 MiB, and 15.6 MiB of 64 MiB is 24.4%. The remaining 8.2 points are the journal and filesystem metadata.

Add 200 more files and it stops being theoretical:

$ for i in $(seq 4001 4200); do printf x > files/f$i; done
$ truncate -s 64M img2 && mkfs.ext4 -q -F -i 16384 -d files img2
mkfs.ext4: Could not allocate inode in ext2 filesystem while populating file system

Yes, it says ext2. That's the underlying library's error string, not a mistake. The image that worked had 84 inodes left and two thirds of its blocks free. Somewhere around file 4,085 the inodes run out, and from there the free blocks stop mattering. On e2fsprogs 1.47.2 the whole sequence runs in under two seconds.

Reverse-engineering the node's mkfs parameter

Back to the real node, and this part is arithmetic rather than a capture, because I can't re-run anything on that cluster.

11M inodes used plus 5.3M free is about 16.3 million inodes in total. The filesystem is roughly 124-133 GiB, working from 103G used plus 22G available and allowing for ext4's reserved blocks. At the stock 16,384 bytes per inode, a filesystem that size only gets 8.1-8.7 million inodes. This one already had 11 million in use, which is more than the stock ratio can even allocate. So it wasn't formatted at 16384.

Divide the size by the inode count and you get roughly 7,500-9,000 bytes per inode, which lines up with 8192. At that ratio the all-small-files case exhausts inodes at 50% of the disk instead of 25%. I can't confirm it directly, but the arithmetic constrains it to about 8192, and on your own nodes tune2fs -l on the root device settles it in one command.

A real node is never all small files. Layer blobs and logs are big, which is why disk was the higher number here. But image GC was waiting for 85%, the file count was climbing at its own pace, and nothing in kubelet compares one against the other.

Finding the millions of files

Knowing the inodes are going doesn't tell you where they went. This does:

du --inodes -xS /var | sort -rh | head -n 20

Both flags earn their place. -x stops du at filesystem boundaries, so it doesn't wander into the tmpfs and volume mounts kubelet sets up under /var/lib/kubelet/pods. -S counts each directory on its own instead of adding in its subdirectories. Leave it off and /var, /var/lib and every other parent sit at the top of the list, above the directory you're actually looking for.

On this node the top of that list wasn't logs and it wasn't pod volumes. It was containerd's snapshot store, /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/, where each numbered directory is one unpacked image layer. Several of those snapshots held the same path: app/node_modules/@mui/icons-material, at 21,553 files each.

That's one package directory out of one image. The image as a whole carried more than 40,000 files: the icon package is over half of it on its own, and the remainder is the rest of the dependency tree, the application source, and the build output, all of which shipped in the same image. Multiply 40,000 by every version a node is still holding, and millions of inodes stop looking mysterious.

It's 21,553 per snapshot, not across all of them. containerd's content store dedupes compressed layers by digest, but the overlayfs snapshotter unpacks each distinct layer into its own directory and shares nothing between them. Two builds that put an identical node_modules into layers with different digests leave two full copies on disk.

node_modules is the well-known offender, but it isn't the only one. Any image that ships a dependency tree full of small files behaves the same way, whether that's a Python virtualenv, vendored PHP packages or a pile of Ruby gems.

The problem was the image, not the cluster

Nothing on the cluster was set up wrong. Kubelet, containerd and the node image were all running on defaults.

The image was a single-stage build, so everything the build needed ended up in what shipped: the source, the build output, and all of node_modules, dev dependencies included, since npm install pulls those in by default. A single-stage build like that usually looks something like this:

FROM node:20
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

Now look at that COPY . . and ask whether the repo has a .dockerignore. If node_modules isn't excluded, COPY . . puts a second copy of it into a layer that changes on every single commit. That's the fastest way to multiply snapshots, and it's easy to miss because the build still works.

Even with a .dockerignore, a CI pipeline that builds without a warm layer cache reruns npm install every time. That produces a new layer digest even when package.json hasn't changed, because the files inside the layer carry new timestamps. Every new digest becomes a new snapshot on every node that pulls it. containerd keeps those snapshots until the image is removed, and kubelet only removes unused images when image GC runs, which brings you straight back to that 85% byte threshold.

It went unnoticed because the number most people watch, disk usage, never crossed the line.

Fixing it

The first fix belongs in the build: a multi-stage Dockerfile. One stage installs dependencies and builds. The final image is a small web server that only receives the compiled output.

FROM node:20 AS build
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

FROM nginx:alpine
COPY --from=build /app/build /usr/share/nginx/html

That should take the deployed image from more than 40,000 files down to the compiled assets plus the base image, which for a React build is typically a few dozen files rather than tens of thousands. I haven't measured both images side by side, so treat that as the expected shape rather than a benchmark. If you try it, docker run --rm <image> find / -xdev -type f | wc -l on each will give you the real before-and-after.

The second fix is monitoring, and credit where it's due: the upstream alert did its job here. NodeFilesystemFilesFillingUp ships with the node_exporter mixin, and it's what caught this at 67%. It's a prediction, though, and predictions go quiet when growth pauses for a while. I'd add a flat threshold alongside it that fires well before kubelet's 95% eviction point:

1 - node_filesystem_files_free{mountpoint="/"}
  / node_filesystem_files{mountpoint="/"}
  > 0.80

Give it for: 30m so a burst of image pulls during a rollout doesn't page anyone. The 80% line is a judgement call; pick whatever leaves you enough room to act before 95%. You won't find anything like it tied to image garbage collection upstream, and that's the distinction worth holding on to: image GC's thresholds are percentages of bytes, and the inode thresholds kubelet does have belong to eviction. They're separate mechanisms, and only one of them runs early.

If you run Prometheus, this one is worth having on a dashboard. It subtracts disk usage from inode usage per node, so anything near the top is burning through inodes faster than bytes, and those are the nodes where the byte thresholds will fail you:

sort_desc(
  (1 - node_filesystem_files_free{mountpoint="/"} / node_filesystem_files{mountpoint="/"})
  -
  (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})
)

Adjust the mountpoint label if your node_exporter reports the root filesystem differently.

Neither fix does anything about what's already sitting on your nodes. Old snapshots stay until their images go. crictl rmi --prune removes every image no container is currently using, and those images simply get pulled again the next time something needs them. You'll also see crictl rm --all suggested for this kind of cleanup. It clears every stopped container and takes kubectl logs --previous with it for anything that restarted recently. Fine in an emergency. I wouldn't put it in a cron job.

If you only remember one thing from this: kubelet's background cleanup runs on bytes, so a workload that's heavy on files rather than bytes gets no early intervention at all. Something does eventually fire, at 5% inodes free, and it fires by evicting pods. That's the emergency brake, not the fix.

Check your own cluster in five minutes

Step one. Run the fleet query above and look at anything positive.

Step two. On the worst node, find the directories holding the most files:

du --inodes -xS /var/lib/containerd | sort -rh | head -n 20

Step three. If node_modules shows up under the snapshot store, list the images on that node:

crictl images

Then go and look at how those images are built.

Reproduce it

The inode ratio. No root needed.

truncate -s 1G img16 && mkfs.ext4 -q -F -i 16384 img16
truncate -s 1G img8  && mkfs.ext4 -q -F -i 8192  img8
tune2fs -l img16 | grep -E '^(Inode count|Block count):'
tune2fs -l img8  | grep -E '^(Inode count|Block count):'

You'll get 65,536 and 131,072 inodes, both on 262,144 blocks.

Running out of inodes with most of the disk free. No root needed.

mkdir files && for i in $(seq 1 4200); do printf x > files/f$i; done
truncate -s 64M img && mkfs.ext4 -q -F -i 16384 -d files img

That fails with Could not allocate inode. Drop the loop to 4,000 files and it succeeds with 84 inodes free and two thirds of the blocks unused.

Snapshot duplication on a node. You'll need a registry and a cluster you can deploy to.

docker build --no-cache -t <registry>/app:v1 .
docker build --no-cache -t <registry>/app:v2 .
docker push <registry>/app:v1 && docker push <registry>/app:v2
# deploy v1, then v2, onto the same node, then on that node:
sudo du --inodes -xS /var/lib/containerd | sort -rh | head -n 20

You should see two node_modules/@mui/icons-material directories with the same file count under two different snapshot IDs.