Deleting the corrupted chunk file made it worse

A Prometheus pod went into CrashLoopBackOff and stayed there. Nine restarts, the same fatal error every time, the first one logged at 10:06 UTC:

level=ERROR source=main.go:1409 msg="Fatal error"
  err="opening storage failed:
       /prometheus/chunks_head/000225: invalid magic number 0"

The pod ran with a PVC-backed TSDB. Something had killed the process while it was mid-write, and one file in chunks_head/ had been created but never got its header written. Zero bytes where the magic number should be.

The fix looks obvious. One file is bad, so delete the bad file. That's what makes this incident worth writing up, because deleting it makes things worse:

level=ERROR source=main.go:1409 msg="Fatal error"
  err="opening storage failed: found unsequential head chunk files
       /prometheus/chunks_head/000224 (index: 224) and
       /prometheus/chunks_head/000226 (index: 226)"

Now the sequence has a hole in it and Prometheus won't open the directory at all. You've traded one broken file for a broken directory.

What's in those files

chunks_head/ holds memory-mapped chunk files, numbered in sequence. They're the on-disk side of Prometheus's in-memory head block: the couple of hours of samples that haven't been compacted into a persistent block yet. Each file grows to at most 128 MiB, and when the next write won't fit, Prometheus cuts a new file with the next number in the sequence. That's why the numbers climb to 000225 on a busy instance rather than sitting at single digits.

Each file opens with an 8-byte header: a 4-byte magic number, a 1-byte version, then 3 bytes of padding. From tsdb/chunks/head_chunks.go:

// MagicHeadChunks is 4 bytes at the beginning of a head chunk file.
MagicHeadChunks = 0x0130BC91

Prometheus preallocates the file before it writes into it, and writes pass through a 4 MiB buffer on the way to disk. So there's a real window in which the file exists and its header doesn't. Kill the process inside that window and you get a file full of zeros. Zero isn't 0x0130BC91, so validation rejects it, and the error prints the value it found in hex. That's the invalid magic number 0. Unless the zeroed file happens to be the newest one, which turns out to matter a lot. More on that below.

Two checks run over that directory at startup, and they are exactly the two failures above. One function walks the files, rejects a sequence with a gap in it, then rejects any file whose magic doesn't match. Both are fatal. The gap check runs first, which is why deleting the bad file swaps one error for the other instead of getting you halfway.

You don't need Prometheus to apply those rules yourself. Forty lines of Python will do it:

#!/usr/bin/env python3
import os, struct, sys

MAGIC = 0x0130BC91  # MagicHeadChunks
HEADER = 8          # 4 magic + 1 version + 3 padding

d = sys.argv[1] if len(sys.argv) > 1 else "chunks_head"
files = sorted(f for f in os.listdir(d) if f.isdigit())

bad, gaps, prev = [], [], None
for f in files:
    path = os.path.join(d, f)
    size = os.path.getsize(path)
    if size < HEADER:
        bad.append((f, f"short file ({size} bytes)"))
    else:
        with open(path, "rb") as fh:
            magic = struct.unpack(">I", fh.read(4))[0]
        if magic != MAGIC:
            bad.append((f, f"invalid magic number {magic:x}"))
    seq = int(f)
    if prev is not None and seq != prev + 1:
        gaps.append((prev, seq))
    prev = seq

print(f"{len(files)} files, {files[0]}..{files[-1]}")
for f, why in bad:
    print(f"  CORRUPT  {f}: {why}")
for a, b in gaps:
    print(f"  GAP      {a:06d} -> {b:06d}: Prometheus will refuse to open this")

Point it at a directory with a zeroed file in the middle and it reports what Prometheus reports:

3 files, 000224..000226
  CORRUPT  000225: invalid magic number 0

Delete the offending file, run it again, and the second failure is waiting:

2 files, 000224..000226
  GAP      000224 -> 000226: Prometheus will refuse to open this

Run this against a copy of the directory before you touch the real one. It tells you whether you're dealing with one bad file or several, and that decides how much you're about to throw away.

How the process died

The incident record doesn't say what killed it. The usual candidates all leave the same file on disk: an OOM kill, which shows up as exit code 137 and is easy to confirm from the pod's last state; a SIGKILL after the termination grace period expired during a rollout or a node drain; a node that lost power or was force-deleted; or kubelet evicting the pod under disk or memory pressure. Any of those can land inside the window between a chunk file being created and its header reaching disk.

Worth checking anyway, because the class of death tells you whether it'll happen again next week. If it was an OOM kill, the memory limit is the real bug and the corrupt chunk file is a symptom. If it was a grace-period timeout, Prometheus needed longer to shut down cleanly than it was given, and raising terminationGracePeriodSeconds is cheaper than another one of these.

Except the obvious story doesn't quite survive contact with the source, and this is the part of the incident I still can't close. When a process is killed abruptly, the file it was in the middle of cutting is by definition the newest file in the directory. Prometheus handles that case on its own, as the next section explains, and nobody ever sees an error. For this to reach a human at all, 000225 needed at least one file after it, and the second error confirms there was one: it names 000226.

So something zeroed a file that wasn't the newest, and a process dying mid-write doesn't explain that. The candidate I'd look at first is the storage layer underneath the PVC losing a writeback that Prometheus believed had landed. That's a hypothesis, not a finding, and I don't have the evidence to settle it. But it does change what you check after clearing the directory. If a not-newest file was zeroed, the interesting question isn't how the pod died. It's whether that volume acknowledges writes it hasn't actually committed.

Why Prometheus didn't fix this itself

Prometheus repairs a lot of its own storage damage, which is why this crashloop is surprising once you know that. It's worth laying out what it does and doesn't handle, because the pattern is not obvious from the outside.

Start with the one that's most relevant here, because it's the reason this incident is odd rather than routine. Before any validation runs, openMMapFiles calls repairLastChunkFile. That function looks at the highest-numbered file only. If it's too short to hold a magic number, or its magic reads as zero, it deletes the file and carries on without logging anything alarming:

// We either don't have enough bytes for the magic number or the magic number is 0.
if size < MagicChunksSize || binary.BigEndian.Uint32(buf) == 0 {
    // Corrupt file, hence remove it.
    if err := os.RemoveAll(files[lastFile]); err != nil {
        return files, fmt.Errorf("delete corrupted, empty head chunk file during last file repair: %w", err)
    }
    delete(files, lastFile)
}

That's the exact failure this article is about, self-healed, as long as the zeroed file is the newest one. Which is the common case, and which is why most people never see this error at all.

Write-ahead log corruption is repaired too. When the head hits a corrupt WAL record during init, Prometheus increments prometheus_tsdb_wal_corruptions_total and calls Repair, which truncates the log at the last good record and carries on.

Corruption found while reading memory-mapped chunk contents is also repaired. Head.Init() loads those chunks, and if the load fails it calls removeCorruptedMmappedChunks, deletes the bad files, and replays the WAL to rebuild what they held. There's a metric for that too, prometheus_tsdb_mmap_chunk_corruptions_total.

Neither path ran here, and the reason is a call-ordering detail. The magic-number and sequence checks don't live in Head.Init(). They run earlier, in openMMapFiles, which is called from NewChunkDiskMapper while storage is still being opened. When they fail, tsdb.Open returns an error and the process exits. That's why the log says opening storage failed rather than anything about chunks. The repair logic sits behind a door that never got opened.

There's a wrinkle worth knowing if you go reading that code. The comment above the magic check says the error is deliberately passed up so the Head's repair mechanism can deal with it. In practice the failure here happened while storage was being opened, which is upstream of that mechanism, so the process died instead. Whatever the intent, the observed behaviour is a crashloop.

So the rule of thumb is four-tiered. A zeroed newest chunk file: deleted quietly before you notice. Bad WAL records: self-healed. Bad chunk contents: self-healed. A chunk file set that fails validation at open, which means a gap, a bad magic number anywhere but the last file, or an unrecognised format version: fatal, and yours to fix by hand. A missing header on a file that isn't last is the fourth kind, and that's this incident.

One thing not to reach for: there is no promtool tsdb repair. The tsdb subcommands are analyze, bench, create-blocks-from, dump, dump-openmetrics and list, and that's true both in current Prometheus and in 3.4.0. promtool tsdb list and analyze are genuinely useful here for confirming your persistent blocks are fine, but nothing in promtool repairs a head chunk.

Recovering it

The order matters, mostly because the operator will fight you.

Pause the Prometheus CR first. Skip this and the operator recreates the pod while you're working on the volume, and you get to do the whole thing twice:

kubectl -n monitoring patch prometheus prometheus \
  --type merge -p '{"spec":{"paused":true}}'
kubectl -n monitoring scale statefulset prometheus-prometheus --replicas=0
kubectl -n monitoring delete pod prometheus-prometheus-0 \
  --force --grace-period=0 --ignore-not-found

Pausing the individual CR beats scaling the prometheus-operator deployment to zero. It isolates the change to one instance instead of freezing reconciliation for every Prometheus in the cluster, which matters if you're running per-tenant instances and only one of them is broken.

Mount the PVC somewhere else. Use a Prometheus image rather than something like busybox, so promtool is there when you want to check the blocks:

apiVersion: v1
kind: Pod
metadata:
  name: prom-fix
  namespace: monitoring
spec:
  containers:
    - name: fix
      image: quay.io/prometheus/prometheus:v3.4.0
      command: ["sleep", "3600"]
      volumeMounts:
        - { name: data, mountPath: /prometheus }
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: prometheus-prometheus-db-prometheus-prometheus-0

Find out where the data actually is. This one costs people ten minutes. If the StatefulSet mounts the volume with a subPath, the TSDB isn't at the root of the PVC. Here the pod used subPath: prometheus-db, so inside the debug pod the data sat at /prometheus/prometheus-db/, with chunks_head/ under that. Check before deleting anything:

kubectl -n monitoring exec prom-fix -- ls -la /prometheus /prometheus/prometheus-db

Clear the whole directory, not the one file. This is the part the first attempt got wrong. Removing 000225 on its own is what produced the sequence gap. Remove all of them and Prometheus rebuilds the head from the WAL at startup. My rule now: clear the directory, never a single file.

kubectl -n monitoring exec prom-fix -- sh -c 'rm /prometheus/prometheus-db/chunks_head/*'

Put it back:

kubectl -n monitoring delete pod prom-fix
kubectl -n monitoring scale statefulset prometheus-prometheus --replicas=1
kubectl -n monitoring patch prometheus prometheus \
  --type merge -p '{"spec":{"paused":false}}'

What it costs

Less than it looks like, which is the good news buried in this.

The memory-mapped chunk files are not the system of record. The WAL is. Anything in chunks_head/ that hasn't been compacted yet is also in the write-ahead log, so clearing the directory and letting the WAL replay reconstructs the head. Your persistent blocks, the ULID-named directories holding the bulk of the history, are never touched by any of this. They were intact here and replayed cleanly.

What you're risking is the window the WAL itself covers, and only if the WAL is also damaged. The head becomes compactable once it spans one and a half times the chunk range, and the chunk range defaults to two hours, so somewhere between two and three hours of samples can be sitting in the head at any moment. That's the rough size of the exposure. In this incident nothing was lost from the head at all. The visible cost was about 30 minutes with no scraping while the pod was down and being fixed, which shows up as a gap in the graphs rather than as missing history.

That trade is worth stating plainly, because it changes how fast you can move. Clearing chunks_head/ is close to free when the WAL is healthy. Confirm the wal/ directory exists and isn't empty, then delete and stop agonising.

I wouldn't delete a PVC for this, however tempting it looks at 3am. It's faster to type and it throws away every block on that volume, which is the one part of the storage that was never in danger. If you find yourself reaching for that, it's usually because the subPath confusion above has convinced you the data is gone. It isn't.

The part that actually needed fixing

Nobody was paged for this. The pod sat in CrashLoopBackOff and somebody noticed during a routine check.

That's the real finding. A Prometheus that is down cannot alert on itself being down, and any rule evaluated by the broken instance is worth nothing while it's broken. The check has to come from outside: a second Prometheus scraping this one, a blackbox probe against its /-/healthy endpoint, or an Alertmanager watchdog wired to a dead man's switch that pages when the heartbeat stops rather than when it arrives.

The cheapest version, if you run kube-state-metrics and have another Prometheus that can see the namespace:

kube_pod_container_status_waiting_reason{
  namespace="monitoring", container="prometheus", reason="CrashLoopBackOff"
} == 1

Give it for: 5m. That metric is a gauge, so compare it against a value and leave it at that. If you'd rather alert on restarts, kube_pod_container_status_restarts_total is a counter and needs increase() over a window, not a bare comparison.

Two more things worth doing while it's fresh. Write the recovery down, because the sequence-gap trap is exactly what gets rediscovered at 3am by someone who reads the first error and reaches straight for rm. And check how the process died, since a corrupt chunk file caused by a memory limit will be back on its own schedule.

The short version

• invalid magic number 0 means a chunk file was created but its header never reached disk.
• Deleting that one file gets you found unsequential head chunk files instead. Clear the whole chunks_head/ directory.
• Prometheus already deletes a zeroed newest chunk file by itself, in repairLastChunkFile. It self-repairs bad WAL records and bad chunk contents too. A chunk file set that fails validation at open is fatal, and that's this one.
• Which raises the question I couldn't answer: a not-newest file was zeroed, so suspect the volume, not the process.
• There is no promtool tsdb repair.
• Pause the Prometheus CR before touching the volume, and check for a subPath before you delete.
• The WAL is what saves you. Persistent blocks are never involved, so don't delete the PVC.
• Alert on Prometheus's own crashloop from outside Prometheus, or you'll find it by accident.