Prometheus data lost after pod restart? How to set kube-prometheus-stack persistent storage (a storageSpec PVC) without losing the emptyDir data

Where kube-prometheus-stack kept 30 days of metrics before and after the move. Measured on our cluster on 2026-11-15.

If Prometheus data was lost after a pod restart, or you are setting up kube-prometheus-stack persistent storage, the cause to check first is this: unless prometheus.prometheusSpec.storageSpec is set, prometheus-operator stores the Prometheus TSDB on an emptyDir, and an emptyDir is deleted together with the pod. The same cause is behind the operator log line recreating StatefulSet because the update operation wasn't possible right after you add storageSpec: if the old pod had an emptyDir, its TSDB was deleted with that pod. Our Prometheus pod had run that way for 349 days, and at the end its emptyDir held 22.7G covering the last 31.5 days of metrics. To check yours: if kubectl get prometheus -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name} storage={.spec.storage}{"\n"}{end}' prints a line that ends in storage= with nothing after it, that Prometheus keeps its TSDB on an emptyDir. To add persistent volume to Prometheus without losing data, create the PVC prometheus-<name>-db-prometheus-<name>-0 and copy the TSDB into its prometheus-db/ folder before adding prometheus.prometheusSpec.storageSpec.volumeClaimTemplate, because the operator deletes the StatefulSet when you add it. In our move, the 30 days of blocks and WAL that we copied were kept, and a deliberate pod delete afterwards lost nothing stored. The samples the old pod wrote after the second copy (00:38:05) were never copied, and Prometheus's scrape of itself has a 2 min 6 s hole around the switch (the cause is our inference from the times, see "What it cost"). The cluster is a 2-node k3s cluster (server-1, agent-1) running kube-prometheus-stack 79.5.0 from Argo CD, prometheus-operator v0.86.2 and Prometheus v3.7.3; all times are UTC, 2026-11-15 unless a date is given.

What we saw: the Prometheus data volume was an emptyDir (prometheus-kube-prometheus-stack-prometheus-db {})

The finding

A full run of kubegap v0.3.0 at 2026-11-14 16:25:29 (the tool is described in the check section below) listed as its problem 2, translated from Korean: "Prometheus data is on an emptyDir." The reason it gave: the moment the Prometheus pod is deleted and created again, 30 days of metrics are gone. Two findings carried it, both fail:

prometheus.storage-risk                 storage emptyDir (PVC None), PVC usage None, TSDB 22.3GiB, retention 30d (size cap 0B),
                                        actual oldest sample 31.2 days, failed compactions over 7 days 0, WAL corruptions 1, corruptions at startup 1
observability.stack-storage-emptydir    emptyDir volumes: config-out, prometheus-kube-prometheus-stack-prometheus-db, replicas 1

The StatefulSet and the Prometheus object

At 00:31:25 we read the objects ourselves. The StatefulSet has no volumeClaimTemplates (the first, empty line of the output), and the data volume is an emptyDir ({}):

$ kubectl -n monitoring get sts prometheus-kube-prometheus-stack-prometheus -o jsonpath='{.spec.volumeClaimTemplates}{"\n"}{range .spec.template.spec.volumes[*]}{.name}{" "}{.emptyDir}{.persistentVolumeClaim}{"\n"}{end}'

config
tls-assets
config-out {"medium":"Memory"}
prometheus-kube-prometheus-stack-prometheus-rulefiles-0
web-config
prometheus-kube-prometheus-stack-prometheus-db {}

$ kubectl -n monitoring get prometheus kube-prometheus-stack-prometheus -o jsonpath='retention={.spec.retention} retentionSize={.spec.retentionSize} storage={.spec.storage} replicas={.spec.replicas}'
retention=30d retentionSize= storage= replicas=1

The container mounted it at /prometheus with no subPath. The pod was 349 days old, the RESTARTS column showed 6 for the pod (it has two containers, 2/2, while step 4 below prints the count of the prometheus container alone), and the pod itself had never been replaced (pod IP removed, node name masked):

$ kubectl -n monitoring get pod prometheus-kube-prometheus-stack-prometheus-0 -o wide
NAME                                            READY   STATUS    RESTARTS      AGE    IP           NODE       NOMINATED NODE   READINESS GATES
prometheus-kube-prometheus-stack-prometheus-0   2/2     Running   6 (23h ago)   349d   <pod IP>     server-1   <none>           <none>

created=2025-11-30T06:09:35Z

How much was in it, and on which disk

$ kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- sh -c 'du -sh /prometheus; df -h /prometheus; ls /prometheus | wc -l'
22.7G   /prometheus
Filesystem                Size      Used Available Use% Mounted on
<node root LVM volume>
                        454.9G    249.7G    185.6G  57% /prometheus
23

The 23 entries are 19 block directories plus chunks_head, lock, queries.active and wal. df shows the node's root filesystem, because an emptyDir lives under the kubelet's directory on the node's disk. Prometheus shared that disk with everything else on server-1.

Prometheus's own metrics at 00:31:36 (labels removed):

prometheus_tsdb_lowest_timestamp_seconds                  1791979200      # 2026-10-14T12:00:00Z
(time() - prometheus_tsdb_lowest_timestamp_seconds)/86400 31.521971956019602
prometheus_tsdb_head_series                               218919
prometheus_tsdb_storage_blocks_bytes                      23835478583
prometheus_tsdb_wal_storage_size_bytes                    393179565
prometheus_tsdb_retention_limit_bytes                     0
prometheus_tsdb_compactions_failed_total                  0
prometheus_tsdb_wal_corruptions_total                     1
prometheus_tsdb_blocks_loaded                             19

prometheus_tsdb_retention_limit_bytes 0 means no size cap: only the 30-day time limit deleted data.

The restart the day before

On 2026-11-14 the node rebooted; the Prometheus containers restarted at 01:12:30 (reason Unknown). Two minutes later Prometheus logged, as collected by the kubegap log scan of that pod:

2026-11-14T01:14:32.965Z level=WARN source=db.go:1019 msg="Encountered WAL read error, attempting repair" component=tsdb err="read records: c…
2026-11-14T01:14:32.965Z level=WARN source=wlog.go:413 msg="Starting corruption repair" component=tsdb segment=8483 offset=19763238

(The err text is cut in the scan; the pod and its full log no longer exist.) This is the WAL corruptions 1 in the finding.

Why: the prometheus operator emptyDir default puts the TSDB in the pod

No storage means emptyDir, in the operator code

kube-prometheus-stack 79.5.0 ships storageSpec: {} in its values.yaml, and our values file never set it (git log -S storageSpec on that file is empty back to its first commit, 2025-11-11). With nothing there, the Prometheus object has no spec.storage, and prometheus-operator v0.86.2 (pkg/prometheus/server/statefulset.go, lines 91–130) does this:

storageSpec := cpf.Storage
switch {
case storageSpec == nil:
    statefulset.Spec.Template.Spec.Volumes = append(statefulset.Spec.Template.Spec.Volumes, v1.Volume{
        Name: prompkg.VolumeName(p),
        VolumeSource: v1.VolumeSource{
            EmptyDir: &v1.EmptyDirVolumeSource{},
        },
    })
...
default: // storageSpec.VolumeClaimTemplate
    pvcTemplate := operator.MakeVolumeClaimTemplate(storageSpec.VolumeClaimTemplate)
    if pvcTemplate.Name == "" {
        pvcTemplate.Name = prompkg.VolumeName(p)
    }
    ...
    statefulset.Spec.VolumeClaimTemplates = append(statefulset.Spec.VolumeClaimTemplates, *pvcTemplate)
}

VolumeName is prometheus-<name>-db (pkg/prometheus/common.go, lines 177–183), which is our prometheus-kube-prometheus-stack-prometheus-db. The operator documentation says the same in one sentence: "By default, the operator configures Pods to store data on emptyDir volumes which aren't persisted when the Pods are redeployed."

What an emptyDir survives, and when "Prometheus data lost after pod restart" is true

The Kubernetes documentation for emptyDir: "When a Pod is removed from a node for any reason, the data in the emptyDir is deleted permanently." and "A container crashing does not remove a Pod from a node. The data in an emptyDir volume is safe across container crashes." The data is deleted when a new pod is created; a restart of the Prometheus container inside the same pod keeps the emptyDir.

Which everyday events keep an emptyDir and which delete it. Both columns happened on this Prometheus, one day apart. The node reboot is in the left column because on 2026-11-14 ours restarted the containers in the same pod and the emptyDir stayed.

Prometheus does not replicate its data. Its storage documentation (v3.7.3) says local storage "is not clustered or replicated" and "should be managed like any other single node database". With one replica and an emptyDir, one pod replacement removes all of it.

Adding storageSpec: "recreating StatefulSet because the update operation wasn't possible"

Kubernetes does not allow volumeClaimTemplates to change on an existing StatefulSet, so when storage appears on a running Prometheus the update is rejected with 422, and the operator (pkg/prometheus/server/operator.go, lines 984–1003) deletes the StatefulSet and creates it again:

if ok && sErr.ErrStatus.Code == 422 && sErr.ErrStatus.Reason == metav1.StatusReasonInvalid {
    ...
    logger.Info("recreating StatefulSet because the update operation wasn't possible", "reason", strings.Join(failMsg, ", "))

    propagationPolicy := metav1.DeletePropagationForeground
    if err := ssetClient.Delete(ctx, sset.GetName(), metav1.DeleteOptions{PropagationPolicy: &propagationPolicy}); err != nil {
        return fmt.Errorf("failed to delete StatefulSet to avoid forbidden action: %w", err)
    }
    continue
}

Foreground propagation deletes the pod before the StatefulSet is gone, and the emptyDir goes with the pod. We saw exactly this log line during the move (next sections). So adding storageSpec and syncing, with nothing else, starts Prometheus on an empty PVC and deletes the old TSDB.

A Red Hat blog post, as quoted in a search result summary (the page itself answered 404 when we fetched it), says "it is not possible to retain data that was written on the default EmptyDir-based installation". Our move contradicts that, and the reason is in how a StatefulSet names its PVCs: the PVC a StatefulSet pod uses is named <volumeClaimTemplate name>-<pod name>, and a PVC that already exists under that name is used as it is. If you create it first and put the TSDB in it, the recreated pod opens the old data.

Where the files have to sit: prometheus-db/

The mount path is /prometheus in both cases, but the subPath is not (pkg/prometheus/common.go, lines 197–204):

func SubPathForStorage(s *monitoringv1.StorageSpec) string {
    if s == nil || s.DisableMountSubPath {
        return ""
    }
    return "prometheus-db"
}

Without storage the emptyDir is mounted at its root; with a volumeClaimTemplate the pod mounts the prometheus-db directory of the PVC. A copy into the root of the PVC would be invisible to Prometheus.

What "Encountered WAL read error, attempting repair" dropped on 11-14

In Prometheus v3.7.3 the repair that follows Encountered WAL read error, attempting repair discards all data after the corruption point: it deletes every WAL segment newer than the corrupted one and rewrites the corrupted one only up to the offset (tsdb/wlog/wlog.go line 399; the log lines at 413, 421 and 443). So the samples after segment 8483, offset 19763238, were lost, while the 19 blocks on disk were not touched. We did not measure how many samples that was.

No size cap on a shared disk

With no size cap (the prometheus_tsdb_retention_limit_bytes 0 above) on a filesystem shared with the rest of server-1, nothing would have stopped the TSDB from filling the node disk; that is why the same change sets retentionSize.

What to check in your cluster now

Steps 1 to 4 are read-only and need kubectl and jq. The names are the kube-prometheus-stack defaults for release name kube-prometheus-stack in namespace monitoring; change them to yours.

1. Where does the TSDB live?

kubectl -n monitoring get prometheus -o jsonpath='{range .items[*]}{.metadata.name}{" storage="}{.spec.storage}{" retentionSize="}{.spec.retentionSize}{"\n"}{end}'
kubectl -n monitoring get sts -l app.kubernetes.io/name=prometheus -o json | jq -r '.items[] | .metadata.name, "  volumeClaimTemplates: \([.spec.volumeClaimTemplates[]?.metadata.name])", (.spec.template.spec.volumes[] | select(.name|test("-db$")) | "  \(.name): \(if .emptyDir then "emptyDir" else "not emptyDir" end)")'

Same problem if: storage= is empty, the -db volume says emptyDir, and volumeClaimTemplates: []. Ours: storage= retentionSize= and prometheus-kube-prometheus-stack-prometheus-db: emptyDir.

2. How much is in it?

kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- sh -c 'du -sh /prometheus; df -h /prometheus'

Same problem if: step 1 said emptyDir; then everything du prints is lost the next time the pod is deleted. df shows which disk it shares, but it cannot tell an emptyDir from a local-path PVC: on k3s the local-path PVC we moved to sits on the same node disk as the emptyDir did. Ours: 22.7G, on a 454.9G root filesystem 57% used.

3. How far back does it go, and is there a size cap?

In the Prometheus UI or with promtool query instant:

(time() - prometheus_tsdb_lowest_timestamp_seconds) / 86400
prometheus_tsdb_retention_limit_bytes

Same problem if: the second value is 0. That means no retentionSize: only time deletes data, and nothing stops the TSDB from filling the disk it shares with the node. Ours: 31.5 days and 0.

4. Has the pod been replaced, and did a restart drop WAL samples?

kubectl -n monitoring get pod prometheus-kube-prometheus-stack-prometheus-0 -o jsonpath='{.metadata.creationTimestamp}{"  restarts: "}{.status.containerStatuses[?(@.name=="prometheus")].restartCount}{"\n"}'
kubectl -n monitoring logs prometheus-kube-prometheus-stack-prometheus-0 -c prometheus | grep -E "attempting repair|Starting corruption repair"

A container restart (an OOMKill, for example) keeps the emptyDir; ours on 2026-11-14 was a node reboot that restarted the containers in the same pod, and the emptyDir stayed. Same problem if: step 1 said emptyDir, whatever the values in this step are; they only show how close you have already been. With an emptyDir, the oldest sample cannot be older than the pod. If the creation time is close to your oldest sample, the pod was replaced then and everything before it was deleted with the old pod. If the creation time is older than your retention, every restart so far kept the data, and the next pod replacement deletes all of it. A repair line in the log means one of those restarts has already dropped the newest WAL samples. Ours: pod created 2025-11-30, last container restart at a node reboot on 2026-11-14, followed by Encountered WAL read error, attempting repair.

Doing this on every run

This problem was found by kubegap, an open-source audit that reads the cluster your kubectl points at and lists operational risks with the evidence for each. kubegap repeats step 1, step 3 and the log check of step 4 on each run: its summary reads the storage kind (emptyDir, or the PVC template and its size), retentionSize, the oldest sample in days, the WAL corruption count and the PVC usage ratio, and reports them as the findings prometheus.storage-risk, observability.stack-storage-emptydir and logs.signatures.prometheus.wal-corrupt. It does not read the pod creation time or restart count of step 4. After the move, a full run at 00:44:32 no longer reported any of the three for Prometheus:

kubegap full runs collected 2026-11-14 16:25 UTC (the report, v0.3.0) and 2026-11-15 00:44 UTC (after the move, one commit after v0.3.0: v0.3.0-1-g8c36a92).

One gap remains in that run: its PVC usage value (pvc_used_ratio) is null, because the kubelet exports no kubelet_volume_stats_* series for this PVC. At 00:50:02 only 2 of the cluster's 39 PVs had one. We did not investigate why the kubelet reports nothing for these local volumes.

What we changed, and what we have not

Changed: kube-prometheus-stack persistent storage, or how to add persistent volume to Prometheus without losing data

Prometheus is installed by the Argo CD Application monitoring from the chart values in the manifests repository, with automated prune: true and selfHeal: true, so the change had to go through Git. We filled the PVC before the commit, because the sync would delete the pod (see "Adding storageSpec" under Why) and with it the only copy of 31 days of metrics.

Where the files went so the recreated pod found them. One copy pod on the same node mounted the old emptyDir from the host, read-only, and the new PVC.

1. The PVC and a copy pod (00:32:52). Applied with kubectl for the move only, not kept in Git (server-1 and the pod uid are masked):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0
  namespace: monitoring
spec:
  storageClassName: local-path-server-1
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 50Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: prometheus-data-copy
  namespace: monitoring
spec:
  nodeSelector:
    kubernetes.io/hostname: server-1
  restartPolicy: Never
  containers:
    - name: copy
      image: alpine:3.20
      command: ["sh", "-c", "apk add --no-cache rsync >/dev/null && mkdir -p /dst/prometheus-db && sleep infinity"]
      volumeMounts:
        - { name: src, mountPath: /src, readOnly: true }
        - { name: dst, mountPath: /dst }
  volumes:
    - name: src
      hostPath:
        path: /var/lib/kubelet/pods/<pod uid>/volumes/kubernetes.io~empty-dir/prometheus-kube-prometheus-stack-prometheus-db
        type: Directory
    - name: dst
      persistentVolumeClaim:
        claimName: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0

The pod uid comes from kubectl -n monitoring get pod prometheus-kube-prometheus-stack-prometheus-0 -o jsonpath='{.metadata.uid}'. The StorageClass local-path-server-1 is a k3s local-path StorageClass: it uses the rancher.io/local-path provisioner, keeps its volumes on the same node disk as the emptyDir, and has binding mode WaitForFirstConsumer, so the PVC was bound only once a pod used it (here the copy pod); the volume was provisioned at 00:33:01.

2. First copy while Prometheus kept running (00:33:20–00:37:53).

$ kubectl -n monitoring exec prometheus-data-copy -- rsync -a --delete --stats --exclude=lock /src/ /dst/prometheus-db/
Number of files: 158 (reg: 116, dir: 42)
Number of regular files transferred: 116
Total transferred file size: 24,396,762,944 bytes
sent 24,414,426,021 bytes  received 2,562 bytes  89,266,649.30 bytes/sec

4 min 33 s at 89 MB/s. The second run used the same command.

3. Second copy right before the switch (00:38:04–00:38:05). Only the WAL and head chunks had changed:

Number of files: 158 (reg: 116, dir: 42)
Total transferred file size: 108,133,626 bytes
22.7G   /dst/prometheus-db

4. The values change (pushed 00:38:05, Argo CD refresh requested 00:38:08). Comment lines omitted:

 prometheus:
   prometheusSpec:
     retention: 30d
+    retentionSize: 40GB
+    storageSpec:
+      volumeClaimTemplate:
+        spec:
+          storageClassName: local-path-server-1
+          accessModes: ["ReadWriteOnce"]
+          resources:
+            requests:
+              storage: 50Gi

5. The operator recreated the StatefulSet (00:38:26), as the source says it would:

ts=2026-11-15T00:38:26.821426997Z level=info caller=/workspace/pkg/prometheus/server/operator.go:996 msg="recreating StatefulSet because the update operation wasn't possible" component=prometheus-controller key=monitoring/kube-prometheus-stack-prometheus statefulset=prometheus-kube-prometheus-stack-prometheus shard=0 reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"

The old pod's containers were stopped (Killing) the same second, and the new pod was created on server-1 at 00:38:30. Its data mount and the new argument:

"dbMount": {
  "mountPath": "/prometheus",
  "name": "prometheus-kube-prometheus-stack-prometheus-db",
  "subPath": "prometheus-db"
}
args: ... "--storage.tsdb.retention.time=30d","--storage.tsdb.retention.size=40GB","--storage.tsdb.path=/prometheus" ...

6. Prometheus opened the copied TSDB. From the new pod's startup log (19 Found healthy block lines, the first shown):

00:38:32.659Z msg="Found healthy block" component=tsdb mint=1791979200038 maxt=1792173600000 ulid=01M53F5PAQ7C40DZXPAKF6RB13
00:38:33.786Z msg="Replaying WAL, this may take a while" component=tsdb
00:39:21.785Z msg="WAL replay completed" component=tsdb checkpoint_replay_duration=2.889331339s wal_replay_duration=45.109577526s ... total_replay_duration=48.613000449s
00:39:22.360Z msg="TSDB started"
00:39:22.867Z msg="Server is ready to receive web requests."

The mint of the first block is the old lowest timestamp, 1791979200. The startup probe failed once with 503 during the replay (00:39:15); the pod was Ready at 00:39:32. At 00:39:55: du -sh /prometheus 22.7G, lowest timestamp 1791979200 (2026-10-14T12:00:00Z), 19 blocks, prometheus_tsdb_retention_limit_bytes 42,949,672,960, prometheus_tsdb_wal_corruptions_total 0. The copy pod was deleted at 00:40:09.

The proof: deleting the pod on purpose

To check that a new pod keeps the data, we deleted the pod at 00:41:41 and compared the same values before and after. Before, at 00:41:40: lowest timestamp 1791979200, 19 blocks, 22.7G.

$ kubectl -n monitoring delete pod prometheus-kube-prometheus-stack-prometheus-0   # 00:41:41
pod "prometheus-kube-prometheus-stack-prometheus-0" deleted from monitoring namespace

The new pod was created at 00:41:43 on the same PVC (Bound, 11 min old), replayed the WAL in 54.4 s, logged TSDB started at 00:42:44 and was ready by 00:43:03. At 00:43:57 all five values were the same as at 00:39:55.

Prometheus at 00:44:17, after both pod replacements: the 5-week graph still starts around Oct 14. The colour changes at the right end because the instance label (the pod IP) changed with each new pod.

What it cost

What the move cost, measured from Prometheus scraping itself every 30 s: two holes in new samples; the stored 30 days stayed.

The self-scrape series up{job="kube-prometheus-stack-prometheus"} has no samples from 00:37:45 to 00:39:51 (2 min 6 s) and from 00:41:21 to 00:42:51 (1 min 30 s). The second hole is the deliberate restart: nothing was scraped while the pod was down, and nothing stored was lost. For the first hole, the inference from the times above: the last sample before the second copy was scraped at 00:37:45 and is in the copy; the next one, due about 30 s later, was written after the copy finished at 00:38:05, while the old pod ran until 00:38:26, so it stayed in the old emptyDir and was deleted with it; after that nothing scraped until the new pod was up. A third rsync between the push and the recreate would have narrowed it; we did not try.

Why 50Gi and retentionSize: 40GB

40GB in retentionSize is 40 GiB: Prometheus documents the units as "Based on powers-of-2, so 1KB is 1024B", the operator's ByteSize type says the same, and the metric shows 42,949,672,960 bytes. That is 80% of the 50Gi request, within the Prometheus documentation's advice "setting the retention size to, at most, 80-85% of your allocated Prometheus disk space". With both limits set, whichever triggers first is used; the current 22.7G is well below either.

On this StorageClass the 50Gi is not a limit. The local-path-provisioner README: "No support for the volume capacity limit currently. The capacity limit will be ignored for now." So retentionSize is the only cap actually enforced, and beyond it only the free space on the node disk limits the TSDB. One more limit from the Prometheus docs: "Only the persistent blocks are deleted to honor this retention although WAL and m-mapped chunks are counted in the total size", so a growing WAL is counted but not shrunk by the cap.

Not changed, and why

• Alertmanager and Grafana are still on emptyDir. kubegap reports observability.stack-storage-emptydir for both. They were not part of problem 2 in the kubegap report; we expect the same copy-then-switch method to work for them but have not tried it.
• The PVC is on the same disk of the same node as the emptyDir was. The data now survives a pod replacement, not the loss of the disk or of server-1. Copies outside the node and the cluster are problem 1 of the same kubegap report and are not covered here. The StorageClass also has reclaim policy Delete; the StatefulSet's persistentVolumeClaimRetentionPolicy is Retain for both deletion and scale-down, so deleting the StatefulSet keeps the PVC, but deleting the PVC by hand deletes the data.
• No volume-filling alert for the new PVC. There is no usage metric for it (the kubelet exports none), so neither a KubePersistentVolumeFillingUp alert nor kubegap sees it fill. retentionSize is the cap that is enforced; the existing node disk alert rules were not changed.
• Prometheus memory and CPU. Memory was 1081Mi of the 1536Mi limit before the move, and the startup probe allows 60 × 15 s = 15 minutes; WAL replays here took 45 s and 54 s. CPU throttling and the memory limit are other items of the same kubegap report.

Where else this happened

The Incident Library, a searchable index of public postmortems on benchlet.com (links below), has no case of a Prometheus TSDB deleted because it ran on an emptyDir. The closest cases are Prometheus restarts that went wrong at the WAL replay or the probes. This move restarts Prometheus twice and replays the WAL each time (45 s and 54 s here, under a startup probe that allows 15 minutes), so if your WAL is larger, check the startup probe before the switch:

• UK Ministry of Justice, 2024-04-15: a planned change restarted Prometheus, the WAL reload took longer than the startup probe allowed, and the platform had no monitoring for 3 h 21 min until the probe was raised to 30 minutes.
• Saifeddine Rajhi, 2024-06-14 (personal blog): after a kube-prometheus-stack upgrade Prometheus was OOMKilled in a loop while replaying its WAL, which needed more than 50Gi of memory against about 30Gi in steady state.
• UK Ministry of Justice, 2023-11-01: Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in the metrics.

More: Prometheus + monitoring down, 8 cases. Cases where data on a pod or node was gone after a change, the outcome this move avoided, are in data loss on Kubernetes, 5 cases.

Timeline

Time (UTC)	Event
2025-11-11 07:49:16	First commit of the chart values file; it never contains storageSpec.
2025-11-30 06:09:35	The Prometheus pod is created on server-1 with its TSDB on an emptyDir.
2026-11-14 01:12:30	Node reboot; the Prometheus containers restart in the same pod (reason Unknown). The emptyDir is kept.
2026-11-14 01:14:32	Encountered WAL read error, attempting repair, Starting corruption repair segment 8483; samples after the corruption point are discarded.
2026-11-14 16:25:29	kubegap v0.3.0 full run reports problem 2: Prometheus data on an emptyDir, 22.3GiB, oldest sample 31.2 days, no size cap.
2026-11-15 00:31:25–00:31:36	Our checks: emptyDir, 22.7G, oldest sample 2026-10-14T12:00:00Z, no size cap.
00:32:52	PVC prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 (50Gi) and copy pod created.
00:33:01	The PVC's volume is provisioned on server-1.
00:33:20–00:37:53	First rsync while Prometheus runs: 24,396,762,944 bytes, 4 min 33 s.
00:37:45	Last self-scrape sample of the old pod.
00:38:04–00:38:05	Second rsync: 108,133,626 bytes changed; 22.7G in the PVC.
00:38:05	Values commit with retentionSize: 40GB and storageSpec.volumeClaimTemplate pushed; Argo CD refresh at 00:38:08.
00:38:26	Operator: recreating StatefulSet because the update operation wasn't possible; old pod's containers stopped.
00:38:30	New pod created on server-1, PVC mounted with subPath: prometheus-db.
00:39:15	Startup probe fails once with 503 during WAL replay.
00:39:22	19 healthy blocks, WAL replay 45.1 s, TSDB started, ready to receive web requests.
00:39:32	Pod Ready.
00:39:51	First self-scrape sample of the new pod (hole of 2 min 6 s).
00:39:55	Size, oldest sample and block count unchanged; size cap 40 GiB.
00:40:09	Copy pod deleted.
00:41:41	kubectl delete pod prometheus-kube-prometheus-stack-prometheus-0; new pod created 00:41:43 on the same PVC.
00:42:44	WAL replay 54.4 s, TSDB started; first self-scrape sample at 00:42:51 (hole of 1 min 30 s); ready by 00:43:03.
00:43:57	Size, oldest sample and block count unchanged.
00:44:17	5-week graph of prometheus_tsdb_head_series still starts around Oct 14.
00:44:32	kubegap full run (one commit after v0.3.0, v0.3.0-1-g8c36a92): the three Prometheus findings are gone; emptyDir still reported for Alertmanager and Grafana.
00:50:02	No kubelet_volume_stats_* series for the new PVC; 2 of 39 PVs have one.

FAQ

prometheus data lost after pod restart

It depends on what restarted, because with no storageSpec the prometheus operator emptyDir default stores the TSDB in a volume that is deleted with the pod, not when a container restarts. A container restart (OOMKill, liveness failure) keeps the emptyDir, so the blocks stay; ours was a node reboot that restarted the containers in the same pod, and the emptyDir stayed. Even then, a WAL repair after an unclean stop can still drop the newest samples. A new pod gets an empty emptyDir: drain, eviction, kubectl delete pod, a chart change to the pod template, or adding storageSpec.

kube-prometheus-stack persistent storage

Set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate with a StorageClass (ours, local-path-server-1, is a k3s local-path StorageClass with the rancher.io/local-path provisioner), access mode and size; the chart default is storageSpec: {}, which leaves the operator's emptyDir. Add retentionSize below the volume size (Prometheus docs: at most 80–85%). On a running Prometheus this change makes the operator recreate the StatefulSet, so copy existing data first.

prometheus operator add persistent volume without losing data

Create the PVC the new StatefulSet will ask for, prometheus-<name>-db-prometheus-<name>-0, with the StorageClass and size you will put in storageSpec. Copy the old emptyDir from the node into prometheus-db/ on it, once while running and once just before the switch. Then commit storageSpec. Our 30 days survived the switch and a deliberate pod delete. The cost was 2 min 6 s of samples.

Encountered WAL read error, attempting repair

Prometheus hit a WAL read error while loading the head at startup. In v3.7.3 the repair deletes every WAL segment newer than the corrupted one and cuts that segment at the corruption offset, so the samples after that point are gone; blocks are not touched. prometheus_tsdb_wal_corruptions_total counts it. Ours followed a node reboot.

retentionSize GB or GiB

GiB. Both Prometheus and prometheus-operator define the units as powers of 2, so 1KB is 1024B and our retentionSize: 40GB shows up as prometheus_tsdb_retention_limit_bytes 42,949,672,960. The cap counts WAL and m-mapped head chunks, but only blocks are deleted to honour it, so leave headroom below the volume.

local-path provisioner PVC size not enforced

Not enforced: the local-path-provisioner README says "The capacity limit will be ignored for now." Our storage: 50Gi does not stop the TSDB from growing, so retentionSize (ours 40GB) is the only cap, and beyond it only the free space on the node disk limits the TSDB.

---

Originally published at benchlet.com.

Comments

[CloudCompare]

This is a really practical walkthrough - the emptyDir default catching people out is such a common trap, and spelling out how to check which mode you are in before changing anything is the part most posts skip.

One datapoint from a small single-node home cluster: once the TSDB is on a PVC, pod restarts and node reboots have been uneventful - the volume just comes back with the pod. The cutover itself is the sharp bit: existing emptyDir series do not migrate, so it is worth framing the switchover as 'accept a small gap of recent data' (or snapshot the WAL first) rather than expecting a seamless handover.

Curious what you settled on for retention as well: we keep a bounded retention on the local PVC and ship blocks out to object storage via a Thanos sidecar, which keeps the local disk comfortably small. When the storageSpec PVC is sized generously, do you find plain retention alone is enough?