Longhorn Read-Only Mounts: Detection, Recovery, and Closing the Silent Failure Window

A Longhorn volume can sit in attached state with robustness: healthy in the UI while the ext4 filesystem inside the pod has been mounted read-only for six hours. The control plane and the filesystem are reporting on two different things, and only one of them knows your application stopped writing.

That's the whole gotcha. Everything below is how to notice it before your users do.

The symptom

The application doesn't crash. It keeps answering HTTP requests, keeps passing its liveness probe, keeps showing up green in whatever dashboard you built. But writes fail with EROFS, and depending on how the app handles that, you get one of three flavors of bad:

• The app logs the error and keeps going (worst case, silent data loss).
• The app buffers writes in memory and grows until OOMKilled.
• The app returns 500s on writes and 200s on reads, so uptime checks stay green.

Postgres is the polite one here: it will refuse to start a new checkpoint and eventually panic. A web app writing uploads to a PVC will happily return "upload complete" while the file goes nowhere. An application writing metrics or logs to a volume just stops writing and nobody notices until someone asks why the graph flatlined.

Inside the container:

$ kubectl -n apps exec deploy/my-app -- touch /data/.probe
touch: cannot touch '/data/.probe': Read-only file system
command terminated with exit code 1

$ kubectl -n apps exec deploy/my-app -- grep /data /proc/mounts
/dev/sdc /data ext4 ro,relatime,stripe=... 0 0

That ro in the mount options is the tell. Nobody set it. The kernel set it.

What I expected

My mental model for a while was that Kubernetes storage failures are loud. A replica dies, Longhorn marks the volume degraded, the alert fires, I go look. If things get bad enough the pod can't mount, it sits in ContainerCreating with a screaming event, and that's also loud.

The middle ground never entered the picture: the volume recovers at the Longhorn layer, but the filesystem does not un-fail itself. Longhorn's job ends at "there's a block device here and replicas are in sync." The filesystem sitting on top of that block device made an independent decision, and Longhorn has no opinion about it.

I wrote about the general version of this gap in Longhorn Volume Health: The Gap Between 'Healthy' and Actually Working. Read-only remounts are the sharpest edge of that gap, because it's the one failure mode where every dashboard you have says everything is fine.

What actually happened

Here's the chain, and each link is boring on its own.

1. The volume's replicas become briefly unreachable. In a homelab this is almost never a disk failure. It's a switch reboot, a node going unresponsive under memory pressure, a kubelet restart, an MTU mismatch after a network change, or a node that hard-froze (I've written about one specific cause of that). Longhorn's engine-replica-timeout setting defaults to 8 seconds. Exceed it and the engine marks that replica errored.

2. The engine loses quorum on writes. With every replica errored, the engine has nowhere to write. The iSCSI target backing /dev/longhorn/pvc-xxxx starts returning I/O errors to the kernel.

3. The kernel does what it was told to do. ext4's default errors=remount-ro behavior kicks in. On the node you'll find something like this in dmesg:

blk_update_request: I/O error, dev sdc, sector 2097168 op 0x1:(WRITE)
Buffer I/O error on dev sdc, logical block 262146, lost async page write
EXT4-fs error (device sdc): ext4_journal_check_start:83: comm postgres: Detected aborted journal
EXT4-fs (sdc): Remounting filesystem read-only

This is correct behavior. The filesystem chose data integrity over availability, which is exactly what you want it to do.

4. Longhorn recovers. The filesystem does not. The network hiccup ends, replicas reconnect, auto-salvage does its thing, and the volume goes back to healthy. The block device is fine now. But a remount to ro is sticky. Nothing in the stack goes back and remounts it rw, because nothing in the stack is watching.

The pod stays running the entire time. Kubelet has no idea. Kubelet's job was to mount the volume, and it did, successfully, hours ago.

And the part that surprised me most: a liveness probe won't save you even if it catches the failure. A failed liveness probe restarts the container, not the pod. The volume mount lives at the pod sandbox level, so the container comes back and lands on exactly the same read-only mount. You get a CrashLoopBackOff that never resolves, and if you're unlucky the restarts are quiet enough that you read the loop as an app bug rather than a storage one.

Detecting it

Three layers, from cheapest to most thorough.

The Longhorn CR is the wrong place to look, but check it anyway

kubectl -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=NAME:.metadata.name,STATE:.status.state,solid:.status.robustness

If a volume shows faulted, you have a different (louder) problem. If it shows healthy, that tells you the storage layer recovered. It does not tell you the filesystem did. This is the check that lies to you, so know what it's actually answering.

node_exporter has the metric, but not for your PVCs

node_filesystem_readonly is exactly the metric you want. There's a catch: node_exporter's default --collector.filesystem.mount-points-exclude regex excludes /var/lib/kubelet/.+, which is where every single CSI volume gets mounted on the host. Out of the box, your PVC mounts are invisible to it.

You can widen the exclusion regex, and it works, but you'll pay for it in cardinality on a cluster with a lot of volumes, and you'll get a metric labeled by an opaque pvc-<uuid> path with no workload context. It's a reasonable backstop. It's a poor primary signal.

Probe from inside the pod, and actually hit the disk

This is the one that works. The naive version:

readinessProbe:
  exec:
    command: ["sh", "-c", "touch /data/.rw-probe && rm -f /data/.rw-probe"]
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 2

touch on a read-only mount fails at the VFS layer with EROFS before any I/O happens, so this catches the remount case immediately. What it doesn't catch is a filesystem that's still nominally rw while the block device underneath has gone away, because the metadata update can land in the page cache and return success.

The version I'd actually ship forces the write to the device:

readinessProbe:
  exec:
    command:
      - sh
      - -c
      - "dd if=/dev/zero of=/data/.probe/rw bs=4k count=1 conv=fsync 2>/dev/null"
  initialDelaySeconds: 20
  periodSeconds: 60
  timeoutSeconds: 10
  failureThreshold: 3

Two things matter in that snippet. conv=fsync is what makes it a real test instead of a page-cache test. And the write goes to a subdirectory, not the data root, because plenty of applications (Postgres among them) will complain loudly about unexpected files in their data directory.

Use a readiness probe, not liveness, for the reasons above. Readiness pulls the pod out of Service endpoints and flips kube_pod_status_ready, which is something you can alert on without a restart loop confusing the picture.

Keep the period generous. A 60-second period with failureThreshold: 3 gives you a three-minute detection window and a handful of 4 KB writes per hour per pod. That's a fine trade.

A canary for the cluster-wide case

Per-pod probes only cover pods you remembered to instrument. A small canary workload with its own PVC per storage class gives you a floor: a CronJob or Deployment whose only job is to write a timestamp, fsync it, read it back, and expose a metric.

Be honest with yourself about what this catches. Read-only remounts are per-volume, so a canary volume won't tell you that your database's volume went read-only. What it catches is the cluster-wide or node-wide version: a Longhorn upgrade that went sideways, a node whose disk filled, a network change that broke replica traffic everywhere at once. It's a smoke detector, not a per-room alarm. Run both.

The alerting rules

groups:
  - name: longhorn-writability
    rules:
      - alert: LonghornVolumeFaulted
        expr: longhorn_volume_robustness == 3
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "Longhorn volume {{ $labels.volume }} is faulted"

      - alert: StorageCanaryWriteFailed
        expr: storage_canary_write_success == 0
        for: 5m
        labels: { severity: critical }

      - alert: PodNotReadyWithPVC
        expr: |
          kube_pod_status_ready{condition="true"} == 0
          and on (namespace, pod)
          kube_pod_spec_volumes_persistentvolumeclaims_info
        for: 10m
        labels: { severity: warning }

longhorn_volume_robustness uses 0 for unknown, 1 healthy, 2 degraded, 3 faulted. I don't page on == 2 (degraded), because degraded is normal and transient during node reboots and rebuilds. That's the difference between an alert you act on and an alert you learn to ignore, which is the whole argument in Prometheus Alerting Rules That Don't Cry Wolf.

Recovering

Confirm the diagnosis first. Map the kernel device back to a PV on the node where the pod is scheduled:

ls -l /dev/longhorn/
# pvc-a1b2c3d4-... -> /dev/sdc
dmesg -T | grep -E 'EXT4-fs (error|warning)|Remounting filesystem read-only'

Then fix it. mount -o remount,rw is not the fix. ext4 sets an error flag in the superblock; a remount without a check either fails or hands you a filesystem with known-bad metadata. What you want is a full detach and reattach, because Kubernetes mount-utils runs fsck -a on ext4 before mounting an existing filesystem.

# 1. Scale the workload to zero. This is what triggers the unmount.
kubectl -n apps scale deploy/my-app --replicas=0

# 2. Wait for Longhorn to actually detach. Don't skip this.
kubectl -n longhorn-system wait --for=jsonpath='{.status.state}'=detached \
  volumes.longhorn.io/pvc-a1b2c3d4-0000-0000-0000-000000000000 --timeout=180s

# 3. Scale back up. fsck -a runs on mount.
kubectl -n apps scale deploy/my-app --replicas=1

Scaling to zero matters more than it looks. Deleting the pod on a Deployment gives you a new pod that may schedule before the old volume finishes detaching, and Longhorn v1 volumes with ReadWriteOnce will just sit in attach/detach limbo. This is the same detach ordering problem that makes node drains hang, which I covered in Pod Disruption Budgets: Why kubectl drain Gets Stuck on Longhorn.

If the pod comes back stuck in ContainerCreating, check the events:

kubectl -n apps describe pod my-app-xxxx | grep -A5 Events
# ... MountVolume.MountDevice failed ... 'fsck' found errors ... exit status 4

Exit status 4 means e2fsck found errors it wouldn't correct unattended. At that point you need a manual pass: attach the volume in maintenance mode from the Longhorn UI (which attaches the block device to a node without a workload), SSH to that node, and run e2fsck -f /dev/longhorn/pvc-<uuid> interactively. Take a Longhorn snapshot before you do that. Manual fsck can and does discard data into lost+found.

And if the answer to fsck is "the metadata is gone," you're in restore territory. That's what Velero + MinIO is for. Longhorn replicas protect you from a disk dying. They don't protect you from a filesystem that corrupted itself and dutifully replicated the corruption three ways.

Why this matters

The general lesson generalizes past Longhorn: health checks that don't exercise the dependency don't check health. An HTTP liveness probe that returns 200 from an in-memory handler tells you the process is scheduled on a CPU. It says nothing about the disk, the database connection, or the message queue. Every storage layer I've run has some version of this gap where the control plane's idea of healthy and the data path's idea of healthy diverge, and the divergence is always quiet.

A few things I'd do differently, or would tell someone setting this up fresh:

• Instrument stateful workloads first, not last. Anything with a PVC that holds data you'd miss gets a write-based readiness probe on day one. It's eight lines of YAML.
• Alert on faulted, not on degraded. Degraded volumes are a normal part of a cluster that reboots nodes. Paging on them trains you to ignore the storage alerts entirely.
• Assume the filesystem won't recover just because the volume did. Build the detach/reattach runbook before you need it, and put the wait --for=jsonpath step in it, because that's the step everyone skips at 2 AM.
• Consider XFS if your failure mode preference is different. XFS shuts the filesystem down on serious errors rather than remounting read-only, which means the application fails harder and faster. Louder failure is easier to catch. It's also harder to recover from. Pick your poison deliberately.

If you're building out monitoring for a bare-metal cluster and want a second set of eyes on where the observability gaps are, that's the kind of thing I do consulting work on. The read-only mount case is a good canary for the broader question: how many of your green dashboards are measuring the thing you actually care about, versus something adjacent to it that's easier to measure?

If you're new to Longhorn on bare metal, Kubernetes Storage on Bare Metal: Longhorn in Practice covers the setup side. This post is the failure mode you'll eventually meet after it's been running a while.