DEV Community

david
david

Posted on • Originally published at woitzik.dev

Redis Killed Nextcloud and Nobody Noticed for Hours

#kubernetes #redis #homelab

Originally published at woitzik.dev

The Nextcloud outage looked like a capabilities problem at first. Kubernetes had just received a securityContext hardening pass — readOnlyRootFilesystem: true across most containers — and the timing fit. Pods were up, they passed readiness checks, but PHP requests were hanging and returning zero bytes. The capabilities hardening had broken things before. It was a reasonable assumption.

It was wrong. The capabilities issue was real, but it wasn't what took Nextcloud down. This is the Redis story.

View the complete homelab infrastructure source on GitHub 🐙

The Setup That Created the Problem

Nextcloud's session data and file lock cache both run through a Redis instance deployed as a sidecar-style container in the same manifest. That Redis container had no PersistentVolumeClaim — it was intentionally ephemeral, purely in-memory, with sessions expected to survive only as long as the pod did.

The problem: Redis doesn't know it's running without a PVC. Its default configuration ships with persistence enabled:

save 3600 1   # write RDB snapshot if 1 key changed in the last hour
save 300 100  # write RDB snapshot if 100 keys changed in the last 5 minutes
save 60 10000 # write RDB snapshot if 10000 keys changed in the last minute
Enter fullscreen mode Exit fullscreen mode

Nextcloud is an active application. It easily generates 100 session and lock writes within 300 seconds of normal use. When the 300-second threshold triggered, Redis attempted to write a snapshot — and hit a read-only filesystem:

Failed opening the temp RDB file temp-repl-16307.rdb (in server root dir /data)
for saving: Read-only file system
Background saving error
Enter fullscreen mode Exit fullscreen mode

That error alone would be logged and ignored. Redis would retry on the next threshold. The issue is what happens after enough background saves fail: Redis enables its stop-writes-on-bgsave-error safeguard, which is yes by default, and which means exactly what it says — Redis stops accepting write commands entirely.

Why the Logs Were Misleading

The Redis log showed the RDB error. But Redis was still running, still responding to pings, still reachable via the network — just refusing writes. From Nextcloud's PHP perspective, the Redis session handler was calling SET and getting back a Redis error object instead of OK. PHP's error handling in that path doesn't surface a visible 500 — it hangs the request or returns an empty response, depending on the session lock timeout.

The result from outside: pages loaded up to the authentication stage, then returned nothing. Not a 500, not a timeout message — nothing. That looks exactly like what capabilities.drop: [ALL] does when it breaks an entrypoint: the process starts, gets partway through initialization, then stops.

The actual diagnostic path that found it:

# Redis container still running, pod shows Running — check the logs directly
kubectl logs nextcloud-<pod> -n apps -c redis-nextcloud --previous

# Key line:
# "Background saving error" / "MISCONF Redis is configured to save RDB snapshots,
# but it's currently not able to persist on disk."

# Confirm Redis is refusing writes:
kubectl exec -it nextcloud-<pod> -n apps -c redis-nextcloud -- redis-cli SET test_key test_val
# → (error) MISCONF Redis is configured to save RDB snapshots...
Enter fullscreen mode Exit fullscreen mode

Once the actual Redis error surfaced, the fix was immediate and obvious.

The Fix: Invoke redis-server Directly with Persistence Off

The fix used the same pattern already in place for the cluster's other ephemeral Redis instances (immich-redis, valkey) — invoke redis-server directly with the relevant flags rather than relying on the image's default config:

# Before: redis-nextcloud relied on image defaults (persistence on)
containers:
  - name: redis-nextcloud
    image: redis:8

# After: explicit command disabling both RDB and AOF
  - name: redis-nextcloud
    image: redis:8
    command:
      - redis-server
      - "--save"
      - ""          # empty string disables all save thresholds
      - "--appendonly"
      - "no"
Enter fullscreen mode Exit fullscreen mode

The same change applied to paperless's Redis, which had the same no-PVC configuration. The authelia Redis was left untouched — it actually has a real PVC (kubernetes/system/redis) so its --appendonly yes configuration is intentional and works correctly.

The Asymmetry That Makes This Hard to Catch

Redis with persistence enabled and no PVC doesn't fail immediately. It runs, accepts connections, handles traffic, and appears healthy in every Kubernetes signal:

  • Pod status: Running
  • Readiness probe: passing (Redis responds to TCP or PING checks)
  • ArgoCD application: Synced/Healthy
  • Restart count: 0

The failure only manifests after the first persistence threshold is hit — which for the 300 100 threshold (100 writes in 5 minutes) could be minutes into normal operation, or could be hours if traffic is light. After that threshold fires and the first failed RDB save triggers stop-writes-on-bgsave-error, the failure is silent at the Redis level and confusing at the application level.

The version of this that would have prevented the outage:

# Before deploying any Redis instance, check whether it has a PVC:
grep -A 20 "redis" kubernetes/apps/nextcloud/nextcloud.yml | grep -i "persistentVolumeClaim\|claimName"
# If nothing → persistence must be explicitly disabled
Enter fullscreen mode Exit fullscreen mode

Or more defensively: any Redis container without a PVC in its volume list gets --save "" --appendonly no as a matter of policy, not an afterthought.

Why This Shows Up in Hardened Clusters More Than Default Ones

The readOnlyRootFilesystem: true security hardening is what surfaced this. Without it, Redis would have written its RDB snapshots to the container's writable root filesystem — wasteful, potentially filling the node's ephemeral storage, but functionally harmless in the short term. The write would succeed, stop-writes-on-bgsave-error would never trigger, and the outage would never happen.

In that sense, the security hardening exposed a pre-existing misconfiguration that was only dormant because the filesystem was writable. The hardening was correct. The Redis configuration was wrong, and the hardening just removed the thing that had been quietly absorbing the mistake.

The same scenario applies to any Redis instance in Kubernetes where sessions, cache, or ephemeral data don't need to survive a pod restart — which, in a cluster with properly-sized application replicas, is most of them. If the Redis deployment doesn't have a PVC, it should have --save "" --appendonly no as explicit arguments rather than inheriting defaults designed for a server with durable storage.

Top comments (0)