"Oops, I Forgot to Tell You That's Dangerous": Claude Code Watched Me Wipe Production Redis - Then Helped Carve It Back Off the Disk"

TL;DR: I enabled Redis AOF persistence by editing redis.conf and restarting the service. On the test environment it worked fine. On production, all three nodes came back with zero keys, and the empty dataset overwrote dump.rdb. The missed step: you must enable AOF on the running instance with CONFIG SET appendonly yes before putting it in the config file. We got the data back by carving the unlinked RDB file out of raw disk blocks with dd. Oh, and the AI assistant that helped me plan the change knew about this trap the whole time - it just didn't mention it until after the wipe. Here's the full story.

Some context first

The stage for this story is a legacy healthcare project that's been running in production for years. Redis sits at the center of it, wearing two hats: it's the cache in front of SQL Server, and it's also the feature flag storage - the backend seeds flags into Redis at startup, and both backend and client read them from there at runtime. Over the years the flag storage format evolved (more on that in a minute), the cache quietly accumulated datasets that exist only in Redis, and the Redis setup itself - a bare tarball install with an untouched config - predates everyone's memory of who set it up. In other words: exactly the kind of system where nobody looks at the persistence settings until something forces them to.

Something did.

It started with a feature flag that "flipped itself"

After we published a new release, one of our feature flags apparently turned itself on. Nobody had touched it. The flag had existed for months with a default of false, and suddenly the client started behaving as if it were true.

I diffed the two release tags first: zero changes to flag definitions or defaults. So I went to where the flags actually live in Redis - and found the flag stored in two places, disagreeing with each other:

redis-cli GET  FeatureFlag:CrossWindowCommands   → "true"    # legacy string key
redis-cli HGET FeatureFlag CrossWindowCommands   → "false"   # current hash field

The current implementation stores flags as fields in a FeatureFlag hash, but there was also a stale legacy string key left over from the old storage format. The false in the hash, it turned out, wasn't the original state at all - a teammate had dug into the same problem before me and manually set it back to false. Which made the mismatch the real hint: the legacy key still held true, untouched, and the values in Redis always win - the startup migration copies legacy keys into the hash with HSETNX, and the config default only applies when the field doesn't exist yet.

So the explanation was less dramatic than a flag flipping itself: the flag had been true in Redis all along. It just didn't matter, because for months no code path ever read it. Then the new client release shipped code that actually did something with the flag, and a value that had been harmlessly wrong the whole time suddenly had teeth. An unused flag quietly became a used one, and it looked like a spontaneous flip.

The real finding: our persistence was a time bomb

While investigating, I checked how the Redis instance itself was configured. It's a redis-stack tarball install running natively on RHEL under a custom systemd unit - no Docker, no operator, one master and two replicas with Sentinel. The persistence settings were all defaults - nothing configured at all:

appendonly no                    # default
save 3600 1 300 100 60 10000     # default snapshot thresholds

At our write volume, that works out to roughly one snapshot an hour. So every restart could silently roll the dataset back by up to an hour - and worse, a rollback could resurrect old values (like stale legacy flag keys) that had been deleted since the last snapshot. That's exactly the kind of environment where flags "change themselves" and nobody can explain why.

So we made a decision: park the whodunit, fix the root cause. Enable AOF (append-only file). With aof-timestamp-enabled yes you even get a timestamped log of every write command - a free audit trail for exactly this class of mystery:

appendonly yes
appendfsync everysec
aof-timestamp-enabled yes
auto-aof-rewrite-min-size 64mb
auto-aof-rewrite-percentage 100

(Honest caveat: it's a what-and-when audit, not a who - the AOF doesn't record which client or user issued a command. Since this instance effectively has a single user anyway, that was good enough for a start.)

Testing: everything green

I tested the change end to end. First in a throwaway Docker container: enabled AOF, wrote some flags, and confirmed the AOF captured every command with #TS: timestamps. I even benchmarked it: 100,000-op redis-benchmark runs against two identical containers came back at ~181k SET/s with AOF versus ~142k without. Yes, the AOF run scored higher - which tells you the difference is pure run-to-run noise. With appendfsync everysec, the write-throughput cost is unmeasurable.

Then on the test environment cluster. I enabled it live on the running instance:

redis-cli CONFIG SET appendonly yes
redis-cli CONFIG SET aof-timestamp-enabled yes

I did that live purely to check the concept: right after flipping it I changed a couple of values and watched them show up in the AOF file. A quick sanity check, nothing more - or so I thought. I had no idea this throwaway "test" step was the one doing the heavy lifting.

I then added the same settings to redis.conf so they'd survive a restart, rebooted the nodes one by one, and verified. Everything held: aof_enabled:1, data intact, dummy writes visible in appendonlydir/*.incr.aof, timestamps and all.

The test environment was perfect.

Production: 0 keys

On production I did what felt like the same change: edited redis.conf on all three nodes, added appendonly yes, stopped the sentinels (deliberately, so no surprise failovers mid-maintenance), and rebooted the nodes one by one.

Why full reboots instead of just restarting the Redis service? Two reasons that felt responsible at the time. All three servers had quietly accumulated 500+ pending OS updates, including some severe security patches - so since I was already in a maintenance window, it seemed silly not to apply them and reboot in one go. And a real restart was part of the plan anyway: the whole point of putting appendonly yes into the config file was to prove the setting survives a node going down, so I wanted to see it hold through a genuine reboot.

Then:

redis-cli DBSIZE
(integer) 0

Empty. Master and both replicas. And dump.rdb on disk? Also empty - about 100 bytes of RDB header and nothing else. Roughly 290,000 production keys, gone: cache, feature flags, and one Redis-only ID-mapping dataset that has no SQL fallback at all.

The missed step

Here's the trap, and if you take one thing from this article, take this:

When Redis 7 starts with appendonly yes, it loads the dataset from the AOF - and ignores dump.rdb.

On the test environment I had run CONFIG SET appendonly yes on the live instance first. That triggers an AOF rewrite that builds the AOF base file from the data currently in memory. By the time the test environment restarted, a fully populated AOF existed on disk, and Redis restored from it.

On production I skipped straight to the config file. So the first startup with appendonly yes found no AOF containing the dataset, initialized an empty one, and came up blank. From there the damage compounds automatically:

1. Redis starts empty and creates an empty AOF base.
2. The next background save writes the empty dataset over dump.rdb - and Redis replaces the file via write-temp-then-rename(), so the original file's inode is unlinked.
3. Replicas reconnect and do a full resync from the empty master, wiping themselves too.

Three nodes of redundancy, all faithfully replicating nothing. (Replication protects you from machine failure. It does not protect you from yourself - it distributes your mistake.)

Full disclosure: I wasn't doing this alone. I had an AI assistant (Claude) in the loop for the whole change - it designed the test rehearsal, wrote the config, and later drove the recovery. It knew about this trap. It just never said it out loud before the prod run. Here's the moment it understood what had happened, opening with its own mea culpa:

This is the classic - and brutal - AOF-enablement trap, and I should have flagged it explicitly for prod: when Redis starts with appendonly yes in the conf but no AOF files exist yet, it loads from the (nonexistent) AOF and ignores dump.rdb entirely - it starts empty and creates an empty AOF. On the test environment this didn't bite because you ran CONFIG SET appendonly yes before rebooting - that built the AOF from the live dataset, so the reboot loaded it. On prod the conf was edited and rebooted directly - first boot came up empty, the next background save overwrote dump.rdb with the empty dataset, and each replica that reconnected did a full resync from the empty master and wiped itself too. That's how all three ended up at 0 keys.

A correct diagnosis in one message - six and a half hours too late to be a warning. (To be fair: I never asked "what could go wrong with this rollout?" either. Neither of us rehearsed the failure mode; we only rehearsed success.)

One honest footnote from the later forensics: block-level evidence showed that at one point during the rollout an AOF base with the full dataset existed on disk for a while, and the actual wipe most likely happened on a subsequent restart in the sequence. The exact fatal moment is unrecoverable; the end state was unambiguous - empty AOF, empty RDB, empty replicas.

And of course: no backup copy of dump.rdb taken before the change, and no off-box backups. (I know. I know.)

The recovery: your data is probably still on the disk

Here's the insight that saved us: because Redis replaces dump.rdb via rename(), the old file wasn't overwritten in place. Its inode was unlinked, but the data blocks were still sitting in the free space of the filesystem, waiting to be reclaimed.

So the plan became: stop all writes, and go dig through the raw block device.

Step 1 - freeze everything. Stop Redis, don't reboot, minimize writes to the filesystem. Every write is a chance for the filesystem to reclaim the very blocks you need.

Step 2 - try an LVM snapshot. The volume group had zero free extents, so no snapshot headroom. Plan B.

Step 3 - scan the block device for RDB signatures. Every RDB file starts with the magic bytes REDIS00. The obvious approach dies immediately on a 2 GB RAM box:

$ sudo grep -abo 'REDIS00' /dev/mapper/rhel-root
grep: memory exhausted

(Block devices have no newlines; grep tries to buffer one infinite "line".) So: a tiny Python scanner that reads the device in 32 MB chunks with an overlap of the pattern length, and prints the absolute byte offset of every match:

DEV   = "/dev/mapper/rhel-root"
PAT   = b"REDIS00"
CHUNK = 32 * 1024 * 1024

import sys
with open(DEV, "rb", buffering=0) as f:
    prev, base = b"", 0
    while True:
        buf = f.read(CHUNK)
        if not buf:
            break
        data  = prev + buf
        start = base - len(prev)
        i = data.find(PAT)
        while i != -1:
            print(start + i, flush=True)
            i = data.find(PAT, i + 1)
        prev  = data[-(len(PAT) - 1):]   # overlap so boundary-spanning hits aren't missed
        base += len(buf)
        if base % (1024**3) < CHUNK:
            print(f"... scanned {base // 1024**3} GB", file=sys.stderr)

Run it with sudo and nohup, go make coffee.

Step 4 - triage the hits. The scan found 10 candidates. Each got a 128-byte peek, read-only:

sudo dd if=/dev/mapper/rhel-root iflag=skip_bytes skip=$OFFSET bs=128 count=1 | od -c

Two hits were, hilariously, the scanner's own output files (the data directory shared the LV we were scanning - which is also why carve output had to be shipped off-box). Four were post-incident debris: the empty AOF bases and near-empty dumps left behind by the wipe. That left four candidates of 35-38 MB with real data, distinguishable by the timestamps embedded in their headers: the last pre-incident dump.rdb, and AOF base files from during the incident window.

Step 5 - carve the candidates to another machine. Never write recovery output to the disk you're recovering from:

sudo dd if=/dev/mapper/rhel-root iflag=skip_bytes skip=59609546752 bs=1M count=64 \
  | ssh user@rescue-box 'cat > /home/redis/cand_D.rdb'

Step 6 - validate. redis-check-rdb on the first candidate reported a CRC error - a block near the tail of the file had already been partially reclaimed - but it still parsed all 290,826 keys. Usable in an emergency, so we set it aside and kept going. The next candidate, the freshest one, came back clean:

$ redis-check-rdb cand_D.rdb
[offset 39730397] Checksum OK
[offset 39730397] \o/ RDB looks OK! \o/
[info] 290834 keys read

Step 7 - verify the contents in a sandbox. Never point production at an unverified file. A throwaway instance on the rescue box, isolated port, no config inheritance:

redis-server --port 6391 --dir /home/redis/rescue-test --dbfilename dump.rdb \
             --appendonly no --daemonize yes
redis-cli -p 6391 DBSIZE
redis-cli -p 6391 HGETALL FeatureFlag
redis-cli -p 6391 shutdown nosave

Real keys, real flag values. We had our database back - carved out of unallocated disk blocks.

Putting it back - carefully this time

Restoring into a Sentinel topology has its own trap: if a sentinel promotes a replica that still holds the empty dataset while you're restoring the master, replication will happily sync the emptiness right back over your restored data. So the order was strict:

1. Confirm sentinels are stopped on all nodes, then stop replicas, then the master.
2. On the master: move the poisoned artifacts aside - never delete evidence mid-incident:

   mv appendonlydir appendonlydir.bad
   mv dump.rdb dump.rdb.empty
   cp cand_D.rdb dump.rdb

1. Set appendonly no in redis.conf - the whole point is to force this boot to load dump.rdb.
2. Start the master and hold your breath:

   redis-cli DBSIZE
   (integer) 290834

1. Start the replicas (they full-resync from the restored master - you do not restore the RDB onto replicas), then the sentinels, last.
2. And only now, enable AOF the right way, on the live instance:

   redis-cli CONFIG SET appendonly yes
   # wait for: INFO persistence → aof_rewrite_in_progress:0
   # verify:   appendonlydir/ base file is megabytes, not ~100 bytes

and then persist appendonly yes into redis.conf. Plus, finally, an immediate scp of the recovered dump to another machine.

290,834 keys - eight more than the older candidate, because the winning file was written a couple of minutes later in the timeline. Full recovery.

One detail I still enjoy: the winning file wasn't the old dump.rdb at all. It was an AOF base file written during the incident - the populated base from that brief window when everything still existed. In Redis 7 the AOF base is itself RDB-format, which is why we could rename it to dump.rdb and boot straight from it.

Lessons

1. Enable AOF on the running instance first. CONFIG SET appendonly yes, wait for the rewrite to finish, verify the AOF base has real size - then edit the config file. Config-file-then-restart is the data-loss path, because Redis with AOF enabled ignores your RDB at boot.
2. "It worked in test" only counts if test rehearsed the same sequence. My test-environment run succeeded because I happened to run CONFIG SET first there. Same change, different order of operations, opposite outcome. Rehearse the runbook, not the end state.
3. Copy the data files before touching persistence settings. A 30-second cp dump.rdb dump.rdb.$(date +%F) (and ideally an scp off-box) would have turned a five-hour incident into a five-minute one.
4. Verify between steps of a rolling change. I rebooted three nodes back to back and only checked DBSIZE at the end. One check after the first node would have contained the blast radius.
5. Replication is not backup. The replicas didn't save the data - they synchronized its destruction.
6. If you do lose a file: stop writes immediately. Deleted ≠ gone. rename()-replaced files leave their blocks in free space, and dd + a signature scan can get them back - but only until something reuses those blocks.
7. Loose RDB-only persistence (save 3600 1, appendonly no) is its own slow-motion incident. Ours had been quietly able to roll back up to an hour of writes on every restart - that's what made a feature flag look haunted in the first place, which is the only reason we went looking.

The irony of the whole story: the change that destroyed the database was the one meant to make it durable. The fix was correct; the order was fatal. In operations, sequence is part of the change.

Have you ever had a "the fix caused the outage" incident? I'd love to hear about it in the comments.