Redis Persistence Horror: I Thought AOF Saved Me — Until It Didn’t

At 2 AM, I was jolted awake by an alert message: “All user sessions invalidated; concurrent re-logins blasted the database connection pool.” I checked and saw that the Redis container had just been killed by the OOM Killer. After restarting, I expected AOF to fully recover the session data, but out of over three million session keys, fewer than 100,000 came back. That was the moment I realized my understanding of Redis persistence was just “I thought.”

Problem Breakdown

We use Redis as a “memory store” — centralizing user sessions, device fingerprints, and temporary authorization codes. The data volume isn’t huge, but losing it triggers a flood of user re-logins that instantly overwhelm the backend. To ensure availability, we had long ago enabled appendonly yes and set appendfsync everysec, thinking this was the most robust official recommendation. So why did we still lose so much data after a container restart?

The root cause lay in two blind spots: First, when the Docker container was killed by OOM, the Redis process never had a chance to execute SHUTDOWN. The last batch of AOF buffer data was lost before it could be fsynced. Second, although we had AOF enabled, we had never done any automated verification — we just manually checked with redis-cli in the dev environment, which can’t simulate extreme scenarios like power loss or process kills. Our regular unit tests completely bypassed the persistence layer; Redis was always mocked, and our production persistence config relied on “faith.” If we didn’t nail this down with automated testing, it would wake me up again in the middle of the night.

Solution Design

The goal was clear: we needed a repeatable, unattended way to verify Redis’s data durability and consistency after various abnormal exits. For the tech stack, I chose Pytest + Docker SDK (docker-py) for three reasons:

1. Why not docker-compose + manual scripts? Manual scripts can’t be integrated into CI, and docker-compose’s stop/kill isn’t granular enough to simulate differences between OOM, kill -9, or power loss.
2. Why not Kubernetes + Chaos Mesh? Our team still deploys with docker-compose. Adopting K8s would be overkill and introduce a bunch of new concepts just to test Redis.
3. Why Pytest? Fixtures make container lifecycle management a breeze. Parametrization easily covers RDB, AOF, and mixed persistence modes. Coupled with --count to repeat tests, it can amplify probabilistic data loss into a reproducible failure.

The architecture is simple: each test case starts a fresh Redis container, writes keys in a fixed pattern, simulates an abnormal exit (docker kill or pkill -9), then starts a new container using the same data volume to check key survival rate and content consistency.

Core Implementation

这段代码解决容器生命周期管理和异常终止模拟

We need a Pytest fixture to provide an isolated Redis container for each test and ensure cleanup no matter what. Using docker-py to talk directly to the Docker daemon is far more reliable than shelling out with subprocess, and it lets us capture exit codes precisely.

import docker
import pytest
import time
import redis

@pytest.fixture(scope="function")
def redis_container(request):
    """
    为每个测试用例创建全新 Redis 容器，挂载临时数据卷
    支持通过 pytest.mark.parametrize 传入持久化配置
    """
    client = docker.from_env()
    # 允许通过标记参数传入自定义 redis.conf 内容
    config_marker = request.node.get_closest_marker("redis_config")
    redis_conf = config_marker.args[0] if config_marker else ""

    container = client.containers.run(
        "redis:7-alpine",
        command=f"redis-server /usr/local/etc/redis/redis.conf" if redis_conf else "redis-server",
        volumes={
            # 把字符串配置写入容器内，避免外部文件依赖
            "/tmp/redis_conf": {"bind": "/usr/local/etc/redis/redis.conf", "mode": "ro"}
        } if redis_conf else {},
        ports={"6379/tcp": None},   # 随机宿主机端口
        detach=True,
        remove=True,                # 停止后自动删除容器，保持环境干净
        tmpfs={"/data": ""},        # 数据写内存，实际场景应改成持久卷
    )
    # 等待 Redis 就绪
    time.sleep(1)
    yield container
    # teardown：无论如何都要移除容器，避免端口冲突
    try:
        container.remove(force=True)
    except Exception:
        pass

这段代码解决“异常退出后数据恢复”的测试逻辑

The test below writes 1000 string keys, uses docker kill -s KILL to simulate a forceful kill, then restarts a new container with the same data volume and counts recovered keys. This is the heart of the automated verification.

import pytest
import redis
import time

@pytest.mark.redis_config("""
appendonly yes
appendfsync always
""")
def test_aof_durability_on_kill(redis_container):
    """
    验证开启 AOF 且 fsync=always 时，docker kill -9 后的数据完整度
    """
    # 获取容器的临时数据卷路径（这里仅作测试，生产场景请用命名卷）
    inspection = redis_container.attrs
    mounts = inspection["Mounts"]
    data_volume = next(m["Source"] for m in mounts if m["Destination"] == "/data")

Comments

[Marcus Kim]

The most important detail is that the test fixture mounts /data on tmpfs while the production failure is specifically about durable recovery; that makes the example useful for kill semantics, but not yet a faithful persistence test. The choice to use Docker SDK and repeat parametrized Pytest runs is solid, especially for distinguishing docker kill from cleaner shutdowns and turning probabilistic loss into a measurable failure. I'd also treat session invalidation as a capacity-planning problem: durability reduces the blast radius, but reconnect backoff and bounded reauthentication are still needed when Redis or its volume genuinely.