How a Redis Config Mistake Deleted 3,000 Records at 3 AM — and the Automated Tests That Saved Me

At 3 a.m., my phone went crazy with vibrations. The alert message was just one line: “MySQL connection pool exhausted, API success rate dropped to 12%.” Hands trembling, I opened the monitoring dashboards — database QPS had skyrocketed from 800 to 20,000, with every request hammering MySQL directly. The culprit? Redis had just been automatically restarted, but its cache was completely empty. Over 3,000 hot product records vanished from memory; the RDB file didn’t load. Our ops team had changed a config the day before, pointing the RDB directory to a non-existent path. After the restart, Redis quietly started with zero data. Our supposedly robust rollback mechanism had gracefully “rolled back to a blank slate.”

Since that night, I made a rule for myself: any change involving persistence or cache recovery must have an automated test pipeline capable of injecting failures as ruthlessly as a hacker, and then telling me bluntly, “Was data lost or not?” That’s why I cobbled together an automated validation approach with Playwright + Python + Docker. Today I’m sharing the full implementation and the hard-earned lessons, hoping to save you some painful debugging nights.

Breaking Down the Problem: Why Your Cache Rollback Tests Are Useless

Many teams test Redis persistence recovery only by: stop Redis → restart Redis → check if keys are still there. That kind of test will never catch production issues, because:

1. The state of persistence files is untrustworthy. RDB files can get corrupted due to a full disk, AOF files can be truncated by bgrewrite, and a wrong config path means the file is never read — these are common failures that manual testing almost never covers.
2. Business logic’s reaction to an empty cache is not verified. Redis recovery fails → cache is empty → does your Go/Python service fall back to MySQL? During loading, can concurrent cache-miss requests overwhelm the database? This requires end-to-end flow testing, not a simple GET of a key.
3. Rollback success is not tied to the frontend experience. When cache is lost, does the user see an error, stale data, or a white screen? What HTTP status code does the API return? Only by exercising the entire chain can we truly validate recovery.

So what we need is not a few unit tests, but an automated solution that can simulate real user operations → inject persistence failures → verify business-layer recovery effects.

Designing the Solution: Why Playwright, Not JMeter or Unit Tests?

We evaluated three paths:

• JMeter/Locust load testing: It can simulate mass requests, but can’t perform user actions like “click button → fill form → submit,” nor verify the values of page elements (e.g., whether the product name was correctly restored).
• Backend unit tests + Mock Redis: They only test logical branches, not real filesystem RDB behavior, process restarts, or I/O load — the kernel-level actions.
• Playwright + Docker + pytest: Playwright controls the browser to simulate full user flows. Pytest’s fixture mechanism lets us arbitrarily stop/destroy Redis containers, delete RDB files, or change configs. Docker guarantees a clean environment for each test. The biggest advantage: fault injection and business validation are closed-loop within the same test case.

The final architecture is simple: docker-compose brings up a minimal business environment (Flask web + Redis + MySQL). Playwright creates data through the browser, pytest executes the Redis “destruction,” and then Playwright reads the page data again to compare against MySQL, determining whether rollback succeeded.

Core Implementation: Building a Fault-Injection Test Framework from Scratch

The following docker-compose.yml is the baseline environment for the entire test suite. Pay attention to the Redis volume configuration — this is the first pitfall: you must explicitly mount the RDB directory, or the file you delete won’t be the one loaded at startup.

# docker-compose.yml
version: '3.8'
services:
  web:
    build: .
    ports:
      - "8000:8000"
    environment:
      REDIS_HOST: redis
      DB_HOST: mysql
    depends_on:
      - redis
      - mysql

  redis:
    image: redis:7-alpine
    # 必须显式挂载 /data，否则RDB文件会写在容器层，重启后丢失路径关系
    volumes:
      - redis_data:/data
    command: redis-server --save 60 1 --loglevel warning

  mysql:
    image: mysql:8
    environment:
      MYSQL_ROOT_PASSWORD: testpass
      MYSQL_DATABASE: shop
    volumes:
      - mysql_data:/var/lib/mysql

volumes:
  redis_data:
  mysql_data:

Next is the overall structure of the pytest test case. This code solves “how to chain user operations, Redis destruction, and recovery verification within a single test function.” We use async functions and Playwright’s async API.

# test_cache_rollback.py
import asyncio
import time
import pytest
import docker
import redis
import pymysql
from playwright.async_api import async_playwright

BASE_URL = "http://localhost:8000"
REDIS_HOST = "localhost"
REDIS_PORT = 6379

@pytest.fixture(scope="module")
def docker_client():
    return docker.from_env()

@pytest.fixture(scope="module")
def redis_conn():
    r = redis.Redis(host=REDIS_HOST, port=REDIS_PORT, decode_responses=True)
    yield r
    r.close()

@pytest.fixture(scope="module")
def mysql_conn():
    conn = pymysql.connect(host='localhost', user='root', password='testpass', database='shop')
    yield conn
    conn.close()

async def test_rollback_after_rdb_loss(docker_client, redis_conn, mysql_conn):
    # 1. 用户通过前端创建商品，缓存写入Redis
    async with async_playwright() as p:
        browser = await p.chromium.launch()