After 3 Years of Redis, I Finally Learned How to Test Cache Consistency

At 2:30 AM, an alarm call yanked me out of deep sleep. The user group had erupted — order statuses didn’t match actual payments. Some users who paid were shown as “pending payment,” while others who hadn’t paid were marked “shipped.” Still half-asleep, I checked the dashboards: the database connection pool was fine, CPU wasn’t spiking, Redis memory looked normal. Only after grepping through the production logs did I find the culprit: during a write operation, after the database was updated but before the cache was deleted, a read request slipped in, read the stale data, and wrote it back to the cache. The window? Less than 20 milliseconds.

That was the final straw. I’d been using Redis for years, yet cache consistency was always ensured through developer self-testing and code reviews — I’d never systematically automated verification. I didn’t want to experience another midnight bug fix. So the next day, I started building a repeatable, automated cache consistency validation suite. If you’ve ever been burned by the gap between cache and database, this approach might help you sleep a little easier.

Why Cache Consistency Is So Hard to Validate

You’ve probably seen this scenario: an e-commerce order service caches order details in Redis and persists them in PostgreSQL. Updates follow the classic Cache-Aside pattern — update the database first, then delete the cache. On a cache miss, the read path loads from the database and writes it back to the cache.

The things that can go wrong are more numerous than you think:

• Concurrent reads and writes: a writer just updated the database but hasn’t deleted the cache yet; meanwhile a reader loads the old data and caches it back.
• Delete failure: the cache deletion fails due to a network hiccup, and without a retry mechanism, the stale data sits in the cache forever.
• Double-write order: some developers take the shortcut of “delete cache first, then update DB,” which lets a read reload the old data before the DB update completes.
• Cache structure changes: the serialization format changes, but an old object still in the cache causes deserialization failures.

All these share a common trait: extremely narrow time windows and complex concurrency timing. Manual testing can rarely reproduce them reliably. You might hit a bug during a load test, but it vanishes on the next run. You need a clean environment where you can precisely control concurrency, repeat execution, and automatically assert correctness.

Design: Pytest + Docker, Clean and Repeatable

My goal: after cloning the repo, anyone can run all consistency scenarios with a single command, and the results are stable and reproducible every time.

The stack:

• Pytest: the de facto standard for Python testing. Fixtures manage resources, conftest shares logic, and parametrization generates multiple scenarios easily.
• Docker Compose: provides a clean Redis 7 + PostgreSQL 15 environment matching production. I avoided testcontainers-python because it pulls a Java container in the background — too heavy for quick verification.
• redis-py + psycopg2: native clients, no ORM, to reduce abstraction overhead and test low-level behavior directly.
• Multi-threading concurrency: Python’s built-in threading allows precise control over read/write timing, so I can make a thread sleep a few milliseconds at a particular step to reliably create race windows.

Why not other approaches?

• Manual testing on real environments: dirty data, leftover caches, manual cleanup — high probability of error, not reproducible.
• Unit tests mocking Redis/DB: mocks bypass the network, so timeouts, serialization, and connection-pool issues go untested. It’s self-deception.
• Integration tests run once: random network and timing can make a test pass today and fail tomorrow, turning it into a flaky test in CI.

I wanted tests that expose problems with clockwork precision in CI, or give you solid confidence when everything is correct.

Core Implementation: Three Scenarios That Stress Cache Consistency

Now I’ll show you the test setup and the code for three key scenarios. Each one tackles a specific problem.

Environment Setup: conftest.py Brings Up Infrastructure with Docker

This code ensures a completely clean Redis + PostgreSQL before each test session, then tears it down, leaving no trace. We use docker-compose to manage the lifecycle and wait for services to be ready.

# conftest.py
import subprocess
import time
import pytest
import redis
import psycopg2

COMPOSE_FILE = "docker-compose.yml"

def wait_for_redis(r, retries=30, delay=0.5):
    for _ in range(retries):
        try:
            if r.ping():
                return
        except Exception:
            time.sleep(delay)
    raise RuntimeError("Redis 未能在预期时间内就绪")

def wait_for_pg(conn_params, retries=30, delay=0.5):
    for _ in range(retries):
        try:
            conn = psycopg2.connect(**conn_params)
            conn.close()
            return
        except Exception:
            time.sleep(delay)
    raise RuntimeError("PostgreSQL 未能在预期时间内就绪")

@pytest.fixture(scope="session")
def docker_services():
    # 启动 compose
    subprocess.run(["docker-compose", "-f", COMPOSE_FILE, "down", "-v"], check=True)
    subprocess.run(["docker-compose", "-f", COMPOSE_FILE, "up", "-d"], check=True)
    # 等待服务健康
    r = redis.Redis(host="localhost", port=6379, decode_responses=True)
    wait_for_redis(r)
    pg_conn = {
        "host": "localhost", "port": 5432, "dbname": "testdb",
        "user": "test", "password": "test"
    }
    wait_for_pg(pg_conn)
    yield {"redis": r, "pg": pg_conn}
    # 结束后彻底清理
    subprocess.run(["docker-compose", "-f", COMPOSE_FILE, "down", "-v"], check=True)

@p