DEV Community

BAOFUFAN
BAOFUFAN

Posted on

Redis Cache Inconsistency: My 48-Hour Debugging Nightmare & The Automated Test That Saved Me

#python #programming

At 1 a.m., an alert call yanked me out of sleep: users were seeing their old nicknames on the page, but after a few refreshes it would fix itself. First thought: the cache wasn’t invalidated. I logged into Redis — the key was still there, but the database already had the updated value. Classic cache-database inconsistency. I assumed it was a small bug. Instead, I spent the next two days hunting this ghost. It was intermittent and only appeared under concurrent updates. Frustrated with debugging by luck, I spent half a weekend building an automated consistency test suite with pytest + Docker. Now it runs before every release and I haven’t been bitten by stale cache data since.

Let me break down the entire approach. If you’ve dealt with similar headaches, you might be able to adopt it directly.

The Problem

The scenario isn’t complicated. A user updates an order status. The code writes to MySQL, then deletes the Redis cache entry. On the next read, the cache is rebuilt. Simple logic, but under concurrency something ugly happens: process A updates the DB and deletes the cache; process B, after A deletes the cache but before A’s transaction commits, reads the old value from the DB and populates it back into the cache. The result? The DB has the new value, the cache holds the old one — only to be fixed when the cache expires. For low-frequency data, this dirty window can stretch for hours.

Standard local tests totally miss this: single-threaded serial execution is always correct. Load-testing tools can trigger concurrency, but they struggle to assert “is the cache consistent with the DB right now?”. You usually only see the final state, and the fleeting intermediate inconsistency slips through. Even worse, these bugs are often introduced after tweaking a SQL query or adding some caching logic — by the time you notice in production, it’s already too late.

The Plan

My core requirement: in a local/CI environment, precisely create cache–database inconsistency with controlled concurrent operations, and assert immediately.

Why I chose what I chose:

  • pytest? Flexible enough to write test cases, fixture management for resources is a breeze, and you can parameterize concurrency combos. Way better than shell scripts and duct-tape tools.
  • Docker? Every test run needs pristine Redis + MySQL instances. Docker Compose spins them up with a single command, locks the versions, and avoids the dreaded “works on my machine” dance.
  • Why not Testcontainers? Not that it’s bad; I just needed finer-grained control over container lifecycles. For example, I wanted to restart Redis mid-test or simulate network glitches. Invoking the Docker CLI directly from a pytest fixture gave me that control.
  • Why not rely on production monitoring? Monitoring only tells you when something broke. I wanted to stop defects at the CI stage.

The overall idea: define the dependent services in docker-compose, use pytest fixtures to start/stop containers, create tables, and seed data. Each test case simulates a specific concurrency model (write-then-delete, delayed double-delete, binlog-subscription refresh, etc.). Concurrency is orchestrated with threading or asyncio. Finally, read from both Redis and MySQL and assert they match.

Core Implementation

The project layout looks like this:

.
├── docker-compose.yml
├── conftest.py
├── test_cache_consistency.py
└── requirements.txt
Enter fullscreen mode Exit fullscreen mode

1. One-Command Environment with Docker Compose

This configuration solves the pain of manually setting up DB and Redis for every test run.

# docker-compose.yml
version: '3.8'
services:
  mysql:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: testpwd
      MYSQL_DATABASE: testdb
    ports:
      - "3306:3306"
    command: --default-authentication-plugin=mysql_native_password
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
      interval: 2s
      retries: 10

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 2s
      retries: 10
Enter fullscreen mode Exit fullscreen mode

Adding healthcheck was the first trap — more on that later.

2. pytest Fixtures: Initialize Connections, Create Tables, Tear Down Data 

# conftest.py
import pytest
import subprocess
import time
import redis
import pymysql

@pytest.fixture(scope="session")
def docker_services():
    # 启动 docker-compose
    subprocess.run(
        ["docker-compose", "-f", "docker-compose.yml", "down", "-v"],
        check=True, stdout=subprocess.DEVNULL
    )
    subprocess.run(
        ["docker-compose", "-f", "docker-compose.yml", "up", "-d"],
        check=True
    )
    # 等待健康检查全部通过，避免服务未就绪就开始测试
    time.sleep(8)
    yield
    subprocess.run(
        ["docker-compose", "-f", "docker-compose.yml", "down", "-v"],
        check=True, stdout=subprocess.DEVNULL
    )

@pytest.fixture(scope="function")
def db_conn(docker_services):
    conn = pymysql.connect(
        host="127.0.0.1",
        port=3306,
        user="root",
        password="testpwd",
        database="testdb",
        autocommit=True  # 后面踩坑会解释
    )
    with conn.cursor() as cur:
        cur.execute("""
            CREATE TABLE IF NOT EXISTS users (
                id INT PRIMARY KEY,
                name VARCHAR(50)
            )
    # ... (the fixture continues to return the connection and clean up after the test)
Enter fullscreen mode Exit fullscreen mode

With this foundation in place, the actual concurrency tests become surprisingly straightforward — but that’s a story for the next section. The key takeaway: if you can reproduce a cache bug deterministically in CI, you’ve already won half the battle.

Top comments (0)