I Spent 4 Hours Debugging a Redis Race Condition. Here's What I Learned.

I was jolted awake at 2 a.m. by an alert call. Users were complaining that their AI chat history was losing chunks midway through conversations. Half-asleep, I combed through the logs—no errors, no crashes. Only that some messages were simply missing from Redis. It wasn’t until I opened the monitoring dashboard and saw the concurrency spikes on that endpoint that I realized: the way we stored session history in Redis was fundamentally broken under concurrent writes.

Breaking Down the Problem

Our setup was typical: each user’s chat memory was stored in Redis as a single String key, serializing the entire conversation into a JSON array. Every time a conversation occurred, the backend would:

1. GET the whole JSON
2. Deserialize, append the new message
3. Serialize, and SET it back

This worked blissfully when QPS was in the single digits. However, with recent user growth and a frontend auto-retry mechanism—meaning the same conversation request could fire twice in rapid succession—the classic read-modify-write race condition appeared. Request A and request B would both read the same old array, each append their own message, and both write back. Inevitably, one request’s message was overwritten and lost.

That’s the root cause of the “eventual inconsistency.” It’s not that Redis is unreliable; we simply expected atomicity beyond what it provides. A plain SET/GET is atomic, but the compound “read‑modify‑write” is not. Adding a distributed lock would certainly fix it, but it would severely degrade throughput. On top of that, locks introduce deadlocks, timeouts, and a host of other headaches. We needed a lock-free atomic operation.

Designing a Solution

Several alternatives were on the table:

• Redis List + RPUSH – Each message would be pushed atomically into a list. No overwrites, naturally append-only. But reading history would require LRANGE to pull the entire list, dragging along the full conversation on every request—not great for long histories.
• Redis Stream – Excellent for message queues, with the ability to persist after consumption. However, introducing consumer groups would complicate history reads and incur a high integration cost with our existing codebase.
• Lua scripting – Execute an atomic script directly on the Redis server that appends an element to the JSON array and writes it back. A single network round trip, lock‑free, and completely atomic. The only cost is moving the JSON logic into Lua; luckily Redis 7 includes the cjson library.

We ultimately chose the Lua script approach. It required the fewest changes: we’d still store the entire session as a String, keep the business‑layer interface unchanged, and simply replace the old “read‑modify‑write” with a single EVAL command. Plus, it allowed a gradual migration with full backward compatibility for existing data.

Core Implementation

Step 1: Reproduce the concurrency bug with Pytest

Before fixing anything, I wrote a test that would fail, so I could reliably reproduce the problem. Using pytest-asyncio, it spins up 20 concurrent coroutines that all append messages for the same user, then checks whether every message survives.

import asyncio
import json
import pytest
import redis.asyncio as aioredis

@pytest.mark.asyncio
async def test_concurrent_append_causes_missing_messages():
    """复现 read-modify-write 导致的消息丢失"""
    r = aioredis.from_url("redis://localhost:6379", decode_responses=True)
    user_key = "user:123:history"
    # 初始化空历史
    await r.set(user_key, json.dumps([]))

    async def bad_append(msg: str):
        # 模拟有问题的做法：GET -> 修改 -> SET
        raw = await r.get(user_key)
        history = json.loads(raw) if raw else []
        history.append(msg)
        await r.set(user_key, json.dumps(history))

    tasks = [bad_append(f"msg-{i}") for i in range(20)]
    await asyncio.gather(*tasks)

    # 验证
    final_raw = await r.get(user_key)
    final_history = json.loads(final_raw)
    # 这里很可能小于 20，因为发生了覆盖
    assert len(final_history) == 20, f"丢失消息！期望 20 条，实际 {len(final_history)}"

This test will fail most of the time, the assertion loudly telling you exactly how many messages were lost. Only by making the bug reproducible can we confidently talk about “eventual consistency.”

Step 2: Write the atomic append Lua script

The Lua script does three things: initialize an empty array if the key doesn’t exist or is empty; parse the JSON with cjson.decode / cjson.encode and append the new element; and optionally return the new length for monitoring.

-- lua/append_history.lua
local key = KEYS[1]
local new_message = ARGV[1]

local raw = redis.call('GET', key)
local history = {}
if raw and raw ~= '' then
    history = cjson.decode(raw)
end
-- 将新消息追加到数组
table.insert(history, new_message)
local new_raw = cjson.encode(history)
redis.call('SET', key, new_raw)
return #history

Step 3: Encapsulate the atomic operation in Python

Use redis-py’s Script object to register the script and call it via evalsha, so we don’t have to resend the script body every time.

import redis.asyncio as aioredis

LUA_APPEND = """
local key = KEYS[1]
local new_message = ARGV[1]
local raw = redis.call('GET', key)
local history = {}
if raw and raw ~= '' then
    history = cjson.decode(raw)
end
table.insert(history, new_message)
local new_raw = cjson.encode(history)
redis.call('SET', key, new_raw)
return #history
"""

class HistoryStore:
    def __init__(self, r: