Distributed Idempotency Key Collisions: Clock Skew Mutex Leaks, Race Conditions, and Double-Execution Vulnerabilities

Wrapping API handlers with naive Redis key locks (SET key val NX EX 30) to enforce idempotency creates an illusion of safety. Under concurrent traffic bursts, worker stalls, network partitions, and non-atomic status releases can cause duplicate executions: charging credit cards twice or creating duplicate database records. Here is an architectural deep dive into diagnosing distributed idempotency lock breaches and engineering atomic state transition engines.

In mission-critical B2B SaaS platforms, especially those handling payment processing, billing generation, or external state provisioning, enforcing strict exactly-once execution via idempotency keys is a core architectural requirement. However, naive Redis lock implementations often contain subtle, high-concurrency race conditions that silently break under network partitions, worker thread stalls, and distributed clock skew.

1. The Fallacy of Naive Redis Mutex Locks

A standard anti-pattern in Node.js and Go microservices is using a simple Redis string set operation to acquire an execution lock prior to invoking business logic:

// NAÏVE ANTI-PATTERN: Vulnerable Idempotency Lock
async function handlePaymentNaive(req: Request, res: Response) {
  const idempotencyKey = req.headers['idempotency-key'];
  const lockKey = `idempotency:${idempotencyKey}`;

  // Attempt to acquire execution lock for 10 seconds
  const acquired = await redis.set(lockKey, 'LOCKED', 'NX', 'EX', 10);

  if (!acquired) {
    return res.status(409).json({ error: 'Concurrent request in progress' });
  }

  // Execute heavy business logic (e.g., Stripe API call + DB mutation)
  const result = await processPayment(req.body);

  // Cache response for future retries
  await redis.set(`response:${idempotencyKey}`, JSON.stringify(result), 'EX', 86400);

  return res.json(result);
}

Three Silent Failure Vectors

• The Worker Stall / GC Pause Breach: If a Garbage Collection (GC) pause, unhandled event loop block, or downstream Stripe API latency keeps the worker busy for longer than 10 seconds, the Redis TTL expires automatically. Redis deletes the lock while the first request is still executing. A concurrent retry attempt instantly acquires the lock and executes the payment a second time.
• The Non-Atomic Release Bug: If a worker finishes execution and calls redis.del(lockKey) or overwrites the key without verifying ownership, it might delete or clobber a lock that was newly acquired by a second worker after the first worker's lock expired, unlocking the door for a third incoming request.
• Clock Skew & Failover Split-Brain: With a single Redis primary, TTLs run on the Redis server's own clock, so application-server skew does not shorten them. Problems appear in distributed setups: multi-master schemes like Redlock depend on bounded clock drift across independent Redis nodes, and Redis replication is asynchronous, so a lock written to a primary can vanish during failover. In both cases, two workers can believe they hold the same lock simultaneously.

2. Architectural Pattern: Atomic Two-Phase State Engine

To close these race conditions, replace primitive boolean lock flags with a 3-state state engine backed by atomic Lua scripts inside Redis. Every state change, both acquiring the key and finalizing it, happens in a single atomic script that checks who owns the key.

State Machine States

• PROCESSING: A request is actively executing. Contains a unique, randomly generated worker UUID (the owner) and an acquisition timestamp.
• COMPLETED: The request finished successfully. Holds the serialized HTTP response status and body payload for replay.
• RELEASED (key deleted): The operation failed with a retryable error. The owning worker deletes the key so the client can retry cleanly.

Phase 1: Atomic Acquisition via Redis Lua Scripting

-- ACQUIRE_LUA_SCRIPT: Atomic Idempotency Lock Acquisition
local lockKey = KEYS[1]
local workerUuid = ARGV[1]
local lockTtlSeconds = tonumber(ARGV[2])

local existingState = redis.call('GET', lockKey)

if not existingState then
    -- Key does not exist: create PROCESSING state with owner UUID
    local payload = cjson.encode({
        status = "PROCESSING",
        owner = workerUuid,
        created_at = ARGV[3]
    })
    redis.call('SET', lockKey, payload, 'EX', lockTtlSeconds)
    return { "ACQUIRED", payload }
else
    -- Key exists: return current state payload to application
    return { "LOCKED", existingState }
end

Phase 2: Ownership-Checked Finalization

Completing or releasing the key must be just as atomic. This script only lets the worker that still owns the PROCESSING lock write the final state. A stale worker whose lock already expired gets 0 back and leaves the key untouched.

-- FINALIZE_LUA_SCRIPT: Only the current owner may complete or release
local lockKey = KEYS[1]
local workerUuid = ARGV[1]
local action = ARGV[2] -- "COMPLETE" or "RELEASE"

local current = redis.call('GET', lockKey)
if not current then return 0 end

local state = cjson.decode(current)
if state.status ~= 'PROCESSING' or state.owner ~= workerUuid then
    return 0 -- Lock expired and was taken over: do not touch it
end

if action == 'COMPLETE' then
    redis.call('SET', lockKey, ARGV[3], 'EX', tonumber(ARGV[4]))
else
    redis.call('DEL', lockKey)
end
return 1

3. Production-Ready TypeScript Middleware Implementation

The application layer acquires the key before running the handler, replays cached responses for completed keys, and finalizes through the ownership-checked script, never with a blind SET or DEL.

// PRODUCTION-GRADE IDEMPOTENCY ENGINE (TypeScript)
import { Request, Response, NextFunction } from 'express';
import { v4 as uuidv4 } from 'uuid';
import { Redis } from 'ioredis';

// ACQUIRE_LUA_SCRIPT and FINALIZE_LUA_SCRIPT hold the two Lua scripts above

export function createIdempotencyMiddleware(redis: Redis) {
  return async (req: Request, res: Response, next: NextFunction) => {
    const idempotencyKey = req.headers['idempotency-key'] as string;
    if (!idempotencyKey) return next();

    const tenantId = req.user?.tenantId || 'global';
    const lockKey = `idempotency:${tenantId}:${idempotencyKey}`;
    const workerUuid = uuidv4();
    const lockTtl = 30; // 30 second execution ceiling

    // 1. Execute Atomic Lua Acquisition Script
    const [action, rawData] = await redis.eval(
      ACQUIRE_LUA_SCRIPT,
      1,
      lockKey,
      workerUuid,
      lockTtl,
      Date.now().toString()
    ) as [string, string];

    if (action === 'LOCKED') {
      const state = JSON.parse(rawData);

      if (state.status === 'PROCESSING') {
        // Request is actively being processed by another worker
        return res.status(409).json({
          error: 'CONCURRENT_REQUEST',
          message: 'An identical request with this Idempotency-Key is currently executing.'
        });
      }

      if (state.status === 'COMPLETED') {
        // Replay identical response directly from cache
        res.setHeader('X-Cache-Lookup', 'HIT-IDEMPOTENT');
        return res.status(state.statusCode).json(state.body);
      }
    }

    // 2. Intercept the response and finalize ONLY if this worker still owns the lock
    const originalJson = res.json.bind(res);
    res.json = (body: any) => {
      const succeeded = res.statusCode >= 200 && res.statusCode < 300;
      const completedPayload = JSON.stringify({
        status: 'COMPLETED',
        statusCode: res.statusCode,
        body
      });

      redis
        .eval(
          FINALIZE_LUA_SCRIPT,
          1,
          lockKey,
          workerUuid,
          succeeded ? 'COMPLETE' : 'RELEASE',
          completedPayload,
          86400 // 24-hour replay retention
        )
        .then((owned) => {
          if (owned === 0) {
            // Our lock expired mid-request and another worker took over.
            // Redis state is protected; the DB constraint below must catch any duplicate side effect.
            console.warn('[IDEMPOTENCY] Lock lost before completion:', lockKey);
          }
        })
        .catch((err) => console.error('[IDEMPOTENCY] Finalize failed:', err));

      return originalJson(body);
    };

    next();
  };
}

This sketch hooks res.json. If your handlers also respond with res.send or res.end, wrap those the same way, or the key will sit in PROCESSING until its TTL expires.

4. Secondary Guardrail: Database-Level Unique Constraints

The ownership check stops a stale worker from corrupting Redis state, but it cannot undo work that worker already did. If a lock expires mid-request, two workers can still both reach the payment step. Redis should act as your fast, first-line filter. For financial mutations, never rely on Redis alone. Always back distributed application locks with strict relational database constraints at the storage engine level.

-- Database-Level Fail-Safe: Unique Idempotency Key Index
CREATE TABLE payment_transactions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL,
    idempotency_key VARCHAR(255) NOT NULL,
    amount_cents BIGINT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT CLOCK_TIMESTAMP(),

    -- Absolute DB Guardrail against double execution
    CONSTRAINT unique_tenant_idempotency UNIQUE (tenant_id, idempotency_key)
);

Insert this row before calling the payment provider, so the reservation happens first. If a Redis lock expires under an extreme 30-second stall, the database will reject the second worker's duplicate insert with a 23505 unique_violation error before any money moves. For the external call itself, pass the same idempotency key to the provider (Stripe, for example, accepts an Idempotency-Key header), so even a duplicate API request is charged only once.

5. Executive Summary & Systems Rules

• Never Use Plain SET NX EX for Idempotency: Always store a structured state payload containing a unique worker UUID to verify lock ownership.
• Never Release Un-Owned Locks: Use Lua scripts to ensure a worker completes or deletes a lock key ONLY if its worker UUID matches the key payload.
• Enforce Storage-Layer Unique Constraints: Always back Redis idempotency checks with unique compound indexes (e.g., (tenant_id, idempotency_key)) inside PostgreSQL or MySQL, and forward the key to external payment providers.

---

Worried about double charges or duplicate records in your payment flows? I help SaaS teams design idempotent APIs, safe distributed locking, and database guardrails that hold up under retries and failures. Book a consultation →

Originally published on ctousman.com.