Refactoring a Distributed Cache for Strong Reads When Eventual Consistency Fails

we hit a subtle bug in our order‑service after a recent traffic surge: occasional duplicate invoices appeared in the admin UI. the root cause? a cache layer that was designed for eventual consistency but was being used for read‑after‑write scenarios.

The Problem

our stack uses a Node.js API gateway, a PostgreSQL write store, and a Redis cluster as a read‑through cache. writes go through a transaction, then an async job publishes an order.updated event to a Kafka topic. the consumer updates the Redis entry. under low load the latency between the DB commit and the cache refresh is negligible, but under load the lag grows to several seconds. the UI, which reads directly from Redis for latency reasons, sometimes shows stale data, leading to duplicate processing downstream.

Why Not Just Switch to Strong Consistency?

we could have forced every read to hit the DB, but that defeats the purpose of the cache. another option is to use Redis' wait command to block until replicas acknowledge the write, but that only helps with replication lag, not with our asynchronous event pipeline. the real challenge is to preserve the low‑latency read path while guaranteeing that any read immediately after a write sees the latest state.

A Pragmatic Refactor

instead of treating the cache as a fire‑and‑forget layer, we turned it into a write‑through store for critical entities. the API now performs the DB write and updates Redis in the same request, using a transaction‑like pattern. we still keep the async Kafka flow for eventual propagation to other services, but the cache is now authoritative for the write‑through path.

// write‑through helper – keeps DB and Redis in sync
async function updateOrder(id: string, payload: Partial<Order>) {
  // 1️⃣ start DB transaction
  await db.transaction(async trx => {
    await trx('orders').where({ id }).update(payload);
    // 2️⃣ update Redis inside the same logical unit
    const cacheKey = `order:${id}`;
    const fresh = { ...(await trx('orders').where({ id }).first()), ...payload };
    await redis.set(cacheKey, JSON.stringify(fresh));
  });
  // 3️⃣ fire async event for downstream consumers
  await kafka.produce('order.updated', { id, ...payload });
}

Trade‑offs & Edge Cases

• Latency vs. Consistency – the write‑through path adds a few milliseconds to the request, but the UI now reads correct data.
• Cache Invalidation – if another service updates the order directly in the DB (bypassing the API), the cache can become stale. we mitigated this by adding a TTL of 30 seconds and a background poller that reconciles mismatches.
• Failure Scenarios – if Redis is unavailable, the transaction still commits to the DB, but we must decide whether to fail the request or proceed without caching. in our case we fail fast, returning a 503, because a stale cache is worse than a brief outage.
• Memory Pressure – keeping a large write‑through cache can increase memory usage. we tuned Redis maxmemory-policy to volatile-lru and scoped the cache to only high‑traffic entities.

The Takeaway

by moving from an eventual‑only cache to a write‑through model we regained strong read guarantees without sacrificing overall performance. the pattern does introduce extra complexity around failure handling and cache hygiene, but those concerns are manageable with clear contracts.

how do you handle read‑after‑write consistency in your high‑traffic services, and what patterns have you found most sustainable?