DEV Community

kholnnaw
kholnnaw

Posted on

avoiding race conditions in async cascade deletes for event‑sourced services

#typescript #microservices #architecture #testing

you've got an event‑sourced aggregate that emits a UserDeleted event, and several downstream services react by cleaning up related rows in their own stores. the cleanup runs in background workers, so a second request that references the same user can slip through the cracks and hit a foreign‑key violation.

the race condition

in a typical setup each service subscribes to the UserDeleted topic and issues a cascade delete against its own tables. because the subscription processing is asynchronous, a request that arrives after the UserDeleted event is published but before the worker finishes will see stale data. the DB then throws an error, and the client gets a 500 instead of a graceful "not found".

a deterministic approach

the trick is to make the delete operation idempotent and observable before the request reaches the service. one pattern is to store a tombstone flag in a shared cache (e.g. Redis) as part of the same transaction that publishes the event. workers check the flag before proceeding, and request handlers read it to short‑circuit.

// publish + tombstone atomically
await redis.multi()
  .set(`tombstone:user:${userId}`, '1', 'EX', 300) // 5 min TTL
  .publish('user.deleted', JSON.stringify({ userId }))
  .exec();
Enter fullscreen mode Exit fullscreen mode

workers then do:

const flag = await redis.get(`tombstone:user:${userId}`);
if (!flag) return; // nothing to clean up
await db.transaction(async trx => {
  await trx('orders').where({ userId }).delete();
  await trx('profiles').where({ userId }).delete();
});
await redis.del(`tombstone:user:${userId}`);
Enter fullscreen mode Exit fullscreen mode

trade‑offs

  • latency vs consistency – adding the cache lookup adds a few milliseconds to every request, but it eliminates sporadic DB errors.
  • cache eviction – the TTL must be long enough to cover the slowest worker, otherwise a stale request could still hit the DB.
  • failure handling – if the worker crashes after deleting rows but before clearing the tombstone, subsequent requests will be rejected even though the data is gone. a periodic reconciler that scans for orphaned tombstones can mitigate this.
  • testing – simulate the race by issuing a delete and a read in parallel within integration tests; assert that the read returns a 404 rather than a DB exception.

when to apply

if your service experiences occasional 500 spikes after bulk deletions, or if you already use a message broker for domain events, the tombstone pattern is a low‑overhead way to enforce eventual consistency without locking the DB.

anyone using a similar approach with different stores (e.g. DynamoDB streams) or who prefers optimistic concurrency controls instead? how do you keep the delete path safe without sacrificing throughput?

Top comments (0)