DEV Community

Rowan Feld
Rowan Feld

Posted on

Queues: at-least-once means never exactly-once

The most useful sentence I've read about queues: at-least-once delivery is easy, exactly-once processing is a property of your handler, not the broker.

Which means every consumer needs to survive a duplicate. The ways I do that, in order of preference:

  1. Idempotent writes. UPDATE ... WHERE version = expected instead of blind writes.
  2. A dedupe key with a unique index. Cheapest and most explicit.
  3. A processed table with a TTL, for handlers with side effects.

The failure mode to design for isn't the duplicate, it's the poison message: one bad event that blocks the partition because the handler retries forever. Give every consumer a max attempt count and a dead-letter queue you actually read.

One more: make the retry delay grow. A flat 1-second retry turns a small outage into a stampede.

What's your dead-letter policy?

Top comments (0)