The most useful sentence I've read about queues: at-least-once delivery is easy, exactly-once processing is a property of your handler, not the broker.
Which means every consumer needs to survive a duplicate. The ways I do that, in order of preference:
-
Idempotent writes.
UPDATE ... WHERE version = expectedinstead of blind writes. - A dedupe key with a unique index. Cheapest and most explicit.
- A processed table with a TTL, for handlers with side effects.
The failure mode to design for isn't the duplicate, it's the poison message: one bad event that blocks the partition because the handler retries forever. Give every consumer a max attempt count and a dead-letter queue you actually read.
One more: make the retry delay grow. A flat 1-second retry turns a small outage into a stampede.
What's your dead-letter policy?
Top comments (0)