Your Message Consumer Needs a Parking Lot
A message can fail because a dependency is temporarily unavailable. It can also fail because its payload is permanently invalid. Treating both cases as "retry forever" is how a queue becomes a warehouse of unresolved incidents.
Transient failures deserve bounded retries. Permanent failures deserve isolation and visibility. A dead-letter queue can hold messages that need investigation, but a dead-letter queue without ownership is just a second place to lose work.
Incoming queue
|
v
Process message
/ \
success failure
| |
acknowledge retry budget
|
v
dead-letter / review
Make processing idempotent. A worker may complete database work and crash before acknowledging the message. Redelivery is normal, not an exceptional corner case. Record stable message IDs and the processing outcome when duplicate side effects would hurt.
Monitoring should include message age, retry counts, dead-letter growth, and the time until a human actually resolves a poison message. Alerting merely when the queue is non-empty produces noise.
The interesting engineering question is not whether messages can fail. They will. It is whether the system preserves work and makes unresolved failures actionable.
Top comments (0)