DEV Community

Aleksandr Argunov
Aleksandr Argunov

Posted on Originally published at argbe.tech AI-assisted

Same message, four fates: SQS, SNS, Kafka and Redis Pub/Sub when a worker crashes

Every broker delivers when nothing breaks. So compare them on the crash.

SQS = a take-a-number ticket. SNS = a group text. Kafka = a tape recorder. Redis Pub/Sub = live radio.

Animated grid titled Same message. Four fates. SQS, SNS, Kafka and Redis Pub/Sub, drawn as a ticket, group text, tape recorder and radio, act out four failure cases.

The grid

Kafka here means ordinary consumer groups. Share groups get their own section below.

Case SQS SNS Kafka Redis Pub/Sub
Who gets it Competing workers; standard can duplicate Every matching subscription Each subscribed group; one owner per partition Matching subscribers connected now
A worker dies Undeleted message becomes visible after timeout Retryable delivery failures get retries Resumes from committed offsets No replay
It keeps failing DLQ with a redrive policy Subscription DLQ for failed delivery, if configured Your code decides Nothing to retry
It arrives twice Standard can; FIFO deduplicates sends within its window and scope Standard can; FIFO deduplicates publishes within its window and scope Uncommitted work can repeat No redelivery; overlapping subscriptions can duplicate
Nobody listening Retained up to 14 days, default 4 Standard keeps nothing for later subscribers Replay while records remain retained Gone

Sources: SQS visibility, SQS quotas, SQS DLQ, SNS filtering, SNS retries, SNS DLQ, Kafka design, Redis Pub/Sub.

SNS retries cover delivery, not your work. Once SNS receives an HTTP endpoint's success response, a later worker crash doesn't trigger a retry. When delivery retries run out, SNS discards the message unless a dead-letter queue is attached.

SNS FIFO can archive and replay when configured. Kafka replay also has limits: deletion and compaction can remove history.

Crash on both sides of the side effect

Say the handler charges a card, without idempotency or a durable handoff. Crash it before the charge, then after the charge succeeds.

For after-work acknowledgement, the crash happens before deletion or offset commit. For before-work acknowledgement, the commit or HTTP success has already been accepted. Assume a healthy broker and retained messages still eligible for recovery.

Setup Crash before the charge Crash after the charge succeeds
SQS, delete after work Eligible for retry after timeout Eligible for retry; charge may repeat
Kafka consumer group, commit after work Re-read on recovery Re-read on recovery; charge may repeat
Kafka consumer group, commit before work Skipped on recovery; charge missed Skipped on recovery; charge already made
SNS to HTTP, success received before work No retry for this crash; charge missed No retry for this crash; charge already made
Redis Pub/Sub, one subscription No replay; charge missed No replay; charge already made

These are failure traces inferred from SQS, Kafka, SNS and Redis docs, not live broker tests. They show recovery paths, not guaranteed charge counts. Expiration, dead-letter routing or further failures can prevent completion.

Acknowledge before the work, and a crash can lose it. Acknowledge after, and a crash can repeat it. Kill your worker at both points to check your handler.

For the charge, reuse a payment-provider idempotency key on retries, within its retention rules (Stripe's example). Writing a local "done" flag after charging leaves the same crash window. Kafka transactions can coordinate Kafka outputs and offsets; an external payment still needs cooperation from the payment system.

SQS FIFO, long-running jobs and the 12-hour limit

An SQS FIFO queue can still run your job twice. Its deduplication window of 5 minutes covers repeated sends with the same deduplication ID, within the configured queue or message-group scope. It doesn't prevent redelivery after a visibility timeout. SNS FIFO has its own five-minute window and topic or message-group scope.

That matters for long AI agent runs. The visibility timeout has a ceiling of 12 hours from SQS receiving the ReceiveMessage request. Extending it doesn't reset that. Once it expires, another worker can receive the undeleted message while the first may still be going. FIFO also blocks further receives for that group while messages are in flight.

For long jobs:

  1. Call ChangeMessageVisibility as a heartbeat while the work runs.
  2. Split the work before the ceiling. AWS suggests smaller steps or Step Functions. Keep progress outside the queue.
  3. Put an idempotency key on each side effect.

Kafka's answer changed

Setup A record that keeps failing
Ordinary consumer group Your code decides.
Share group, 4.2 group.share.delivery.count.limit caps attempts: default 5, range 2 to 10. That isn't a DLQ.
Share group, 4.4 Configurable DLQs. Enable share.version=2 and set the group's errors.deadletterqueue.topic.name (Apache).

That topic name must use the configured prefix, dlq. by default. Auto-creation is off by default: create the topic and set its errors.deadletterqueue.group.enable=true, or enable cluster auto-creation. The DLQ stores metadata by default. Set the group's errors.deadletterqueue.copy.record.enable=true to include the original record (Apache).

Different clocks

Clock Value
SQS retention 4 days by default, 14 at most
SQS visibility timeout 30 seconds by default, 12 hours per receive at most
SQS FIFO deduplication 5 minutes
SNS FIFO deduplication 5 minutes
SNS retries to SQS or Lambda 100,015 attempts over 23 days for retryable delivery errors
Kafka share-group delivery limit 5 attempts by default

How to pick

  • Jobs that must survive a crash go on SQS.
  • Long AI agent runs too, with a heartbeat and checkpoints.
  • One event for many services goes to SNS, often fanned out into SQS queues.
  • History you want to replay lives in Kafka.
  • For order per customer, use SQS FIFO message groups, or a Kafka key that consistently maps to one partition. The handler must preserve that order. Moving a failed FIFO message to a DLQ can break it.
  • Live updates you can afford to miss, like typing indicators, fit Redis Pub/Sub. If you need them kept, use Redis Streams, with durability determined by persistence and replication settings.

Every broker works on a good day. Pick the one whose bad day you can live with.

The maintained article is on argbe.tech.

Top comments (0)