DEV Community

finnmorgan226
finnmorgan226

Posted on

Apache Kafka Retry Patterns vs AWS SQS for Logistics SaaS Job Retries

#kafka #sqs #reliability #sre

Short answer: choose the retry mechanism by the deadline, replay shape, and acknowledgement boundary, not by the fact that both systems can hold a failed job. For a logistics SaaS renewal reminder, a retry must never outlive the business deadline, and a successfully acknowledged message must have a durable record of the next attempt. Kafka gives a log-oriented replay model; SQS gives a queue-oriented redelivery model; RabbitMQ exposes acknowledgement semantics that make the same boundary explicit.

The example uses a reminder due at 09:00 local time, a 15-minute processing budget, and at most three attempts. Those numbers are policy inputs, not universal defaults. The SLO is that 99.9% of eligible reminders reach dispatch before the deadline, while expired work is marked and measured instead of being retried forever.

What failure did the deadline expose?

I treat this as a bounded production scenario: a carrier's renewal reminder is scheduled for a business deadline, the downstream billing lookup times out, and the retry worker wakes up after the reminder is no longer useful. The incident is not that delivery was late. The incident is that the system spent capacity on work whose value had already become zero.

A retry policy needs two clocks. The first is the attempt clock: back off after a transient error, perhaps 30 seconds, then 2 minutes, then 10 minutes. The second is the business clock: stop before 09:00, including queue delay, consumer startup, network time, and the time needed to record terminal state. A visibility timeout or consumer acknowledgement does not replace that business clock.

Deadlines are unforgiving.

The preventative path is a small decision function, surrounded by idempotency and durable state:

package retry

import "time"

type Job struct {
    ID          string
    DueAt       time.Time
    Attempt     int
    MaxAttempts int
}

func nextAction(job Job, now time.Time, retryAfter time.Duration) string {
    if !now.Before(job.DueAt) {
        return "expired"
    }
    if job.Attempt >= job.MaxAttempts {
        return "dead-letter"
    }
    if now.Add(retryAfter).Add(2 * time.Minute).After(job.DueAt) {
        return "expired"
    }
    return "retry"
}
Enter fullscreen mode Exit fullscreen mode

The two-minute reserve is a capacity-planning decision: it covers state persistence and dispatch handoff, then gets validated against latency histograms. In a real implementation, the worker claims the job with a lease, performs an idempotent operation keyed by Job.ID, and writes the next attempt or terminal result before acknowledging the transport message. The code does not apply unchanged when late delivery still has value; those workflows need a different state machine.

How should Apache Kafka and AWS SQS handle retry patterns?

The important difference is ownership of progress. Kafka consumers track offsets in partitions, so a consumer can replay records by controlling the committed position; that is useful when a whole stream, partition, or time range must be reprocessed. The cost is operational: retry ordering, poison records, partition head-of-line blocking, and offset commit timing become application concerns. A retry topic can make backoff visible, but it adds routing and retention decisions.

SQS presents a queue with visibility timeout and redelivery. A worker receives a message, keeps it invisible for a bounded interval, and either deletes it after successful processing or allows it to return. A dead-letter queue and maximum receive count provide a terminal path. Visibility timeout is a lease on delivery, not proof that the business operation completed. If processing can exceed the timeout, duplicate delivery is expected and idempotency is mandatory.

RabbitMQ's acknowledgement model is another useful comparison. Its consumer acknowledgement documentation distinguishes automatic acknowledgement from explicit acknowledgement, and explicit acknowledgement lets a consumer acknowledge after work completes or reject a message for another disposition. That makes the transport boundary easy to reason about, but it does not make a non-idempotent handler safe.

Decision concern Log-oriented stream Queue-oriented delivery
Replay unit Offset range, partition, or retained record Individual message, subject to retention and redelivery policy
Retry ownership Application routing and offset discipline Visibility, delete, and redelivery policy
Main deadline risk A blocked record or retry lane delays later work A lease expires while a worker is still active
Good fit Auditable event history and deliberate reprocessing Isolated jobs with explicit attempt limits

This is a buy-vs-build decision in operational terms. Building delay queues and replay controls around a log can preserve flexibility while increasing on-call surface. Buying queue semantics from a managed service reduces maintenance work while increasing dependence on its delivery and retention model. Neither choice establishes the SLO by itself.

The limitation is not a missing feature in one transport. A log-oriented design is a poor fit when the team needs isolated job retries without partition head-of-line effects; a queue-oriented design is a poor fit when retained history and deliberate range replay are first-class requirements. That trade-off should be tested against the on-call roster, recovery procedure, and deadline SLO before a platform choice is made.

How should the retry budget be measured?

Start with a budget equation:

deadline - enqueue_time = queue_wait + processing_time + retry_backoff + persistence_margin

Measure each term separately. A single end-to-end latency number cannot tell an SRE whether the consumer is underprovisioned, the backoff is too generous, or the database write is the new bottleneck. I want dashboards for eligible jobs, expired jobs, attempts per job, age of the oldest pending job, duplicate execution, and terminal failures. Alert on error-budget burn and deadline misses, not merely on queue depth. Queue depth can look healthy while the oldest job is already useless.

The retry budget belongs in capacity planning. If 1,000 reminders are due in a five-minute window and 8% enter a second attempt, the worker pool must absorb 1,080 executions plus database and downstream-call headroom. A design that works at the average arrival rate can still fail the deadline SLO during the burst. Load tests should inject timeouts, process pauses, duplicate deliveries, and a clock near the deadline; clean success traffic tests the least interesting path.

What should the deployment and failure policy include?

Make the policy data-driven and versioned. Store the due time, attempt count, next eligible time, terminal reason, and idempotency key with the job. Emit transitions for scheduled, attempted, succeeded, retryable_failure, and expired. During deployment, drain or lease-transfer workers so a process restart cannot turn an in-flight acknowledgement into an accidental success.

Separate transient transport failures from permanent validation failures. Apply jitter to retries so a shared dependency does not receive a synchronized wave, cap the retry horizon at the business deadline, and send terminal jobs to a reviewable store. A dead-letter queue is an inspection boundary, not a substitute for ownership: someone still needs a runbook for replay, suppression, and customer-visible correction.

The advice changes for ordered stream processing across many entities, first-class audit replay, or a late reminder with legal or financial meaning. Preserve the event history and model deadline expiry as a domain event in those cases. For isolated renewal jobs, keep the message contract small and make the terminal decision explicit.

The neutral decision rule is this: use the delivery model that makes the business deadline, acknowledgement point, replay scope, and duplicate behavior easiest to observe and test. Kafka and SQS can both carry a retry workflow, and RabbitMQ can make explicit acknowledgement boundaries clear; the hard part remains the same state machine around them.

Sources

Top comments (0)