DEV Community

AbernathyCross6857
AbernathyCross6857

Posted on

Apache Kafka Retry Patterns vs AWS SQS: Choose Jobs for Property Updates

#scheduling #kafka #retry

TL;DR: For a property-management system fanning a shipment update to many subscribers, a queue with explicit visibility timeout and dead-letter handling is the safer default for failed jobs. A replay log becomes the better fit when the update is an event stream that several independent consumers must reprocess. The choice is a latency-versus-cost decision, but retry ownership is the part that prevents duplicate texts and silent delivery gaps.

A shipment update sounds harmless until one parcel status has to reach residents, property staff, maintenance partners, and an audit pipeline. One subscriber may be slow. Another may return a transient 429. An SMS provider may accept the request while the worker loses its connection before recording success. The failure is not one thing, so a single retry loop is a poor boundary.

That boundary matters.

Should Apache Kafka or AWS SQS own retry patterns for job updates?

Start with the unit of work: one subscriber notification, not one shipment. Store an idempotency key such as shipment_id:subscriber_id:status_version. The dispatcher creates one durable job per destination, and the worker records the provider result alongside that key. A retry can then repeat a request without turning a timeout into two messages.

I initially treated the shipment event as the retry unit. That made the first implementation attractive: consume one event, loop over recipients, and retry the loop. The flaw appeared in the failure matrix. A slow email endpoint held up SMS recipients, and a single process restart replayed work that had already succeeded. Splitting the fanout made the queue larger, but it made ownership and recovery legible.

The latency target should decide how much machinery is justified. If a resident needs a delivery update within seconds, keep the first attempt close to the event and use short, bounded backoff for transient failures. If the update is informational and a delay of several minutes is acceptable, a cheaper delayed retry path can absorb provider rate limits. Do not make every failure urgent. A design that optimizes every notification for the lowest possible latency will spend capacity on stale or already superseded statuses.

The useful state machine is small:

from dataclasses import dataclass
from datetime import datetime, timedelta


@dataclass
class Job:
    key: str
    attempts: int
    available_at: datetime


def next_attempt(job: Job, now: datetime, retry_after: int | None) -> Job:
    if retry_after is not None:
        delay = min(retry_after, 900)
    else:
        delay = min(2 ** job.attempts, 300)

    return Job(
        key=job.key,
        attempts=job.attempts + 1,
        available_at=now + timedelta(seconds=delay),
    )
Enter fullscreen mode Exit fullscreen mode

The worker should classify failures before applying that schedule. A 429 or a network timeout is usually retryable; an invalid phone number, revoked destination, or rejected payload needs a terminal path. The exact provider response vocabulary varies, so keep the classifier behind an interface and test it with recorded responses. Never retry an authentication failure at full speed. That is how a small incident becomes a rate-limit storm.

What do logs and queues actually guarantee?

Apache Kafka and Amazon SQS solve adjacent problems. Kafka is a partitioned append-only log: consumers track offsets and can reread a range when replay is part of the business workflow. SQS presents messages to consumers, hides an in-flight message for a visibility timeout, and can route messages that exceed a receive limit to a dead-letter queue. RabbitMQ uses acknowledgements and negative acknowledgements to decide whether a message is accepted, requeued, or discarded.

Those mechanics create different retry costs. A log replay is efficient when the same shipment event must feed a new audit consumer or rebuild a derived view. It also means each consumer owns offset progress, duplicate suppression, and the policy for a poison event. A queue retry is easier to reason about for a single job: receive, process, acknowledge only after durable success, and let the broker make an unacknowledged job visible again. The simplicity has a price: long retention, large fanout, and per-message delay policies can raise operational cost. The queue is a poor fit if every team needs an independently replayable history; a log is a poor fit if the only requirement is a bounded, per-recipient attempt schedule.

None of these systems makes an external notification exactly once. The broker can redeliver after a worker succeeds but before its acknowledgement is observed. The provider can accept a request before the worker sees the response. Exactly-once claims inside a transport do not erase that boundary. Treat delivery as at-least-once and make the receiver operation idempotent.

Concern Replay-log approach Queue-job approach
Primary unit Event and consumer offset Individual subscriber job
Recovery Seek and replay a range Redelivery after visibility or nack
Fanout Consumers derive their own work Dispatcher creates explicit jobs
Best latency shape Stream processing with active consumers Bounded work with backoff
Main trap Replaying successful side effects Hidden expiry or duplicate delivery

This is why I would choose the queue boundary for the property-management shipment workflow when each recipient has a different endpoint and retry policy. I would choose the log boundary when the shipment history itself is the product: several teams need independent replay, and the notification jobs are derived projections rather than the canonical record. That is a boundary condition, not a brand preference.

Keep that boundary boring.

How should a worker behave under pressure?

Backpressure is part of correctness. A worker pool that grows until it matches the backlog can turn a provider slowdown into a larger outage. Cap concurrency per destination class, reserve capacity for high-priority resident updates, and expose queue age rather than only queue depth. A queue with 10,000 fresh jobs is healthier than one with 100 jobs that have been waiting for 40 minutes.

Use a monotonic attempt count and persist it with the job. The count should survive a process restart. Record first_seen_at, last_attempt_at, next_attempt_at, the failure class, and the destination class. Those fields answer the operational questions that matter: are retries delayed by policy, by capacity, or by a provider?

Dead-lettering is not a trash can. Set a bounded attempt policy, preserve the original payload and failure history, and give operators a replay action that checks idempotency before resubmitting. A poison payload should become visible for diagnosis without blocking unrelated subscriber jobs. For OTP or security-sensitive messages, the policy may be stricter: late delivery can be actively harmful, so expiration belongs in the job contract.

Test the awkward transitions. Kill a worker after the external call and before acknowledgement. Return a 429 with a long retry hint. Let the visibility timeout expire while a downstream call is still running. Publish the same shipment status twice. These cases are more valuable than a happy-path test that proves a message can travel from producer to consumer.

The sharp edge is acknowledgement timing. Suppose the worker sends delivered to a subscriber, then stalls for 4 seconds before acknowledging. If visibility is 3 seconds, a second worker can send the same status while the first is still alive. Extending visibility reduces that race, but a long extension holds work hostage when the first worker really has died. The idempotency key closes the gap; the timeout only controls when another worker gets a chance. That distinction belongs in the runbook and in the failure-injection test, because operators need to know which duplicate is expected and which one indicates a broken key.

The limitation is deliberate: a queue is not a substitute for an event history, and a replay log is not a free per-recipient scheduler. If the business needs audit reconstruction, retain the canonical shipment event separately. If it needs only notification delivery, adding independent offset management creates a real maintenance tradeoff.

A practical rollout for the first 30 days

Begin with a durable job schema and an idempotency store independent of the broker API. Run the dispatcher in shadow mode so it can measure fanout size, predicted queue age, and destination mix without sending. Then enable one low-risk notification class, with a short maximum retry horizon and a dead-letter review queue.

The useful dashboard is narrow: successful first attempts, retryable failures, terminal failures, oldest job age, duplicate-suppression count, and provider response class. Add alerts for age and failure rate, not just backlog. A quiet queue can still hide a worker that stopped acknowledging.

After the first week, compare measured latency with the business threshold. If the queue is mostly empty but the system needs cross-team replay, introduce a log as the event history and keep the job queue as the side-effect boundary. If replay is never used and per-subscriber scheduling dominates, keep the design focused on jobs. Avoid paying the complexity cost of two retry authorities unless each has a clear owner.

The durable choice is the one that makes a failed shipment notification explainable: who owns the retry, what makes it safe to repeat, when it expires, and where an operator can recover it. Choose the replay log for independent historical reconstruction. Choose the queue for bounded subscriber work. In both cases, the external call remains at-least-once, so idempotency and observability carry the final responsibility.

Sources

Top comments (0)