DEV Community

ArthurFinley2291
ArthurFinley2291

Posted on

Message Queues for High-Scale Reminders: 4 Retry-Safe Shipment Stages

Short answer: Message queues are preferred over direct HTTP calls for high-scale user reminders when a scheduled game shipment update must reach many subscribers, because queues isolate the producer from slow recipients and make retryable delivery explicit.

A scheduled shipment update changes the delivery problem before any code is written: one producer must notify many gaming subscribers, while the producer cannot afford to wait for each downstream HTTP response. The practical choice is to record the event once, enqueue delivery work, and let independently retryable consumers reconcile their results.

The queue is not a guarantee of exactly-once delivery. It is a boundary that makes delayed work, bounded concurrency, and recovery explicit. Exactly-once is a useful mindset for the business effect, not a promise that a network will deliver one packet only once.

Why are message queues preferred over direct HTTP calls here?

Consider a game platform that schedules shipment updates for physical collector items, tournament merchandise, or an in-game purchase that has a warehouse status. At 09:00, one shipment update may need to reach an account timeline, a customer notification service, a fulfillment dashboard, and an audit stream. These subscribers have different latency, availability, and retry behavior.

A direct HTTP fan-out makes the scheduler own all of those properties. It must decide how long to wait, how many connections to hold, which failures are transient, and what to do after the process dies after calling subscriber three. A slow subscriber can consume the same execution budget as a failed one. At larger fan-out counts, the producer becomes a traffic coordinator by accident.

A queue changes the ownership boundary. The scheduling transaction records a durable shipment event and publishes a delivery job; workers claim jobs at a controlled rate. Each worker can acknowledge a job only after the subscriber result has been classified. The scheduler can finish its own work without pretending that every recipient is already current.

Keep it boring.

That boundary is the point. The scheduler owns the fact that a shipment changed, while the delivery workflow owns the work of reaching subscribers whose response times and availability it cannot control. In a high-scale reminder run, this distinction prevents a single slow customer-notification endpoint from holding open the scheduling path for every account, but it does not erase the need to set a freshness target, limit outstanding work, and tell operators when the target has been missed. Those policies belong beside the queue because the queue only stores work; it does not decide what “late,” “duplicate,” or “complete” means for the game.

This is the decision rule I use: if downstream work can be delayed and retried without changing the meaning of the scheduled event, put that work behind a durable queue. Keep direct HTTP for the small set of calls whose response is part of the initiating transaction.

Why do retries need an identity, not just a timeout?

Retries are where an apparently reliable fan-out becomes a duplicate-delivery system. A worker can receive a job, send the HTTP request, and lose its connection before reading the response. Repeating the request may be correct, or it may create two notifications, two fulfillment actions, or two ledger entries. The timeout does not tell us which case occurred.

Give the business event a stable identity, such as shipment-evt-8f42, and give each subscriber attempt a deterministic key derived from that event and subscriber identity. The receiving endpoint must treat a repeated key as the same operation, returning the stored result or safely confirming that the effect already exists. This is the useful form of idempotency: duplicate transport attempts do not duplicate the business outcome.

The queue message can carry a compact envelope like this:

type ShipmentUpdate struct {
\tEventID       string `json:"event_id"`
\tShipmentID    string `json:"shipment_id"`
\tStatus        string `json:"status"`
\tOccurredAt    time.Time `json:"occurred_at"`
\tSubscriberID  string `json:"subscriber_id"`
\tIdempotencyKey string `json:"idempotency_key"`
}
Enter fullscreen mode Exit fullscreen mode

The fields are deliberately boring. They identify what happened, when it happened, who should receive it, and how a repeated attempt is recognized. They do not encode an instruction such as “send this exactly once,” because a message broker cannot enforce the subscriber's business semantics.

The receiving service should persist the idempotency key with the result of the operation, under a uniqueness constraint. If the operation updates its own database and emits a follow-up event, those writes need a transactional outbox or an equivalent atomic publication design; acknowledging the queue before either durable write is complete creates a silent loss window.

Which failure belongs to the queue, and which belongs to the subscriber?

The useful separation is not “queue good, HTTP bad.” It is a classification of failure ownership.

A connection refusal, timeout, or server-side 5xx usually deserves a retry with exponential backoff and jitter. A malformed payload, an unknown subscriber, or a validation response should normally go to a dead-letter path with enough context for correction. A rate-limit response needs a delay that respects the receiver's signal, rather than a tight loop that turns one outage into a larger one.

Backoff alone is insufficient. The worker needs a maximum attempt policy, a visibility or lease interval longer than the expected processing time, and an explicit terminal state. Otherwise a poison message can cycle forever, while operators see a queue that is technically active but not making progress.

Here is a small Go decision function. It is an example of policy, not a complete queue client; the important part is that classification is separate from transport code.

type Action int

const (
\tRetry Action = iota
\tAcknowledge
\tDeadLetter
)

func classify(status int, attempt int) Action {
\tif status == 408 || status == 429 || status >= 500 {
\t\tif attempt < 6 {
\t\t\treturn Retry
\t\t}
\t\treturn DeadLetter
\t}
\tif status >= 400 {
\t\treturn DeadLetter
\t}
\treturn Acknowledge
}
Enter fullscreen mode Exit fullscreen mode

The number six is an explicit example policy, not a universal constant. Teams should choose it from the subscriber's recovery characteristics, notification freshness window, and operational capacity, then test the resulting backlog behavior. The point is to make the decision reviewable.

I initially treated the queue as the reliability mechanism. Later I found that assumption incomplete: the queue supplied durable work and isolation, but the audit trail supplied the answer to the harder question: did the subscriber accept this event, did it reject it, or did the worker lose the result? Without those states, retries merely hide uncertainty.

How should the audit trail shape the worker?

For each event-subscriber pair, keep a durable delivery record with the event ID, subscriber ID, attempt count, last response class, next eligible attempt time, and final disposition. Do not rely on worker logs as the source of truth; logs are excellent evidence during an incident, but they are a poor reconciliation database.

A worker's state transition can be described compactly:

  1. Claim one eligible delivery record and extend its lease while processing.
  2. Send the request with the deterministic idempotency key and a bounded timeout.
  3. Store the response classification and the next action before acknowledging the queue message.
  4. Acknowledge successful or terminal work; release retryable work for a future attempt.

The ordering matters. If acknowledgement comes first, a process crash can lose the delivery record. If the database update comes first and the acknowledgement is lost, the message may be redelivered; the idempotency key makes that duplicate harmless. This is why I prefer a design that tolerates at-least-once delivery over one that quietly assumes exactly-once transport.

The audit record also supports reconciliation. A scheduled report can find shipments whose subscriber set is incomplete, compare the expected event IDs with durable delivery records, and enqueue only the missing pairs. That repair job should be safe to run repeatedly. It is an operational control, not an emergency script that depends on a developer remembering a command.

What should be measured before increasing concurrency?

The first dashboard should show age of the oldest pending delivery, retry rate by subscriber, terminal failures, lease expirations, and the difference between produced events and completed delivery records. Queue depth alone is ambiguous: a growing queue may mean a burst is being drained successfully, while a flat queue can conceal a worker repeatedly failing the same message.

Load testing should use the actual fan-out shape. Five hundred subscribers receiving one event is different from five hundred events each reaching one subscriber, even if both create five hundred HTTP calls. Include slow responses, dropped connections after request transmission, malformed payloads, rate limits, and a worker restart between the subscriber response and queue acknowledgement. Verify the audit table after each case.

There is a real trade-off. A queue adds a broker, worker deployment, delivery records, and operational vocabulary. It also adds delay: a notification is no longer coupled to the scheduler's request path. For a small, synchronous operation where the caller needs an immediate, authoritative response, that cost may not be justified. For scheduled fan-out, the decoupling usually reflects the actual business boundary: the shipment update is accepted first, and delivery is a process with its own lifecycle.

The implementation should remain portable at the boundary. Define an internal producer interface, keep message serialization versioned, and make the worker's acknowledgement semantics visible in tests. The underlying queue can change; the event identity, retry policy, and audit contract should not.

A compact rollout plan

Start with one subscriber and one durable delivery record per event. Run the worker in shadow mode if the subscriber supports it, or send to a test endpoint that records idempotency keys without producing user-visible effects. Then inject the failure cases that matter: timeout after transmission, duplicate message delivery, expired lease, and a subscriber that returns a permanent validation error.

Add subscribers only after the first contract can be reconciled from stored state. During rollout, cap concurrency per subscriber rather than using one global limit, because the slowest or most restrictive recipient should not determine every other recipient's throughput. Keep a bounded dead-letter workflow with ownership and replay rules.

The finished system is not “a queue instead of HTTP.” It is a scheduled event, a durable delivery record, a retry policy, and an idempotent subscriber contract connected by a queue. That combination turns fan-out from a long synchronous call chain into an observable workflow, while preserving the audit evidence needed to explain every eventual outcome.

Sources

Top comments (0)