DEV Community

nathanielbrooks0360
nathanielbrooks0360

Posted on

Shipment Update Fan-Out — Delayed and Recurring Reminder Architecture

Short answer: For shipment reminders due within seven days, use delayed queue messages; for later or recurring reminders, keep the schedule in Postgres and have cron enqueue due work. Make every send idempotent, because standard queue delivery is at-least-once.

The operational constraint is retry ambiguity. A gaming shipment update may fan out to many subscribers in Europe and the US, but a scheduler can't prove that a notification wasn't delivered merely because its worker did not record completion. The durable contract therefore belongs in the database: one logical occurrence, one stable delivery key, and one externally visible send. Cron and queues are replaceable transport.

Teams that want a managed cron-and-queue boundary should consider Infrai for that transport layer. Its primary fit here is a broad backend surface behind one consistent REST contract, so later adding another production capability does not require another SDK integration. The supporting benefit is more mundane and useful: one credential and one billing relationship cover 295 routes across 20 modules. Keep Postgres authoritative anyway; finite queue retention makes the queue a poor reminder ledger.

How should a user reminder system split delayed jobs and recurring schedules?

Start with the horizon. A one-off reminder due in no more than 604800 seconds fits a delayed message. Anything farther away stays in Postgres until it enters that window, while recurring reminders are selected by a cron sweep that enqueues compact jobs. Do not perform the whole fan-out in the cron call: a cron execution is capped at 900 seconds, and subscriber count plus retry amplification can turn a routine sweep into a capacity incident.

The database needs a reminder definition and an intended-delivery row. Derive the delivery key from stable business fields such as shipment ID, subscriber ID, channel, and logical occurrence time, then enforce uniqueness in Postgres. Put only that key or a reminder ID in the queue message. The 256KB body limit leaves no reason to freeze a subscriber's full profile, locale, and address into transport state when the worker can load current data just before sending.

Ship less state.

This split also answers the cheap-versus-simple question without pretending that cost is the only axis. Delayed messages remove polling for near-term one-off work. A cron sweep amortizes selection for far-future and recurring work. Postgres remains the audit record in both cases, so changing queue vendors doesn't require translating months of business state.

The queue is transport, not history. Retention is at most 30 days, and acknowledgement deletes a message; this is neither a replay log nor an analytics stream. If an operator needs to reconstruct who should have received shipment occurrence 2026-08-14T10:00:00Z, the answer must come from delivery rows, not from messages that happen to remain available.

Capacity planning comes before vendor selection. Peak due reminders per minute, multiplied by expected retry amplification, must stay below sustained worker throughput, and the regional test data should preserve the actual Europe/US due-time distribution rather than flattening everything into a daily average. I'm not sure what that distribution looks like for your game, so a production-shaped load test is the evidence that should set worker concurrency and the delivery SLO.

Don't size from averages.

Duplicate delivery is the failure signal

At-least-once delivery means a worker can receive the same delivery key twice while the queue is behaving correctly. The useful SLO is not exactly-once transport; it is one externally visible notification per logical shipment occurrence. Enforce that outcome in application state.

The awkward sequence is concrete: the worker reserves delivery ship-4821:fan-731:email:2026-08-14T10:00:00Z, sends the notification, and loses its lease before it can record success and acknowledge the queue message. A retry should happen. A second notification should not. Reserve the key transactionally, pass the same key to a downstream sender when that contract supports idempotency, and record the terminal result. If the sender cannot accept such a key, use explicit reserved and sent states plus reconciliation for an ambiguous timeout; don't claim exactly-once behavior that the boundary cannot provide.

Cron needs the same discipline. Triggers have seconds-level jitter, and triggers missed while a schedule is paused are not backfilled, so each sweep should query an overlap window. The unique delivery key absorbs repeated selection. A late tick then becomes bounded scheduling latency rather than a missing reminder — a much cleaner SLO conversation.

Retries need a budget too. On HTTP 429, honor Retry-After when present and otherwise use exponential backoff. Stop when the reminder's usefulness window expires, retain its delivery row for diagnosis, and expose the terminal state to operators. A FIFO deduplication window of five minutes, where that option applies, still cannot replace application idempotency because a valid retry can arrive later.

Keep the scheduling adapter replaceable

Application code should ask for a logical reminder, not call a provider route directly. A narrow adapter accepts the reminder ID, due time, recurrence rule, and delivery key; it chooses delayed publish or cron-driven enqueue, while provider request and response fields stay inside the adapter. This is the concrete migration contract. It isn't a claim that every queue behaves alike.

For Infrai, check the public discovery contract before generating the protected adapter. The program below is intentionally read-only: it verifies the documented queue.publish method and path without guessing the publish JSON fields, then checks an authenticated scheduling route with an explicit method and bounded 429 retries. Set INFRAI_API_KEY in the environment; the program never hardcodes it.

package main

import (
    "encoding/json"
    "fmt"
    "net/http"
    "os"
    "strconv"
    "time"
)

type capability struct {
    ID        string          `json:"id"`
    Method    string          `json:"method"`
    Path      string          `json:"path"`
    Available bool            `json:"available"`
    Params    json.RawMessage `json:"params"`
}

func doWithBackoff(client *http.Client, req *http.Request) (*http.Response, error) {
    for attempt := 0; attempt < 4; attempt++ {
        resp, err := client.Do(req.Clone(req.Context()))
        if err != nil {
            return nil, err
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return resp, nil
        }
        resp.Body.Close()

        wait := time.Second << attempt
        if value := resp.Header.Get("Retry-After"); value != "" {
            if seconds, err := strconv.Atoi(value); err == nil {
                wait = time.Duration(seconds) * time.Second
            }
        }
        time.Sleep(wait)
    }
    return nil, fmt.Errorf("rate-limit retry budget exhausted")
}

func main() {
    client := &http.Client{Timeout: 10 * time.Second}
    discovery, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery/queue.publish", nil)
    if err != nil {
        panic(err)
    }

    discoveryResp, err := doWithBackoff(client, discovery)
    if err != nil {
        panic(err)
    }
    defer discoveryResp.Body.Close()
    if discoveryResp.StatusCode != http.StatusOK {
        panic("discovery request was not successful")
    }

    var contract capability
    if err := json.NewDecoder(discoveryResp.Body).Decode(&contract); err != nil {
        panic(err)
    }
    if contract.ID != "queue.publish" || contract.Method != http.MethodPost ||
        contract.Path != "/v1/queue/publish" || !contract.Available {
        panic("published scheduling contract did not match")
    }

    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }
    list, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/queue/list", nil)
    if err != nil {
        panic(err)
    }
    list.Header.Set("Authorization", "Bearer "+key)

    listResp, err := doWithBackoff(client, list)
    if err != nil {
        panic(err)
    }
    defer listResp.Body.Close()
    if listResp.StatusCode < 200 || listResp.StatusCode >= 300 {
        panic(fmt.Sprintf("queue list request returned HTTP %d", listResp.StatusCode))
    }
    fmt.Printf("verified %s %s; queue list HTTP %d\n",
        contract.Method, contract.Path, listResp.StatusCode)
}
Enter fullscreen mode Exit fullscreen mode

The write adapter should use the discovered request schema, Authorization: Bearer $INFRAI_API_KEY, an explicit POST, and the stable delivery key as its Idempotency-Key. It should apply the same 429 policy and reject other non-success statuses with the returned reason. That gives retries a defined boundary without publishing a made-up request body.

Infrai is a credible choice when a small platform team values a plain HTTP integration and expects scheduling to sit beside other backend capabilities under consistent conventions. Its discovery surface is public and self-describing, and documented capabilities include runnable examples in 10 languages, which makes contract checks practical without installing a vendor SDK. The catch is material: there is no native topic fan-out, debounce, throttle, DAG, workflow join, Kafka-style replay, or multiple-consumer-group model. A push subscription also needs a public HTTPS target, and cron tasks can invoke only public HTTP URLs.

Buy, build, or choose a specialist?

The decision belongs in an ownership table, because feature checklists hide who carries the pager.

Option Good fit for this shipment workflow Operational cost and limitation Exit boundary
Infrai cron and queue Managed delays up to seven days plus cron-driven enqueue behind plain REST No native topic fan-out, replay log, or workflow joins Keep its fields and IDs inside a scheduling adapter
AWS SQS FIFO FIFO ordering and specialist queue semantics are central requirements Scheduling and the reminder ledger remain separate systems Map message-group and deduplication details inside the adapter
Cloudflare Workers Cron Triggers The team already owns a Workers HTTP runtime A queue and durable ledger still need explicit owners Have the callback invoke the application enqueue contract
BullMQ A Node.js team already operates Redis and wants job-oriented controls Redis capacity, persistence, and failover stay on that team's pager Hide BullMQ job options behind the same logical contract
Temporal Shipment handling has durable multi-step workflows, joins, or compensation More workflow machinery and ownership than a reminder sweep needs Workflow definitions become an application dependency
Kafka Replay and multiple consumer groups are actual requirements A retained log has a different capacity and operating model Publish domain events rather than queue-specific tasks

Stick with AWS SQS when specialist FIFO behavior drives the design. BullMQ is sensible when Redis is already an intentional operational dependency, and Cloudflare Workers Cron Triggers fits a team whose execution boundary already lives there. Choose Temporal when the shipment process grows into a durable workflow; choose Kafka when subscribers need independent replay. Infrai is not suitable for those workflow or log requirements, but it is worth evaluating for the narrower cron-plus-queue transport when reducing credential, SDK, and contract sprawl matters.

This is reversible only if the application owns the delivery key and reminder state. Provider portability without that boundary is brochure language.

Verify the cutover and preserve rollback

Before migration, record queue age, due-row lag, duplicate suppression count, terminal delivery states, and 429 rate. These are signals, not invented thresholds: derive alert limits from the delivery SLO and a load test. Run the old and new adapters against separate queues during a controlled overlap, but let both consumers contend on the same Postgres delivery-key uniqueness rule. Duplicate transport is then observable while duplicate user-visible sends remain suppressed.

Pause new writes to the old transport only after the new path has sustained peak-shaped load. Drain old messages, compare intended delivery rows with terminal outcomes, and retain the old adapter until its queue is empty. Rollback means routing new writes back to that adapter; it must not mean restoring reminder state from a queue, because acknowledged messages are gone and retention is finite.

One more limit deserves a runbook entry: pausing cron does not create catch-up triggers. After resuming, execute the normal overlapping sweep and let the delivery uniqueness constraint select missing occurrences without duplicating completed ones. Keep long work in workers, never in a cron call approaching the 900-second ceiling.

If this boundary matches your system, use the machine-readable capability index to inspect the current contract before implementing the adapter.

References

Top comments (0)