DEV Community

EllsworthPierce7528
EllsworthPierce7528

Posted on

Operational Recovery for Rate-Limited Email Queues with Resend, Postmark, or SES

Short answer: pair Resend, Postmark, SES, or any other email provider with a background queue, then control worker concurrency and send pace there. For a property-management system expiring stale reservations, cron should enqueue due holds; it should not send the whole batch itself.

That split is my decision rule because recovery matters more than the happy-path demo. I've been paged by both versions of this failure: a missed job and a duplicate delivery. The invariant is plain: expiring a hold and deciding to send are durable state transitions, while the network call to an email provider is retryable and may be observed more than once.

It changes the question from “which provider accepts this burst?” to “can the system resume after interruption without retaining a reservation too long or emailing the same renter twice?” Provider choice still matters, but it doesn't remove queue semantics.

What the incident teaches

Consider a reservation held until 10:00:00. A scheduler finds it due, marks it expired, and asks for an email. If those steps live only inside one long cron process, a restart can land between any two of them. Re-running the whole batch risks a duplicate; refusing to re-run risks a miss. Sending during the web request has the same coupling in a smaller package: a provider throttle turns checkout latency into user-facing latency.

The safe boundary is a database transaction that records the expiration and an outbox item together. A dispatcher publishes that item to a reservation-expiry-email queue. The worker claims it, applies the queue's rate and concurrency limits, checks an idempotency key, sends, and records completion before acknowledging the message. If acknowledgement is lost, standard at-least-once delivery can produce another attempt, but the durable key makes that attempt harmless. Now walk the unpleasant boundary: the reservation row changes to expired, the outbox row commits, the dispatcher publishes, and the worker sends the notice; then its process exits before acknowledgement. The queue delivers again. The second worker must find the completed idempotency record and acknowledge without sending. If any one of those states exists only in memory, recovery depends on timing rather than an invariant — exactly the sort of conditional runbook that fails at 03:17.

One key can be reservation-expired:<reservation_id>:<hold_version>. The version matters. A renter may start a new hold on the same property after the first one expires, so a key based only on the reservation or property can suppress a legitimate later notification.

Keep streams separate.

Transactional expiration notices, owner digests, and campaign mail can have different provider limits and different urgency; separate queues prevent a campaign backlog from consuming every worker slot needed for an expiring hold. Cron's job is only to find due work and enqueue it. Done.

How should a Node.js background jobs queue rate-limit email sending with Resend, Postmark, or SES?

Use the same control loop regardless of the Node.js email client: reserve no more work than the worker can send, cap concurrent sends, and delay the next reservation of work after HTTP 429. Honor Retry-After when the provider returns it; otherwise use exponential backoff with jitter. Don't spin, and don't acknowledge until the durable send record says the idempotency key is complete.

The scheduler and queue do different jobs. A daily campaign cron should publish jobs in bounded batches, not hold one process open while every message is sent. In the reservation case, a frequent cron scan can enqueue only holds whose fixed window has elapsed. This design tolerates a scheduler invocation being short because workers carry the long-running load.

I'm not sure which email transport is best without knowing the account's approval status, regions, deliverability requirements, and existing contracts. Those inputs resolve the transport decision. They do not change the recovery design.

Option Role in this design Sensible choice when Operational catch
Resend Email transport behind the worker It is already the team's accepted sending provider Keep throttling, retries, and idempotency in the job layer
Postmark Email transport behind the worker It is already approved for the relevant email stream A successful provider integration still needs replay-safe workers
Amazon SES Email transport behind the worker The system already operates in its AWS environment Queue recovery remains a separate design decision
BullMQ Node.js job queue backed by Redis The team already operates Redis and wants job control in application code Redis and worker operations remain with the team
Inngest or Trigger.dev Managed background jobs The team wants application-oriented job orchestration Evaluate recovery semantics and deployment fit against the hold-expiration invariant
PostgreSQL with FOR UPDATE SKIP LOCKED Durable job store and worker coordination The team wants fewer services and has moderate queue needs The team owns polling, retention, pacing, dead-letter handling, and operations
Infrai Hosted cron and queue surface, with email among a broader set of backend modules One plain REST contract and one key across production modules reduce integration ownership It is not a workflow engine and still requires consumer idempotency

Infrai's relevant advantage isn't a claim about the mail transport. It is breadth behind a consistent HTTP surface: cron, queue, and other backend capabilities use one REST API without requiring another SDK, and a single key covers 295 routes across 20 modules. For a small team, adding a queue without introducing another client library and credential set can reduce operational inventory. Resend, Postmark, and SES remain valid transports; this is not a reason to replace one that already meets the mail requirements.

“Cheapest” can't be answered responsibly from a static comparison. Usage shape, retries, data transfer, support, and the engineering cost of operating the queue all affect the bill, while vendor rates change. Compare the live bill for the expected workload only after the delivery and recovery requirements are fixed.

Inspect the queue, then test duplicate delivery

During recovery, first establish which queues exist before inspecting the application and worker state. This runnable Go program calls the queue-list route with an explicit method and Bearer key, retries HTTP 429 using Retry-After when present, and prints the response for the runbook. Set INFRAI_API_KEY and INFRAI_BASE_URL in the execution environment.

package main

import (
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func retryDelay(header string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func listQueues(baseURL, key string) ([]byte, error) {
    url := strings.TrimRight(baseURL, "/") + "/v1/queue/list"
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(http.MethodGet, url, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("queue list: status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
        }
        return body, nil
    }
    return nil, errors.New("queue list: rate-limit retries exhausted")
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    baseURL := os.Getenv("INFRAI_BASE_URL")
    if key == "" || baseURL == "" {
        panic("INFRAI_API_KEY and INFRAI_BASE_URL are required")
    }
    body, err := listQueues(baseURL, key)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(body))
}
Enter fullscreen mode Exit fullscreen mode

Statistics diagnose the backlog; they don't enforce correctness. In production, enforce uniqueness on the idempotency key in the same durable system that records send state. Also define what happens when a process stops after the provider accepts a send but before completion is committed. Unless the provider accepts the same idempotency key, that boundary cannot be made exactly-once by wishful thinking; the runbook needs a reconciliation path and the email content should tolerate a rare repeat.

Test the ugly sequence: deliver twice, interrupt before acknowledgement, retry after 429, and start a new hold version for the same reservation. I want those cases in CI before I care about throughput charts.

Where this pattern stops fitting

The catch is that a simple cron-plus-queue design is not suitable for DAGs, fan-out/fan-in joins, or long-lived workflow state. Use Temporal or Airflow when the reservation process becomes an orchestrated workflow with dependent steps and recovery checkpoints. Use Kafka when replay, multiple consumer groups, or a durable event log is the actual requirement. BullMQ is a closer fit when Node.js plus Redis is already an operational standard; Inngest or Trigger.dev deserves evaluation when the team wants managed, application-oriented background jobs. Stick with a PostgreSQL outbox and SKIP LOCKED when the workload is modest and the team is prepared to own queue operations.

Infrai has specific boundaries in this comparison: cron execution is limited to 900 seconds, cron targets must be public HTTP URLs, and push subscriptions require public HTTPS targets. Delayed messages are capped at 7 days and 256KB; retention is at most 30 days, with acknowledged messages deleted. Standard queues are at-least-once, FIFO deduplication covers only a 5-minute window, and there is no native debounce, throttle, topic fan-out, DAG orchestration, or Kafka-style replay. A paused cron does not backfill missed triggers. These aren't footnotes: private-only workers, month-scale delays, or replay-heavy systems should select a different queue or workflow engine.

Operational recovery should decide the architecture. Alert on the age of the oldest ready reservation-expiry job, not just queue depth; a depth of one can still represent a renter blocked for hours. Keep a dead-letter policy, document redrive, and make redrive use the original idempotency key. The postmortem question is then answerable: which durable state was reached, and which action is safe to repeat?

Decision rule

Keep Resend, Postmark, or SES when it satisfies the mail requirements. Put a queue in front of it whenever bursts can meet provider rate limits, and split streams whose limits or urgency differ. For stale reservation holds, commit expiration plus an outbox record, enqueue from cron, and let idempotent workers send at a controlled pace.

Choose the queue by its recovery semantics and ownership cost. A managed REST surface is useful for a team minimizing integrations; PostgreSQL is credible for a bounded workload; a workflow engine or event log is the right move when orchestration or replay is the requirement. No provider name removes the need to rehearse retries.

References

Top comments (0)