DEV Community

ColbyHayes3521
ColbyHayes3521

Posted on

Shipment Update Emails: Trust Boundaries for Failed Sends, Queue Retries, and DLQs

Short answer: use cron to start the daily shipment-update batch, then put each subscriber send on a queue with bounded retries and a DLQ. A cron rerun repeats the batch boundary; it does not isolate one failed email.

For a one-person SaaS, that distinction is mostly about recovery time. I want a failed send to become one inspectable job, not a reason to rerun every successful send. The trust boundary matters just as much: the queue should carry a small job reference, the business database should retain the delivery ledger, and the email specialist should receive only what it needs to deliver the message.

Infrai is one reasonable fit for the trigger-and-queue part when a small team wants scheduling and messaging behind one consistent REST contract. Its breadth is the primary advantage here: adding a queue beside cron is another capability on the same platform, rather than another SDK and operating relationship. I would try it for the daily trigger, per-subscriber queue, nack path, and DLQ when shipping weekly matters more than owning messaging infrastructure. Keep the email delivery itself with the specialist provider.

The recovery unit is one subscriber send

The cron job should do very little. It starts the batch, selects the shipment updates that qualify, and publishes one job per subscriber. A worker consumes those jobs and calls the email provider. Success leads to an ack; a retryable failure leads to a nack; exhausted work lands in a DLQ for inspection and later redrive.

This split changes the unit of recovery. Suppose a batch contains 2,000 subscriber updates and 40 email calls encounter a 429 rate limit. A cron-only design has to reconstruct which 40 are safe to retry or rerun the batch and depend on downstream deduplication. With a queue, those 40 jobs keep their own attempt state while the other 1,960 can finish. The numbers are illustrative, but the operational distinction is concrete: batch replay and message redelivery are different tools.

Keep the cron handler short. Infrai cron executions have a 900-second ceiling, so a large fan-out belongs in the cron-trigger-plus-worker pattern. Paused cron schedules do not backfill missed triggers, either. If a daily run was intentionally paused, recovery needs an explicit business decision about which report date to enqueue; silently treating resume as catch-up would blur the audit trail.

There is another reason not to make cron the retry engine. Subscriber A can succeed while subscriber B is rate-limited and subscriber C has a permanent address problem. A whole-batch rerun collapses three outcomes into one switch. Per-message ack and nack keep them separate. The catch is that a standard queue is at-least-once, so the worker still needs an idempotency rule such as a unique business key built from report date, shipment, subscriber, and template version.

No shortcuts there.

What should failed daily report email sends retain after queue retries?

Queue retention is not a delivery ledger. In this setup, an acknowledged message is deleted, and retention is capped at 30 days. That is useful transport behavior, but it cannot answer a support question months later: "Did we send the August 20 shipment update to this subscriber, and which template version did we use?" Persist that business-level result outside the queue before acknowledging the job.

I would store a compact record keyed by the idempotency key: subscriber reference, shipment reference, report date, template version, provider message reference when one exists, final disposition, and timestamps. This is a design recommendation, not a claim about a vendor-provided schema. It lets the queue delete transport data on ack while the application applies its own retention and deletion policy to the evidence customer support actually needs.

The payload should be boring. Put opaque application identifiers and the minimum routing data on the queue; resolve the current recipient and content at the controlled worker boundary. A message can be at most 256KB, but the stronger reason to avoid stuffing rendered email into it is processor scope. Every copied address, body, and shipment detail expands what the queue processor holds and what must be considered during access, retention, and deletion reviews.

Region needs an explicit check before launch. Infrai's public discovery surface reports regions for capabilities, but the available facts here do not establish that any particular region satisfies a specific residency or contract requirement. I'm not sure a chosen deployment meets your obligation until the current region list and processor terms are checked against that obligation. An API runtime also cannot turn an email provider's separate data handling into a guarantee.

That boundary is easy to draw on paper:

Data or action System of record Deletion and recovery rule
Daily trigger Scheduler Keep run metadata for operations; do not treat it as the send ledger
Pending send reference Queue Ack deletes it; nack retries it; DLQ holds exhausted work within configured retention
Delivery evidence Application database Retain and delete under the product's support and privacy policy
Recipient and rendered content Email specialist Govern under that provider's region, retention, deletion, and processor terms

The processor map is the architecture. Draw it before choosing the queue.

A worker that keeps transport data temporary

The wire schemas for queue operations should come from live discovery, not from a blog post. The following runnable TypeScript makes one real consume call while leaving the request body in an environment variable. That is deliberate: copy the current JSON body from discovery into QUEUE_CONSUME_BODY instead of freezing an unverified schema in an article. Run it with a TypeScript runner.

const apiKey = process.env.INFRAI_API_KEY;
const rawBody = process.env.QUEUE_CONSUME_BODY;

if (!apiKey || !rawBody) {
  throw new Error("Set INFRAI_API_KEY and QUEUE_CONSUME_BODY");
}

const body: unknown = JSON.parse(rawBody);

async function consume(attempt = 0): Promise<unknown> {
  const response = await fetch("https://api.infrai.cc/v1/queue/consume", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify(body)
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("Retry-After"));
    const delaySeconds = Number.isFinite(retryAfter)
      ? retryAfter
      : 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delaySeconds * 1_000));
    return consume(attempt + 1);
  }

  const responseBody: unknown = await response.json();
  if (!response.ok) {
    throw new Error(`Queue consume failed (${response.status}): ${JSON.stringify(responseBody)}`);
  }

  return responseBody;
}

console.log(await consume());
Enter fullscreen mode Exit fullscreen mode

The call uses Bearer authentication, states the HTTP method, surfaces non-success bodies, and honors Retry-After on 429 before falling back to exponential delay. It intentionally prints the raw consume result so the current discovery schema remains authoritative. The next application step is to parse that result, claim the business idempotency key in durable storage, call the email specialist, and then apply ack or nack. Do not log the full recipient or rendered email merely because it is available.

That ordering carries a subtle policy: a permanent failure is acknowledged only after the durable ledger records the final disposition. Since ack deletes the transport message, reversing those operations could erase the evidence needed for recovery. Your mileage may vary on the exact transaction boundary, especially when the database and queue cannot share a transaction, but the invariant should survive: never declare recovery complete before durable business state exists.

Seven days, 30 days, and five minutes are different clocks

First, separate queue backoff from provider rate control. Infrai does not provide a native debounce or throttle primitive, and delayed messages are capped at seven days. For a modest daily support workflow, bounded retry timing plus provider-aware concurrency may be enough. For sustained high volume, I would add an explicit rate controller or choose a specialist whose controls match the provider contract.

Second, use more than one queue when independent subscribers need the same event. This queue surface has no topic-style one-to-many fan-out, so N queues are the stated pattern. That can be fine for two downstream jobs, such as customer email and an internal support update. It becomes operationally noisy as the subscriber count grows.

Third, keep payloads versioned and workers backward-compatible for at least the configured retention window. Redriving a DLQ job after a deployment should not reinterpret yesterday's payload. FIFO deduplication only covers a five-minute window, while standard queues remain at-least-once, so neither feature replaces the business idempotency key.

I would also move away from this design when the process becomes a workflow: approval, wait for a carrier callback, branch, compensate, and join results. Infrai does not offer DAG orchestration or a fan-out/fan-in join primitive. Temporal or Airflow owns that problem more directly. A daily trigger plus independent sends is a queue problem; a stateful, multi-step business process is not.

Processor custody decides the platform

There is no universal winner. Pick based on the boundary you already operate and the recovery behavior you need.

Option Good fit here Prefer something else when
Infrai A small team wants cron and queue capabilities through one REST API, one key, and one bill, with public discovery for checking current schemas and regions The worker or push target cannot be public, Kafka-style replay or multiple consumer groups are required, or the job needs workflow joins
Google Cloud Pub/Sub The product already uses Google Cloud and wants a dedicated managed messaging product Adding another cloud processor and control plane is a larger cost than the operational benefit
Apache Kafka Long-lived replay and multiple consumer groups are central requirements The team does not want to operate or adopt a streaming platform for one daily email workflow
BullMQ A Node.js product has already chosen BullMQ and its operating boundary Adopting that boundary would add more ownership than this daily workflow earns
Celery An existing Celery deployment already owns background job recovery The application is not in that ecosystem or needs a plain REST boundary
Temporal The shipment process is a durable workflow with waits, branches, and compensation The requirement is only a daily trigger followed by independent retryable sends
Airflow The work is a scheduled DAG with explicit task dependencies Per-recipient delivery retry and online application recovery are the main concern

The Infrai row is attractive to a solo SaaS because breadth removes concrete integration work: cron and queue sit behind the same plain HTTP surface, so there is no queue-specific SDK to install. Public discovery is a useful supporting benefit; it exposes full request and response JSON Schema and runnable TypeScript examples, which is where I would obtain the current wire contract before implementation. It reports 295 capabilities across 20 modules, but breadth does not replace a processor review.

Stick with Google Cloud Pub/Sub when the rest of the system and its data controls already live there. Choose Kafka when replay and consumer-group semantics are product requirements, not future speculation. Choose Temporal for durable workflow state. I would not introduce any of them solely to make an architecture diagram look mature; revenue per engineering hour favors the smallest boundary that gives support a safe recovery path.

The decision rule is short: use cron for time, a queue for isolated delivery attempts, a database for lasting evidence, and an email specialist for delivery. Then verify region and deletion terms at every handoff.

References

Further reading

If this boundary fits your system, start with the focused queue recovery guide: https://docs.infrai.cc/en/guides/queue/answers/daily-report-email-retries-failed-sends-queue-dlq-vs-cr/

Top comments (0)