Short answer: use one daily scheduler to create durable delivery records, then let a rate-aware queue push each record to an idempotent public HTTPS subscriber; choose an in-process scheduler only when losing or manually replaying a run is acceptable.
That split gives a fintech team a clean recovery unit: one report delivery, with one stable identity. Scheduling decides when work exists. The queue decides when work may run. The subscriber decides whether that delivery has already completed. Keep those decisions separate and a provider slowdown becomes a controlled backlog, not a duplicate-email incident.
What must a public HTTPS subscriber reveal about a rate-limited daily report email queue?
Start with the smallest design whose failure mode the team can operate:
| Design | Pick this when | Recovery story | Main catch |
|---|---|---|---|
| Scheduler calls the sender | The batch is small and a missed run can be replayed by hand | Rerun the dated batch | Process restarts can blur which recipients completed |
| Scheduler fills a local durable queue | One service owns dispatch and has stable storage | Resume pending records after restart | Scaling and storage maintenance stay with that service |
| Scheduler fills a queue that pushes to a public HTTPS subscriber | Dispatch needs independent pacing, retries, and worker deployment | Redeliver the same stable delivery identity | The public ingress needs authentication, capacity controls, and its own monitoring |
For a regulated daily report, the third row is usually the practical default because operational recovery is the primary decision axis. It is not automatically the best choice. A team sending a tiny internal digest may gain nothing from a separately operated queue and public endpoint. The local option is easier to reason about when a single owner, a single process, and a manual replay are genuinely enough.
Here is the diagram in words: daily clock -> batch manifest -> durable delivery record -> rate gate -> HTTPS subscriber -> mail provider -> completion record. The manifest answers “who should have received the report for this business date?” The completion record answers “who did?” Those are much better recovery questions than “did yesterday's cron run?”
The distinction matters. A scheduler can fire exactly on time while half the sends remain unfinished. Conversely, a delayed worker can safely finish yesterday's batch if every delivery retains its business date and identity. Don't make clock success stand in for delivery success.
What privacy and retention rules belong in the dispatch contract?
The recovery contract starts with data ownership. Decide which system is authoritative for the recipient list, which immutable fields form a delivery identity, which fields may enter the queue payload, and how long each state remains queryable. A queue message should carry the identifiers needed to locate approved report content, not become an ungoverned copy of the report. The operations view can expose delivery state without automatically exposing message contents.
Write the operator's allowed actions into that contract. Can an operator replay one recipient, a tenant, or an entire report date? Does replay require approval? Which audit fields record the request and its reason? These questions are independent of transport. Answering them first prevents a “retry all” button from becoming the only recovery tool after a partial dispatch.
The manifest is the useful boundary. Persist it before any delivery attempt and give it an explicit business date. A direct scheduled job can then mark recipients complete. A queued design can materialize one delivery record per manifest row. Both designs retain the same evidence: intended recipients, observed outcomes, and authorized replays.
Short jobs are lovely.
How can retries preserve one report delivery identity?
A task queue exists to distribute units of work to workers, and workers can monitor a queue for new work. That basic separation, described in the Celery introduction, is useful even when the implementation is not Celery. The important unit here is report-delivery, not daily-report-batch. Small units let the dispatcher pace sends, retry one recipient, and inspect exactly what remains.
Model every delivery with a stable ID derived from immutable business inputs such as tenant, report date, report kind, and recipient. Enforce that identity with a unique constraint in durable storage. A retry then carries the same ID; it does not create a fresh send request. This is the core duplicate-defense boundary. The queue may present the work again, and two subscriber requests may overlap, but only one state transition may claim an uncompleted delivery.
Pacing belongs before the provider call. A worker first acquires capacity from a shared rate gate, then attempts the send. If capacity is unavailable, it reschedules without counting that wait as a provider attempt. After a retryable response, exponential backoff spreads later attempts over time rather than immediately adding pressure. Add jitter so a large set of records does not wake at the same instant. The exact base delay and cap are policy decisions; I'm not sure one default can fit both a two-minute operational report and a report whose usefulness lasts all day. The report's delivery objective resolves that choice.
Public HTTPS push changes one boundary, not the whole design. The queue initiates the request, so the subscriber must authenticate it, reject stale or replayed signatures according to the team's security policy, and return only after it has durably decided what happened to the delivery. A pull worker avoids public ingress and may suit a private network better. Stick with pull when public reachability conflicts with the threat model or when workers need long-lived local access to internal data.
Watch the queue as a timeline, not a single depth number. Useful signals include age of the oldest ready delivery, ready count by report date, retry count by reason, completions per minute, and the gap between scheduled and completed deliveries. Alert on the user-facing objective: an old undelivered report. A deep queue that is draining within its delivery window can be healthy; a shallow queue containing one stuck high-value report can be urgent.
Test ambiguous outcomes with executable interfaces
The following TypeScript keeps infrastructure behind small interfaces. It is intentionally the subscriber's decision core, not a complete web server. Authentication and body-size enforcement should happen before this function, while storage supplies atomic claim and completion operations.
type Delivery = {
id: string;
tenantId: string;
reportDate: string;
recipient: string;
attempt: number;
};
type Claim =
| { kind: "acquired" }
| { kind: "already-complete" }
| { kind: "busy" };
interface DeliveryStore {
claim(id: string): Promise<Claim>;
complete(id: string, providerMessageId: string): Promise<void>;
release(id: string, reason: string): Promise<void>;
}
interface RateGate {
take(tenantId: string): Promise<boolean>;
}
type SendResult =
| { kind: "sent"; providerMessageId: string }
| { kind: "retry"; reason: string }
| { kind: "rejected"; reason: string };
interface ReportSender {
send(delivery: Delivery): Promise<SendResult>;
}
interface RetryQueue {
later(delivery: Delivery, delayMs: number): Promise<void>;
deadLetter(delivery: Delivery, reason: string): Promise<void>;
}
const retryDelayMs = (attempt: number): number => {
const baseMs = 1_000;
const capMs = 15 * 60_000;
const exponential = Math.min(capMs, baseMs * 2 ** attempt);
return Math.floor(exponential * (0.75 + Math.random() * 0.5));
};
async function deliverReport(
delivery: Delivery,
store: DeliveryStore,
rateGate: RateGate,
sender: ReportSender,
retries: RetryQueue,
): Promise<"complete" | "deferred" | "rejected"> {
const claim = await store.claim(delivery.id);
if (claim.kind === "already-complete") return "complete";
if (claim.kind === "busy") return "deferred";
if (!(await rateGate.take(delivery.tenantId))) {
await store.release(delivery.id, "local-rate-limit");
await retries.later(delivery, 1_000);
return "deferred";
}
const result = await sender.send(delivery);
if (result.kind === "sent") {
await store.complete(delivery.id, result.providerMessageId);
return "complete";
}
await store.release(delivery.id, result.reason);
if (result.kind === "rejected") {
await retries.deadLetter(delivery, result.reason);
return "rejected";
}
await retries.later(
{ ...delivery, attempt: delivery.attempt + 1 },
retryDelayMs(delivery.attempt),
);
return "deferred";
}
There is a sharp edge in any “send, then mark complete” flow: the provider can accept a message before the completion write is visible. Retrying blindly can duplicate the email. The cleanest resolution is to pass the stable delivery ID through a provider-supported idempotency mechanism. If that mechanism is unavailable, the application cannot honestly promise exactly-once external delivery; it can reduce risk with a durable send ledger, reconciliation against provider message identifiers, and an operator review state for ambiguous outcomes. This limitation should appear in the runbook, not hide behind the queue abstraction.
Test the awkward orderings. Invoke two deliveries with the same ID concurrently and assert that only one sender call occurs. Simulate capacity denial and confirm that the attempt count does not rise. Exercise a retryable result, then a successful result, and verify that a later duplicate becomes already-complete. Finally, freeze time around the business-date boundary. A report identity based on local wall-clock time can quietly move into the wrong date during timezone or daylight-saving changes; pass the intended business date into the batch explicitly.
Deployment deserves the same care. Ship the schema and read path before enabling the scheduler. Start workers with sending disabled, observe that manifests and delivery records look right, then enable a small tenant cohort. A crisp before/after dashboard should show scheduled records becoming ready, ready records becoming active, and active records becoming complete. Keep dead-letter replay a deliberate command that preserves the original delivery ID.
Roll out the smallest operable system
The catch is that a queue does not create an exactly-once email channel, decide which failures are retryable, or define the acceptable delivery window. Those are application policies. A public HTTPS subscriber also introduces certificate, authentication, ingress, and request-capacity work. If the endpoint cannot be exposed under the team's security rules, use a pull worker on the private side.
Start the rollout with manifest creation and sending disabled. Inspect one report date, verify the intended recipient count, and exercise an authorized replay in a non-delivering environment. Enable a small cohort next, watching oldest-delivery age and ambiguous outcomes before expanding. The rollback is simple: stop creating new sendable records while preserving the manifest and delivery ledger for inspection.
Prefer direct scheduling when the batch is small, visible, and safely replayed by a person. Prefer a local durable queue when one service can own persistence and dispatch. Prefer queue-to-HTTPS push when independent pacing and deployment are worth the public boundary. In every version, the recovery test is the same: can an operator name one business date, see every intended recipient, distinguish complete from ambiguous work, and replay only what is safe?
That's the decision rule.
Top comments (0)