For a fintech shipment update fanout, the least complex reliable choice is a nightly cron trigger that batches pending webhooks into a durable queue, then lets workers publish with idempotency and backoff. Keep the trigger boring. Put the cleverness in observability and recovery.
Short answer: use a queue when delayed webhook delivery can tolerate seconds or minutes of waiting; use a direct publish path only when the subscriber contract demands sub-second latency and you can absorb its cost.
The dispatch decision in one table
| Pattern | Pick this when | Main cost | Failure to watch |
|---|---|---|---|
| Direct publish from the transaction | One or two subscribers, strict latency, short request budgets | Coupled availability and request time | A slow endpoint delays the database request |
| Transactional outbox plus queue | Events must not disappear when a commit succeeds | A relay and queue to operate | Duplicate delivery after a retry |
| Nightly cron batch into a queue | Pending work is acceptable overnight and spend is the primary constraint | Delivery is intentionally delayed | A growing backlog hides until morning |
| Workflow scheduler | Delays, branches, and human review are part of the process | More state and operational surface | State-machine complexity exceeds the job |
The table is a decision aid, not a leaderboard. A hosted task service, BullMQ backed by Redis, and Temporal-style workflows all solve different slices of the same problem. Their APIs and durability guarantees differ, so compare the contract you need rather than the logo on the dashboard.
How should a cron trigger queue pending webhooks for a nightly batch?
Start with a durable record. When the shipment status changes, write an outbox row in the same database transaction as the status update. A separate publisher claims rows that are ready, creates a queue message, and marks the row as enqueued. This is the transactional outbox pattern: the business write and the intent to publish commit together, while delivery happens later.
The cron trigger is only a nudge. It should select a bounded batch, such as 500 rows, and enqueue work with a stable message key. It must be safe to run twice. A lock or lease prevents two schedulers from claiming the same rows, but the worker still needs idempotency because queues commonly provide at-least-once delivery.
Here is a small Node.js example. It uses generic interfaces so the same shape can sit over a cloud task queue, Redis-backed queue, or an internal broker.
type PendingWebhook = {
id: string;
shipmentId: string;
endpoint: string;
payload: unknown;
attempts: number;
};
interface PendingStore {
claimReady(limit: number, leaseSeconds: number): Promise<PendingWebhook[]>;
markEnqueued(id: string, messageId: string): Promise<void>;
markDelivered(id: string): Promise<void>;
reschedule(id: string, nextAttempt: Date): Promise<void>;
}
interface Queue {
enqueue(messageId: string, body: PendingWebhook): Promise<void>;
}
export async function nightlyEnqueue(store: PendingStore, queue: Queue) {
const batch = await store.claimReady(500, 300);
for (const item of batch) {
const messageId = `shipment-webhook:${item.id}`;
await queue.enqueue(messageId, item);
await store.markEnqueued(item.id, messageId);
}
}
One small detail matters: the message ID comes from the outbox ID, not from the current time. That gives you a deduplication key you can inspect in logs. If enqueue succeeds and the process dies before markEnqueued, the next cron run may publish the same item again. That is expected. Make the receiver—or your delivery ledger—treat item.id as an idempotency key.
What does the worker publish, retry, and measure?
The worker owns the network call. It sets a bounded timeout, sends an idempotency key, and records the response class. A 2xx response marks delivery. A timeout or 5xx response gets exponential backoff with jitter. A 4xx response usually needs quarantine and an operator decision; retrying a permanently invalid signature forever just burns queue capacity.
export async function deliver(
item: PendingWebhook,
store: PendingStore,
fetcher: typeof fetch,
) {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), 8_000);
try {
const response = await fetcher(item.endpoint, {
method: "POST",
headers: {
"content-type": "application/json",
"idempotency-key": item.id,
},
body: JSON.stringify(item.payload),
signal: controller.signal,
});
if (response.ok) {
await store.markDelivered(item.id);
return;
}
if (response.status >= 400 && response.status < 500) {
await store.reschedule(item.id, new Date("2099-01-01T00:00:00Z"));
return;
}
const delaySeconds = Math.min(3_600, 2 ** item.attempts * 30);
await store.reschedule(item.id, new Date(Date.now() + delaySeconds * 1000));
} finally {
clearTimeout(timer);
}
}
The 2099 date is a readable quarantine sentinel for this example; production code should use an explicit dead-letter state. Do not hide that distinction in a generic retry helper.
Measure the path as a timeline: outbox_created, queue_enqueued, worker_started, request_finished, and delivered. The useful metrics are age of the oldest pending row, batch size, enqueue lag, delivery latency, retry count, and dead-letter count. Alert on age and error rate, not on a single slow request. Logs should carry shipment ID, webhook ID, message ID, attempt, and a trace ID. That lets an on-call engineer answer “where did this update stop?” without replaying the entire night.
Which queue shape fits latency versus cost?
If customers watch a shipment screen in real time, a nightly batch is the wrong shape. Use the outbox relay continuously and let workers drain small messages; the extra queue operations buy lower latency. If the webhook is a reconciliation hint consumed by a back-office job, a 15-minute or nightly batch can be rational. Your mileage may vary when subscribers are spread across time zones or have contractual delivery windows.
A queue service usually charges for requests and retained work; Redis-backed queues shift cost toward always-on capacity and operations; workflow engines charge for state transitions and add visibility for multi-step processes. I’m not sure which line item dominates for your traffic until you chart messages per shipment, retry probability, and retention hours. Cost belongs beside latency in the design review, never in place of a delivery guarantee.
The catch is operational ownership. A cron plus queue is not suitable when each webhook needs human approval, calendar-aware waits, or a long saga; stick with a workflow engine then. Conversely, a workflow engine is excessive for one POST and one retry policy. Keep the state model proportional to the failure modes you can actually explain.
Limits and a practical rollout
Begin in shadow mode: enqueue copies, measure lag, and compare the worker's result with the existing publisher. Then enable one subscriber group, cap concurrency, and document the replay command. Test duplicate messages, a full queue, a rotated signing key, and a database lease that expires halfway through a batch. These are ordinary paths, not exotic chaos tests.
Measure twice.
For a useful rehearsal, freeze the subscriber endpoint for ten minutes while a batch is in flight. The queue should retain the messages, the worker should record timeouts without losing payloads, and the age-of-oldest metric should climb. Restore the endpoint, watch retries drain, and verify that the delivery ledger contains one terminal result per webhook ID even if the same message was attempted several times. Run that exercise with 50 records first, then with the largest batch your database lease can hold; the point is to validate the handoff and the alert, not to manufacture a benchmark.
There is no exactly-once network delivery. There is only an idempotent receiver, a durable intent, and enough evidence to replay safely. If the business cannot accept delayed updates at all, move the decision earlier: publish from the committed outbox continuously and design the subscriber contract around bounded latency.
References
Further reading:
Top comments (0)