Short answer: accept each card-shipment webhook once, expand its subscriptions into a durable queue, and push to every subscriber's HTTPS endpoint with rate limiting; ACK only after delivery is recorded, and NACK permanent failures into a dead-letter state.
For a fintech team sending shipment updates, the latency-versus-cost decision belongs at the delivery boundary. A synchronous fan-out gives the first few subscribers a quick notification, but one slow HTTPS endpoint makes the producer wait and turns subscriber count into request latency. A durable queue adds storage and worker work, yet it lets each subscriber move at a pace that respects its own limit.
The decision here is specific: use a durable fan-out queue for normal delivery, an explicit ACK/NACK contract for worker outcomes, and a scheduled repair pass only for records that have stopped making progress. The three boundaries are acceptance, delivery, and recovery. Mixing them is how duplicate messages, retry storms, and missing notifications hide inside an apparently simple webhook handler.
The invariants and failure boundaries
The source event is a shipment transition such as card_shipment.dispatched, not an instruction to send an HTTP request immediately. Give it a stable event ID, the card shipment ID, an occurred-at timestamp, and a non-secret payload suitable for every authorized subscriber. The producer transaction stores that event once. A separate expansion step creates a delivery record for every subscription that was active at the chosen membership cutoff.
That cutoff matters. Suppose a card leaves the fulfillment center at 14:03, a subscriber is disabled at 14:04, and expansion runs at 14:05. Sending according to current subscription state may suppress an event the subscriber was entitled to receive; sending according to event-time state requires a versioned subscription snapshot. There isn't a universal answer. The API contract must choose one meaning and preserve the data needed to reproduce it.
Four invariants keep the design honest:
- One source event can create at most one delivery record for a given subscription.
- A delivery can be leased by only one worker at a time, but an expired lease can be reclaimed.
- A successful subscriber response is recorded before the delivery is considered complete.
- A permanent failure is retained with a reason and payload reference rather than silently discarded.
Exactly-once HTTP delivery isn't the promise. The useful promise is durable at-least-once delivery plus a stable idempotency key, so a subscriber can recognize a repeat after an ambiguous timeout. Keep the event ID in a header and the body; signatures should cover the bytes actually sent, a timestamp, and enough context to prevent a valid signature from being reused for a different delivery. Don't put a full card number, authentication secret, or one-time passcode in a shipment payload. A retry queue is long-lived data, and compliance scope follows the data into logs, backups, and dead-letter storage.
Duplicates happen.
The failure boundaries are equally concrete. Producer failure must not leave a committed business transition without its durable event. Expansion failure must be resumable without creating a second delivery. Worker failure must release or expire a lease. Subscriber rejection must become a classified outcome, not a generic exception. A recovery scan may repair stranded state, but it must never become the main delivery engine.
Keep those boundaries boring.
How should an HTTPS endpoint ACK or NACK a rate-limited queue push?
Define ACK and NACK as application outcomes, then map them to the HTTPS exchange. In this design, an ACK means the subscriber accepted responsibility for the event. A NACK means the attempt did not complete and the delivery record needs either another eligible time or a terminal state. HTTP status alone can't tell the worker whether an arbitrary 4xx is permanent, so the subscription contract needs a small, documented classification policy.
One workable policy treats 200 through 299 as ACK, 408, 429, and network timeouts as retryable NACK, and other 400 through 499 responses as permanent NACK after the endpoint and credentials have been checked. A 429 response may include Retry-After; the worker should combine that subscriber signal with its own attempt ceiling and scheduling rules. This is a policy, not a law of HTTP, and teams should change it when their subscriber agreement defines different semantics.
The awkward case is a timeout after the remote endpoint has persisted the event. Retrying can duplicate the side effect; marking it complete can lose the event. There is no observation available to the sender that distinguishes those two worlds. The idempotency key resolves the ambiguity at the receiver, while the sender retries conservatively. Short and sharp: timeout means unknown.
Rate limiting belongs per subscriber, not only at the worker pool. A global worker ceiling protects the sender's CPU and sockets, while a token bucket or next-eligible timestamp per subscription prevents one high-volume tenant from consuming every delivery slot. Add bounded random jitter to retry times so thousands of deliveries rejected in the same second don't return in the same second. The exact retry delay depends on the subscriber agreement and measured recovery behavior; I'm not sure a fixed sequence is defensible without those two inputs.
The dead-letter rule also needs precision. Exhausted attempts, an explicitly permanent response, an invalid endpoint configuration, or a payload that cannot satisfy the agreed schema can move a delivery to a terminal review state. Dead-lettering isn't deletion. Store the event identity, subscription identity, classification, attempt count, and last response metadata with strict retention and access controls. For fintech payloads, response bodies deserve the same scrutiny as request bodies because downstream systems sometimes echo submitted data.
Comparing the dispatch mechanisms
The options differ less by syntax than by who owns backpressure and recovery. Use the table as an architecture record, not a universal ranking.
| Mechanism | Latency profile | Cost profile | Failure ownership | Best fit |
|---|---|---|---|---|
| Synchronous fan-out in the producer | Immediate for early calls; total time grows with slow or numerous endpoints | No separate queue, but producer capacity waits on remote I/O | Producer must classify, retry, and persist partial progress | A tiny, fixed subscriber set with a strict low-latency need |
| Durable push-delivery queue | Adds a queue hop; workers can dispatch as capacity becomes available | Queue storage and delivery attempts are explicit | Queue and worker own leases, pacing, retries, and terminal outcomes | Variable fan-out with independent subscriber limits |
| Database-backed pull workers | Poll interval can add delay; polling can be tuned | Reuses database operations but adds scans, locks, and retained rows | Application owns claiming, leases, cleanup, and retry scheduling | Moderate volume when operational simplicity outweighs specialized queue features |
| Scheduled batch sweep | Latency follows the schedule interval | Work is concentrated into periodic batches | Scheduler and batch job own the whole retry window | Non-urgent reconciliation and repair |
The cost model should count attempts, not events. If one shipment event has 800 subscribers and 5% of destinations need one retry, the system performs 840 outbound attempts. That number is example arithmetic, not a benchmark. It shows why subscriber fan-out, retry rate, payload bytes, queue retention, and worker time belong in the estimate. Latency should be measured as a distribution from event commit to subscriber ACK, with queue wait separated from network time; a single average hides a quiet subscriber stuck behind a noisy one.
For the primary path, the durable push-delivery queue wins this record because subscriber count varies and each endpoint has an independent rate limit. The catch is operational overhead: it requires durable storage, concurrency controls, retry policy, and dead-letter review. A synchronous path is still valid when there are perhaps two tightly controlled internal subscribers, every call fits inside the producer's latency budget, and partial progress can be represented safely. A database queue is attractive when the team already operates the database well and the load is moderate, but polling and cleanup need capacity planning.
The critical path in Python
The following worker code is intentionally generic. DeliveryStore owns atomic leasing, Limiter owns subscriber pacing, and HttpClient owns the HTTPS exchange. The separation makes the two acknowledgements visible: the queue lease is completed only after the remote subscriber has ACKed and the result has been persisted.
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Protocol
class Outcome(Enum):
ACK = "ack"
RETRY = "retry"
DEAD_LETTER = "dead_letter"
@dataclass(frozen=True)
class Delivery:
delivery_id: str
event_id: str
subscription_id: str
endpoint: str
payload: bytes
attempt: int
@dataclass(frozen=True)
class Response:
status: int
retry_after_seconds: int | None = None
class DeliveryStore(Protocol):
def lease_next(self, now: datetime) -> Delivery | None: ...
def mark_acked(self, delivery_id: str, status: int) -> None: ...
def retry_at(self, delivery_id: str, when: datetime, reason: str) -> None: ...
def dead_letter(self, delivery_id: str, reason: str) -> None: ...
class Limiter(Protocol):
def next_slot(self, subscription_id: str, now: datetime) -> datetime: ...
class HttpClient(Protocol):
def post(self, endpoint: str, body: bytes, headers: dict[str, str]) -> Response: ...
def classify(response: Response) -> Outcome:
if 200 <= response.status < 300:
return Outcome.ACK
if response.status in {408, 429}:
return Outcome.RETRY
if 400 <= response.status < 500:
return Outcome.DEAD_LETTER
return Outcome.RETRY
def deliver_one(
store: DeliveryStore,
limiter: Limiter,
client: HttpClient,
now: datetime,
) -> bool:
delivery = store.lease_next(now)
if delivery is None:
return False
eligible_at = limiter.next_slot(delivery.subscription_id, now)
if eligible_at > now:
store.retry_at(delivery.delivery_id, eligible_at, "subscriber_rate_limit")
return True
headers = {
"Content-Type": "application/json",
"Idempotency-Key": delivery.event_id,
"X-Event-Type": "card_shipment.dispatched",
}
response = client.post(delivery.endpoint, delivery.payload, headers)
outcome = classify(response)
if outcome is Outcome.ACK:
store.mark_acked(delivery.delivery_id, response.status)
elif outcome is Outcome.DEAD_LETTER:
store.dead_letter(delivery.delivery_id, f"permanent_http_{response.status}")
else:
delay = response.retry_after_seconds or min(15 * (2 ** delivery.attempt), 3600)
store.retry_at(
delivery.delivery_id,
now + timedelta(seconds=delay),
f"retryable_http_{response.status}",
)
return True
def run_once(store: DeliveryStore, limiter: Limiter, client: HttpClient) -> None:
now = datetime.now(timezone.utc)
while deliver_one(store, limiter, client, now):
now = datetime.now(timezone.utc)
Production code needs two refinements around this skeleton. First, network exceptions belong in the retryable unknown class, with the lease allowed to expire if the process dies before it can record the retry. Second, the attempt ceiling must be enforced atomically by the store, because two workers can otherwise make different decisions around an expired lease. The database claim can use row locking with SKIP LOCKED so concurrent workers skip rows already held by another transaction; the PostgreSQL documentation also warns that this produces an inconsistent view, which is acceptable for queue-like access but not for general reporting.
Observability should follow delivery identity across boundaries. Record queue age, attempt count, subscriber pacing delay, request duration, ACK class, and dead-letter reason. Alert on age and terminal-rate changes rather than raw queue depth alone: a large queue draining within its objective may be healthy, while a small old queue can represent a stuck subscription. Logs should use endpoint identifiers instead of full URLs when URLs contain tenant-specific paths or credentials.
Deploy policy changes like code. A new retry classification can replay traffic that was previously terminal, so stage it against captured response metadata, cap concurrency, and watch age by subscriber cohort. Contract tests should cover duplicate event IDs, timeouts after acceptance, 429 with and without Retry-After, lease expiry, a worker crash after remote ACK, and replay from dead-letter storage. Those are the places where the tidy happy path stops helping.
Why a cron-driven dispatcher was rejected
Cron is a time-based job scheduler. It is useful here for a periodic reconciliation task that finds deliveries whose lease expired or whose state violates an invariant. It is not suitable as the primary dispatcher when shipment updates must fan out promptly, because delivery latency becomes coupled to the schedule interval and each batch competes for a burst of outbound capacity.
Stick with a scheduled batch when the notification is genuinely non-urgent, the subscriber set is small, and the business accepts latency bounded by the run interval. Daily shipment summaries are a plausible example. For event-by-event card dispatch updates, keep scheduling in the recovery plane and let the queue expose backpressure continuously.
The final decision rule is plain: choose the durable queue when variable fan-out and per-subscriber rate limiting dominate; choose synchronous delivery only for a tiny controlled set with a proven latency budget; choose database pull workers when their operational trade-off is understood; and reserve cron for repair or deliberately delayed batches. ACK after durable acceptance, NACK ambiguous or transient outcomes into a bounded retry, and retain permanent failures for controlled review.
Top comments (0)