Short answer: treat a webhook subscription as an at-least-once delivery contract, ACK only after a shipment update is durably accepted, NACK overload with a bounded retry delay, and move terminal or exhausted deliveries to a dead-letter queue. For an edtech shipment fan-out, this keeps one slow school district endpoint from blocking every family waiting for a workbook update.
The central choice is the delivery guarantee. Exactly-once delivery across a queue and somebody else's HTTPS endpoint isn't a realistic transport promise: the consumer can commit an update and lose its response. Aim for at-least-once transport plus idempotent processing. Keep a stable event ID, track attempts per subscription, and make rate limiting part of the contract rather than an accidental timeout policy.
Duplicates are normal.
How should a webhook queue push to a rate-limited HTTPS endpoint consumer?
Picture one shipment.updated event for a box of lab kits. The publisher writes it once. A fan-out step creates one delivery record per active subscription. Workers lease due records, send each payload to its subscriber endpoint, and interpret the response as ACK, retryable NACK, or terminal rejection. The original event and each subscriber delivery need separate state; otherwise, adding a new subscriber or retrying a slow one can disturb deliveries that already succeeded.
The scheduler should only identify work that is due. It shouldn't hold a database transaction open while making an HTTPS call. In a Postgres implementation, workers can select a small due batch with FOR UPDATE SKIP LOCKED, mark those rows with a short lease, commit, and then perform network I/O. PostgreSQL documents that SKIP LOCKED gives an inconsistent view and is suitable for avoiding lock contention with multiple consumers on a queue-like table. That is exactly the narrow job here; it isn't a general-purpose consistency mode.
I use four durable identifiers in the eval fixture: event_id, subscription_id, delivery_id, and attempt. The first three don't change across a retry. The attempt count does. This is a small detail, but it stops dashboards from counting three attempts as three workbook shipments.
The state transition is compact:
| Consumer result | Queue action | Why |
|---|---|---|
| Contract ACK | Mark this delivery complete | The consumer durably accepted the event |
| Explicit overload NACK | Schedule another attempt after the declared delay | Backpressure is temporary |
| Other retryable outcome | Apply capped exponential backoff with jitter | Prevent synchronized retry waves |
| Invalid or unauthorized delivery | Dead-letter immediately | Repetition won't repair the request |
| Attempt budget exhausted | Dead-letter with the last outcome | Human or automated review now has context |
Don't let a bare timeout erase that distinction. A timeout means the result is unknown, so retrying is valid only because the consumer is idempotent.
A runnable Python consumer before the policy details
The following consumer is deliberately small. Put it behind a TLS-terminating reverse proxy so subscribers receive an HTTPS URL. Its local HTTP boundary keeps the sample focused on the delivery contract: validate the envelope, ACK duplicates, NACK excess load with 429, and return 204 only after acceptance. In production, replace the in-memory seen set with a durable unique key on (subscription_id, event_id) and hand accepted work to durable storage before responding.
import json
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
class TokenBucket:
def __init__(self, rate_per_second: float, capacity: int) -> None:
self.rate = rate_per_second
self.capacity = capacity
self.tokens = float(capacity)
self.updated_at = time.monotonic()
self.lock = threading.Lock()
def take(self) -> bool:
with self.lock:
now = time.monotonic()
elapsed = now - self.updated_at
self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
self.updated_at = now
if self.tokens < 1:
return False
self.tokens -= 1
return True
bucket = TokenBucket(rate_per_second=5, capacity=10)
seen: set[tuple[str, str]] = set()
seen_lock = threading.Lock()
class ShipmentWebhook(BaseHTTPRequestHandler):
def do_POST(self) -> None:
if self.path != "/webhooks/shipments":
self.send_error(404)
return
if not bucket.take():
self.send_response(429)
self.send_header("Retry-After", "2")
self.end_headers()
return
try:
length = int(self.headers.get("Content-Length", "0"))
event = json.loads(self.rfile.read(length))
required = {"event_id", "subscription_id", "type", "shipment"}
if not required.issubset(event) or event["type"] != "shipment.updated":
raise ValueError("invalid event envelope")
except (ValueError, json.JSONDecodeError):
self.send_error(400)
return
key = (event["subscription_id"], event["event_id"])
with seen_lock:
if key not in seen:
seen.add(key)
print(json.dumps(event["shipment"], sort_keys=True))
self.send_response(204)
self.end_headers()
def log_message(self, message: str, *args: object) -> None:
return
if __name__ == "__main__":
server = ThreadingHTTPServer(("127.0.0.1", 8080), ShipmentWebhook)
server.serve_forever()
For a Node.js consumer, the same observable contract applies even though the framework syntax changes: cap concurrent work, deduplicate before side effects, send the overload response with a retry hint, and ACK after durable acceptance. The sender shouldn't infer success from a TCP connection or from bytes leaving its process.
There is one sharp edge in this tiny program. It reserves a rate-limit token before discovering a duplicate. That is a conservative default because duplicates still consume request parsing and connection capacity, but a consumer dominated by legitimate retries may prefer a cheap authenticated deduplication lookup before the expensive-work bucket. Put both orderings in the load eval; don't choose by taste.
ACK, NACK, and dead-letter behavior are application contracts
ACK and NACK are queue vocabulary, not magic properties of HTTPS. Define their mapping explicitly. In this design, 204 is an ACK, 429 is a retryable NACK, an invalid envelope is terminal, and a connection with no usable response leaves the outcome unknown. Your mileage may vary if an existing subscriber platform owns those meanings. What matters is that the publisher and consumer share one versioned table of outcomes.
Retries need an attempt ceiling and an age ceiling. An attempt ceiling prevents one poisoned delivery from consuming capacity forever; an age ceiling recognizes that a workbook's “out for delivery” update may have little value after the shipment arrives. Store the next-attempt timestamp rather than sleeping inside a worker. Then a process restart, deployment, or scaling event doesn't reset the schedule. Backoff should be capped and include jitter. Suppose a district endpoint allows five accepted events per second and 600 students receive the same update. The first few deliveries consume the available bucket, while the rest receive the explicit overload NACK. If every worker retries exactly two seconds later, the queue has merely moved the spike; jitter distributes those due times, and a per-subscription concurrency limit prevents this district from taking every worker slot while other schools wait. A global cap still matters because connection pools and file descriptors are shared even when subscriber quotas aren't. During the run, some deliveries may reach the attempt or age ceiling. Those become dead letters, and dead letters are data, not a trash can: save the immutable event ID, subscription ID, attempt count, first and last attempt times, a bounded response classification, and the policy version that made the decision. Avoid storing secrets or an unrestricted response body. A replay operation should create a new delivery attempt linked to the dead-letter record, not edit history until the dashboard looks green. This single burst test checks pacing, fairness, expiry, evidence, and replay without pretending that a successful HTTP request proves the whole workflow.
This is also where prompt-cost awareness belongs in an AI-assisted operations stack. Don't send every retry body to a model for classification. Deterministic status mapping handles the normal path; a sampled, redacted dead-letter set can feed an eval or triage workflow. The model should inspect ambiguity, not meter routine backpressure.
Scheduling, leasing, and the failure window
Cron can wake a dispatcher, but cron alone isn't the queue. It is a time-based job scheduler, and the queue still needs durable due times, leases, and ownership. A one-minute cron cadence may be acceptable for a nightly course-material shipment digest; it is a poor fit for near-real-time tracking notifications. Use the latency objective to choose the wake-up mechanism, then keep the delivery state machine the same.
The hard failure window sits between the consumer's durable acceptance and the sender recording the ACK. If the sender loses the response, the lease expires and another worker pushes the event again. This is why the stable idempotency key is part of the guarantee, not optional cleanup. No amount of sender-side locking can prove what a remote consumer committed.
Late can be wrong.
Leases also need fencing. A worker that pauses past its lease must not overwrite the result written by a newer lease holder. Compare a lease token or version in the completion update. If zero rows match, discard the stale worker's result and let the current owner decide. Short leases improve recovery but create more false expiry under slow endpoints; long leases reduce duplicate overlap but delay recovery. I'm not sure there is a universal ratio between request timeout and lease duration. Resolve it from observed tail latency, deployment pause time, and the time required for a replacement worker to claim work.
The catch is that push delivery isn't suitable when subscribers cannot expose a stable HTTPS endpoint, need long offline periods, or require strict control over when they spend compute. Offer a pull queue in that case. Stick with a direct synchronous call when there is one internal consumer, the caller can wait, and failure should abort the originating transaction; fan-out machinery would add state without buying a useful guarantee.
Operate the guarantee, not just the endpoint
Before deployment, run a deterministic eval matrix against the sender and consumer together. Include a duplicated event_id, a burst above the five-per-second fixture limit, a delayed response that outlives a lease, malformed JSON, an expired subscription, and a dead letter replay. Assert durable state transitions and side-effect counts, not log text. One duplicate should still produce one shipment mutation. That's the invariant.
In production, graph queue age and due-delivery count by subscription class, then separate retryable NACKs from terminal dead letters. Attempt count alone is misleading: a healthy burst under a published subscriber limit can create retries without threatening the latency objective. Alert on sustained age against the shipment update's usefulness window, sudden terminal classifications, and lease-expiry overlap. Cap label cardinality; raw event IDs belong in traces or searchable logs, not metric labels.
Deployment should preserve policy compatibility. Workers running two versions may otherwise disagree about whether the same response is retryable. Store a policy version with each delivery, drain old workers, and rehearse replay with redacted fixtures. Keep payload size limits, authentication, signature verification, secret rotation, and retention in the same release checklist as rate limits. Delivery guarantees fail at boundaries — queue, network, consumer, and operator — so the evidence has to cross those boundaries too.
The final decision rule is brief. Choose at-least-once push when subscribers can run HTTPS endpoints and accept idempotency work; pair per-subscription rate limits with durable retries and inspectable dead letters. Choose pull when subscriber-controlled pacing matters more. Either way, measure the age of useful work, because a perfectly retried shipment update can still arrive too late to help a learner.
Top comments (0)