The short answer: put the HTTP cron task behind a durable work queue, make the consumer enforce the receiver's rate limit, and persist an idempotency record before acknowledging a successful delivery. Retries belong to the queue; duplicate prevention belongs to the application. Mixing those responsibilities is how an edtech enrollment event gets sent twice while the queue still reports healthy work.
This is a delivery problem.
For an enrollment system, the useful unit is an event such as enrollment.updated, identified by an immutable event_id. The scheduler should enqueue that event and return quickly. A worker claims it, waits for the next permitted send time, posts the payload, and records the receiver's result. If the worker disappears after the remote server accepts the request but before the local acknowledgement is durable, the next attempt must be allowed to repeat the HTTP call. The receiver, or a shared idempotency store, must make that repetition harmless.
How should I configure a queue for rate-limited webhook processing?
A cron schedule is a trigger, not a throughput policy. A task that runs every minute can release hundreds of enrollment callbacks at once if a previous run accumulated work. The queue absorbs that burst, but it does not automatically know whether the receiving learning platform permits 10 requests per second, 10 concurrent requests, or a different limit per tenant. Those are separate constraints.
I initially treated rate-limiting as a worker-count setting. Later I found that a fixed worker count controls concurrency, not the number of requests released over time. An HTTP-based webhook consumer needs a rate limiter, a concurrency limit, and a retry policy that can be inspected independently.
I start with three quantities: the maximum request rate, the maximum in-flight requests, and the retry delay after a refusal. A token bucket is a useful mental model for the first quantity. Tokens arrive at a fixed rate up to a bounded capacity; a send consumes one token. A semaphore models the second. The worker needs both when the receiver limits requests per second and concurrent connections independently.
The queue should therefore contain intent, not a precomputed series of future HTTP calls. A compact message might carry the event identifier, tenant identifier, destination key, attempt number, and a creation timestamp. The destination key selects the correct limiter. One noisy tenant should not consume the entire allowance for every school.
That is the trap.
No queue fixes that.
A dead-letter path matters here. A message that has exhausted its retry policy needs a visible terminal state, not an endless loop that hides a bad payload. Amazon's dead-letter queue guidance describes moving messages after a configured receive threshold so the failed work can be isolated for inspection. That mechanism does not decide whether a callback is safe to repeat; it only gives operations a boundary around repeated failure.
Why does a successful HTTP request still need a retry?
Because delivery has two acknowledgements: the remote HTTP response and the local record that says the response was handled. They are separated by a failure window. The worker can receive a 2xx response, lose its process, and fail to delete or acknowledge the message. On redelivery, treating the event as new creates a duplicate. Treating every redelivery as completed creates a lost notification when the first request never arrived.
This is the usual at-least-once trade-off. A queue can make work durable and retryable, but it cannot make a network call and a database write one atomic transaction. A practical design accepts possible repetition and makes the side effect idempotent. The event ID is the stable key; the attempt ID is not.
The receiving endpoint should store the event ID before applying a non-repeatable side effect, or perform both operations in one transaction when its storage model supports that. If the ID already exists, it can return the previously recorded outcome or a deliberate duplicate response. The exact response contract belongs to the two systems, but the invariant is simple: a retry of the same event must not create a second enrollment, invoice, or access grant.
Here is the shape I use for a worker boundary. It is intentionally generic. The limiter, queue client, and delivery store are interfaces because their durability and scope are architectural decisions, not properties to assume from an HTTP cron service.
from dataclasses import dataclass
from datetime import datetime, timezone
@dataclass
class Callback:
event_id: str
tenant_id: str
destination_key: str
payload: dict
attempt: int
def deliver(callback, limiter, delivery_store, http_client, queue):
limiter.wait(callback.destination_key)
if delivery_store.is_terminal(callback.event_id):
queue.ack(callback.event_id)
return
delivery_store.mark_attempt(
event_id=callback.event_id,
attempt=callback.attempt,
started_at=datetime.now(timezone.utc),
)
response = http_client.post(
callback.destination_key,
json=callback.payload,
headers={"Idempotency-Key": callback.event_id},
)
if 200 <= response.status_code < 300:
delivery_store.mark_terminal(callback.event_id, response.status_code)
queue.ack(callback.event_id)
return
if response.status_code == 429 or response.status_code >= 500:
queue.retry(callback, retry_after=response.retry_after)
return
delivery_store.mark_terminal(callback.event_id, response.status_code)
queue.ack(callback.event_id)
The ordering is deliberate. A terminal record is checked before work, an attempt is recorded before the network call, and the message is acknowledged only after the result is durable. The code still cannot prove that a 2xx response and mark_terminal happened as one transaction. That is why the remote side must honor the same event identity, or the system must place the side effect behind a transactional inbox.
How should rate limits and retry delays interact?
A limiter controls normal admission. A retry policy controls what happens after a request has already failed. If a worker sleeps inside a queue lease until a long backoff expires, the queue may redeliver the message while the first worker still owns it, or tie up a worker slot for no useful work. Prefer scheduling a future retry message, extending visibility only when the queue contract requires it, and keeping the delay observable.
For HTTP responses, distinguish the receiver asking for slower traffic from the receiver rejecting the request permanently. A 429 response is a rate signal; its Retry-After value can inform the next eligible time when present. A 5xx response usually calls for bounded exponential backoff with jitter. A 4xx response caused by an invalid payload should normally move toward a terminal failure instead of consuming the same retry budget. These rules need an explicit policy table, because a single “retry on error” branch is too blunt for enrollment data.
The queue's visibility or lease timeout should exceed the normal processing time, but it should not be used as the backoff clock. I would measure the 95th and 99th percentile processing duration, add room for the HTTP timeout, and then set a bounded lease. The exact value is workload-specific; a number copied from another system is not an engineering decision.
A useful retry envelope has four fields: event_id, attempt, next_attempt_at, and last_error_class. It lets dashboards separate rate pressure from bad data and lets a deployment resume work without reconstructing state from logs. Keep the original event creation time too. An event that has waited 45 minutes should not look fresh merely because its fifth attempt was scheduled now.
Which failure signals should operators trust?
Queue depth alone is weak evidence. A growing queue can mean the consumer is slow, the destination is throttling, one tenant is monopolizing tokens, or a payload is failing repeatedly. I want at least these correlated measurements: age of the oldest message, attempts per event, terminal successes and failures, 429 and 5xx counts by destination, limiter wait time, and the number of events in the dead-letter path.
The cardinality trap is real. Logging a full payload or an unbounded destination URL makes observability expensive and may expose student data. Log the event ID, tenant-safe identifier, destination key, status class, attempt, and timing fields. Store payloads under the data-retention rules for the system, with access controls separate from routine worker logs.
The alert should be based on user impact. For example, a rising oldest-message age combined with increasing limiter wait time points to capacity or an external rate limit. A stable queue with repeated terminal 4xx responses points to validation or contract drift. Those require different owners.
Testing should exercise the failure window rather than only the happy path. Inject a process stop after the remote mock records the request and before the queue acknowledgement. Deliver the same event again and assert one receiver-side effect. Then test a timeout where the mock cannot tell the worker whether it accepted the request. Finally, test a 429 with Retry-After, a permanent 400, and a message that reaches the dead-letter threshold. The important assertions are about state transitions, not just response codes.
A compact rollout plan for an existing cron task
First, assign every outbound event a stable ID at creation time and make the receiver's side effect idempotent. Without that contract, adding a queue changes timing but does not solve duplicate delivery.
Next, have the cron task enqueue bounded batches and record the enqueue result. A partial enqueue must be retryable without creating a second event, so the enqueue operation needs its own idempotency key or an outbox record. Keep the old direct sender disabled only after the new worker has a clear ownership boundary; two active senders are an easy way to double traffic.
Then introduce the limiter with a conservative ceiling, measure actual receiver responses, and raise capacity only when the evidence supports it. Partitioning by destination or tenant is usually more useful than one global bucket, but it increases configuration and monitoring work. Choose that complexity when isolation is part of the requirement.
Finally, rehearse dead-letter recovery. A replay tool should require an explicit event selection and preserve the original event ID. Replaying with a new ID defeats duplicate protection. Keep the retry and terminal records long enough to investigate a delivery dispute, subject to the application's retention policy.
The decision rule is straightforward: use the queue to preserve intent, the limiter to admit work, retry state to explain delay, and an idempotency key to make repetition safe. A cron trigger can start the process, but it should not carry the reliability policy.
Top comments (0)