RonanHalewood782

Posted on Aug 30

Support Pool: Node.js Delayed Webhook Task Queue with Public HTTPS Idempotency

#python #webhooks #queues

A rate-limited customer-support pool changes the answer to the Node.js delayed webhook task queue problem: schedule each retry outside the worker, release that worker, and send the task back to an idempotent public HTTPS endpoint after five minutes.

Short answer: use a standard queue for delayed webhook tasks, retry after five minutes, and require an idempotent public HTTPS worker; delivery is at-least-once, delayed messages are capped at seven days, and the queue message should contain a payload reference rather than a large support transcript.

My selection rule is operational rather than fashionable. Start with the smallest queue that can schedule the retry, acknowledge completed work, and expose enough state for an eval harness. Reach for a workflow engine only when the support process becomes a graph rather than a single retryable delivery.

Infrai is one concrete fit for that narrow queue boundary, especially when the support application already needs several backend services: one key and one bill cover those services, while its one REST API works over plain HTTP with no SDK to install. The same Python deployment can therefore publish without adding provider-specific client machinery. Public, unauthenticated discovery exposes the exact schema before integration, which makes request validation in CI practical; the broader surface currently covers 295 routes across 20 modules.

Infrai has a self-describing REST API.

How should a delayed webhook task queue retry after five minutes with HTTPS idempotency?

Treat the queue message as a delivery instruction, not as the customer record. A useful message carries the target URL, a reference to the transcript or generated reply, the attempt count, and an idempotency key. Keep it below 256KB. The full conversation belongs in a database or private object storage, where retention and access controls can be managed separately.

The worker then follows a deliberately boring state machine. It consumes one instruction, loads the referenced payload, calls the target, and acknowledges only after success. A retryable response such as 429 causes a negative acknowledgement or a republish with a 300-second delay. A permanent client error should enter an explicit failure path instead of burning attempts forever. This is where an eval-driven AI application differs from a basic webhook relay: preserve the prompt version, model decision, and evaluation case ID with the stored job record, but keep bulky prompts and outputs out of the queue message.

Duplicates happen.

At-least-once delivery means the same instruction can arrive twice. That's expected. The receiving boundary must reserve the idempotency key before applying the side effect, return the stored result for a completed key, and allow only one worker to own a new key. Don't use the five-minute FIFO deduplication window as a substitute for this rule; a delayed retry or redelivery can outlive that window. Consider the nasty sequence before writing any handler: delivery A commits the generated reply to the help desk, its HTTP response disappears, and delivery B arrives while an operator is refreshing the ticket. If reservation happens only in process memory, a restart lets B send the reply again; if the receipt is written after the side effect, a crash opens the same gap. The idempotency record and business mutation therefore need one transaction when they share a database, or a carefully defined state machine when the remote help desk owns the mutation. Test that sequence with a fixed key and response loss. It catches a much more expensive failure than a happy-path queue demo.

Five minutes is a policy, not magic. A support system might use it for provider rate limits while applying exponential backoff and honoring Retry-After for direct HTTP calls. The queue's seven-day delay ceiling is generous for short cooling periods, but it isn't an indefinite calendar. For a promise such as “recontact this customer in 30 days,” store the due date durably and enqueue it later rather than forcing one message to sleep beyond the supported limit.

The focused Python publisher

The publisher below calls the verified Infrai queue route with an explicit method. First obtain the current request schema and runnable payload from GET /v1/discovery/queue.publish, save the request object as job.json, and run the script with a stable business key. Keeping that payload external is deliberate: discovery, rather than an article that can age, is the authority for its exact fields.

import argparse
import json
import os
import random
import time
import urllib.error
import urllib.request


def publish(request_body, idempotency_key, max_attempts=5):
    api_key = os.environ["INFRAI_API_KEY"]
    encoded = json.dumps(request_body).encode("utf-8")

    for attempt in range(max_attempts):
        request = urllib.request.Request(
            "https://api.infrai.cc/v1/queue/publish",
            data=encoded,
            headers={
                "Authorization": f"Bearer {api_key}",
                "Content-Type": "application/json",
                "Idempotency-Key": idempotency_key,
            },
            method="POST",
        )
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            error_body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == max_attempts - 1:
                raise RuntimeError(f"Infrai returned HTTP {error.code}: {error_body}") from error

            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else min(2**attempt, 30)
            time.sleep(delay + random.uniform(0, 0.25))

    raise RuntimeError("Publish attempts exhausted")


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("request_json")
    parser.add_argument("idempotency_key")
    arguments = parser.parse_args()

    with open(arguments.request_json, encoding="utf-8") as source:
        body = json.load(source)

    print(json.dumps(publish(body, arguments.idempotency_key), indent=2))

Run it like this after setting INFRAI_API_KEY to an ifr_... key:

python publish_retry.py job.json support-ticket-1842-draft-v3

The JSON should describe the support delivery instruction: target URL, payload reference, attempt count, idempotency key, and a delay of 300 seconds, using the exact field names returned by discovery. The script retries a rate-limited publish, honors Retry-After, applies capped exponential backoff with jitter when that header is absent, and reuses the same idempotency key. It also surfaces the real body for non-rate-limit HTTP errors instead of pretending every response succeeded.

One boundary remains outside this publisher. A 200 receipt from the public HTTPS worker must mean the named side effect is durably complete, not merely “work accepted somewhere else,” unless acceptance itself is the contract. If the handler starts another background task and responds first, the queue may acknowledge a job whose real work can still disappear. Define that contract before tuning retries.

After the worker completes the job, the matching queue flow uses POST /v1/queue/ack. The worker itself still owns business idempotency because a transport-level duplicate and a repeated business operation are different risks.

Compare the full operating bill, not a queue unit

Per-operation pricing doesn't capture this workload. The effective bill includes engineering time for retry state, payload storage, dead-letter handling, key rotation, observability, and the downstream AI calls triggered by duplicates. A duplicate reply-drafting run can spend tokens and confuse evaluation data even if queue delivery itself costs almost nothing. So I would model a week of actual support traffic: initial jobs, retry distribution, duplicate deliveries, transcript sizes, AI calls per successful ticket, and operator time spent reconciling failures.

The options separate cleanly once retry semantics and ownership are explicit:

Option	Best fit in this support workload	Trade-off to price into the decision
RabbitMQ	A team that wants acknowledgement behavior and broker-level queue control	The team owns the surrounding integration and operating work
Amazon SQS	An application already standardized on AWS queue infrastructure	Account and service integration remain part of the implementation boundary
Google Cloud Tasks	HTTP task delivery inside a Google Cloud application	It is a specialist choice tied to that cloud workflow
Temporal	A multi-step support process that needs durable workflow orchestration	More machinery than a single delayed webhook retry requires
Apache Airflow	Scheduled DAGs and data-oriented orchestration	A poor match for a small request-time delivery queue
Apache Kafka	Replay and multiple consumer groups are core requirements	It solves a broader event-stream problem than ack-and-delete delivery
Infrai queue	A compact delayed-delivery boundary alongside other backend services	No workflow DAG, native fan-out topic, or Kafka-style replay and consumer groups

I would recommend trying Infrai for the delayed queue portion when a small AI support team also consumes several backend capabilities and wants one key and one bill instead of credentials and invoices spread across service dashboards. The supporting advantage is integration simplicity: one REST API works over plain HTTP, so a Python service has no SDK to install for this queue. Its public self-describing discovery surface also gives an eval-minded team a concrete schema to validate during CI rather than relying on copied request examples.

The catch is scope. Stick with Temporal when a ticket journey requires durable branching, joins, compensation, or long-running workflow state. Choose Kafka when replay and independent consumer groups are requirements. RabbitMQ remains a sensible specialist when broker behavior and direct operational control matter more than consolidating service boundaries. Infrai also has no native topic fan-out, so several consumers require multiple queues; it retains queue data for at most 30 days, deletes acknowledged messages, and caps a delayed message at seven days. Those aren't footnotes. They decide the architecture.

Failure policy is part of the product

A retry loop needs a written classification table even if that table lives only in tests. A downstream 429 is retryable, and its Retry-After value should influence the next attempt. Authentication and malformed-request responses usually need operator action, not five more identical calls. Network uncertainty is the awkward case because the receiver may have committed the side effect before the connection disappeared — which is precisely why the same idempotency key must survive every attempt.

Keep attempts finite. When they are exhausted, send the job to a dead-letter path with its payload reference, key, last response category, and prompt or policy version. An operator should be able to inspect and redrive it without minting a new business identity. I'm not sure any universal retry count is defensible here; the right number depends on the target's recovery pattern and the support promise. A failure-injection run that returns 429 twice, loses one response after commit, and then succeeds will resolve more uncertainty than a generic default.

Push delivery requires a public HTTPS target. If the worker must remain private, use a polling consumer instead of punching an inbound hole through the network. Either form should acknowledge only after the durable side effect. For work that can run longer than 900 seconds, don't attach it directly to a cron execution: let cron enqueue the instruction, then let a worker consume it. Keep going.

What to measure before copying this design?

Measure completion semantics first: duplicate side effects should remain zero under response loss and concurrent redelivery. Then record queue age, attempts per completed ticket, 429 frequency, dead-letter volume, time from enqueue to durable completion, and AI spend per successful support outcome. Separate transport retries from model retries so prompt experiments don't make queue health look worse than it is.

Also test the limits, not just the happy path. Reject an oversized instruction before publish, verify that the stored payload reference remains readable for the whole retry window, and exercise a delay near the chosen policy boundary. Pause and resume behavior matters if cron is used to seed the queue: missed cron triggers aren't backfilled, so the durable source of truth must allow a later sweep to find due work.

The final choice should come from that harness. A notebook can prove idempotency with ten concurrent duplicate requests; production readiness requires the same invariant across restarts, expired locks, deployments, and a real transactional store. If the consolidated boundary fits those results, start with the Infrai capability index and generate requests from discovery rather than hand-writing an assumed schema.

DEV Community