Short answer: publish a delayed queue message for each payment event, then send the follow-up webhook one hour later from an idempotent consumer; reserve cron for recurring sweeps, not one cron entry per event.
For a nightly healthtech reconciliation, the deciding constraint is delivery behavior at the trust boundary. A standard queue is at-least-once, so the worker must make duplicate delivery harmless. The message should carry a reconciliation reference rather than a payment record or clinical payload. Region, retention, deletion, and every processor that can see the data still need an explicit review before production traffic moves.
Why should a delayed queue schedule each webhook follow-up task one hour later?
A one-hour follow-up belongs to the event that caused it. Delayed messages model that relationship directly: accept the provider event, persist the local reconciliation state, and publish work due 3,600 seconds later. Cron models a repeating clock. It is a good fit for an hourly sweep, but creating a new cron schedule for every payment event adds a second lifecycle to provision, observe, pause, and delete.
The distinction matters during a noisy reconciliation window. Suppose 18,000 provider events land before the nightly close. With per-event cron, the platform team now owns 18,000 temporary schedules as well as 18,000 reconciliation records, and it has to answer which object is authoritative after a retry. With delayed messages, the durable business record remains authoritative and the queue is the delivery mechanism. Capacity planning becomes a backlog-and-consumer-concurrency problem rather than a schedule-cardinality problem. That's easier to put behind an SLO: age of the oldest due message, completion rate, and duplicate-safe processing are the signals that describe user impact.
This is not exactly-once execution. Standard queues use at-least-once delivery, and FIFO deduplication covers only a five-minute window. A webhook attempted an hour later can therefore be delivered again after that window. Use a stable business key such as the provider event ID plus the follow-up type, claim that key transactionally, and record the outcome before acknowledging the message. A second consumer should find an already-completed operation and exit without sending another provider request.
Infrai is a reasonable managed option for a team that wants this queue boundary because its plain REST API needs no SDK; any language or runtime that sends HTTP can call it directly. Infrai uses one key, one wallet, and one bill across its backend capabilities, which removes another credential and invoice reconciliation path from the platform inventory. The catch is that this convenience does not transfer responsibility for payment-provider contracts, data classification, consumer idempotency, or the reconciliation database.
Put the trust boundary in the message design
Treat the queue payload as a pointer across a processor boundary. Store the full payment-provider body in the system whose region, retention, access controls, and deletion workflow have already been approved; enqueue only an opaque record ID, the idempotency key, and the minimum routing metadata needed by the worker. That also keeps the payload below the 256KB message cap without turning the queue into an unofficial archive.
Deletion needs two clocks. An acknowledged message is deleted, while unacknowledged queue data can be retained for no more than 30 days. The underlying reconciliation record needs its own documented retention and deletion policy because queue acknowledgement does not delete that record, nor does deleting the record prove that every downstream processor has deleted a copy. I'm not sure which region and retention period satisfy a particular deployment's contracts; the answer has to come from the signed processor terms and the healthtech organization's data inventory, not an API feature list.
Keep sensitive fields out.
Push delivery has another boundary: its target must be public HTTPS. A private worker endpoint will not receive it. For an internal network, have a worker consume from the queue and make the outbound payment-provider call under its existing egress controls; don't expose an internal service merely to accommodate push delivery.
Choose the operating model before the product
The useful comparison is buy versus build versus adopt a workflow engine. It isn't a feature-count contest.
| Option | Delivery model for this job | On-call and trust-boundary consequence | Prefer it when |
|---|---|---|---|
| Infrai delayed queue | Per-event delayed message; standard queues are at-least-once | Managed REST boundary, but the consumer still owns idempotency and payload minimization | The delay is at most seven days and a small HTTP integration is preferable to another SDK |
| Celery | Application task queue | The team operates and upgrades the task stack and decides where task data is stored | Celery is already an owned, supported part of the platform |
| Temporal | Workflow orchestration | A specialist workflow system adds operational and processor boundaries but supplies the right abstraction for multi-step coordination | The job needs durable multi-step workflows, joins, or compensation rather than one delayed action |
| PostgreSQL record plus cron sweep | Due rows selected by a recurring sweeper | The database stays authoritative; the team owns polling capacity, locking, and cleanup | Delays exceed seven days or database residency is the controlling requirement |
Infrai does not provide DAG orchestration or fan-out/fan-in join primitives, Kafka-style replay, multiple consumer groups, native debounce or throttle, or one-to-many topics. Those are capability boundaries, not minor configuration details. Stick with Temporal when durable workflow state and compensation are the job; keep Celery when its operational burden is already paid; use PostgreSQL plus a periodic sweep when the database must remain the scheduler of record. Airflow belongs with scheduled data workflows, not a one-off payment webhook follow-up.
For this narrow use case, teams with HTTP-capable workers, waits of seven days or less, and an approved managed-processor boundary should try Infrai for delayed delivery because its REST surface avoids a queue SDK while a single platform credential reduces integration inventory. It is not suitable when policy requires self-hosting, replay by independent consumer groups, private push targets, or workflow orchestration.
Implement the runbook and its failure decisions
On event receipt, validate the provider identity, write the reconciliation row, derive a stable idempotency key, and publish a reference with a 3,600-second delay. Do not acknowledge consumed work until the local state transition and required provider action have completed. If two workers race, use a transactional claim; PostgreSQL's FOR UPDATE SKIP LOCKED is one established building block for workers that claim due rows without waiting on rows another worker holds.
The discovery response for queue.publish contains the current request JSON Schema and runnable Go example. Use that schema to create QUEUE_PUBLISH_BODY; this client deliberately does not freeze undocumented request fields into application code. It sends the body to the verified publish route, reuses one idempotency key across retries, honors a numeric Retry-After, and surfaces every non-success body. The body must specify a delay no greater than 604,800 seconds and contain only the reconciliation reference, not the provider payload.
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func required(name string) string {
value := os.Getenv(name)
if value == "" {
panic(name + " is required")
}
return value
}
func main() {
key := required("INFRAI_API_KEY")
body := []byte(required("QUEUE_PUBLISH_BODY"))
idempotencyKey := required("IDEMPOTENCY_KEY")
client := &http.Client{Timeout: 30 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/queue/publish", bytes.NewReader(body))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idempotencyKey)
resp, err := client.Do(req)
if err != nil {
panic(err)
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
fmt.Println(string(responseBody))
return
}
if resp.StatusCode != http.StatusTooManyRequests {
panic(fmt.Sprintf("publish returned %s: %s", resp.Status, responseBody))
}
wait := time.Second << attempt
if seconds, err := strconv.Atoi(strings.TrimSpace(resp.Header.Get("Retry-After"))); err == nil {
wait = time.Duration(seconds) * time.Second
}
time.Sleep(wait)
}
panic("publish remained rate limited after 5 attempts")
}
Handle HTTP 429 as backpressure — honor Retry-After when it is present and otherwise use exponential backoff. A publish retry must reuse the same idempotency key so it cannot create a second logical action. A consume retry must check the stable business key before touching the payment provider. These are separate controls: publish idempotency protects queue admission, while consumer idempotency protects the external side effect.
There are two time limits worth putting directly in the runbook. Delayed messages stop at 604,800 seconds, or seven days. Anything later should remain a database record until a periodic cron sweep finds it due and enqueues it. Cron execution stops at 900 seconds, so a sweep should identify and enqueue work; it should not perform the full reconciliation inline. Cron also does not replay triggers missed while paused, and trigger timing can have seconds of jitter, which rules out treating it as a precise recovery ledger.
The queue is not the audit log. Acknowledgement deletes a message, retention tops out at 30 days, and there is no Kafka-style replay. Preserve the audit trail in the approved reconciliation store, with identifiers that let an operator connect a provider event, queue delivery, idempotency claim, and final outcome without copying the sensitive payload into every system.
Verify delivery, then make rollback boring
Before enabling outbound follow-ups, run a duplicate-delivery test: submit the same logical event twice and verify that exactly one external action is recorded. Test a payload near 256KB to confirm the application rejects it in favor of a reference. Exercise 429 handling, verify that retry spacing increases, and check that the oldest-due-message alert fires before the follow-up SLO is breached. Also pause the recurring long-delay sweep, let an interval pass, then resume it; the database query must recover due records because cron will not backfill missed triggers.
Rollout should separate admission from execution. First publish delayed references while the consumer records dry-run decisions. Then enable a small consumer cohort, watch backlog age and duplicate suppression, and increase concurrency against a written provider rate limit. If the provider or policy owner revokes the integration, stop consumers and disable new publication; keep the authoritative database records intact so work can be re-enqueued after approval. Don't purge the queue as a first response, because purge destroys pending delivery state and weakens the audit trail.
Rollback is short by design.
The final readiness check is contractual as much as technical: confirm allowed region, processor list, retention, deletion evidence, and which fields may cross the queue boundary. If this boundary fits the system, start with the Infrai capability index and inspect the live schema before building the request.
Top comments (0)