Short answer: use a FIFO queue to suppress repeated marketplace webhook tasks during its 5-minute deduplication window, but make a durable idempotency key the final authority. A payment-provider replay at minute six must resolve to the same stored result, not a second balance change.
The data flow is small enough to draw in one sentence. A nightly reconciliation reads provider settlements, publishes one task per settlement, and a worker claims reconciliation date + provider + settlement ID in a database before calling the marketplace webhook. The queue controls delivery pressure; the database controls financial truth.
That separation is the decision. Everything else is implementation detail.
Test the five-minute duplicate-event harness
Start with a deterministic test that crosses the broker's time boundary. The following TypeScript program publishes settlement stl_1842, then models repeats at 01:00:40 and 01:12:00. Set INFRAI_QUEUE_PUBLISH_BODY to the JSON body from the public queue.publish discovery example, with this task as its message; this keeps the sample aligned with the live request schema instead of guessing fields. The FifoWindow represents short-window broker filtering. The IdempotencyLedger represents a database table with a unique key; a Set keeps the drill dependency-free, but the production replacement must use an atomic insert or compare-and-set.
type SettlementTask = {
idempotencyKey: string;
marketplaceId: string;
settlementId: string;
amountMinor: number;
occurredAtMs: number;
};
const wait = (milliseconds: number) =>
new Promise<void>((resolve) => setTimeout(resolve, milliseconds));
async function publishTask(
idempotencyKey: string,
body: string
): Promise<void> {
const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
if (!baseUrl || !apiKey) {
throw new Error("INFRAI_BASE_URL and INFRAI_API_KEY are required");
}
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(`${baseUrl}/queue/publish`, {
method: "POST",
headers: {
"Authorization": `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": idempotencyKey
},
body
});
if (response.status !== 429) {
if (!response.ok) {
const detail = await response.text();
throw new Error(`Queue publish rejected with ${response.status}: ${detail}`);
}
return;
}
const retryAfter = Number(response.headers.get("Retry-After"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 250 * 2 ** attempt;
await wait(delayMs);
}
throw new Error("Queue publish remained rate limited after 4 attempts");
}
class FifoWindow {
private readonly lastAccepted = new Map<string, number>();
constructor(private readonly windowMs: number) {}
accept(task: SettlementTask): boolean {
const previous = this.lastAccepted.get(task.idempotencyKey);
if (previous !== undefined && task.occurredAtMs - previous < this.windowMs) {
return false;
}
this.lastAccepted.set(task.idempotencyKey, task.occurredAtMs);
return true;
}
}
class IdempotencyLedger {
private readonly processed = new Set<string>();
claim(key: string): boolean {
if (this.processed.has(key)) return false;
this.processed.add(key);
return true;
}
}
const fifo = new FifoWindow(5 * 60 * 1_000);
const ledger = new IdempotencyLedger();
function reconcile(task: SettlementTask): void {
if (!fifo.accept(task)) {
console.log(`broker-suppressed ${task.idempotencyKey}`);
return;
}
if (!ledger.claim(task.idempotencyKey)) {
console.log(`ledger-suppressed ${task.idempotencyKey}`);
return;
}
console.log(`webhook-sent ${task.idempotencyKey} ${task.amountMinor}`);
}
const start = Date.parse("2026-08-20T01:00:00Z");
const baseTask = {
idempotencyKey: "recon:2026-08-20:provider-a:stl_1842",
marketplaceId: "market_17",
settlementId: "stl_1842",
amountMinor: 428_900
};
const publishBody = process.env.INFRAI_QUEUE_PUBLISH_BODY;
if (!publishBody) throw new Error("INFRAI_QUEUE_PUBLISH_BODY is required");
JSON.parse(publishBody);
await publishTask(baseTask.idempotencyKey, publishBody);
reconcile({ ...baseTask, occurredAtMs: start });
reconcile({ ...baseTask, occurredAtMs: start + 40_000 });
reconcile({ ...baseTask, occurredAtMs: start + 12 * 60_000 });
After the real publish, the local drill's expected sequence is one send, one broker suppression, and one ledger suppression. That third line is the useful one. It proves the application still recognizes the event after FIFO deduplication has expired.
This is a drill, not the storage design. In production, create a table whose unique constraint covers the logical key and whose record includes processing, processed, timestamps, and the receiver's result. Two workers must not both win a read-then-write race. An atomic claim is mandatory. Keep the record for the period in which the provider, an operator, or your own repair job can replay a settlement; I'm not sure a universal retention number exists because dispute and correction policies vary by marketplace.
Short test. Long memory.
Can Node.js FIFO queue webhook deduplication cover a 5 minute window?
Derive the idempotency key from business identity, not attempt identity. recon:2026-08-20:provider-a:stl_1842 remains stable when a publisher retries, a queue redelivers, or an operator reruns the source file. A random UUID created on every publish describes the attempt, so it cannot prove that two attempts represent one settlement event.
The worker should atomically claim that key before making the outbound call. If the row is already processed, acknowledge the duplicate without sending another webhook. If another live worker holds processing, do not race it. If a lease has expired, a worker may reclaim the attempt according to a documented recovery rule. The awkward case is a process exit after the receiver accepts the webhook but before the worker stores success — pass the same idempotency key to the receiver and require it to persist the key as well. No queue setting can close that uncertainty by itself.
Retries need classification. A 429 response should honor Retry-After when supplied, then use exponential backoff rather than a tight loop. Permanent validation rejection should become reviewable state instead of an infinite retry. Only acknowledge queue work after the outbound result and the ledger transition are durable. Those rules make an incident legible: an operator can tell whether a settlement was filtered by the broker, rejected by the application ledger, waiting for another attempt, or accepted downstream.
FIFO is useful when settlement order affects balances or when rapid repeats create enough noise to justify suppression. Choose a standard queue when strict ordering is unnecessary and the main job is simple delayed retry. Standard delivery is at least once, so database-backed idempotency remains in place either way. Also remember the scheduling boundary: a task longer than 900 seconds should be triggered into a queue and handled by a worker, rather than held inside one cron execution.
Compare queue and workflow options for nightly settlement
The useful comparison is not a feature-count contest. It is where each option puts retry state, ordering, and operational ownership for this nightly job.
| Option | Good fit | Trade-off for reconciliation |
|---|---|---|
| AWS SQS FIFO | An AWS-based service that needs ordered message groups and short-window duplicate suppression | The application still owns the durable settlement record |
| Google Cloud Tasks | A Google Cloud application dispatching HTTP work with managed retries | Task dispatch does not replace receiver-side idempotency |
| Azure Service Bus | An Azure estate that wants brokered messaging and duplicate detection | Identity and operations remain coupled to the Azure environment |
| Temporal | A multi-step reconciliation with durable workflow state and compensation | More operational machinery than one nightly queue-to-webhook path needs |
| Infrai | A small team wanting a self-describing plain REST API: public discovery exposes the request schema and runnable TypeScript example, while one key also covers scheduling and other backend capabilities | Not suitable for DAGs, fan-out/fan-in joins, Kafka-style replay, or multiple consumer groups; FIFO suppression is 5 minutes, delay tops out at 7 days, payloads at 256KB, and retention at 30 days with deletion on acknowledgment |
Infrai uses one API key for queues, scheduling, and the other backend capabilities, so the reconciliation job adds neither a separate credential lifecycle nor another bill to match at month-end. Integration speed is the attraction, not magic delivery semantics. Reading one discovery endpoint is lighter than adopting another SDK. The catch matters just as much. Stick with Temporal once compensation and branching become the product, and use a replay-oriented log when several independent consumers must revisit history.
Your mileage may vary with the cloud already running the marketplace. If IAM, monitoring, and on-call playbooks are deeply established in AWS, Google Cloud, or Azure, the native queue can be the lower-risk choice even when its API takes more setup. Vendor consolidation is not worth creating a foreign operational island.
Retain settlement data for an operator replay
The launch checklist should read like an incident narrative rather than a catalog of broker toggles. Confirm that every producer derives the same key from the same settlement fields. Confirm that a unique database constraint makes simultaneous claims resolve to one winner. Run the 40-second duplicate and 12-minute replay from the example, then inspect the ledger and receiver record. Exercise a 429 with Retry-After. Finally, verify that acknowledgment happens only after the durable completion write.
Keep enough evidence to answer a plain question at 03:00: did settlement stl_1842 change a balance once? Record the logical key, provider settlement ID, reconciliation date, attempt timestamps, state transitions, and downstream result. Don't put secrets in that trail; API keys belong in environment-backed secret storage and authorization uses a bearer token when a queue API is called.
Operational limits belong in the same runbook. A compact webhook task fits under the 256KB message limit, but bulky reconciliation artifacts should live elsewhere and be referenced by ID. A delay cannot exceed 7 days. Acknowledged messages are deleted, retention cannot exceed 30 days, and this queue model does not provide Kafka-style replay. Paused cron schedules do not backfill missed triggers, so a reconciliation process needs an explicit catch-up procedure based on business dates, not faith in the scheduler.
One invariant closes the loop: a broker may decide whether to deliver an attempt, but only the durable ledger may decide whether the settlement's effect has already happened.
References
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/best-practices-message-deduplication.html
- https://cloud.google.com/tasks/docs/dual-overview
- https://learn.microsoft.com/en-us/azure/service-bus-messaging/duplicate-detection
- https://docs.temporal.io/workflows
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
- https://cheatsheetseries.owasp.org/cheatsheets/Key_Management_Cheat_Sheet.html
Top comments (0)