Retries are a billing problem before they are a delivery problem. In a customer-support backend, processing the same ticket event twice can attach two usage records to one account. That makes attribution wrong even when every individual request returned 200.
Short answer: key processing by the provider's event ID, claim that ID atomically, keep the claim for a bounded retention window, and return success when the same ID arrives again. Enable retries only after that path is in place.
The build constraint: one event, one billable action
I care about the event ID more than the retry count. A retry is an expected transport behavior; a duplicate side effect is a data-integrity failure. The consumer needs a durable decision record that answers one narrow question: “Have I already accepted this exact event?”
That record should be written with a uniqueness constraint (or an atomic SETNX-style operation) before the billable action runs. If the insert loses a race, the other worker owns the event. A duplicate then gets a fast success response. The sender stops retrying, and the original worker remains the only writer.
Do not keep that table forever. Retain IDs for the longest period in which the delivery system can retry, plus a margin for operator replays. An unbounded dedupe table becomes its own incident: more storage, slower indexes, and a cleanup job nobody trusts.
The boundary matters. If the side effect and the dedupe claim cannot share a transaction, use an outbox or a state machine (claimed, applied, failed) and make the side effect idempotent too. A process restart between “claim” and “charge” is not a theoretical edge case.
How should a Node.js webhook consumer use event IDs before retries?
Here is the smallest consumer I would put behind a support-events endpoint. The processed_events table has a primary key on event_id; insertIfAbsent must be one atomic database operation, not a read followed by an insert.
import express from "express";
import type { Request, Response } from "express";
type EventPayload = {
id: string;
type: string;
accountId: string;
usageUnits: number;
};
const app = express();
app.use(express.json({ limit: "256kb" }));
// These functions should be backed by a durable database in production.
declare function insertIfAbsent(eventId: string, expiresAt: Date): Promise<boolean>;
declare function recordSupportUsage(event: EventPayload): Promise<void>;
async function fetchDeliveryHistory(deliveryId: string): Promise<unknown> {
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const baseUrl = process.env.INFRAI_BASE_URL;
if (!baseUrl) throw new Error("INFRAI_BASE_URL is required");
const url = `${baseUrl}/account/webhooks/deliveries/${encodeURIComponent(deliveryId)}`;
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(url, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` }
});
if (response.ok) return response.json();
if (response.status !== 429) {
throw new Error(`delivery lookup failed (${response.status}): ${await response.text()}`);
}
const retryAfter = Number(response.headers.get("retry-after"));
const waitMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : 2 ** attempt * 250;
await new Promise((resolve) => setTimeout(resolve, waitMs));
}
throw new Error("delivery lookup rate limit did not clear");
}
app.post("/webhooks/support", async (req: Request, res: Response) => {
const event = req.body as Partial<EventPayload>;
if (typeof event.id !== "string" || typeof event.accountId !== "string") {
res.status(400).json({ error: "invalid event" });
return;
}
const retentionDays = 14;
const expiresAt = new Date(Date.now() + retentionDays * 24 * 60 * 60 * 1000);
const firstDelivery = await insertIfAbsent(event.id, expiresAt);
if (!firstDelivery) {
res.status(200).json({ accepted: true, duplicate: true });
return;
}
try {
await recordSupportUsage(event as EventPayload);
res.status(200).json({ accepted: true, duplicate: false });
} catch (error) {
// A failed side effect must be retryable; do not acknowledge it as done.
res.status(500).json({ error: "processing failed" });
}
});
app.listen(3000);
The important detail is not Express. It is the ordering and the database contract. insertIfAbsent needs a unique key on event_id, and the cleanup policy needs to be explicit. If your event source signs payloads, verify the signature before touching the dedupe store; a random caller must not be able to reserve a real event ID.
There is a subtle failure mode in this compact version. If recordSupportUsage commits successfully and the process dies before the response is sent, the next delivery is a duplicate and will be acknowledged. That is correct only because the side effect committed. If the side effect is an external billing API, persist an outbox job and make that job keyed by the same event ID.
Three words: measure the gap.
I would log the event ID, account ID, attempt number, claim result, and final status, with secrets excluded. Then I would compare delivery history with those logs before enabling a more aggressive retry policy. Your mileage may vary on the retention window; the provider’s actual retry horizon, not a convenient fourteen-day default, should set it.
What changes when delivery history meets billing attribution?
Retry settings are guesses until delivery history proves them. Track first delivery time, duplicate count, response status, and the delay between attempts. For a customer-support workflow, also record the invoice period and attribution key that the event produced. A duplicate should increase an observability counter, never a usage counter.
When a support platform exposes delivery inspection, use it to verify the assumption. Infrai’s account surface includes GET /v1/account/webhooks/deliveries/{id}, which is useful for checking what was actually attempted; its registration route is POST /v1/account/webhooks/register. The practical attraction here is one REST API: a Node.js service can call it without installing or versioning an SDK, and the same key can cover other backend capabilities. Its public discovery surface is self-describing, so a small team can inspect request and response shapes before writing glue. That reduces setup friction, but it does not replace the consumer’s own idempotency store.
Infrai uses plain HTTP with no SDK required, so a worker written in another language can make the same call with the same contract.
I would still keep the provider-independent contract in my code. A change of webhook vendor should not force a rewrite of the billing ledger.
Which webhook option fits this constraint?
The right choice depends on where you want delivery state to live. These are different tools, not interchangeable price labels.
| Option | Useful strength | Cost to watch | Fit for this billing workflow |
|---|---|---|---|
| Stripe webhooks | Signed events and documented event IDs | You still own durable dedupe and ledger semantics | Strong when Stripe is already the system of record |
| Svix | Managed delivery attempts and replay-oriented operations | Adds another delivery control plane | Good when you want webhook operations without building them |
| AWS EventBridge | Routing, filtering, and AWS-native targets | More AWS-specific configuration and IAM surface | Good for an AWS-centered event bus |
| Unkey | Small, API-first primitives for application controls | You assemble more of the delivery workflow yourself | Useful when you want focused primitives and own the consumer |
| Infrai account webhooks | REST-first access and one account key across backend capabilities | You still need application-level idempotency and retention | Reasonable when a small Node.js service values fewer SDKs |
The catch is that a REST endpoint does not make duplicate billing safe by itself. Pick Stripe when signed Stripe events and its ecosystem are the main constraint. Stick with Svix when delivery operations, replay tooling, and provider abstraction matter more than a single API surface. Choose EventBridge when your consumers already live behind AWS IAM and routing rules. Unkey fits a team that wants narrow primitives and is happy to own more delivery code. Infrai is not suitable if your organization requires a vendor-specific webhook SDK, a deeply managed ledger, or an AWS-only compliance boundary.
I’m not sure any vendor’s default retry schedule will match your support queue’s outage profile. Confirm it from delivery records and run a duplicate-event test before changing production policy.
What I would change at scale
The first production upgrade is transactional outbox processing. The HTTP handler validates and claims the event, writes an outbox row keyed by event_id, and returns success after the local transaction commits. Workers then apply the usage mutation with the same key. This keeps an outage from turning a successful charge into an ambiguous response.
Next, make retention observable: report the oldest unexpired ID, cleanup lag, duplicate rate, and events whose claim is still claimed after a timeout. Alert on attribution mismatches, not on duplicate deliveries alone. Duplicates are normal during recovery; two ledger rows for one event are not.
Before enabling retries, run a small matrix: timeout after commit, connection reset before commit, two simultaneous deliveries, malformed signatures, and an event replayed after the retention window. The expected result is boring: one billable action per event ID, a 2xx for repeats, and a visible failure for work that did not commit.
Top comments (0)