Short answer: assign one dedupe key to each failure episode, make every retry carry that key, and alert on a state transition instead of every polling sample. Keep the raw attempts in metrics and logs so a fintech team can reconstruct what happened across tenant cohorts.
Incident reconstruction starts with the control-point table
| Control point | Pick this when | Trade-off |
|---|---|---|
| Sender-side idempotency key | You own the Node.js webhook sender and can persist delivery state | Requires durable state and a retention policy |
| Receiver-side deduplication | Receivers can enforce a unique event identifier | A misbehaving sender can still create noisy traffic |
| Poller state machine | You discover failures by polling metrics or job status | More state to reason about, but better incident timelines |
| Alert grouping window | You need a quick noise reduction for legacy paths | It can hide distinct failures inside one window |
The practical rule is to use sender idempotency plus receiver checks when both sides are under your control. For a third-party receiver, the sender key and a bounded retry ledger are the part you can guarantee. Start there, before tuning alert thresholds.
How can polling errors and webhook retries share one dedupe key?
Treat an alert as an episode, not a request. A polling cycle may see the same failed payment experiment ten times; those observations should point to one episode ID. The delivery attempts remain separate events.
Here is a small TypeScript shape for that contract. Derive it deterministically from the business identity and the failure boundary, never from the current attempt number.
type FailureEpisode = {
tenantId: string;
experimentId: string;
cohort: "control" | "treatment";
metric: string;
observedAt: string;
dedupeKey: string;
};
function makeDedupeKey(input: Omit<FailureEpisode, "dedupeKey">): string {
return [input.tenantId, input.experimentId, input.cohort, input.metric].join(":");
}
Do not include a timestamp in this key. A timestamp belongs on each observation; adding it turns one ongoing incident into a stream of unique alerts.
That distinction is easy to miss.
Instrument the sender so the timeline survives a retry
The ledger needs three facts for each key: the latest status, the next eligible retry time, and the attempt count. Store the outbound event body or a content hash too. That lets an incident reviewer answer “did we send the same payload twice?” without guessing from logs.
type Delivery = {
key: string;
payloadHash: string;
attempts: number;
state: "pending" | "sent" | "acknowledged" | "exhausted";
nextAttemptAt: number;
};
async function sendWithRetry(
delivery: Delivery,
post: (body: string, headers: Record<string, string>) => Promise<number>,
body: string
): Promise<Delivery> {
const status = await post(body, {
"Idempotency-Key": delivery.key,
"Content-Type": "application/json"
});
delivery.attempts += 1;
delivery.state = status >= 200 && status < 300 ? "acknowledged" : "pending";
delivery.nextAttemptAt = Date.now() + Math.min(60_000, 2 ** delivery.attempts * 1_000);
return delivery;
}
The retry decision must be explicit. A network timeout means the receiver may have accepted the event, so retry with the same key. A validation response in the 400 range usually needs correction, not another attempt. Keep that classification in code and emit it as a metric label; otherwise dashboards collapse meaningful failure modes together. For a payment experiment, record the poll sequence, worker ID, HTTP status, and ledger transition in one structured log event. A reviewer can then line up the metric spike with the exact attempt that changed state, even after a process restart.
Use counters for attempts and outcomes, a gauge for the number of open episodes, and a histogram for acknowledgement latency. A useful minimum is webhook_attempts_total{result}, webhook_episode_open, poll_errors_total{kind}, and webhook_ack_seconds. Keep tenant_id out of unbounded metric labels; put it in structured logs and traces instead. Node.js workers should also expose a process restart counter and a queue age measurement. Those two signals explain why a retry appeared late without pretending that lateness was a new incident.
The alert query should require a transition. For example, fire when webhook_episode_open rises above zero for five minutes, then suppress repeats while the same dedupe keys remain open. Recovery should close the episode and produce one resolution event. This preserves a clean before/after in the incident timeline.
A common failure looks like this: a five-minute grouping window is in place, yet duplicate pages continue because two workers use different keys and one includes the poll timestamp. Derive the key from tenant, experiment, cohort, and metric instead. In one review, a 30-second clock skew also made retries look late, so the ledger stored server time and original observation time separately. Your mileage may vary when a receiver expires keys sooner than your retry horizon.
Don't treat status codes as decoration. A 408 or a connection timeout is ambiguous and normally safe to retry with the same key. A 429 asks for backoff and often includes a server-provided limit. A 400 caused by schema validation needs a corrected payload, not another delivery. Recording those branches makes the dashboard useful to the on-call engineer instead of merely colorful.
Limits and decision checks
This design is not suitable when the receiver treats idempotency keys as opaque data and offers no retention guarantee; use a receiver-owned unique event ID or a queue with transactional handoff in that case. It is also a poor fit for alerts that must fire for every individual event, such as per-transaction fraud review. Stick with per-event alerting there and put deduplication in the notification layer.
Before shipping, test a timeout after the server commits, two pollers racing on the same episode, a process restart with a pending ledger row, and a receiver returning a permanent 4xx. Verify that one episode yields one opening alert, N observable attempts, and one recovery. Short tests. Clear evidence.
Top comments (0)