Design a Reliable Webhook Consumer with Signatures, Replay Protection and Idempotency

A webhook endpoint looks simple: accept an HTTP request, parse JSON and update a record. Production failures usually appear in the gaps between those steps. Providers retry deliveries, requests arrive out of order, workers crash after completing an action, and attackers can replay a captured request. A reliable consumer treats delivery as an untrusted, repeatable event rather than a one-time command.

This guide develops a concrete Node.js pattern around four controls: verify the signature over the raw body, enforce a short timestamp window, claim a stable event identity atomically, and acknowledge only after durable state has been recorded.

Verify the exact bytes that were signed

Most providers calculate an HMAC over the original request bytes. If middleware parses and serialises the JSON first, whitespace or property ordering can change and a legitimate signature may fail.

Capture the raw body and compare signatures in constant time:

import {createHmac, timingSafeEqual} from "node:crypto";

function verifyHmac({rawBody, suppliedHex, secret}) {
  const expected = createHmac("sha256", secret)
    .update(rawBody)
    .digest();

  const supplied = Buffer.from(suppliedHex, "hex");
  if (supplied.length !== expected.length) return false;
  return timingSafeEqual(supplied, expected);
}

The real provider may include a version prefix, timestamp or delivery ID in the signed payload. Follow that provider's documented construction exactly. Do not invent a generic signature header, and never log the shared secret or full authentication header.

Reject malformed signatures before parsing the body. Return a clear 4xx status so the provider does not interpret an authentication failure as a transient server problem.

Add a replay window

A valid HMAC proves that somebody possessed the secret when the message was signed. It does not prove that a captured message is fresh.

If the provider signs a Unix timestamp, require it to be close to the server clock:

function timestampIsFresh(headerValue, nowMs = Date.now()) {
  const seconds = Number(headerValue);
  if (!Number.isInteger(seconds)) return false;

  const ageMs = Math.abs(nowMs - seconds * 1000);
  return ageMs <= 5 * 60 * 1000;
}

Five minutes is an example, not a universal standard. Choose the provider's recommended tolerance and keep server time synchronised. A timestamp check reduces the replay window, but it does not replace event-level deduplication because the same delivery can be retried legitimately inside that window.

Claim a stable event ID atomically

Use the provider's immutable event or delivery ID. Do not hash only the JSON body: two different legitimate events can contain identical data. Scope the identity by provider and account when IDs are not globally unique.

Create a table with a unique constraint:

CREATE TABLE webhook_deliveries (
  provider TEXT NOT NULL,
  account_id TEXT NOT NULL,
  event_id TEXT NOT NULL,
  state TEXT NOT NULL,
  payload_sha256 TEXT NOT NULL,
  received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  processed_at TIMESTAMPTZ,
  last_error TEXT,
  PRIMARY KEY (provider, account_id, event_id)
);

At ingress, insert a row in state received. If the insert conflicts, load the existing row. A previously completed event should return a successful acknowledgement without repeating its side effect. An event still being processed should not be run concurrently.

Store a digest of the raw payload. If the same event ID later arrives with a different digest, quarantine it and alert an operator. Silently accepting two payloads under one identity weakens the audit trail.

Separate receipt from processing

Slow business logic should not keep the provider's HTTP request open. After verification and the durable insert, enqueue the database key and return an accepted response. The queue message is only a wake-up signal; the database row remains the source of truth.

The worker claims the event with a conditional update:

UPDATE webhook_deliveries
SET state = 'processing'
WHERE provider = $1
  AND account_id = $2
  AND event_id = $3
  AND state IN ('received', 'retryable');

Continue only when one row changed. This protects against queue duplication and overlapping workers.

For higher throughput, add an expiring lease rather than leaving crashed rows permanently in processing. An expired lease should move to reconciliation, not blindly repeat an external action.

Make side effects idempotent

Deduplicating deliveries does not automatically make every downstream action safe. Consider a payment event that grants account credit. If the worker updates the balance and crashes before marking the delivery complete, a retry can grant credit twice.

Prefer a single database transaction containing both the business change and completion marker:

BEGIN;

INSERT INTO account_credits(account_id, source_event_id, amount)
VALUES ($1, $2, $3)
ON CONFLICT (source_event_id) DO NOTHING;

UPDATE webhook_deliveries
SET state = 'completed', processed_at = now()
WHERE provider = $4 AND account_id = $1 AND event_id = $2;

COMMIT;

The unique constraint on source_event_id is the decisive guard. An application-level “already processed” check alone has a race between reading and writing.

When the side effect calls another API, use that API's documented idempotency key if available. Persist the remote request key and returned resource ID before optional work. If a timeout leaves the outcome uncertain, query the destination by that key or ID before retrying.

Handle ordering explicitly

Webhook arrival order is not business order. A delayed “updated” event can arrive after “deleted”, and retries can overtake newer messages.

If the payload provides a monotonically increasing version, update the aggregate only when the incoming version is newer. If it provides an event time, remember that clocks and delivery delays can still make timestamps ambiguous. For critical state, fetch the current object from the provider instead of reconstructing truth from event order.

Treat deletion carefully. A tombstone with a version is safer than removing all evidence and later accepting an older update as new.

Classify failures

Use distinct outcomes:

• completed: the business effect is durably confirmed.
• retryable: a temporary dependency failure occurred before an uncertain external write.
• uncertain: a write may have succeeded; reconciliation is required.
• rejected: the payload is authentic but invalid for a permanent reason.
• quarantined: the identity or payload conflicts with recorded evidence.

Retry only retryable work with exponential backoff and jitter. Put a finite limit on automatic attempts, then move the item to a review queue with non-secret diagnostics.

Test the boundaries

A useful test suite sends the same event concurrently and asserts exactly one business effect. It also covers a valid request with an expired timestamp, an invalid signature, the same ID with a changed payload, a worker crash after the side effect, and an external API timeout followed by reconciliation.

Do not stop at HTTP status assertions. Count durable rows and downstream writes. The safety property is not “both requests returned 200”; it is “one intended event produced at most one side effect, and every uncertain outcome remained visible”.

A robust webhook consumer accepts that networks retry and processes crash. Signatures establish authenticity, replay controls establish freshness, atomic identities prevent duplicate claims, and idempotent side effects make recovery safe.