Debugging webhooks in Node: failed signatures, retries and duplicate events

Webhook bugs have a talent for not reproducing on your laptop. Three of the usual suspects:

• A signature check that passes nine times and fails the tenth.
• The same event processed twice, so one order ships twice or one email goes out twice.
• A slow deploy, a pile of timeouts, and a retry storm right behind it.

None of these throw a useful stack trace. Each one is easy to fix once you can see the exact request the provider sent. This post covers the fix for each, then how to get that request in front of you.

1. Signatures: hash the bytes you received, not the JSON you rebuilt

Providers sign the exact bytes they send. If your handler re-serializes the parsed body and hashes that, it is hashing a different string.

Here's how that goes wrong. One team on dev.to wrote up a marketplace webhook that started failing about one time in ten after they moved it to a Node gateway. The marketplace sent amounts like 24.50. JSON.parse turns that into 24.5, JSON.stringify writes it back one character shorter, and the HMAC no longer matches. Every order with a trailing zero got a 401. Their tests passed because the fixtures had been generated with JSON.stringify too.

The broken version usually looks like this:

// Don't: this hashes your serialization, not the provider's bytes
const digest = crypto
  .createHmac('sha256', secret)
  .update(JSON.stringify(req.body))   // "24.50" has already become 24.5
  .digest('hex');

The fix is to keep the raw body. You can use express.raw() on the webhook route, or keep express.json() and save the bytes with its verify hook, so req.body is still parsed for the rest of your app:

const express = require('express');
const crypto = require('node:crypto');

const app = express();

app.use(express.json({
  limit: '1mb',                        // the default is 100kb; some providers send more
  verify: (req, res, buf) => {
    req.rawBody = buf;                 // the exact bytes, before parsing
  },
}));

function verifySignature(req, secret) {
  // Header name and format vary by provider. This is Meta's: "sha256=<hex>"
  const received = Buffer.from(req.get('x-hub-signature-256') || '');
  const expected = Buffer.from(
    'sha256=' + crypto.createHmac('sha256', secret).update(req.rawBody || '').digest('hex')
  );
  // timingSafeEqual throws on different lengths, so check that first
  return received.length === expected.length && crypto.timingSafeEqual(received, expected);
}

Two details that matter:

• Compare with crypto.timingSafeEqual, not ===, and check lengths first because it throws when they differ.
• Raise the body limit. The WhatsApp Cloud API write-up notes payloads of up to 3 MB against Express's 100 KB default. An oversized payload gets a 413, the provider retries, and it fails again.

Build your test fixtures from real payloads captured from the provider's sandbox, including an amount that ends in zero. Fixtures your own code generated will always agree with your own code.

2. Retries and duplicates: answer fast, then make it safe to run twice

Providers retry when they don't get a 2xx in time. Meta, for example, "retries with decreasing frequency for up to 7 days" and warns that retries can produce duplicate notifications. The marketplace in the trailing-zero story retried each rejected event for three days.

So assume every event can arrive more than once, and design for it:

1. Verify, store, return 200. Don't call your payment provider, send email or generate a PDF before you answer. Slow handlers are what turn a deploy into a retry storm.
2. Deduplicate on the provider's event ID, not on a hash of the body. A retry can arrive in a differently shaped payload.
3. Let the database enforce it. A unique key and ON CONFLICT DO NOTHING is a race-free claim; a SELECT followed by an INSERT is not.

app.post('/webhooks/orders', async (req, res) => {
  if (!verifySignature(req, process.env.WEBHOOK_SECRET)) {
    return res.sendStatus(401);
  }

  const event = req.body;

  // webhook_events.id is the provider's event ID, with a unique constraint
  const { rowCount } = await db.query(
    `INSERT INTO webhook_events (id, type, payload)
     VALUES ($1, $2, $3)
     ON CONFLICT (id) DO NOTHING`,
    [event.id, event.type, event]
  );

  // rowCount 0 means we've seen it: answer 200 so the provider stops retrying
  res.sendStatus(200);
});

A worker then picks up rows that haven't been processed (SELECT ... WHERE processed_at IS NULL ... FOR UPDATE SKIP LOCKED), does the real work, and sets processed_at in the same transaction as the business write. Storing the event before answering means a crash after the 200 loses nothing. If the insert itself fails, Express returns a 500 and the provider tries again later, which is what you want.

If you'd rather not run a worker, the idempotency-key write-up covers the same claim pattern for synchronous handlers, including what to do when a second request arrives while the first is still in progress (409) and how long to keep the keys (longer than the longest retry window).

3. Debugging: you need the request the provider actually sent

Every bug above is invisible in your own logs if they only record POST /webhooks/orders 401. To fix a signature mismatch you need the exact headers and body. To prove a duplicate you need both deliveries side by side.

There are two ways to get them:

• Log inbound webhook requests in full, with secrets and personal data redacted before they're written anywhere.
• Point the provider's test or sandbox webhook at a capture URL that records the request and answers it, so you can read the payload before you've written any code for it.

Either way, scrub credentials first. A webhook log full of customer emails is a new problem, not a fix.

4. Doing this with ApiTraffic

If you already run the ApiTraffic SDK, webhooks are inbound requests like any other: the request log records their headers and body, the status you answered, and how long you took.

const apitraffic = require('@apitraffic/node');

app.use(express.json({ limit: '1mb', verify: (req, res, buf) => { req.rawBody = buf; } }));
app.use(apitraffic.express());          // after body parsers, before routes

app.post('/webhooks/orders', async (req, res) => {
  apitraffic.tag('eventId', String(req.body.id));
  // ...verify, store, 200, as above
});

The order matters: ApiTraffic records the body your parser produced, and the firewall can only inspect a body that's already been parsed. Tagging the event ID turns "did we process this twice?" into a search:

tag:eventId == "evt_8f2c"

Both deliveries come back, with their timestamps, statuses and durations. A slow first attempt followed by a retry a second later is the duplicate, explained. For the signature problem, url_path like "/webhooks/*" AND status_code == 401 lists every rejected delivery. Open one and compare the body you received with the string you hashed.

Redaction. The SDK masks credential headers such as Authorization, Cookie and X-Api-Key, and credential-shaped keys in bodies, from the first request. Check how your provider's signature header shows up, and add rules for anything personal in the payload (emails, addresses). Rules set to run at the API are applied before data leaves your server, and they're pushed from the dashboard without a deploy.

Captures. Every bucket has a capture URL. Send a request to it and ApiTraffic records the method, path, headers and body, answers 200, and forwards nothing. Point a provider's test webhook at it to see the real payload before you write the handler, or to build fixtures from real bytes. Two cautions:

• The provider gets a 200, so it considers the event delivered. Use a test or sandbox endpoint, not production.
• Anyone who knows the capture domain can send to it. Share it only where you need it, and switch the webhook back when you're done.

Checklist for your PR

- [ ] Signature is computed over the raw request bytes (express.raw() or the express.json verify hook)
- [ ] Comparison uses crypto.timingSafeEqual with a length check
- [ ] Body limit raised to cover the provider's largest payload
- [ ] Test fixtures are real provider payloads, including an amount ending in 0
- [ ] Handler verifies, stores and returns 2xx before doing slow work
- [ ] Events deduplicated on the provider's event ID with a unique constraint
- [ ] Side effects run once per event (processed_at set in the same transaction)
- [ ] Event ID tagged or logged on every delivery
- [ ] Alert on 401s from webhook routes
- [ ] Secrets and personal data redacted from webhook logs

Signature failures, duplicates and retry storms all come down to the same gap: you can't see the request the provider sent. Close that gap first and the fixes are short.

See what's hitting your API: apitraffic.io