DEV Community

Rasika Dangamuwa
Rasika Dangamuwa

Posted on

Why Webhooks Fail in Production: 5 Engineering Traps and How to Fix Them

#javascript #webdev #security #api

When integrating webhooks from Stripe, GitHub, Shopify, or Twilio, local development usually feels straightforward: you configure an endpoint, trigger a test event in the dashboard, parse the payload, and return a 200 response.

Then you push to production. Under real-world network conditions and traffic spikes, subtle distributed systems issues emerge: customer cards get billed twice, database queries time out during delivery spikes, webhook retries swamp your servers, and attackers replay intercepted payloads.

Webhooks are deceptively simple—they are just HTTP POST requests. But unlike synchronous REST APIs, webhooks operate over asynchronous boundaries with at-least-once delivery guarantees. Here are five engineering traps that consistently break webhook architectures in production, along with practical ways to fix them.

1. The Synchronous Processing Trap (and Retry Storms)

Most webhook senders enforce strict HTTP response timeouts. Stripe times out after 10 seconds; GitHub and Shopify expect responses within 5 to 10 seconds.

If your webhook handler executes heavy database transactions, sends emails, or calls third-party APIs synchronously within the HTTP request cycle, any upstream latency spike will push execution beyond the timeout limit. When your endpoint fails to acknowledge with a 2xx within that window, the provider assumes a failure and triggers an exponential backoff retry.

If fifty events fail simultaneously, fifty retries arrive a few minutes later—right when your database is already struggling. This creates a cascading retry storm that takes down your application.

The Fix: Decouple ingestion from execution. Your webhook route should perform only two tasks: verify the cryptographic signature and push the raw payload onto an asynchronous queue (such as Redis BullMQ, SQS, or RabbitMQ). Return HTTP 200 OK or 202 Accepted immediately:

app.post("/webhooks/stripe", express.raw({ type: "application/json" }), async (req, res) => {
  const sig = req.headers["stripe-signature"];

  // Verify signature quickly
  let event;
  try {
    event = stripe.webhooks.constructEvent(req.body, sig, endpointSecret);
  } catch (err) {
    return res.status(400).send(`Webhook Error: ${err.message}`);
  }

  // Push to background queue and exit immediately
  await jobQueue.add("process-stripe-event", { eventId: event.id, payload: event });
  return res.status(200).json({ received: true });
});
Enter fullscreen mode Exit fullscreen mode

2. Timing Attacks on Signature Verification

Webhook providers sign payloads using HMAC-SHA256 with a shared secret, passing the signature in an HTTP header (such as Stripe-Signature or X-Hub-Signature-256).

A frequent implementation mistake is comparing the computed HMAC digest against the incoming header using standard string comparison operators:

// VULNERABLE: standard string equality leaks timing information
if (computedSignature === headerSignature) {
  processEvent();
}
Enter fullscreen mode Exit fullscreen mode

Standard string equality checks (===) compare characters sequentially and return false at the first mismatch. An attacker who repeatedly sends payloads and measures execution time with microsecond precision can infer characters one by one.

The Fix: Always use constant-time byte comparisons:

const crypto = require("crypto");

function safeCompare(a, b) {
  const bufA = Buffer.from(a, "utf8");
  const bufB = Buffer.from(b, "utf8");
  if (bufA.length !== bufB.length) return false;
  return crypto.timingSafeEqual(bufA, bufB);
}
Enter fullscreen mode Exit fullscreen mode

3. Missing Idempotency Guards

In distributed computing, webhooks follow at-least-once delivery. If network connectivity drops right after your server returns 200 OK but before the provider receives the TCP ACK, the provider will assume delivery failed and send the event again.

Furthermore, out-of-order delivery is common. A customer subscription cancellation event (customer.subscription.deleted) can occasionally arrive before the creation event (customer.subscription.created).

If your consumer processes events without deduplication, accounts will receive duplicate credits or conflicting states.

The Fix: Enforce idempotency using the provider’s unique event identifier (e.g., evt_1O...). In your background worker, check an atomic store before executing business logic:

INSERT INTO processed_events (event_id, processed_at) 
VALUES ($1, NOW()) 
ON CONFLICT (event_id) DO NOTHING;
Enter fullscreen mode Exit fullscreen mode

If the row already exists, acknowledge the job as completed and immediately discard the duplicate.

4. Replay Attacks and Unchecked Timestamps

If an attacker intercepts a legitimate webhook payload and its valid signature, they can resend that exact HTTP request to your endpoint days or weeks later. Because the cryptographic signature matches the payload, signature verification alone will pass.

To guard against this, modern webhook providers include a Unix timestamp alongside the signature hash (for example, t=1757121600,v1=...).

The Fix: Validate the timestamp header against your server clock and reject requests with timestamps outside an acceptable tolerance window—typically 5 minutes (300 seconds):

const tolerance = 300; // 5 minutes
const currentTime = Math.floor(Date.now() / 1000);

if (currentTime - eventTimestamp > tolerance) {
  throw new Error("Webhook timestamp expired (possible replay attack).");
}
Enter fullscreen mode Exit fullscreen mode

5. Raw Body Mutation and Header Parsing Gotchas

To verify an HMAC signature, your cryptographic function must hash the exact, unparsed raw byte stream sent over the wire.

In Node.js, Express, Next.js, and Python frameworks, global JSON parsing middleware parses the body into an object. Re-stringifying this object (JSON.stringify(req.body)) alters spacing, reorders object keys, or alters Unicode escapes, causing signature verification to silently fail every time.

Testing these raw delivery streams during local development can be tricky without inspecting the verbatim incoming HTTP headers and byte streams. Using an interactive bin like Nutilz Webhook Tester gives you a temporary capture URL to inspect incoming POST requests, headers, query parameters, and raw payloads before you configure local tunnels and endpoint parsers.

Summary

Building robust webhook consumers requires designing for the realities of distributed systems:

  1. Acknowledge fast: Return 200 OK immediately after signature verification and offload work to background queues.
  2. Prevent timing leaks: Use crypto.timingSafeEqual() instead of === for HMAC checks.
  3. Idempotency is mandatory: Track processed event IDs atomically to survive duplicate deliveries.
  4. Enforce timestamp windows: Discard events older than 300 seconds to eliminate replay vulnerabilities.
  5. Inspect the raw stream: Preserve raw buffers for cryptographic checks, and verify payloads with tools like Nutilz Webhook Tester before wiring complex pipeline logic.

Top comments (0)