DEV Community: EventDock

What is the worst production bug you have hit on Cloudflare Workers?

EventDock — Tue, 31 Mar 2026 17:02:41 +0000

I recently shipped a webhook relay service on Cloudflare Workers and hit some genuinely painful production bugs — the kind where everything looks fine but data is silently disappearing.

The worst one: an async function that was not awaited in a queue consumer. In Node.js, the fire-and-forget promise would probably complete. On Cloudflare Workers, the runtime kills the execution context once the handler returns, so the promise just... vanishes. Events that needed retries silently stopped existing.

Another fun one: a cron-based recovery system that was supposed to catch exactly these failures — except the cron trigger in wrangler.toml was commented out. The safety net did not exist for months.

I wrote up all four bugs in detail here: I Built a Webhook Relay on Cloudflare Workers. Here Are the Bugs That Killed It

What is the worst production bug you have hit on Cloudflare Workers (or any edge/serverless platform)? The "distributed systems fail in ways that look like success" category especially — where metrics look fine but data is quietly being lost.

Best Webhook Monitoring Tools in 2026: Complete Comparison

EventDock — Wed, 25 Mar 2026 11:20:16 +0000

The webhook tooling landscape in 2026 ranges from simple inspection tools to full infrastructure platforms. Choosing the right tool depends on what you're trying to do: debug during development, monitor in production, or guarantee delivery at scale. This guide compares seven popular tools across these dimensions.

I'll be honest upfront: I built EventDock, so I have a bias. I'll try to be fair about where each tool shines and where it falls short — including EventDock's limitations.

Quick Comparison

Tool	Primary Use Case	Retries	DLQ	Local Dev	Free Tier
EventDock	Production reliability	7 retries, exp. backoff	Yes + replay	No	5K events/mo
Hookdeck	Event gateway + routing	Configurable	Yes	CLI tunnel	100K events/mo
Svix	Sending webhooks (provider-side)	Yes (outbound)	Yes	Self-host option	Limited
RequestBin	Inspection & debugging	No	No	Yes (web)	Free (public)
webhook.site	Quick inspection	No	No	Yes (web)	Free (limited)
ngrok	Local tunnel for dev	No	No	Yes (primary use)	Free (1 tunnel)
Stripe CLI	Stripe webhook testing	No	No	Yes (Stripe only)	Free

EventDock — Production Webhook Reliability

What it does: EventDock is a webhook proxy that sits between your webhook providers (Stripe, Shopify, GitHub, etc.) and your application. It receives webhooks at the edge, stores them durably, and delivers to your endpoint with automatic retries, dead letter queue, and a real-time dashboard.

Best for: Teams that need production webhook reliability without building the infrastructure themselves. If you've lost webhook events due to deploys, timeouts, or server outages, EventDock prevents that.

Strengths:

Sub-50ms acknowledgment from the nearest edge (Cloudflare network)
7 retries with exponential backoff over 2+ hours
Dead letter queue with one-click replay
Real-time delivery dashboard with full payload inspection
Simple setup: change the webhook URL, done

Limitations:

No local development tunnel (use ngrok or Hookdeck CLI for local dev)
Newer product with a smaller community compared to Hookdeck or Svix
No webhook transformation or routing rules (it's a relay, not a gateway)
Free tier is 5K events/month (lower than Hookdeck's 100K)

Pricing: Free tier with 5,000 events/month and 3 endpoints. Paid plans for higher volume.

Hookdeck — Event Gateway and Routing

What it does: Hookdeck is a full-featured webhook event gateway. It receives webhooks, applies transformations, routes them to destinations based on rules, and provides monitoring and replay. It's the most feature-rich tool in this comparison.

Best for: Teams with complex webhook routing needs — multiple sources going to multiple destinations, event filtering, payload transformation, fan-out patterns.

Strengths:

Powerful routing and transformation rules
Generous free tier (100K events/month)
CLI tool for local development tunneling
Excellent dashboard with filtering and search
Event replay and bulk retry
Well-documented, active community

Limitations:

More complex setup for simple use cases (sources, connections, destinations)
Pricing scales with features — advanced routing on paid plans
Can be overkill if you just need "receive webhook, deliver reliably"

Pricing: Free tier with 100K events/month. Team plan starts at $75/month.

Svix — Webhook Sending Infrastructure

What it does: Svix solves the opposite problem from the other tools here. Instead of receiving webhooks, Svix helps you send them. If you're building an API or platform that needs to notify customers via webhooks, Svix provides the delivery infrastructure, retry logic, and management portal.

Best for: SaaS companies and API providers that need to send webhooks to their customers reliably. Think of it as "Stripe's webhook infrastructure, as a service."

Strengths:

Purpose-built for webhook sending (the provider side)
Customer-facing webhook management portal
Open-source core (self-hostable)
SDKs for multiple languages
Handles signature generation, retries, and endpoint management

Limitations:

Doesn't help with receiving webhooks — it's for the sending side
Not a monitoring or debugging tool
Self-hosting requires significant infrastructure

Pricing: Free tier with limited messages. Paid plans from $75/month. Open-source version available for self-hosting.

RequestBin — Quick Webhook Inspection

What it does: RequestBin (now part of Pipedream) gives you a temporary URL that captures and displays any HTTP request sent to it. You point your webhook at the RequestBin URL, trigger an event, and see exactly what was sent — headers, body, method, everything.

Best for: Development and debugging. When you're integrating a new webhook provider and want to see the exact payload format before writing handler code.

Strengths:

Instant setup — no account required for public bins
Clear, readable display of requests
Part of Pipedream ecosystem (can chain into workflows)
Good for one-off debugging

Limitations:

No retries, no delivery guarantees
Public bins are publicly accessible (don't send sensitive data)
Not designed for production use
Bins expire after a period of inactivity

Pricing: Free for public bins. Pipedream paid plans for private bins and workflows.

webhook.site — Instant Webhook Testing

What it does: Similar to RequestBin — gives you a unique URL that captures and displays incoming requests in real-time. Slightly simpler UX, focused purely on inspection.

Best for: Quick testing and debugging during development. The simplest possible way to see what a webhook provider is sending.

Strengths:

Extremely simple — visit the site, get a URL, done
Real-time updates in the browser
Supports custom responses (return specific status codes)
No account needed

Limitations:

Development tool only — not for production
No retries, forwarding, or reliability features
Limited history on the free tier
URLs are temporary

Pricing: Free tier with limited requests. Pro plan at $9/month for more features.

ngrok — Local Development Tunnels

What it does: ngrok creates a public URL that tunnels traffic to your local development machine. You run your webhook handler on localhost:3000, ngrok gives you a public URL like https://abc123.ngrok.io, and you point your webhook provider at that URL.

Best for: Local development. When you want to test webhook handlers on your laptop without deploying to a server.

Strengths:

Essential for local webhook development
Works with any webhook provider
Request inspection in the ngrok dashboard
Replay requests from the dashboard
Mature, reliable product

Limitations:

Development tool — not for production
Free tier URLs change every session (need paid plan for stable URLs)
No retry logic or reliability features
Adds latency (traffic routes through ngrok's servers)

Pricing: Free tier with 1 tunnel. Paid plans from $8/month for stable domains and more tunnels.

Stripe CLI — Stripe-Specific Webhook Testing

What it does: The Stripe CLI's stripe listen command forwards Stripe webhook events to your local development server. It can also trigger test events (stripe trigger payment_intent.succeeded) so you don't need to create real transactions during development.

Best for: Stripe-specific development. If you're building a Stripe integration and need to test webhook handlers locally.

Strengths:

Official Stripe tool — always up to date with Stripe's API
Can trigger specific event types on demand
Handles signature verification automatically
Free, no account setup beyond Stripe

Limitations:

Stripe only — doesn't work with Shopify, GitHub, etc.
Local development only — not for production monitoring
No retry logic, DLQ, or reliability features

Pricing: Free (part of Stripe's developer tools).

Which Tool Should You Use?

Most teams need different tools for different stages:

Stage	Recommended Tool	Why
Local development	ngrok or Stripe CLI	Tunnel webhooks to localhost for testing
Debugging payloads	webhook.site or RequestBin	See exactly what the provider sends
Production reliability	EventDock or Hookdeck	Retries, DLQ, monitoring, guaranteed delivery
Sending webhooks	Svix	If YOU are the webhook provider
Complex routing	Hookdeck	Multiple sources/destinations, transformations
Simple reliability	EventDock	Proxy setup, no routing complexity needed

EventDock vs. Hookdeck: The Main Decision

For production webhook reliability, EventDock and Hookdeck are the two primary options. The choice comes down to complexity vs. simplicity:

Choose Hookdeck if you need routing rules, payload transformations, fan-out to multiple destinations, or if the generous free tier matters (100K vs. 5K events/month).
Choose EventDock if you want the simplest possible setup (change one URL), don't need routing or transformations, and want edge-based delivery with sub-50ms acknowledgment.

Both tools solve the core problem: making sure webhook events don't get lost. The difference is in how much additional functionality you need around that core.

Frequently Asked Questions

What is the best webhook monitoring tool in 2026?

It depends on your stage: ngrok for local dev, webhook.site for quick inspection, EventDock or Hookdeck for production reliability, Svix for sending webhooks. Most teams use 2-3 tools across different stages.

Do I need a webhook monitoring tool for production?

If webhooks drive critical functionality (payments, orders, notifications), yes. Without monitoring, failures are invisible until customers complain. A reliability layer adds retries, DLQ, and visibility.

What is the difference between EventDock and Hookdeck?

Both provide webhook reliability. Hookdeck is a full event gateway with routing, transformations, and a larger free tier (100K events/month). EventDock is a simpler relay focused on fast edge delivery and zero-config setup (5K events/month free). Choose based on whether you need routing complexity.

Is webhook.site safe for production webhooks?

No. It's a development tool with no retries, no delivery guarantees, and temporary URLs. Never use it for production webhooks.

---

Try EventDock free

Webhook reliability in 2 minutes. Point your provider at EventDock, point EventDock at your server. Automatic retries, dead letter queue, and a real-time dashboard.

5,000 events/month, 3 endpoints, no credit card.

    [Start Free](https://dashboard.eventdock.app/login)

Exactly-Once Webhook Processing: The Pattern Every Developer Gets Wrong

EventDock — Wed, 25 Mar 2026 11:19:21 +0000

Every major webhook provider — Stripe, Shopify, GitHub, Twilio — delivers webhooks with at-least-once semantics. That means duplicates aren't a bug; they're a guarantee. Your system will receive the same event two, three, or more times. If you process each delivery independently, you'll charge customers twice, send duplicate emails, or create duplicate records.

Most developers know they need idempotency. The problem is that the most common idempotency pattern has a race condition that fails under real production load. This post covers the wrong way, the right way, and the battle-tested patterns for exactly-once webhook processing.

Why Webhook Duplicates Are Inevitable

Duplicates happen for multiple reasons, and you can't prevent any of them:

Provider retries: Your server returned 200 but the response was lost in transit (network blip, load balancer timeout). The provider sees no response and retries.
At-least-once queues: If the provider uses a message queue internally (most do), the queue may deliver the same message twice. This is a fundamental property of distributed message queues.
Webhook replay: Someone clicks "resend" in the Stripe dashboard, or an automated recovery system replays events.
Multiple delivery paths: Some systems have both real-time delivery and a recovery/catch-up mechanism that can send the same event through different paths.

You cannot make duplicates stop happening. You can only make your system handle them correctly.

The Pattern Everyone Uses (And Why It Breaks)

Here's the "obvious" idempotency pattern — check if you've seen the event, skip if you have:

// THE WRONG WAY: check-then-act (has a race condition)
app.post('/webhook', async (req, res) => {
  const eventId = req.body.id; // e.g., evt_abc123

  // Step 1: Check if already processed
  const existing = await db.processedEvents.findUnique({
    where: { eventId }
  });

  if (existing) {
    return res.json({ duplicate: true }); // Already handled
  }

  // Step 2: Process the event
  await processEvent(req.body);

  // Step 3: Mark as processed
  await db.processedEvents.create({
    data: { eventId, processedAt: new Date() }
  });

  res.json({ received: true });
});

This looks correct. It checks for duplicates, processes the event, records that it was processed. What could go wrong?

The Race Condition

Imagine two deliveries of the same event arrive 50ms apart (this happens regularly under load):

Timeline:
  T=0ms   Request A: SELECT * FROM processed_events WHERE event_id = 'evt_abc123'
  T=5ms   Request A: Result: no rows → not a duplicate
  T=10ms  Request B: SELECT * FROM processed_events WHERE event_id = 'evt_abc123'
  T=15ms  Request B: Result: no rows → not a duplicate  (!!!)
  T=20ms  Request A: processEvent(event)  → charges customer
  T=25ms  Request B: processEvent(event)  → charges customer AGAIN
  T=50ms  Request A: INSERT INTO processed_events (event_id, ...)
  T=55ms  Request B: INSERT INTO processed_events → duplicate key error (too late)

Both requests pass the duplicate check because neither has written to the database yet when the other checks. The customer gets charged twice. The second INSERT might fail on a unique constraint, but by then the damage is done.

This is the classic check-then-act race condition (also called TOCTOU — Time Of Check to Time Of Use). It's the same bug that causes double-spend in payment systems, double-voting in election software, and overselling in e-commerce. In webhook processing, it's everywhere.

The Correct Patterns

Pattern 1: Database Constraint as the Lock (Recommended)

Instead of checking then acting, use the database's unique constraint as an atomic lock. Insert first, process only if the insert succeeds:

// THE RIGHT WAY: insert-first with unique constraint
app.post('/webhook', async (req, res) => {
  const eventId = req.body.id;

  // Attempt to claim this event atomically
  try {
    await db.processedEvents.create({
      data: {
        eventId,
        status: 'processing',
        receivedAt: new Date()
      }
    });
  } catch (err) {
    if (isDuplicateKeyError(err)) {
      // Another request already claimed this event
      return res.json({ duplicate: true });
    }
    throw err; // Some other database error
  }

  // If we get here, we "own" this event — no race condition
  try {
    await processEvent(req.body);
    await db.processedEvents.update({
      where: { eventId },
      data: { status: 'processed', processedAt: new Date() }
    });
  } catch (err) {
    await db.processedEvents.update({
      where: { eventId },
      data: { status: 'failed', error: err.message }
    });
    // Don't rethrow — we still return 200 so the provider doesn't retry
    // The failed event needs manual investigation or a retry job
  }

  res.json({ received: true });
});

function isDuplicateKeyError(err: any): boolean {
  // Prisma
  if (err.code === 'P2002') return true;
  // PostgreSQL
  if (err.code === '23505') return true;
  // MySQL
  if (err.errno === 1062) return true;
  // SQLite
  if (err.code === 'SQLITE_CONSTRAINT_UNIQUE') return true;
  return false;
}

The key insight: the INSERT with a unique constraint on eventId is atomic at the database level. Two concurrent requests can't both succeed. One will insert, the other will get a duplicate key error. No race condition possible.

Pattern 2: SELECT ... FOR UPDATE (Pessimistic Locking)

If you need more flexibility (e.g., the idempotency key isn't the primary key), use row-level locking:

// Pessimistic locking with a transaction
app.post('/webhook', async (req, res) => {
  const eventId = req.body.id;

  await db.$transaction(async (tx) => {
    // Lock the row (or lock nothing if it doesn't exist yet)
    const existing = await tx.$queryRaw`
      SELECT * FROM processed_events
      WHERE event_id = \${eventId}
      FOR UPDATE
    `;

    if (existing.length > 0) {
      return; // Already processed, skip
    }

    await tx.processedEvents.create({
      data: { eventId, status: 'processing' }
    });

    await processEvent(req.body);

    await tx.processedEvents.update({
      where: { eventId },
      data: { status: 'processed' }
    });
  });

  res.json({ received: true });
});

Tradeoff: This holds a database lock for the entire duration of processing. Fine for fast operations, but problematic if processEvent() takes seconds or calls external APIs. The lock blocks all other requests for the same event.

Pattern 3: Idempotency Key in a Separate Store (Redis/KV)

For high-throughput systems where database transactions are expensive, use an atomic set-if-not-exists operation in a fast key-value store:

// Redis-based idempotency with atomic SETNX
app.post('/webhook', async (req, res) => {
  const eventId = req.body.id;
  const lockKey = `webhook:lock:\${eventId}`;

  // SETNX: set only if key doesn't exist (atomic)
  // EX 86400: expire after 24 hours (cleanup)
  const acquired = await redis.set(lockKey, 'processing', 'EX', 86400, 'NX');

  if (!acquired) {
    // Another process already claimed this event
    return res.json({ duplicate: true });
  }

  try {
    await processEvent(req.body);
    await redis.set(lockKey, 'processed', 'EX', 86400);
  } catch (err) {
    // Release the lock so a retry can attempt processing
    await redis.del(lockKey);
    throw err;
  }

  res.json({ received: true });
});

Tradeoff: Redis is fast but not as durable as a database. If Redis restarts between the lock acquisition and processing completion, you might process the event twice. For most webhook use cases, this is acceptable — Redis rarely restarts, and the 24-hour TTL keeps the keyspace clean.

Which Pattern Should You Use?

Pattern	Best For	Tradeoff
Unique constraint (Pattern 1)	Most applications	Requires database write before processing
SELECT FOR UPDATE (Pattern 2)	Complex idempotency keys	Holds lock during processing
Redis SETNX (Pattern 3)	High-throughput systems	Less durable than database

For 90% of webhook handlers, Pattern 1 (unique constraint) is the right choice. It's simple, correct, and uses infrastructure you already have.

Edge Cases That Still Bite You

Processing Succeeds But Status Update Fails

You process the event (charge the customer), but the database update to mark it as "processed" fails (network error, database full). On retry, you see the event in "processing" state. Is it safe to reprocess? You need a way to check the actual side effect (was the charge created?) rather than just the status field.

Non-Deterministic Processing

If processEvent() creates resources with random IDs, processing the same event twice creates two different resources. True idempotency means the same input always produces the same output. Use deterministic IDs derived from the event ID where possible.

Multi-Step Processing

If processing involves multiple steps (update DB, send email, call API), partial failure means some steps completed and some didn't. On retry, you need to skip completed steps. This is the saga pattern, and it's significantly more complex than single-step idempotency.

How EventDock Handles Deduplication

EventDock deduplicates at the infrastructure level before events reach your handler. Each event gets a unique ID, and the delivery pipeline uses KV-based atomic dedup to ensure your endpoint receives each event exactly once — even if the webhook provider delivers it multiple times.

This means your webhook handler doesn't need any of the patterns above. You write simple processing logic, and EventDock guarantees you won't see the same event twice. If you want defense-in-depth, you can still add application-level idempotency using Pattern 1 — but you won't need it for correctness.

Frequently Asked Questions

Why do webhooks get delivered more than once?

At-least-once delivery is a fundamental property of distributed systems. If the provider doesn't receive your 200 response (network blip, load balancer timeout, process crash), it retries. Some providers also use internal queues that may deliver the same message twice. You cannot prevent duplicates — only handle them.

What is the check-then-act race condition?

It's when you check if an event was processed (SELECT), then process it, then mark it done (INSERT). Two concurrent requests can both pass the check before either writes, causing double processing. Fix it by inserting first with a unique constraint — the database enforces atomicity.

What is an idempotency key for webhooks?

A unique identifier for a webhook event used to detect duplicates. Stripe provides event.id, Shopify provides X-Shopify-Webhook-Id, GitHub provides X-GitHub-Delivery. Store these with a unique database constraint.

How do I make my webhook handler idempotent?

Use the insert-first pattern: INSERT the idempotency key with a unique constraint. If the insert succeeds, process the event. If it fails with a duplicate key error, skip it. This is the only pattern that's both correct and race-condition-free without requiring explicit locks.

---

Skip the dedup complexity

EventDock deduplicates webhook events at the infrastructure level. Your handler receives each event exactly once — no idempotency keys, no race conditions, no duplicate processing.

5,000 events/month free. No credit card required.

    [Start Free](https://dashboard.eventdock.app/login)

Shopify Webhook Reliability: Why Orders Go Missing and How to Fix It

EventDock — Wed, 25 Mar 2026 11:19:20 +0000

If you run a Shopify store with custom integrations, you've probably hit this: an order comes in, but your fulfillment system, inventory tracker, or CRM never gets notified. The order exists in Shopify, but it's invisible to the rest of your stack. The culprit is almost always a failed webhook.

Shopify's webhook system has specific constraints that make it surprisingly easy to lose events. This guide covers the exact failure modes, how to verify webhook signatures properly, and how to guarantee you never miss an order again.

Shopify's Webhook Constraints (The Ones That Bite You)

Shopify's webhook delivery system has three properties that combine to create reliability problems:

1. The 5-Second Timeout

Shopify gives your endpoint 5 seconds to respond with a 2xx status code. If your server takes longer — due to a slow database query, a cold start, or downstream API calls — Shopify marks the delivery as failed.

Five seconds sounds generous until you realize what happens during real production conditions:

Your server is restarting during a deploy (10-30 second gap)
Your database connection pool is exhausted during a flash sale
You're calling a third-party API (shipping provider, ERP) synchronously in the handler
Lambda/serverless cold start adds 2-3 seconds, leaving you 2 seconds for actual work

2. Limited Retry Window

When delivery fails, Shopify retries up to 19 times over 48 hours with increasing delays. After that, the webhook subscription is automatically marked as degraded. If failures continue, Shopify may delete your webhook subscription entirely — meaning you stop receiving ALL events of that type, not just the one that failed.

This is the most dangerous behavior: a temporary outage on your end can permanently break your integration if you don't catch it quickly.

3. Mandatory Webhook Topics

Starting in 2025, Shopify requires apps to subscribe to certain webhook topics (like app/uninstalled and compliance-related webhooks). If your app doesn't handle these properly, it can fail app review or lose API access. You can't afford to miss these events.

Verifying Shopify Webhook Signatures (HMAC-SHA256)

Every Shopify webhook includes an X-Shopify-Hmac-SHA256 header — a Base64-encoded HMAC digest of the request body, signed with your app's shared secret. You must verify this before processing any webhook.

Here's the correct implementation in Node.js:

import crypto from 'crypto';

function verifyShopifyWebhook(
  rawBody: Buffer,
  hmacHeader: string,
  secret: string
): boolean {
  const generatedHash = crypto
    .createHmac('sha256', secret)
    .update(rawBody)
    .digest('base64');

  // Use timingSafeEqual to prevent timing attacks
  try {
    return crypto.timingSafeEqual(
      Buffer.from(generatedHash),
      Buffer.from(hmacHeader)
    );
  } catch {
    return false; // Lengths don't match
  }
}

// Express middleware
app.post('/webhooks/shopify',
  express.raw({ type: 'application/json' }),
  async (req, res) => {
    const hmac = req.headers['x-shopify-hmac-sha256'] as string;

    if (!verifyShopifyWebhook(req.body, hmac, process.env.SHOPIFY_WEBHOOK_SECRET!)) {
      console.error('Shopify webhook signature verification failed');
      return res.status(401).send('Unauthorized');
    }

    const event = JSON.parse(req.body.toString());
    const topic = req.headers['x-shopify-topic'] as string;

    // ACK immediately — process async
    res.status(200).send('OK');

    // Process in background
    try {
      await processShopifyEvent(topic, event);
    } catch (err) {
      console.error('Failed to process Shopify webhook:', topic, err);
      // Without a reliability layer, this event may be lost
    }
  }
);

Critical detail: You must use the raw request body for HMAC verification, not a parsed-and-re-serialized version. If you use express.json() middleware and then JSON.stringify(req.body), the signature won't match because JSON serialization isn't deterministic (key ordering, whitespace).

The 5 Ways Shopify Webhooks Go Missing

1. Deploy Windows

Every time you deploy, there's a window where your server is unavailable. If Shopify sends an orders/create webhook during that window, it fails. With blue-green deployments you can minimize this, but you can't eliminate it entirely — especially with database migrations.

2. Synchronous Processing

The most common mistake: doing all your work inside the webhook handler before returning 200. If you're updating inventory, notifying a warehouse, and sending a customer email — all before responding — you'll blow past the 5-second timeout.

// BAD: synchronous processing
app.post('/webhooks/shopify/orders', async (req, res) => {
  const order = req.body;
  await updateInventory(order);        // 1.5s
  await notifyWarehouse(order);        // 2s (external API)
  await sendConfirmationEmail(order);  // 1s
  await updateCRM(order);             // 1.5s
  res.status(200).send('OK');         // Total: 6s — TIMEOUT
});

// GOOD: ACK first, process async
app.post('/webhooks/shopify/orders', async (req, res) => {
  const order = req.body;
  await db.webhookQueue.insert({ topic: 'orders/create', payload: order });
  res.status(200).send('OK');  // <100ms

  // Process via background job/queue worker
});

3. Unhandled Errors Crashing the Process

If your handler throws an unhandled exception, the HTTP connection drops without a response. Shopify sees this as a failure. If your error is deterministic (like a missing field in a new webhook format), every retry will fail the same way.

4. Scaling Issues During Flash Sales

Shopify can send hundreds of orders/create webhooks per minute during a flash sale. If your server can't keep up — connection pool exhaustion, memory pressure, CPU saturation — webhooks start timing out. Shopify's retry mechanism doesn't back off fast enough to help you recover, and the retries add even more load.

5. Webhook Subscription Deletion

This is the silent killer. If your endpoint fails consistently, Shopify doesn't just stop retrying the individual events — it removes the webhook subscription. New events aren't even attempted. You have to re-register the webhook, and the events that occurred during the gap are permanently lost.

Making Shopify Webhooks Bulletproof

There are two approaches: build it yourself or use a reliability layer.

DIY Approach

If you're building it yourself, you need:

ACK immediately: Return 200 within 1 second, process asynchronously via a job queue (BullMQ, SQS, etc.)
Idempotency: Use the X-Shopify-Webhook-Id header as a dedup key — Shopify may deliver the same event multiple times
Reconciliation job: Periodically poll the Shopify Orders API to find orders that your system missed, comparing Shopify's records against your database
Subscription monitoring: Check your webhook subscriptions daily using the Admin API — if one disappeared, re-register it and backfill from the API
Dead letter queue: Events that fail processing after N retries need a place to go where you can inspect and replay them manually

That's a solid approach, but it's easily 2-3 weeks of engineering work to build and maintain properly.

Reliability Layer Approach

Put a webhook proxy between Shopify and your server. Instead of pointing Shopify at your endpoint directly, point it at a reliability layer that:

Responds to Shopify within milliseconds (never times out)
Stores every event durably before forwarding
Retries delivery to your server with exponential backoff
Provides a dead letter queue for manual inspection and replay
Never lets Shopify see your endpoint's failures

This is what EventDock does. Because EventDock responds to Shopify in under 50ms from the nearest edge node, Shopify never sees a timeout. Your webhook subscription stays healthy even if your server has a bad day.

# Setup:
# 1. Create endpoint in EventDock dashboard
#    Provider: Shopify
#    Destination: https://yourapp.com/webhooks/shopify
#    You get: https://in.eventdock.app/ep_xyz789
#
# 2. In Shopify Admin → Settings → Notifications → Webhooks:
#    Point all webhook topics at: https://in.eventdock.app/ep_xyz789
#
# Now: Shopify → EventDock (instant ACK, durable storage)
#       → Your server (retries, DLQ, monitoring)
#
# Your server can be slow, restart, or even go down.
# EventDock holds the events and delivers when you're ready.

Frequently Asked Questions

How long does Shopify retry failed webhooks?

Shopify retries up to 19 times over 48 hours with increasing delays. After all retries fail, the event is permanently lost. Worse, persistent failures can cause Shopify to delete your webhook subscription entirely, meaning you stop receiving all future events of that type.

Why is my Shopify webhook subscription disappearing?

Shopify automatically removes webhook subscriptions that consistently fail. If your endpoint returns errors or times out repeatedly, the subscription gets marked as degraded and eventually deleted. Using a webhook reliability layer prevents this by always responding to Shopify instantly, keeping your subscription healthy.

What is the Shopify webhook timeout limit?

5 seconds, strictly enforced and not configurable. Your endpoint must respond with a 2xx status within 5 seconds or the delivery is marked as failed. This is why synchronous processing in webhook handlers is dangerous.

How do I verify Shopify webhook signatures in Node.js?

Compute an HMAC-SHA256 digest of the raw request body using your app's shared secret, Base64-encode it, and compare it against the X-Shopify-Hmac-SHA256 header using crypto.timingSafeEqual(). The key detail is using the raw body — not a parsed and re-serialized version.

Can I recover missed Shopify webhooks?

You can poll Shopify's Admin API for resources (orders, products, etc.) and reconcile against your database. But this is a workaround, not a solution. It only works for resources with API endpoints, adds API rate limit pressure, and requires building a custom reconciliation system. Better to prevent loss in the first place with a reliability layer.

---

Never miss a Shopify order webhook again

EventDock sits between Shopify and your server, storing every event durably and delivering with automatic retries. Your Shopify webhook subscriptions stay healthy, even when your server doesn't.

5,000 events/month free. Set up in 2 minutes.

    [Start Free](https://dashboard.eventdock.app/login)

I Built a Webhook Relay on Cloudflare Workers. Here Are the Bugs That Killed It

EventDock — Mon, 23 Mar 2026 20:16:52 +0000

Four production bugs. All invisible in staging. Each one silently dropping data while every dashboard metric looked perfectly healthy. Here is what I learned building a webhook relay on Cloudflare Workers — and why distributed systems fail in ways that look like success.

I built EventDock, a webhook reliability layer that sits between webhook providers (Stripe, GitHub, etc.) and your application. The idea is simple: accept the webhook instantly, store it durably, and deliver it to your endpoint with retries, logging, and a dead letter queue.

I chose Cloudflare Workers as the platform. Edge compute seemed like the perfect fit — webhook providers have short timeouts (Stripe gives you ~20 seconds), so you want to ACK as fast as possible. A Worker can respond in under 50ms from the nearest edge node. No cold starts, no servers to manage, global by default.

The architecture works beautifully on paper. Getting it to work reliably in production required finding and fixing bugs that were invisible in development and staging. Here are the four that almost killed the project.

The Architecture

Before diving into the bugs, here's the flow:

Provider (Stripe, GitHub, etc.)
  → CF Worker (ingest) — validates, stores to D1, enqueues
    → CF Queue — at-least-once delivery semantics
      → CF Worker (delivery) — fetches payload, delivers to customer endpoint
        → Customer's app

The supporting cast:

D1 (SQLite at the edge) — stores event metadata, delivery state, and retry counts
KV — idempotency keys and deduplication
R2 — payload storage for large webhook bodies
Cron triggers — a recovery mechanism that finds stuck events and requeues them

The key design decision: the ingest worker does minimal work. Accept the webhook, write to D1 and the queue, return 200. Everything else happens asynchronously. This keeps the p99 response time under 100ms, which matters when Stripe is waiting for your response.

Bug #1: The Unwaited Retry

This one was subtle and devastating. In the queue consumer (the delivery worker), when a delivery attempt failed and needed to be retried, the code looked like this:

// The queue consumer processes batches of messages
async queue(batch: MessageBatch, env: Env) {
  for (const msg of batch.messages) {
    const event = msg.body;
    const result = await deliverWebhook(event, env);

    if (!result.ok) {
      // Schedule retry with exponential backoff
      handleRetry(event, env);  // ← THE BUG
    }

    msg.ack();
  }
}

handleRetry() is an async function. It writes to D1 to update retry count and schedules the next attempt. But it wasn't being awaited.

In Node.js, this would be a fire-and-forget — the retry would probably still complete in the background. In Cloudflare Workers, it's a death sentence. Workers are request-scoped. Once you call msg.ack() and the queue batch handler returns, the runtime can (and will) terminate the execution context. The unresolved promise from handleRetry() just... disappears.

The fix was one word:

// Before (broken)
handleRetry(event, env);
msg.ack();

// After (fixed)
await handleRetry(event, env);
msg.ack();

Why it was hard to find: Events appeared to be processing. The first delivery attempt worked fine. It was only events that needed retries that silently vanished. And in development, you're usually testing against endpoints that succeed. The failure mode only appeared under real production conditions — intermittent endpoint failures, timeouts, 500s.

The monitoring showed events being consumed from the queue (good!) but some never reaching a final delivered/failed state. They just sat in "delivering" forever. Classic fire-and-forget symptom, but only obvious in retrospect.

Bug #2: Queue Send Without Try-Catch

The ingest handler had a straightforward flow: validate the webhook, store the event in D1, then send it to the queue for delivery.

// Ingest handler (simplified)
async function handleWebhook(request: Request, env: Env) {
  const payload = await request.json();
  const event = createEvent(payload);

  // Step 1: Save to D1 (durable state)
  await env.DB.prepare(
    'INSERT INTO events (id, endpoint_id, payload, status) VALUES (?, ?, ?, ?)'
  ).bind(event.id, event.endpointId, event.payload, 'pending').run();

  // Step 2: Enqueue for delivery
  await env.QUEUE.send({
    eventId: event.id,
    endpointId: event.endpointId
  });
  // ↑ No try-catch. If this throws, the event is saved but never queued.

  return new Response('OK', { status: 200 });
}

The problem: QUEUE.send() can fail. Cloudflare Queues are generally reliable, but they're a distributed system — transient errors happen. When the queue send failed, the error propagated up and the handler returned a 500 to the webhook provider. But the D1 write had already committed.

So now you have an event in the database with status "pending" that will never be picked up by the queue consumer. The customer sees it in their dashboard as "received" but it never gets delivered. It's a zombie event.

// After: catch queue failures, mark for cron recovery
await env.DB.prepare(
  'INSERT INTO events (id, endpoint_id, payload, status) VALUES (?, ?, ?, ?)'
).bind(event.id, event.endpointId, event.payload, 'pending').run();

try {
  await env.QUEUE.send({
    eventId: event.id,
    endpointId: event.endpointId
  });
} catch (e) {
  // Event is in D1 — the cron recovery job will find it
  // and requeue it. Log the error but return 200 to the provider.
  console.error('Queue send failed, event will be recovered by cron', event.id, e);
}

// Always return 200 — the event is durably stored regardless
return new Response('OK', { status: 200 });

The insight: the D1 write IS the source of truth. If the event is in the database, the system should eventually deliver it. The queue is an optimization for fast delivery, not the guarantee. The guarantee comes from the cron recovery job that scans for stuck events.

Which brings us to bug #3.

Bug #3: The Safety Net That Never Ran

From the very beginning, I built a cron-triggered recovery system. Every few minutes, a Worker runs, queries D1 for events stuck in "pending" or "delivering" for too long, and requeues them. This was supposed to be the safety net for exactly the kind of failure in Bug #2.

The code was solid. Tested in development. Ready to go.

One problem: the cron trigger in wrangler.toml was commented out.

# wrangler.toml (production env)

[env.production]
name = "eventdock-worker-prod"
# ... other config ...

# [env.production.triggers]
# crons = ["*/5 * * * *"]

I had commented it out during early development (probably debugging something) and never uncommented it. The deployment pipeline didn't warn about missing triggers. There's no "expected crons" health check. The recovery system silently didn't exist for months.

During those months, any event that hit Bug #2's failure mode (or any other edge case where the queue send didn't fire) was just... gone. Saved in D1 but never delivered and never recovered.

The fix was uncommenting two lines. The lesson was harder: your safety nets need their own monitoring. If the recovery cron hasn't run in the last 10 minutes, that itself should be an alert.

[env.production.triggers]
crons = ["*/5 * * * *"]

A note on Wrangler v3 syntax: cron triggers for specific environments need the [env.production.triggers] block with a crons = [...] array. Not [triggers] at the top level (that's the default env), and not cron = (singular). This particular config syntax isn't well-documented, and getting it wrong means your crons silently don't deploy.

Bug #4: The Ghost Consumer

This one was the most disorienting to debug. Events were being enqueued correctly (I could see the queue depth increasing and then dropping back to zero), but some events were never delivered. The delivery worker's logs showed nothing — no attempts, no errors, nothing. Events just vanished from the queue without a trace.

The cause: my wrangler.toml had a queue consumer binding in the default (top-level) environment, not just in [env.production].

# wrangler.toml (the problem)

# Default env — leftover from initial setup
[[queues.consumers]]
queue = "eventdock-deliveries"
max_batch_size = 10

# Production env — the "real" consumer
[env.production]
[[env.production.queues.consumers]]
queue = "eventdock-deliveries"
max_batch_size = 10

When I deployed with wrangler deploy --env production, it deployed the production worker. But the default environment's consumer config meant there was a previous worker deployment (from running wrangler deploy without --env during early development) that was ALSO registered as a consumer on the same queue.

Cloudflare Queues distributes messages across all registered consumers. So roughly half the events went to the production worker (which delivered them correctly) and half went to a stale, unmonitored worker that consumed them and did nothing useful.

The fix was removing the default environment's consumer config and deleting the stale worker deployment. But finding it required going to the Cloudflare dashboard, looking at the queue's consumer list, and realizing there were two workers consuming from the same queue.

The debugging process took hours because every metric looked "kind of right." Queue depth went up, queue depth went down, delivery rate was non-zero. It was only when I compared "events enqueued" to "events delivered" over a 24-hour window that the ~50% loss rate became obvious.

Edge Compute Gotchas (Things I Wish I'd Known)

Beyond the specific bugs, here are the platform-level lessons from running a production system on Cloudflare Workers:

D1 is SQLite, and that matters. It's excellent for reads and great for the kind of workload a webhook relay generates (mostly inserts and point lookups). But SQLite has write contention characteristics you need to understand. Concurrent writes to the same database serialize. Under high throughput, your D1 write latency can spike. Batch your writes where possible.

CF Queues are at-least-once, not exactly-once. This is documented, but you need to internalize what it means: your queue consumer MUST be idempotent. The same message can be delivered multiple times. If your delivery handler isn't idempotent, you'll send duplicate webhooks to your customers. We use KV-based deduplication keyed on event ID.

No setTimeout, no background work. Workers are request-scoped. When the request handler (or queue batch handler) returns, your execution context can be terminated. Any work you haven't awaited is at risk. This is fundamentally different from a long-running Node.js server where fire-and-forget async calls will "probably" complete. On Workers, they probably won't.

Wrangler config is your infrastructure-as-code. Unlike traditional IaC tools (Terraform, Pulumi), Wrangler doesn't have a plan/apply cycle. wrangler deploy just does it. There's no diff, no confirmation, and misconfigured environments are silent failures. Treat wrangler.toml with the same rigor you'd treat a Terraform file — code review every change.

What I'd Do Differently

If I were starting over, I'd invert the delivery architecture. Instead of queue-based delivery as the primary path with cron recovery as a safety net, I'd make the cron the primary mechanism.

Here's why: the cron recovery pattern is inherently reliable. It scans D1 for undelivered events, attempts delivery, and updates state. It doesn't depend on queue health, consumer registration, or message acking semantics. It's a simple polling loop backed by a durable database.

The queue would become a performance optimization — a fast path that delivers events within seconds instead of waiting for the next cron tick. But the system would be correct and complete with ONLY the cron. The queue just makes it faster.

This is the same insight behind patterns like the transactional outbox pattern: write to the database first, process asynchronously second. The database is the source of truth, and the async mechanism is an optimization.

Where It Stands Now

After fixing all four bugs, the system works well:

Ingest latency: sub-100ms p99 (usually under 50ms)
Delivery: 7 retries with exponential backoff over 2+ hours
Recovery: cron scans every 5 minutes for stuck events
Dead letter queue: events that exhaust all retries go to DLQ for manual inspection
Idempotency: KV-based dedup prevents duplicate deliveries

The common thread through all four bugs: distributed systems fail in ways that look like success. Events appeared to be processing. Queue depth looked healthy. The dashboard showed events being received. But under the surface, promises weren't being awaited, queues were silently failing, safety nets weren't running, and ghost workers were eating messages.

The only way to find these bugs was to compare inputs to outputs at every stage: events received vs. events enqueued vs. events delivered. Any discrepancy means something is silently dropping data. If you're building on Cloudflare Workers (or any edge compute platform), build that end-to-end observability from day one. You'll need it.

EventDock is a webhook reliability layer for teams that can't afford to lose events. If you've hit these kinds of problems and don't want to build the infrastructure yourself, check out eventdock.app.

EventDock vs Svix: They Solve Different Problems

EventDock — Mon, 23 Mar 2026 20:08:14 +0000

If you're researching webhook infrastructure, you've probably seen both Svix and EventDock. They sound similar — both deal with webhooks, both handle retries, both have APIs. But they solve fundamentally different problems.

This is one of the most common confusions in the webhook space, so let's clear it up.

The Key Difference: Sending vs. Receiving

The entire difference comes down to one question: are you sending webhooks or receiving them?

	Svix	EventDock
Core function	Send webhooks to your users	Receive webhooks from providers
You are...	A SaaS building webhook delivery	A developer consuming Stripe/Shopify/GitHub
Direction	Your app → Customer servers	Provider servers → Your app
Problem solved	How to reliably notify your users	How to reliably receive notifications

When You Need Svix

Svix is the right choice when you are the webhook sender. Specifically:

You're building a SaaS product and need to notify customers when things happen (new order, payment processed, status change)
Your customers expect webhook integrations — they want to receive events from your platform in their own systems
You need a management portal where your customers can configure their webhook endpoints, see delivery logs, and retry failures
You want to offer webhook delivery as a feature without building the infrastructure yourself

Svix handles the hard parts of webhook sending:

// Using Svix to SEND webhooks to your customers
import { Svix } from 'svix';
const svix = new Svix('your-auth-token');

// When something happens in your app, send a webhook
await svix.message.create('app_customer123', {
  eventType: 'order.completed',
  payload: {
    orderId: 'ord_456',
    amount: 9900,
    currency: 'usd',
  },
});

// Svix handles:
// - Delivery to the customer's endpoint
// - Retries if their server is down
// - Signature generation for verification
// - A portal for customers to manage their endpoints

This is a real engineering problem. Sending webhooks reliably to thousands of customer endpoints with different uptimes, response times, and configurations is complex. Svix is good at it.

When You Need EventDock

EventDock is the right choice when you are the webhook receiver. Specifically:

You integrate with Stripe, Shopify, GitHub, or any provider that sends you webhooks
You've had webhooks fail silently during deploys, outages, or bugs
You need retries, dead letter queues, and monitoring for incoming webhooks
You want automatic signature verification for major providers
You want to stop worrying about whether your server was up when Stripe sent that payment event

EventDock sits between the provider and your server:

# Without EventDock:
Stripe --webhook--> Your Server
                    (down during deploy? event lost.)

# With EventDock:
Stripe --webhook--> EventDock (edge, always up)
                        |
                        +--> Your Server (retries, DLQ, monitoring)

# Setup:
# 1. Create endpoint at dashboard.eventdock.app
# 2. Get URL: https://in.eventdock.app/ep_abc123
# 3. Point Stripe at that URL
# 4. Set your server as the destination
# Done. 2 minutes.

Can You Use Both?

Yes — and some teams do. They're complementary, not competing.

If you're building a SaaS that both receives webhooks from providers (Stripe for payments, GitHub for CI) AND sends webhooks to your own customers (notifying them of events in your platform), you might use:

EventDock on the receiving side — making sure you never miss a Stripe payment event or a GitHub push notification
Svix on the sending side — reliably delivering your own webhooks to your customers' endpoints

They operate on opposite sides of the webhook flow and don't overlap.

Detailed Feature Comparison

Feature	Svix (Sending)	EventDock (Receiving)
Retries with backoff	Yes (for outgoing deliveries)	Yes (for incoming deliveries to your server)
Dead letter queue	Yes	Yes, with one-click replay
Signature handling	Signs outgoing webhooks	Verifies incoming signatures automatically
Dashboard	Customer-facing management portal	Developer-facing delivery monitoring
Provider integrations	N/A (you are the provider)	Stripe, Shopify, GitHub, and more
Edge-native	No	Yes (Cloudflare Workers, global PoPs)
Self-hostable	Yes (open source)	No (managed service)
API-first	Yes (SDK for sending)	Yes (API for management, URL-based for receiving)
Idempotency	Event-type-based dedup	Built-in 24h deduplication window
Setup time	Hours (SDK integration in your codebase)	~2 minutes (URL swap)

Why People Confuse Them

The confusion makes sense. Both services:

Use the word "webhook" in their marketing
Offer retries and reliability
Have APIs and dashboards
Solve real infrastructure problems

But the direction is opposite. A simple test:

"I need to send webhooks to my customers" → Svix
"I need to receive webhooks from Stripe/Shopify/GitHub" → EventDock
"I need both" → Use both

What About Other Options?

For Sending (Svix alternatives)

If you're looking at Svix for sending webhooks, other options include building it yourself (risky), using a message queue like SQS/RabbitMQ with custom delivery code (complex), or services like Hookdeck's outbound features.

For Receiving (EventDock alternatives)

If you're looking at EventDock for receiving webhooks, other options include Hookdeck (more features, more complex, usage-based pricing) or building your own reliability layer with queues and retry logic (weeks of work).

Frequently Asked Questions

Is EventDock a Svix alternative?

Not exactly. They solve different problems. Svix helps you send webhooks to your customers. EventDock helps you receive webhooks reliably. They're complementary, not competitors.

Can I use Svix and EventDock together?

Yes. Use EventDock on the receiving side (Stripe, Shopify, GitHub webhooks coming into your app) and Svix on the sending side (your app notifying your customers).

What's the difference between sending and receiving webhooks?

Sending: your app notifies other systems when events happen. Receiving: your app listens for events from other services. Different infrastructure is needed for each direction.

Is Svix open source?

Yes, Svix offers an open-source webhook server you can self-host. EventDock is a managed service running on Cloudflare's edge network.

Do I need a webhook service if I only receive from one provider?

Even with one provider, webhooks fail during deploys and outages. EventDock's free tier (5,000 events/month) covers most single-provider setups.

Receive webhooks reliably. EventDock adds retries, DLQ, signature verification, and monitoring to your incoming webhooks. Set up in 2 minutes, free for 5,000 events/month.

I Vibe-Coded a SaaS and My Webhooks Were Silently Failing — Here's What I Missed

EventDock — Mon, 23 Mar 2026 20:07:28 +0000

You built an entire SaaS in a weekend. Cursor, Copilot, or Claude wrote most of the code. It works, it's deployed, users are signing up. That's genuinely impressive. But there's one thing AI code generators almost always get wrong: webhooks.

This isn't about AI-generated code being bad. It's about a specific blind spot. When you ask an AI to "add a Stripe webhook endpoint" or "handle Shopify order notifications," you get code that works in the happy path. But webhooks aren't a happy-path problem — they're an infrastructure reliability problem. And AI tools don't think about infrastructure reliability unless you explicitly ask.

What AI Actually Generates for Webhooks

Here's what you typically get when you ask Cursor or Copilot to add a webhook handler:

// AI-generated webhook handler (typical output)
app.post('/webhooks/stripe', async (req, res) => {
  const event = req.body;

  if (event.type === 'checkout.session.completed') {
    const session = event.data.object;
    await db.users.update({
      where: { email: session.customer_email },
      data: { plan: 'pro', stripeId: session.customer },
    });
  }

  res.json({ received: true });
});

This code works. It handles the event, updates the database, returns 200. Ship it.

But here's what's missing:

No signature verification. Anyone can POST to your webhook URL and fake a payment event. AI tools rarely add stripe.webhooks.constructEvent() unless you specifically ask.
No retry handling. If your database is slow or your server restarts during a deploy, the webhook fails. There's no retry logic.
No idempotency. Stripe may deliver the same event twice. Without deduplication, you'll grant access twice, send two confirmation emails, or double-charge.
No error handling for the webhook itself. If db.users.update() throws, the entire handler crashes. Stripe gets a 500 and retries, but your code will crash again the same way.
No dead letter queue. After Stripe gives up retrying (3 days), the event is gone. Your customer paid but never got access. You have no record of the failure.
No monitoring or alerts. You won't know webhooks are failing until a customer emails you.

Why AI Code Generators Miss This

It's not a bug in the AI — it's a natural limitation of how code generation works:

AI optimizes for the prompt. "Add a Stripe webhook" means: create an endpoint that handles Stripe events. The AI delivers exactly that. It doesn't add infrastructure it wasn't asked for.
Training data bias. Most webhook tutorials and Stack Overflow answers show the happy path. The AI learned from code that doesn't include retry logic because most example code doesn't either.
Reliability is cross-cutting. Retry logic, DLQ, monitoring — these aren't features of your webhook handler. They're infrastructure concerns that span your entire system. AI generates code file-by-file, not architecture-by-architecture.
The failure mode is silent. A missing button is obvious. A webhook that fails during a deploy at 3 AM is invisible. AI can't warn you about problems it can't detect.

Real Failure Scenarios in Vibe-Coded Apps

These are the patterns we see repeatedly in apps built with AI assistance:

The Deploy Gap

You push a new version. Your server restarts. During those 10-30 seconds, Stripe sends a payment_intent.succeeded event. Your server is down. Stripe retries a few times, then gives up after 3 days. The customer paid $99 but never got access to your product.

The Unhandled Exception

Your AI-generated handler works for checkout.session.completed but crashes on customer.subscription.updated because the payload structure is slightly different. Every subscription update webhook returns a 500. Stripe retries 16 times over 3 days, then stops. You don't find out until users report they can't cancel.

The Spoofed Payment

Without signature verification, someone discovers your webhook URL (it's usually predictable — /webhooks/stripe) and sends a fake checkout.session.completed event. Your code grants them Pro access without payment.

The Silent Drift

Your app works fine for weeks. Then you add a feature that makes the webhook handler 200ms slower. Then another. Eventually it times out under load. Stripe starts retrying. You don't notice because there are no alerts. By the time a customer reports a problem, dozens of events have been lost.

What Proper Webhook Handling Actually Looks Like

Here's the full version of what your webhook code should do:

import Stripe from 'stripe';

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY);

app.post('/webhooks/stripe',
  express.raw({ type: 'application/json' }),  // Raw body for signature verification
  async (req, res) => {
    // 1. Verify signature (reject spoofed events)
    let event;
    try {
      event = stripe.webhooks.constructEvent(
        req.body,
        req.headers['stripe-signature'],
        process.env.STRIPE_WEBHOOK_SECRET
      );
    } catch (err) {
      console.error('Signature verification failed:', err.message);
      return res.status(400).send('Invalid signature');
    }

    // 2. Check idempotency (skip duplicates)
    const alreadyProcessed = await db.webhookEvents.findUnique({
      where: { eventId: event.id }
    });
    if (alreadyProcessed) {
      return res.json({ received: true, duplicate: true });
    }

    // 3. Store the event (so you have a record even if processing fails)
    await db.webhookEvents.create({
      data: { eventId: event.id, type: event.type, payload: event }
    });

    // 4. Acknowledge fast — return 200 BEFORE processing
    res.json({ received: true });

    // 5. Process asynchronously (failures won't affect the response)
    try {
      await processWebhookEvent(event);
      await db.webhookEvents.update({
        where: { eventId: event.id },
        data: { status: 'processed' }
      });
    } catch (err) {
      console.error('Processing failed:', err);
      await db.webhookEvents.update({
        where: { eventId: event.id },
        data: { status: 'failed', error: err.message }
      });
      // TODO: alert, retry, or add to DLQ
    }
  }
);

That's a lot more code. And it still doesn't include retries, a dead letter queue, monitoring dashboards, or alerting. For a solo developer or small team shipping fast, building all of this is a week of work that pulls you away from your actual product.

The Shortcut: Add a Reliability Layer in 2 Minutes

Instead of building all that infrastructure yourself, you can put a reliability layer between the webhook provider and your server. This is exactly what EventDock does.

Here's the setup:

# 1. Sign up at dashboard.eventdock.app (free, no credit card)

# 2. Create an endpoint:
#    Provider: Stripe (or Shopify, GitHub, etc.)
#    Destination: https://yourapp.com/webhooks/stripe
#    → You get: https://in.eventdock.app/ep_abc123

# 3. Point Stripe at your EventDock URL instead of your server:
#    Stripe Dashboard → Webhooks → Add endpoint
#    URL: https://in.eventdock.app/ep_abc123

# Done. Now you get:
# ✓ Automatic signature verification
# ✓ 7 retries with exponential backoff
# ✓ Dead letter queue with one-click replay
# ✓ Real-time delivery dashboard
# ✓ Slack/email alerts on failures

Your AI-generated webhook handler stays simple — because EventDock handles the reliability part. You focus on business logic; EventDock makes sure the events actually arrive.

Frequently Asked Questions

Do AI code generators add webhook retry logic?

No. AI code generators like Cursor, Copilot, and ChatGPT typically generate webhook handlers that handle the happy path. They rarely add retry logic, dead letter queues, or idempotency handling unless explicitly prompted.

What happens when my server is down and a webhook is sent?

The webhook provider (Stripe, Shopify, etc.) receives an error and retries with exponential backoff. Stripe retries for about 3 days. If your server is still failing after all retries, the event is permanently lost. A reliability layer like EventDock stores events durably and retries independently.

Is vibe coding bad for production apps?

No. AI-assisted development is excellent for building features fast. The gap is in infrastructure reliability — retries, queues, monitoring — which AI tools don't add automatically. The fix isn't to stop using AI; it's to add a reliability layer for critical points like webhooks.

How do I add webhook reliability without rewriting my code?

Use a webhook reliability layer like EventDock. Point your provider at EventDock instead of your server. EventDock handles retries, DLQ, and signature verification. Your existing code doesn't change.

Does Cursor or Copilot add webhook signature verification?

Sometimes, but inconsistently. Specific prompts like "secure Stripe webhook handling" may include it. Default prompts usually don't, leaving your app vulnerable to spoofed events.

Ship fast, don't lose webhooks. EventDock adds retries, DLQ, and monitoring in 2 minutes. Free for 5,000 events/month.

Why Your Stripe Webhooks Are Failing (And How to Fix It)

EventDock — Fri, 20 Mar 2026 19:35:02 +0000

You're losing Stripe webhooks and you don't even know it.

Stripe's webhook system is solid, but your receiving end isn't. Here are the 5 most common failure scenarios and exactly how to fix them.

1. Your Server Is Down During Deploys

The most common cause. You deploy, your server restarts, and during that 10-30 second window, Stripe sends a webhook. Your server returns a 502. Stripe will retry, but if your deploys are frequent or your downtime is longer, you're in trouble.

Stripe's retry behavior: 16 attempts over approximately 3 days, with exponential backoff.

After that? The event is gone. You'd need to manually poll the Events API to recover it.

Fix: Use a webhook proxy that accepts webhooks even when your server is down, then delivers them when it's back up.

2. Your Handler Times Out

Stripe expects a response within 20 seconds. If your webhook handler does heavy processing (database writes, external API calls, email sends), you might exceed this.

Stripe sees a timeout as a failure and retries. Now you're processing the same event multiple times.

Fix:

// Bad: Process everything synchronously
app.post('/webhook', async (req, res) => {
  await processPayment(req.body);    // 5s
  await updateDatabase(req.body);     // 3s  
  await sendEmail(req.body);          // 8s
  await notifySlack(req.body);        // 4s
  res.sendStatus(200);                // Total: 20s - too slow!
});

// Good: Acknowledge immediately, process async
app.post('/webhook', async (req, res) => {
  res.sendStatus(200);  // Respond in <100ms

  // Process in background
  queue.add('process-webhook', req.body);
});

3. Wrong Response Codes

Your server returns 301 Moved Permanently because you have HTTP→HTTPS redirects. Or it returns 401 Unauthorized because your auth middleware intercepts the webhook endpoint.

Stripe only counts 2xx as success. Everything else triggers a retry.

Fix: Ensure your webhook endpoint:

Is served over HTTPS directly (no redirects)
Is excluded from auth middleware
Returns 200 or 204 explicitly

4. Signature Verification Fails

You're comparing the wrong payload. Common mistakes:

// Wrong: Using parsed JSON body
const event = stripe.webhooks.constructEvent(
  JSON.stringify(req.body),  // This re-serializes differently!
  sig,
  secret
);

// Right: Using the raw body
const event = stripe.webhooks.constructEvent(
  req.rawBody,  // Exact bytes Stripe sent
  sig,
  secret
);

Express, Next.js, and most frameworks parse the body before your handler sees it. You need the raw body for signature verification.

5. Firewall or WAF Blocks

Your CDN or WAF (Cloudflare, AWS WAF) blocks Stripe's webhook requests because they look suspicious — high volume POST requests from unknown IPs.

Fix: Whitelist Stripe's webhook IPs or, better yet, use signature verification instead of IP whitelisting (IPs can change).

What Happens After Stripe Stops Retrying?

After 16 failed attempts over 3 days, Stripe marks the event as failed and stops trying. Your options:

Manual recovery: Use the Stripe Dashboard → Events to find and manually resend
API polling: Periodically call stripe.events.list() to find events you missed
Webhook proxy: Use a service that sits between Stripe and your server, guaranteeing delivery

Option 3 is the only one that doesn't require ongoing manual work.

The Reliability Layer Approach

Instead of pointing Stripe directly at your server, point it at a webhook proxy:

Stripe → EventDock (always up) → Your Server (might be down)

EventDock accepts the webhook immediately (sub-100ms, runs on Cloudflare's edge), stores it, verifies the Stripe signature, and then delivers it to your server. If your server is down, EventDock retries with exponential backoff for hours.

Setup takes 2 minutes:

Create an account at eventdock.app
Create an endpoint pointing to your server
Copy the EventDock URL into Stripe's webhook settings

You get:

Automatic retries (7 attempts over 2+ hours)
Dead letter queue for permanently failed webhooks
One-click replay from the dashboard
Full request/response logging
Real-time failure alerts

Prevention Checklist

Before your next deploy, verify:

[ ] Webhook endpoint returns 200 in <5 seconds
[ ] Raw body is preserved for signature verification
[ ] Endpoint is excluded from auth middleware
[ ] No HTTP→HTTPS redirects on the webhook path
[ ] Firewall allows Stripe's IPs
[ ] Idempotency keys prevent duplicate processing
[ ] You have monitoring/alerts on webhook failures

Originally published at eventdock.app/blog

What's the worst webhook failure you've experienced? Drop a comment below.