Building a Production-Ready Webhook Delivery System in 5 Minutes

Building a webhook delivery system from scratch is deceptively complex. You need URL verification, HMAC signatures, retry logic, health monitoring, and scalable architecture to handle thousands of subscribers without blocking your API.

Most teams spend weeks building this infrastructure. But what if you could deploy a production-ready webhook system in minutes?

This post shows you how to build a complete outgoing webhook delivery system with queue-based architecture, automatic retries, security best practices, and enterprise-grade reliability—using just a few hundred lines of code.

How It Works: The Flow

Key Features:

• ⚡ Instant response (202 Accepted)
• 🔄 Parallel delivery to all subscribers
• 🔐 HMAC SHA-256 signatures on every webhook
• ♻️ Auto-retry with exponential backoff (1s, 2s, 4s)
• 📊 Health monitoring and auto-disable after 10 failures

What Are Outgoing Webhooks?

Most developers are familiar with receiving webhooks from services like Stripe, GitHub, or Slack. But what about sending them?

Outgoing webhooks (also called "reverse webhooks") let your application notify external services when events occur. When something happens in your system—a user signs up, an order ships, a payment completes—your app sends HTTP POST requests to registered webhook URLs.

This is crucial for building integrations, enabling real-time notifications, and allowing customers to build on top of your platform.

The Challenge: Building It Right Is Hard

Building a robust webhook delivery system from scratch involves solving many complex problems:

• URL Verification: How do you validate that webhook URLs are legitimate?
• Security: Payloads need HMAC signatures to prevent tampering
• Reliability: What happens when a webhook endpoint is down?
• Scale: How do you deliver to thousands of subscribers without blocking?
• Monitoring: You need delivery stats, failure tracking, and auto-disabling of broken webhooks
• Retry Logic: Exponential backoff, maximum attempts, timeout handling

Most teams spend weeks building this infrastructure. But there's a better way.

Queue-Based Architecture: The Secret Sauce

The key to scalable webhook delivery is decoupling event triggering from delivery. Here's how it works:

// 1. Store the event for audit trail
await conn.insertOne('events', eventData);

// 2. Mark all matching webhooks with the event ID
await conn.updateMany('webhooks',
  { status: 'active', $or: [{ events: eventType }, { events: '*' }] },
  { $set: { pendingEventId: eventData.id } }
);

// 3. Queue ALL webhooks in one atomic operation
await conn.enqueueFromQuery('webhooks',
  { status: 'active', pendingEventId: eventData.id },
  'webhook-delivery'
);

The magic is in enqueueFromQuery. Instead of loading webhooks into memory and looping through them, it:

• Queues everything in a single database operation
• Handles thousands of webhooks without memory overhead
• Returns immediately (202 Accepted)
• Processes deliveries in parallel via worker functions

This architecture delivers instant response times even with massive subscriber lists.

Security First: HMAC Signing

Every webhook payload includes a cryptographic signature so receivers can verify authenticity:

function generateSignature(payload, secret) {
  const timestamp = Math.floor(Date.now() / 1000);
  const sigBasestring = `${timestamp}.${payload}`;
  const signature = crypto
    .createHmac('sha256', secret)
    .update(sigBasestring, 'utf8')
    .digest('hex');
  return { signature: `v1=${signature}`, timestamp };
}

Headers sent with each webhook:

• X-Webhook-Signature: HMAC SHA-256 signature
• X-Webhook-Timestamp: Unix timestamp (prevents replay attacks)
• X-Webhook-Id: Subscription ID

Receivers validate signatures and reject old timestamps (>5 minutes) to prevent replay attacks.

Automatic URL Verification

Before accepting webhook registrations, the system verifies URLs using industry-standard methods:

Stripe-style verification: Sends a test payload with a verification token and expects HTTP 200.

Slack-style challenge: Sends a random challenge string and expects it echoed back in the response.

Verification happens asynchronously in a worker queue, so registration returns immediately while verification runs in the background.

Smart Retry Logic

Failed deliveries trigger automatic retries with exponential backoff:

// Retry configuration
{
  maxRetries: 3,
  backoffIntervals: [1s, 2s, 4s],
  timeout: 10 seconds,
  autoDisableAfter: 10 consecutive failures
}

A cron job runs every 30 minutes to retry webhooks that have been failing for over an hour, preventing hammering of broken endpoints.

Event Flexibility

Unlike some webhook systems that force you to predefine event types, this system supports any event name:

# E-commerce events
POST /events/trigger/order.placed

# IoT events
POST /events/trigger/sensor.temperature.high

# Custom business events
POST /events/trigger/report.generated

Subscribers can register for specific events or use "*" as a wildcard to receive everything.

Production-Grade Monitoring

The system tracks detailed statistics for each webhook:

{
  "deliveryCount": 1247,
  "consecutiveFailures": 0,
  "lastDeliveryAt": "2025-01-15T10:30:00Z",
  "lastDeliveryStatus": "success",
  "status": "active"
}

After 10 consecutive failures, webhooks are automatically disabled to prevent wasting resources on dead endpoints. Users can manually retry with POST /webhooks/:id/retry.

The Complete API

The system provides a full CRUD API for webhook management:

# Create webhook (with automatic verification)
POST /webhooks
{
  "clientId": "customer-123",
  "url": "https://example.com/webhook",
  "events": ["order.placed", "order.shipped"],
  "verificationType": "stripe"
}

# List webhooks
GET /webhooks?status=active&event=order.placed

# Trigger events (queues delivery to all subscribers)
POST /events/trigger/order.placed
{
  "orderId": "123",
  "total": 99.99,
  "customer": "[e-mail]"
}

# Check delivery stats
GET /webhooks/:id/stats

# Manually retry failed webhook
POST /webhooks/:id/retry

Idempotent Registration

The system uses clientId + url as a composite key for upsert logic:

• First registration creates the webhook
• Subsequent registrations with the same clientId + url update it
• Prevents duplicate webhooks
• Preserves HMAC secrets across updates
• Perfect for multi-tenant SaaS applications

Real-World Integration

The system integrates seamlessly with workflow tools:

n8n: Create a webhook trigger, register the URL, and your workflows activate automatically.

Zapier: Use "Webhooks by Zapier" with Catch Hook, register the URL with verificationType: "stripe", and your Zaps come alive.

Custom apps: Implement signature verification in any language and start receiving events.

See It In Action

Want to try it yourself? The complete implementation is available as a Codehooks.io template:

coho create mywebhooks --template webhook-delivery
cd mywebhooks
npm install
coho deploy

You'll have a production-ready webhook delivery system running in minutes, not weeks.

Key Takeaways

Building webhook delivery infrastructure is complex, but the right architecture makes it manageable:

1. Queue-based delivery scales to thousands of subscribers
2. HMAC signatures ensure payload authenticity
3. Automatic verification validates webhook URLs before accepting them
4. Smart retries handle transient failures gracefully
5. Health monitoring auto-disables broken endpoints
6. Event flexibility supports any event naming scheme

Whether you're building a SaaS platform, IoT system, or integration hub, having solid webhook infrastructure lets your customers build powerful automations on top of your product.

The architecture patterns shown here work with any backend stack—Node.js, Python, Go, or whatever you prefer. The key is separating event triggering from delivery, implementing proper security, and handling failures gracefully.

---

The complete source code for this webhook delivery system is available at https://github.com/codehooks-io/codehooks-io-templates/tree/main/webhook-delivery