Designing a Bulletproof Webhook Ingestion System in Ruby on Rails

As your Rails application grows and begins integrating with external platforms—think Stripe, Shopify, or GitHub—handling incoming webhooks efficiently becomes critical.

It’s easy to spin up a quick controller action, parse some JSON, and update a database record. But what happens when an external service suddenly floods your server with thousands of concurrent requests? Or worse, what if your third-party provider experiences network instability and drops connection mid-flight?

If your webhook endpoint performs heavy database writes, executes API callbacks, or sends emails synchronously, you are asking for trouble. Today, we will build a resilient, decoupled, and production-ready webhook ingestion system using Rails 7/8, Solid Queue (or Sidekiq), and database-backed idling.

The Blueprint: Decouple Fast, Process Later

The absolute golden rule of webhooks is: Acknowledge receipt immediately, handle processing asynchronously. Your endpoint should do exactly three things:

1. Verify the request signature (security first!).

2. Persist the raw payload to an inbound webhooks table.

3. Enqueue a background job and instantly return a 200 OK.

By moving all business logic out of the request-response cycle, you keep your database locks brief, protect your web workers from timed-out connections, and ensure zero data loss.

Step 1: Data Architecture & Security

First, let's create a dedicated table to house our raw webhook data. This gives us an immutable audit log and allows for seamless job retries if background workers fail.

rails generate model InboundWebhook status:integer provider:string payload:jsonb error_message:text
rails db:migrate

We will use an enum to keep track of the webhook lifecycle:

# app/models/inbound_webhook.rb
class InboundWebhook < ApplicationRecord
  enum :status, { pending: 0, processing: 1, completed: 2, failed: 3 }, default: :pending

  validates :provider, :payload, presence: true
end

Step 2: The High-Speed Controller

Our controller needs to be incredibly lean. It should strictly handle verification and record creation, then pass the heavy lifting off to an ActiveJob.

# app/controllers/webhooks/stripe_controller.rb
module Webhooks
  class StripeController < ApplicationController
    skip_before_action :verify_authenticity_token # Webhooks don't have CSRF tokens
    before_action :verify_signature!

    def create
      webhook = InboundWebhook.create!(
        provider: 'stripe',
        payload: JSON.parse(request.body.read)
      )

      # Hand off to background workers immediately
      ProcessWebhookJob.perform_later(webhook)

      render json: { success: true }, status: :ok
    rescue JSON::ParserError => e
      render json: { error: "Malformed payload" }, status: :bad_request
    end

    private

    def verify_signature!
      endpoint_secret = Rails.application.credentials.dig(:stripe, :webhook_secret)
      signature = request.env['HTTP_STRIPE_SIGNATURE']
      payload = request.body.read

      # Use Stripe's official SDK tool to verify the event authenticity safely
      Stripe::Webhook.construct_event(payload, signature, endpoint_secret)
    rescue Stripe::SignatureVerificationError => e
      render json: { error: "Invalid signature" }, status: :unauthorized
    end
  end
end

Step 3: Resilient Background Processing

​Now that the request has safely closed with a fast 200 OK, our background architecture takes over. If the underlying logic fails (due to a third-party API outage, a database deadlock, etc.), our system marks the webhook as failed and saves the stack trace instead of silently swallowing the error.

# app/jobs/process_webhook_job.rb
class ProcessWebhookJob < ApplicationJob
  queue_as :default

  # Automatically retry network or concurrency issues before marking as failed
  retry_on ActiveRecord::Deadlocked, wait: 2.seconds, attempts: 3

  def perform(webhook)
    return if webhook.completed?

    webhook.processing!

    # Routable processing based on payload types
    case webhook.provider
    when 'stripe'
      process_stripe_event(webhook.payload)
    else
      raise "Unknown webhook provider: #{webhook.provider}"
    end

    webhook.completed!
  rescue => e
    webhook.update!(status: :failed, error_message: "#{e.class}: #{e.message}")
    raise e # Re-raise to let your error tracker (Sentry/Honeybadger) catch it
  end

  private

  def process_stripe_event(payload)
    case payload['type']
    when 'charge.succeeded'
      # Implement your transaction tracking or ledger updates here
      # Invoice.payment_received!(payload['data']['object'])
    when 'customer.subscription.deleted'
      # Handle subscription cancellations gently
    end
  end
end

Step 4: The Superpower — Idempotency Guardrails

Webhooks are guaranteed to be delivered at least once. This means your application will eventually receive the exact same webhook payload twice. If you don't account for this, you risk double-charging clients or duplicating inventory data.

To make our processing layer strictly idempotent, we can utilize database uniqueness constraints or Redis locks based on the provider's unique event ID.

# Prevent duplicate processing using Stripe's unique event ID
def process_stripe_event(payload)
  event_id = payload['id']

  # Use an atomic lock or transaction mapping to prevent race conditions
  return if InboundWebhook.where(status: :completed)
                          .where("payload->>'id' = ?", event_id).exists?

  # Proceed with processing safely...
end

Wrapping Up
​By decoupling webhook storage from execution, your Rails app can handle sudden traffic spikes without flinching.

​The Win: Web servers spend a fraction of a millisecond handling external requests.

​The Peace of Mind: If your worker crashes, you still have the full history of payloads sitting securely in your database ready to be rerun with a quick webhook.reload.process_later Rake task.

​How are you currently scaling your webhook consumers? Are you using Redis-backed Sidekiq or looking into Rails 8's native Solid Queue? Let's discuss in the comments below!