Muhammad Ali Mughal

Posted on Nov 9

Webhooks That Stopped Arriving: Gap Detection & Reconciliation

#laravel #webhooks #architecture #devops

A customer emails: "I cancelled my plan three weeks ago, why was I charged again?" You check the billing provider. The cancellation is there. You check your database. The subscription is still active. Your webhook handler has no errors, because for three weeks it received nothing to fail on. The provider disabled your endpoint after a run of failed deliveries, sent one email to an address nobody reads, and moved on.

I wrote this with help from AI tools and checked every migration, query and command myself. I am a Principal Software Engineer in Karachi with 9+ years across PHP/Laravel, Vue, Node and .NET microservices, and at Loop Origin we build and maintain SaaS backends that depend on payment, shipping and identity webhooks. This is Part 2 of the series Silent Failures in Production SaaS. Part 1 covered cron jobs that never ran; webhooks are the same class of problem pointed the other way. Your code is fine. The input just stopped.

📚 Series Roadmap — Silent Failures in Production SaaS

Part 1: Cron Jobs That Never Ran: Heartbeats & Dead-Man Switches for SaaS

Part 2 (This Post): Webhooks That Stopped Arriving: Gap Detection & Reconciliation

Part 3: Email That Never Landed: SPF/DKIM/DMARC Drift & Bounce Pipelines

Part 4: Config & Feature Flags That Quietly Diverged Across Environments

Part 5: Backups That Never Restored: Automated Restore Drills

Part 6: Synthetic Checks & SLOs for the Paths Nobody Clicks

Related earlier posts: Webhook Security in Production: HMAC-SHA256 Signatures & Replay Protection · Idempotent Jobs & Exactly-Once Effects · Dead Letter Queues, Poison Messages & Safe Replay Playbooks

How do you detect webhooks that stopped arriving?

Direct answer: Record every received event with its provider event ID, then alert when the receive rate for a source drops below its normal baseline for that time window. Because webhooks are not guaranteed, also run a scheduled reconciliation job that lists recent events or object states from the provider's API and processes anything you missed.

Why webhook gaps are invisible

A webhook integration fails silently in more ways than it fails loudly:

Endpoint disabled by the provider. Many providers retry for hours or days, then stop or disable the endpoint after repeated failures. Some, such as GitHub, do not retry automatically at all.
Signing secret rotated on one side only. Every request now returns 400 from your signature check, which you may log at debug level.
WAF or bot protection starts challenging the provider's IPs after a rule update.
Route or domain moved during a refactor, and the provider still posts to the old URL, which now returns a 404 from a catch-all page.
New event types were never subscribed, so the feature built on them never receives anything.
Received but not processed. The controller stored the payload and queued a job, but the queue worker is down.

None of these throw an exception inside your application. Your error tracker stays green.

Step 1: store every receipt first

Treat the webhook endpoint as an inbox. Verify the signature, store the raw event with a unique constraint, return 2xx fast, and process asynchronously. This gives you the data you need to detect gaps.

// database/migrations/2026_11_09_000000_create_webhook_receipts_table.php
Schema::create('webhook_receipts', function (Blueprint $table) {
    $table->id();
    $table->string('provider', 32);            // stripe, shopify, github...
    $table->string('event_id', 128);
    $table->string('event_type', 128);
    $table->timestampTz('provider_created_at')->nullable();
    $table->timestampTz('received_at')->useCurrent();
    $table->timestampTz('processed_at')->nullable();
    $table->string('source', 16)->default('push');  // push | reconcile
    $table->jsonb('payload');
    $table->unique(['provider', 'event_id']);
    $table->index(['provider', 'received_at']);
});

// app/Http/Controllers/StripeWebhookController.php
public function __invoke(Request $request)
{
    try {
        $event = \Stripe\Webhook::constructEvent(
            $request->getContent(),
            $request->header('Stripe-Signature'),
            config('services.stripe.webhook_secret'),
        );
    } catch (\Throwable $e) {
        // Signature failures must be visible: a rotated secret looks exactly like this.
        Log::warning('stripe_webhook_rejected', ['reason' => $e->getMessage()]);
        return response('invalid', 400);
    }

    $inserted = DB::table('webhook_receipts')->insertOrIgnore([
        'provider' => 'stripe',
        'event_id' => $event->id,
        'event_type' => $event->type,
        'provider_created_at' => Carbon::createFromTimestamp($event->created),
        'payload' => json_encode($event->toArray()),
    ]);

    if ($inserted) {
        ProcessWebhookReceipt::dispatch('stripe', $event->id);
    }

    return response()->noContent();
}

The unique index makes duplicate deliveries and reconciliation overlaps harmless. That idempotency is what lets you reconcile aggressively later without double-charging or double-provisioning anyone.

Step 2: alert on missing volume, not just errors

The simplest useful detector compares the last hour against the same hour-of-week over the previous four weeks. Webhook traffic is usually seasonal, so a flat threshold either misses gaps at peak or cries wolf at 3 AM.

-- Postgres: sources whose last-hour volume fell below 30% of their hour-of-week baseline
WITH recent AS (
  SELECT provider, count(*) AS n
  FROM webhook_receipts
  WHERE source = 'push' AND received_at >= now() - interval '1 hour'
  GROUP BY provider
),
baseline AS (
  SELECT provider, count(*) / 4.0 AS avg_n
  FROM webhook_receipts
  WHERE source = 'push'
    AND received_at >= now() - interval '29 days'
    AND received_at <  now() - interval '1 day'
    AND extract(isodow FROM received_at) = extract(isodow FROM now())
    AND extract(hour   FROM received_at) = extract(hour   FROM now() - interval '1 hour')
  GROUP BY provider
)
SELECT b.provider, coalesce(r.n, 0) AS last_hour, round(b.avg_n, 1) AS expected
FROM baseline b
LEFT JOIN recent r USING (provider)
WHERE b.avg_n >= 5
  AND coalesce(r.n, 0) < b.avg_n * 0.3;

The avg_n >= 5 guard skips low-volume sources, where an empty hour is normal. For those, use a simpler "no events in N hours" rule, plus the reconciliation in step 3. Two more signals catch the "received but not processed" case and the rotated-secret case:

-- Stored but not processed for more than 15 minutes
SELECT provider, count(*) FROM webhook_receipts
WHERE processed_at IS NULL AND received_at < now() - interval '15 minutes'
GROUP BY provider;

For signature rejections, count the stripe_webhook_rejected log line in your log platform and alert on any sustained non-zero rate.

Step 3: reconcile against the provider

Detection tells you something is wrong. Reconciliation repairs it without anyone replaying events by hand. Most providers offer either an events list or a way to fetch current object state. Stripe, for example, lets you list events from roughly the last 30 days.

// app/Console/Commands/ReconcileStripeEvents.php
public function handle(\Stripe\StripeClient $stripe): int
{
    $since = now()->subHours((int) $this->option('hours') ?: 6)->timestamp;
    $recovered = 0;

    $events = $stripe->events->all([
        'created' => ['gte' => $since],
        'types' => config('services.stripe.handled_events'),
        'limit' => 100,
    ]);

    foreach ($events->autoPagingIterator() as $event) {
        $inserted = DB::table('webhook_receipts')->insertOrIgnore([
            'provider' => 'stripe',
            'event_id' => $event->id,
            'event_type' => $event->type,
            'provider_created_at' => Carbon::createFromTimestamp($event->created),
            'source' => 'reconcile',
            'payload' => json_encode($event->toArray()),
        ]);
        if ($inserted) {
            ProcessWebhookReceipt::dispatch('stripe', $event->id);
            $recovered++;
        }
    }

    Log::info('stripe_reconcile', ['recovered' => $recovered, 'since' => $since]);
    return self::SUCCESS;
}

// routes/console.php
Schedule::command('webhooks:reconcile-stripe --hours=6')
    ->hourly()->onOneServer()->withoutOverlapping(30)
    ->pingOnSuccess(env('HB_STRIPE_RECONCILE_URL'));

The overlapping window (six hours, run hourly) means a short outage is covered several times. The recovered count is the most honest health metric you have: in a healthy system it is near zero. When it jumps, push delivery is broken even if nothing else looks wrong. And yes, the reconcile job itself gets a heartbeat, from Part 1.

If a provider has no events API, reconcile on state instead: page through the provider's subscriptions, orders or users updated since the last run and compare against your copies.

Choosing a strategy per provider

Provider capability	Detection	Reconciliation	Notes
Events list API with IDs	Volume baseline + recovered count	Replay missed events by ID	Best case; idempotent insert does the work
Object state API only	Volume baseline	Compare updated-since objects to local state	Requires reliable `updated_at` filters
Delivery log with manual redeliver	Volume baseline + provider dashboard checks	Redeliver via API or UI	Usually limited history; act quickly
No API at all	"Silence for N hours" rule	Periodic full export or manual review	Document the risk with the business owner

Rolling it out

When we add this to a client system at Loop Origin, the order is: inventory every inbound webhook, add the receipts table and idempotent insert, wire the volume and unprocessed alerts, then add reconciliation for the integrations that touch money or access. If your stack needs this kind of plumbing, our custom web application development team designs integrations that assume delivery will fail, and our Laravel development service covers receivers, queues and reconciliation commands.

Diagnostic checklist

[ ] Every inbound webhook source is listed with its owner and the events it should send.
[ ] Receipts are stored with a unique (provider, event_id) constraint before processing.
[ ] Signature failures are logged at warning level and alerted on.
[ ] Volume alerts compare against an hour-of-week baseline, not a fixed number.
[ ] Low-volume sources have a "no events in N hours" rule.
[ ] Stored-but-unprocessed receipts alert after a short window.
[ ] A scheduled reconciliation job exists for every money- or access-related integration.
[ ] The reconciliation job reports a recovered count and has its own heartbeat.
[ ] Provider notification emails go to a monitored shared inbox.
[ ] I tested the alerts by disabling a webhook endpoint in staging.

FAQ

Isn't reconciliation enough on its own?

It repairs data but adds delay equal to the reconcile interval. Push webhooks give speed; reconciliation gives correctness. You want both.

How far back should reconciliation look?

Several times the run interval, and never beyond what the provider retains. Hourly runs with a six-hour window are a practical default.

Will reconciled events be processed twice?

Not if processing keys off the unique event ID and handlers are idempotent. That is why the receipts table comes first.

What about event ordering?

Do not assume order. Handlers should fetch current object state or compare timestamps before applying older events.

Should I return 200 for events I do not handle?

Yes, after verifying the signature. Returning errors for unhandled types can lead providers to disable the endpoint.

How do I notice a disabled endpoint quickly?

The volume alert usually fires first. Also route provider account emails to a shared, monitored inbox, and check endpoint status in the provider dashboard during incident reviews.

Next in this series, Part 3 covers email that never landed: SPF, DKIM and DMARC drift, and building bounce pipelines you can trust.

DEV Community