The Webhook Race Condition That Changed How I Design Backend State

A support ticket comes in: "I paid, but your site still says my order is processing." You check the payment provider's dashboard — the charge went through, cleanly, seconds ago. You check your database — the order is still sitting in processing. Nobody made a mistake. Two pieces of correct code just disagreed about which one was telling the truth.

This is one of the most common bugs in any product that combines a payment provider with a frontend that polls for status, and it's almost never caused by bad code. It's caused by an architecture that never decided, on purpose, which system owns the truth.

How it actually happens

Here's the sequence. A user submits a payment. Your frontend gets an immediate response and starts polling an endpoint, or waiting on a websocket event, for the order to flip to paid. Meanwhile, the payment provider processes the charge asynchronously and fires a webhook back to your server once it's done — completely independent of whatever your frontend is currently doing.

Under normal conditions, the webhook arrives fast enough that nobody notices the gap. Under real-world conditions — a slow network hop, a queued webhook, a frontend polling on too long an interval — the user sees "processing" for a few extra seconds, and if anything else goes wrong in that window, "processing" can persist far longer than it should.

The deeper issue isn't the delay. It's that two independent systems — your frontend's local assumption of state, and your backend's actual record of state — were never explicitly told which one wins when they disagree.

Webhooks are also not exactly-once

It gets worse before it gets better. Most payment providers explicitly document that webhooks can be delivered more than once — a timeout on their end, a retry policy, a network blip, any of it can cause the same event to hit your endpoint twice. If your webhook handler isn't written to expect that, a duplicate delivery can double-charge a customer, double-provision a subscription, or double-send a confirmation email.

A naive handler looks like this:

Route::post('/webhooks/payment', function (Request $request) {
    $event = verifyAndParse($request);

    Order::find($event->orderId)->markAsPaid();

    return response()->noContent();
});

This works exactly once, and silently breaks the second time the same event arrives.

Designing for a single source of truth

The fix has two parts, and both matter.

First: pick one system as the source of truth, and make the other one reconcile against it. In practice, this almost always means the backend owns the real state, and the frontend is a view into that state — never the other way around. The frontend shouldn't assume "processing" just because it hasn't heard otherwise; it should be actively reconciling against whatever the backend actually reports.

Second: make webhook handlers idempotent by design, not by hoping the provider never retries. The standard pattern is to record the event ID the first time you see it, and check for that record before doing anything else:

Route::post('/webhooks/payment', function (Request $request) {
    $event = verifyAndParse($request);

    $alreadyProcessed = DB::table('processed_webhook_events')
        ->where('event_id', $event->id)
        ->exists();

    if ($alreadyProcessed) {
        return response()->noContent();
    }

    DB::transaction(function () use ($event) {
        DB::table('processed_webhook_events')->insert(['event_id' => $event->id]);
        Order::where('id', $event->orderId)->update(['status' => 'paid']);
    });

    return response()->noContent();
});

Wrapping the "mark as processed" insert and the actual state change in the same transaction matters — it's what stops a crash between the two steps from leaving you in a half-updated state where the event is marked processed but the order never actually got updated, or vice versa.

Why this generalizes

This isn't really a payment-specific problem. The same pattern shows up anywhere two systems need to agree on state asynchronously — a subscription platform reconciling billing status, a fulfillment system waiting on a shipping provider's callback, an AI pipeline waiting on a long-running job to finish. The fix is the same every time: decide explicitly which system owns the truth, treat every external callback as something that might arrive twice, and make the handler safe to run more than once.

Once that's in place, "I paid but it still says processing" stops being a bug you fix reactively and becomes a race condition that structurally can't happen anymore.

---

I'm Faisal Nadeem, a full-stack engineer with 6+ years shipping production SaaS platforms where getting this exact pattern right was non-negotiable — a car-insurance subscription platform where a double-charge or a lapsed policy isn't just a bug, it's a real problem for a real customer. If you're dealing with a similar state-consistency issue, or need a senior generalist who designs the backend and frontend together instead of negotiating between them, details are at faisalnadeem.net/hire-a-senior-full-stack-developer.