Reconciling Payments When Webhooks Never Arrive

Designing a recovery path for asynchronous transactions that remain stuck in processing.

Webhooks provide a fast way to learn that an external payment changed state. They do not guarantee that every event will reach the application.

A callback can be delayed by a network failure, rejected because of a signature or configuration problem, or lost during an outage. The provider may complete the payment while the local transaction remains in processing.

Without a second source of confirmation, the transaction can remain stuck indefinitely even though the money has already moved.

Reliable payment systems therefore need active reconciliation in addition to passive webhook handling.

Record enough information to ask again

Reconciliation starts when the transaction is created. The local record should preserve:

• the internal business reference;
• the provider used for this particular attempt;
• the provider transaction identifier;
• the current normalized state;
• the request and response timestamps; and
• a sanitized provider-response summary.

The recorded provider matters. Querying whichever provider is currently configured can produce an incorrect result after the platform switches rails. A processing transaction must always be reconciled against the provider that originally accepted it.

Reconcile outside the customer request

Opening a transaction history screen should not block while the server calls several external providers. Instead, the API can return the current local state and enqueue a background refresh for eligible processing transactions.

def request_status_refresh(payment):
    lock_key = f"status-refresh:{payment.id}"

    if not cache.add(lock_key, "queued", timeout=30):
        return False

    reconcile_payment.delay(payment.id)
    return True

The short-lived cache lock prevents repeated screen refreshes from creating a burst of identical provider calls. The worker removes the lock after processing so a later request can check again if the payment remains unresolved.

Use one normalization path

Webhook processing and active reconciliation should not maintain separate definitions of success.

Both paths should translate the external result into the same internal vocabulary and call the same guarded transition service:

def apply_provider_result(payment, external_result):
    normalized = normalize_status(external_result)

    if normalized == "completed":
        complete_once(payment, external_result)
    elif normalized == "failed":
        fail_once(payment, external_result)
    else:
        keep_processing(payment, external_result)

This avoids a dangerous situation where a webhook considers one status successful while the polling worker considers it pending.

Bound retries without inventing failure

A provider can continue returning pending for a legitimate transaction. After several checks, the worker may stop retrying automatically, but it should not convert uncertainty into failure without evidence.

Useful outcomes include:

• terminal success: complete the local transaction once;
• terminal failure: apply the defined failure or refund path once;
• still processing: retry with delay up to a defined limit;
• unavailable provider: record the operational error and retry later;
• unknown status: preserve processing and flag the transaction for review.

Retry exhaustion means automated reconciliation paused. It does not prove that the payment failed.

Make reconciliation safe to repeat

The provider may send the missing webhook while the status worker is running. Both paths can discover the same terminal result at nearly the same time.

The final transition must therefore lock the transaction and protect its financial side effect with an idempotency key. Reconciliation should be safe to run repeatedly, including from a staff action.

Give operations controlled visibility

Some transactions will require manual investigation. Staff need access to:

• local and provider references;
• the provider recorded on the transaction;
• the latest normalized and external statuses;
• timestamps for prior checks;
• the last safe provider error; and
• an action to request another status refresh.

Provider errors should remain internal. Customers need a stable, understandable transaction state rather than raw infrastructure messages.

The design principle

Webhooks are the low-latency path. Reconciliation is the completeness path.

Using both creates a convergent lifecycle: whether the platform learns the result through a callback, a background poll, or an authorized staff check, the same protected transition brings the transaction to the same local state.

This does not make external systems reliable. It makes their unreliability manageable.