Recovering asynchronous payments safely when callbacks are delayed, duplicated, or missing.
Webhooks are asynchronous evidence, not guaranteed delivery.
They may arrive twice, arrive out of order, or never arrive. A reliable payment platform therefore needs several ways to discover the same final state:
- verified webhook ingestion;
- scheduled polling for processing transactions;
- controlled staff status checks;
- reconciliation against provider records; and
- provider webhook replay where available.
Every path must converge on one idempotent update function:
def apply_provider_status(payment_id, event):
with transaction.atomic():
payment = Payment.objects.select_for_update().get(id=payment_id)
target = normalize_status(event)
transition_payment(payment, target, evidence=event)
The function should ignore regressions, recognize terminal states, and separate provider acknowledgement from local ledger settlement.
Polling should run in background workers with bounded retries and observable failure reporting. Staff tools should call the same service instead of directly editing statuses.
This design allows multiple imperfect signals to converge safely. Missing one callback no longer means leaving customer money permanently stuck.
Verify before processing
Webhook endpoints should authenticate the source using the provider's documented signature scheme. Verification must use the raw request representation required by that scheme, a constant-time comparison, and an acceptable timestamp window when timestamps are signed.
Return quickly after durable ingestion. Expensive business work belongs in a background job so temporary ledger or notification failures do not cause the provider to repeatedly deliver the same event while the request remains open.
Store the event before applying it
Persist a sanitized event record containing the provider event identifier, transaction reference, event type, received time, verification result, and payload fingerprint. A unique constraint on the provider event identifier makes duplicate delivery harmless.
The event record and transaction state serve different purposes. The event proves what arrived. The transaction represents the application's accepted financial truth.
Poll selectively
Polling every transaction forever wastes capacity and can hit provider limits. Poll only non-terminal transactions that have passed an appropriate delay, use exponential backoff, and stop after a defined recovery window. Separate “still processing” from “status check failed” so network problems do not become payment failures.
Manual status checks should enqueue the same background task where possible. That keeps staff requests responsive and prevents the dashboard from implementing a second reconciliation path.
Observe unresolved work
Track the age and count of processing transactions, polling failures, webhook verification failures, duplicate events, and transitions recovered without a webhook. Alert on growing queues rather than individual routine retries.
A webhook integration is complete only when the system also knows what to do when the webhook does not arrive.
Implementation checklist
Give every recovery mechanism clear ownership and limits. Define when polling begins, how often it retries, when staff intervention becomes available, and when an unresolved payment becomes a reconciliation exception. Record which signal caused the final transition.
Create operational views for aging processing transactions and failed status checks. Include a safe replay mechanism for individual transactions and controlled batches, with permissions and audit records.
Finally, simulate missing, duplicated, reversed, and out-of-order callbacks. A recovery path that has never been exercised is only a diagram. Reliable systems continuously prove that alternative paths converge on the same result.
Top comments (0)