A valid payment webhook can still be dangerous to process twice

Payment providers retry. That is normal. What is not normal is what most applications do when the retry arrives.

A valid Stripe, Midtrans, Tripay, or Xendit webhook can still be dangerous to process twice. The signature checks out. The payload is well formed. The provider is doing exactly what it promised. And the order state gets corrupted anyway.

I built a small local lab to reproduce that failure and to test the fixes. This post walks through what actually breaks and how I would approach it in a real integration.

What actually fails

Five things show up again and again in production payment webhooks.

Duplicate delivery. The provider sends the same event twice because your endpoint returned a 500 the first time. The first request succeeded in the database but the response failed. Now you have two charges or two fulfillment triggers.

Out of order delivery. payment.succeeded arrives before order.created. Your state machine treats this as an unknown order and either drops the event or creates a phantom record.

Malformed payload. Someone changes the payload schema on the provider side and a required field is missing. If you trust the shape, you write half a record and fail on commit.

Timeout during processing. You accept the webhook, start processing, and the process dies. The provider retries. Now you have two half-writes.

Upstream failure after partial commit. You wrote the event record and then the downstream fulfillment call failed. The retry now sees the event record and thinks it is already processed.

Each of these is a small bug. Together they are the reason payment integrations break at 3 AM.

The four things that actually fix it

Not a framework. Not a message queue. Four concrete decisions.

1. Verify the signature against the raw body, not the parsed JSON

Most frameworks hand you a parsed object. By the time you re-serialize it to check the HMAC, key order and whitespace have changed and the signature fails. You end up disabling verification or comparing against a hash of the wrong bytes.

The fix is boring: read the raw request body first, check the HMAC against it, then parse.

raw = request.body
if not verify_hmac(raw, headers['x-signature'], secret):
    return 401
payload = json.loads(raw)

2. Idempotency on event ID plus payload hash, not event ID alone

Event ID alone is not enough. If the same event ID arrives with different data, that is either a provider bug or an attacker. Either way, the safe answer is to fail closed.

Store the event ID and a hash of the payload. On retry, compare both. Same ID, same hash: return the previous result. Same ID, different hash: reject and alert.

3. Explicit order state, not implicit

Do not let the webhook handler decide order state from context. Define the states, define which transitions are legal, and reject anything that does not fit. payment.succeeded on an order that is already fulfilled should not move it back to pending.

4. Bounded retries with a review queue

If a webhook fails, retry it. But only a bounded number of times. After three attempts, stop and move the event to a queue a human can inspect. Auto-retry forever is how a broken downstream eats your API quota.

What the lab actually does

I built this as a small local project with synthetic fixtures. No real payment provider. No real money. Just the failure modes and the recovery paths.

It has 18 deterministic tests covering the failure matrix: duplicate events, invalid signatures, malformed payloads, timeouts, upstream failures, partial writes, and out-of-order updates. It exposes a local HTTP API, stores events in SQLite, and has a reset command so you can rerun the demo cleanly.

There is a replay path too. When an operator authorizes a replay, the event goes through the same validation and state machine as a live event. No special bypass. Just a signed authorization on top.

What this is not

It is not a production payment system. It does not connect to Stripe or any live provider. It does not replace your message queue.

What it does is let you see the failure modes without a payment account, without risk, and with a test suite that proves the fixes work.

When this pattern helps

If your payment webhook has ever caused a duplicate charge, a missing fulfillment, or a stale order state, this is the shape of the fix. If you are about to add payments to a product for the first time, this is what you want to have in place before the first customer.

The code is at github.com/glatinone/payment-webhook-repair-lab. The pattern is the interesting part. The implementation is intentionally small.

If you are dealing with a payment webhook that breaks in ways you cannot reproduce, I take short implementation sprints on exactly this. kielltampubolon.id