Never Trust the Client to Create the Order: Making Payment the Source of Truth

TL;DR — If order creation happens on the client after payment, a crashed phone means money captured with no order. Move creation to the payment.capturedwebhook so the durable, retrying, server-side actor owns the write. That creates a client-vs-webhook race, which takes three layers to kill: a UNIQUE(txn_id)constraint (the DB has the final say), an "order already exists" guard, and treating duplicate webhooks as expected no-ops. Verify signatures, always 200what you won't act on, and ship it behind a flag.

The bug that doesn't show up in your tests

Here's the order flow almost every food app ships first, because it's the obvious one:

1. Customer taps Pay.
2. The payment gateway's SDK opens, they pay, it succeeds.
3. The SDK returns control to your app.
4. Your app calls POST /createOrder with the cart.
5. The order lands in the database. The kitchen sees it.

Every step works. The demo works. The staging test works. And then you go to production, and a handful of times a day, a customer's money leaves their account and no order is ever created.

Look at the flow again. The order is created in step 4 — by the client, after payment. Everything between "money captured" and "order written" is running on a phone you don't control:

• The app crashes right after the gateway returns.
• The OS kills the backgrounded app before step 4 fires.
• The customer's train goes into a tunnel and the /createOrder request never lands.
• Someone force-quits because the spinner "looked stuck."

The payment is real. The order is not. You've built a dual-write : two systems (the payment provider and your database) that must both commit, coordinated by the least reliable computer in the entire system — the customer's phone. There is no transaction spanning both. When the second write is skipped, you don't get an error. You get silence, and a support ticket that reads "I paid and nothing happened."

You cannot fix this by adding retries on the client. The client is the thing that's gone.

The fix: make the money the source of truth

The payment provider already knows something happened — that's what a webhook is for. When a payment is captured, the gateway makes a server-to-server call to a URL you own. That call doesn't depend on the customer's phone at all. It retries on its own if your server is briefly down.

BEFORE — client is responsible for the write that matters
                                          ✗ crash / tunnel / OS-kill
  [phone] ──pay──▶ [gateway] ──ok──▶ [phone] ──✗──▶ [POST /createOrder]
                                                          money taken, no order

AFTER — durable server-side actor owns the write
  [phone] ──pay──▶ [gateway] ──payment.captured──▶ [your webhook] ──▶ createOrder()
     ▲ (retries on its own) │
     └────────────── order_confirmed (push) ─────────────────────────────┘

So the order should be created there — on the payment.captured webhook, server-side — not on the client's say-so.

The one thing this requires: at the moment you initiate payment, you have to stash everything needed to build the order somewhere durable, so the webhook can reconstruct it later without the client. I persist the entire order payload onto the transaction row when payment starts:

// At payment initiation — BEFORE the customer pays.
// The order doesn't exist yet, but everything to build it is now durable.
await db.transactions.insert({
  txn_id: txnId,
  status: 'pending',
  payment_type: 'online',
  order_payload: orderBody, // the full cart, exactly as /createOrder expects it
  customer_id: customerId,
});

Now the webhook has everything. The client's job shrinks to "start the payment and then listen for a push telling you the order exists." It is no longer responsible for the order's existence.

// The webhook the payment provider calls. No phone involved.
router.post('/payment/order-webhook', async (req, res) => {
  // 1. Verify it's really from the gateway (see "the boring parts" below)
  if (!verifySignature(req.rawBody, req.headers['x-signature'])) {
    return res.status(400).json({ ok: false });
  }

  const { event, payload } = req.body;
  if (event !== 'payment.captured') {
    return res.status(200).json({ ok: true }); // ack, don't act
  }

  const txn = await db.transactions.findByTxnId(payload.txn_id);

  // 2. Build the order from the payload we stashed at initiation
  const order = await createOrder(txn.order_payload);

  // 3. Push to the customer's app — it's been waiting, not driving
  io.toCustomer(txn.customer_id).emit('order_confirmed', { orderId: order.id });

  return res.status(200).json({ ok: true });
});

That's the whole idea in five lines. The interesting part is everything that idea breaks.

One order, two callers, zero divergence

The moment you have a webhook creating orders, you have two things that create orders: the webhook, and the old client route (which you can't just delete — cash orders, wallet orders, and manual/counter orders still come in through it, and you want a staged rollout, not a big-bang cutover).

Two code paths that build the same object will drift. One will validate stock and the other won't. One will apply the subsidy math slightly differently. Six weeks later they disagree about what a valid order even is, and you're debugging why webhook orders have a different total than client orders.

The fix is to make "create an order" exactly one function — pure business logic, no HTTP, no framework objects, callable from anywhere:

// One function. No req, no res. Throws typed errors so any caller
// can map them to its own transport (HTTP status, socket error, log).
async function createOrder(params) {
  if (!params.txnId) throw orderError('VALIDATION', 'txnId is required');
  if (params.total <= 0) throw orderError('VALIDATION', 'total must be positive');

  const items = await validateAndFetchItems(params.items); // stock, availability
  const outlet = await fetchOutlet(params.outletId);
  enforceMinimumOrderValue(params, outlet); // server-authoritative
  const pricing = applySubsidies(params, outlet); // same math, always

  return db.orders.insert(buildOrderRow(params, pricing));
}

function orderError(type, message) {
  // type ∈ {'VALIDATION','NOT_FOUND','INTERNAL'} → caller maps to 400/404/500
  const e = new Error(message); e.type = type; return e;
}

The HTTP route becomes a thin adapter — parse the request, call the service, map the typed error to a status code. The webhook is another adapter that does the same, minus the response body. The order semantics live in exactly one place , and both entry points are provably identical because they are literally the same function call.

A quiet but important detail: things like the minimum-order-value check and the pricing/subsidy math live inside this service, not in the client. The client can compute a total to show the user, but the number that hits the database is computed server-side, every time. Once the server owns order creation, it should own every rule that decides whether the order is valid — otherwise a modified client is a discount generator.

The race you just created

Here's the trap. You didn't replace the client flow — you added the webhook alongside it. So on a normal, healthy order, both fire :

• The client's /createOrder lands (the happy path still works fine).
• The payment.captured webhook also lands, a second or two later.

Two writers, same order, racing. Without a guard you get two orders for one payment — which is strictly worse than the bug you started with, because now the kitchen makes the dish twice and settlement is wrong.

The tempting fix is "check if the order exists, and if not, create it." That check-then-act is itself a race: both callers can check "no order" in the same instant and both proceed. You cannot fix a concurrency bug with an if statement in application code.

So I layered three guards, weakest to strongest.

Layer 1 — the database has the final say

Every order carries the payment's transaction id. A single unique constraint makes a second order for the same payment physically impossible, no matter how the code races:

ALTER TABLE orders ADD CONSTRAINT orders_txn_id_unique UNIQUE (txn_id);

Postgres allows multiple NULLs in a unique column, so orders that legitimately have no payment yet (pending, cash-on-delivery being assembled) are unaffected. Whoever inserts second gets a constraint violation instead of a duplicate. This is the backstop that does not depend on me being clever — the other two layers just make it rare and graceful instead of an exception in the logs.

Before you can add this constraint, you have to prove there are no existing duplicates. That one-line GROUP BY txn_id HAVING COUNT(*) > 1 is not a formality — run it, and you'll often find the historical duplicates the bug already created. Fix those first, then constrain.

Layer 2 — the webhook recognizes "the client already won"

Before creating anything, the webhook checks whether an order for this payment already exists. If it does, the client beat it here — so the webhook does not create a second order. It just reconciles the payment status and re-notifies the customer:

const existing = await db.orders.findByTxnId(txnId);
if (existing) {
  await db.transactions.markSuccess(txnId); // just reconcile the money
  io.toCustomer(customerId).emit('order_confirmed', { orderId: existing.id });
  return res.status(200).json({ ok: true });
}

Layer 3 — duplicate webhooks are expected, not exceptional

Payment providers guarantee at-least-once delivery. They will send you the same payment.captured twice, and if your endpoint is slow to 200, they'll retry it as a matter of policy. So "I've seen this transaction already" has to be a no-op, not a second order:

if (txn.status === 'success' && txn.payment_id) {
  // We already fully processed this one. Re-emit the confirmation
  // (the customer may have reconnected) and ack. Do not re-create.
  io.toCustomer(customerId).emit('order_confirmed', { orderId: existing.id });
  return res.status(200).json({ ok: true });
}

And every status write is conditional — UPDATE ... WHERE status = 'pending' — so replaying a webhook can never move a row backward. Conditional updates are the cheapest idempotency tool there is; use them everywhere a webhook writes.

The boring parts that aren't optional

Verify the signature. Your order-creation endpoint is now a public URL that creates orders. Anyone who finds it could POST a fake payment.captured and get free food. Every provider signs its webhooks — an HMAC of the raw body against a shared secret. Verify it against the raw bytes, before any parsing, and reject on mismatch. This is the one place I return 400.

Return 200 for everything you're not going to act on. Wrong event type? 200. Not a payment you care about? 200. Feature flag off? 200. A non-2xx tells the provider "retry me," and a webhook that returns 500 on messages it simply doesn't handle will get itself hammered by the retry policy it triggered. Acknowledge fast; act only on what's yours.

Ship it behind a flag. This changes how money becomes orders — the single most sensitive path in the system. I put the whole webhook flow behind an environment flag, defaulted off, so the existing client flow was untouched until I was ready. Even disabled, the endpoint returns 200 so the provider isn't building a retry backlog while I'm still testing. Flip it on for one outlet, watch, widen.

What actually broke in production

I'd be lying if I framed this as a clean win. Moving the source of truth to the server surfaced a class of bug that the client flow had been hiding: status reconciliation for the paths that don't go through the gateway at all.

Cash and counter/manual orders never produce a payment.captured webhook — there's no online payment to capture. Their transaction rows were being created as pending and then... nothing flipped them to success, because the thing that flips them is the webhook that never fires for cash. For a while, revenue reports undercounted because a slice of perfectly-good cash orders looked unpaid.

The fix was two-part, and it's the honest shape of most production work:

1. Forward fix — the code that creates a manual/cash order now writes status = 'success' at creation time, since for cash there is no later confirmation step to wait for.
2. Backfill — a migration to reconcile the rows already stuck in pending, run as preview first (a read-only SELECT of exactly what will change), then the UPDATE inside a transaction I could ROLLBACK. You do not run a bare UPDATE against a money table and hope.

The lesson I keep relearning: when you make one path authoritative, you inherit responsibility for every path that isn't on it. The webhook made online orders bulletproof and immediately made me care about the orders that have no webhook.

What I'd still improve

• The client flow is still allowed to write orders. Long-term, I'd rather the client only ever initiates and the server is the sole creator for online payments too — collapsing three idempotency layers back toward one. The layers exist because I'm running both flows during migration; they're a bridge, not the destination.
• Reconciliation should be a scheduled job, not a lucky webhook. At-least-once is a floor, not a ceiling — a provider can drop a webhook entirely. A periodic sweep that asks the gateway "what did you capture in the last hour that I have no order for?" turns "we lost an order" from an incident into a self-healing background task. That's the next thing I'm building.

The one idea to take away

If two systems must both commit and no transaction spans them, do not let the unreliable one go second. Put the durable, retrying, server-side actor in charge of the write that matters, stash everything it needs ahead of time, and assume it will be told to do the job more than once. Everything above is just the working-out of that one sentence.