Idempotency in Payment Systems: Lessons From Building POS Checkout Flows

If you've built a payment flow for any kind of point-of-sale system, you've probably had this moment: a customer swipes their card, the screen freezes for a second, the cashier taps "retry" out of instinct, and now you're staring at two charges on one order. Multiply that by a busy Friday dinner rush across a few hundred terminals, and idempotency stops being a nice-to-have and becomes the thing standing between you and a very uncomfortable support queue.

This post walks through what we learned building checkout flows for a restaurant POS system, why "just don't double-submit" is not a real strategy, and how we actually solved it.

The Problem, Concretely

A checkout flow in a restaurant POS looks simple on paper:

1. Cashier finalizes the order.
2. POS sends a charge request to the payment processor.
3. Processor returns success or failure.
4. POS updates the order status and prints a receipt.

The failure modes live entirely in step 2 and 3. Networks are unreliable, especially in restaurants where a terminal might be on shaky Wi-Fi or a spotty cellular connection. Here's what actually goes wrong in production:

• The charge request reaches the processor and succeeds, but the response never makes it back to the POS (timeout). The POS assumes failure and retries. Now you've charged the customer twice.
• The cashier gets impatient during a slow response and manually hits "retry" before the first request has resolved.
• A terminal crashes or reboots mid-transaction, and on restart, a queued request gets replayed.
• A background job retries a failed webhook delivery from the payment processor, and your backend processes the same "payment succeeded" event twice.

None of these are exotic edge cases. If you're running any real transaction volume, you will hit all of them, regularly.

Why "Check If It Already Happened" Isn't Enough

The naive fix is to query your own database before charging: "has this order already been paid?" This helps, but it's not sufficient on its own, for two reasons.

First, there's a race condition. If two requests for the same order arrive close together, both can pass the "not yet paid" check before either one updates the order status. You need atomicity, not just a check.

Second, this only protects against duplicates within your own system. It does nothing to prevent your payment processor from seeing two separate charge requests and treating them as two legitimate transactions, because from the processor's point of view, they're just two API calls with no inherent relationship to each other.

This is exactly what idempotency keys solve.

Idempotency Keys: The Core Mechanism

Most modern payment processors (Stripe, Adyen, Braintree, and others) support idempotency keys natively. The idea is simple: you generate a unique key per logical operation, and you send that same key with every retry of that operation. The processor recognizes the key and returns the result of the original request instead of creating a new charge.

// Generate the idempotency key once, when the checkout attempt begins
const idempotencyKey = `order_${orderId}_attempt_${attemptId}`;

async function chargeOrder(order, idempotencyKey) {
  const response = await paymentProcessor.charges.create(
    {
      amount: order.totalCents,
      currency: "usd",
      source: order.paymentToken,
      metadata: { orderId: order.id },
    },
    {
      idempotencyKey, // Same key on retry = same result, no duplicate charge
    }
  );

  return response;
}

The critical detail here is how you generate that key. It needs to be:

• Stable across retries of the same logical attempt. If your retry logic regenerates a new UUID every time it retries, you've defeated the entire mechanism.
• Unique per distinct checkout attempt. If a cashier voids a failed transaction and genuinely starts a new one, that should get a new key — otherwise a legitimately different charge could get silently deduplicated.

We landed on keying off the order ID plus an "attempt" counter that only increments when a human explicitly starts a new transaction (not on automatic network retries). That distinction — automatic retry vs. new attempt — is where most idempotency bugs actually hide.

Idempotency Isn't Just at the Payment Processor Layer

Here's the part that's easy to miss: even with a solid idempotency key strategy at the processor level, you still need idempotency inside your own backend. Your system has to safely handle:

• The same webhook event delivered more than once (processors generally guarantee at-least-once delivery, not exactly-once).
• A client retrying an API call to your own order service, independent of what happens downstream with the processor.

For webhooks, we store the event ID from the processor and check it against a table before processing:

async function handlePaymentWebhook(event) {
  const alreadyProcessed = await db.query(
    "SELECT 1 FROM processed_webhook_events WHERE event_id = $1",
    [event.id]
  );

  if (alreadyProcessed.rowCount > 0) {
    return; // Already handled, safely ignore
  }

  await db.transaction(async (tx) => {
    await tx.query(
      "INSERT INTO processed_webhook_events (event_id, received_at) VALUES ($1, now())",
      [event.id]
    );
    await applyPaymentResult(event, tx);
  });
}

The insert and the business logic update happen in the same transaction. This matters — if you check for the event ID first and insert it after processing, a crash between those two steps reintroduces the exact race condition you were trying to avoid.

Handling the Order State Machine

Idempotency at the payment layer only gets you halfway. The order itself needs a well-defined state machine so that duplicate or out-of-order events don't corrupt its status.

A simplified version of what we use:

pending → charging → paid
pending → charging → failed → pending (retry allowed)
paid → refunded

The key rule: transitions are only valid from specific prior states. If an event says "mark this order paid" but the order is already in the paid state, that's treated as a no-op, not an error and not a re-processing.

async function markOrderPaid(orderId, tx) {
  const result = await tx.query(
    `UPDATE orders
     SET status = 'paid', paid_at = now()
     WHERE id = $1 AND status = 'charging'
     RETURNING id`,
    [orderId]
  );

  if (result.rowCount === 0) {
    // Either already paid, or in an unexpected state — log and investigate,
    // but don't blindly reapply the transition
    logger.warn(`Skipped paid transition for order ${orderId}, unexpected state`);
  }
}

That WHERE status = 'charging' clause is doing the real work here. It makes the update conditional on the current state, so a duplicate event simply fails to match any rows instead of double-applying a transition.

What About Offline Mode?

Restaurant POS systems often need to function during connectivity drops — you don't want a terminal going fully dark because Wi-Fi hiccupped mid-shift. This adds another layer to the idempotency problem: transactions queued locally on a terminal and synced later.

Our approach was to generate the idempotency key client-side, at the moment the transaction is initiated, rather than server-side. That way, even if a terminal queues a transaction offline and doesn't sync it until connectivity returns 20 minutes later, the key travels with the transaction the entire time and any retries during that sync window resolve to the same result.

// Generated locally on the terminal, before any network call is attempted
function createLocalTransaction(order) {
  return {
    orderId: order.id,
    idempotencyKey: `${order.id}_${crypto.randomUUID()}`,
    createdAt: Date.now(),
    status: "queued",
  };
}

The important discipline here is that this key is generated exactly once and persisted locally before the first network attempt — not regenerated each time the sync job runs.

Lessons That Generalize Beyond POS

A few things we'd tell anyone building payment flows, POS-specific or not:

• Idempotency keys need a clear ownership boundary. Decide explicitly what counts as "the same operation" versus "a new one," and enforce that distinction in code, not just convention.
• State machines with conditional transitions prevent more bugs than they look like they should. A single WHERE status = X clause on an update is often cheaper and more reliable than complex deduplication logic.
• At-least-once delivery is the default assumption, not the exception. Design webhook and event handlers to be safely replayable from the start, rather than bolting on deduplication after the first incident.
• Client-generated keys matter for offline-tolerant systems. If your system needs to function without a live connection, idempotency has to be decided at the point of origin, not assigned later by a server that might not see the request until much later.

None of this is unique to restaurant POS software — it applies to e-commerce checkout, subscription billing, or any system where money changes hands over an unreliable network. But restaurants make a particularly unforgiving testing ground: high transaction volume, flaky in-store connectivity, and cashiers who will absolutely hit "retry" the moment a screen looks frozen. If your idempotency strategy survives a Friday night dinner rush, it'll survive just about anything else you throw at it.