🎃 The Real Nightmare Isn't a Bug. It's a Bug Nobody Notices. 🎃

Imagine deploying a new feature.

All tests pass. CI is green. No errors in the logs. Everything looks fine.

Then a user reports that their payment was processed twice.

Here's a simplified example of what can go wrong:

async function processPayment(order) {
  try {
    return await paymentAPI.charge({
      amount: order.total,
      customerId: order.customerId
    });
  } catch (error) {
    if (error.code === "TIMEOUT") {
      return await paymentAPI.charge({
        amount: order.total,
        customerId: order.customerId
      });
    }

    throw error;
  }
}

Looks reasonable, but there's a problem.

The first request might have reached the payment service and completed successfully. Only the response got lost.

The retry sends another charge request.

Now imagine this happening with a real customer.

One way to prevent this is to use an idempotency key:

async function processPayment(order) {
  return await paymentAPI.charge({
    amount: order.total,
    customerId: order.customerId,
    idempotencyKey: `payment-${order.id}`
  });
}

Now the payment service can recognize repeated requests for the same operation and return the original result instead of charging the customer again.

Of course, this depends on the payment provider correctly supporting and implementing idempotency.

That's what makes production bugs interesting. Sometimes the code works exactly as written. The problem is that nobody considered what would happen under real-world conditions.

A green test suite is good news, but it doesn't mean you've tested every possible failure.

Sometimes the scariest bugs are the ones that never show up in your test results. 👻

Happy Halloween! 🎃

Comments

[Steven Mock]

The double-payment example lands because the damage is not technical, it is trust. A customer who gets charged twice does not think "retry logic bug" — they think the business cannot be trusted with their money, and that is very hard to win back.

I liked the shift from "did the code run?" to "did the outcome happen exactly once?" That is the check that matters for anything involving money, bookings, or enquiries. Green tests and a quiet log can still hide a failure that only shows up when a real person is affected, which is exactly why it is so easy to miss.

[Sina Rezaei]

That's an important distinction. There's another layer to this problem that often gets overlooked: even if the payment provider handles idempotency correctly, your own backend can still introduce duplicate side effects.

For example, Stripe may deliver the same webhook more than once if your endpoint doesn't acknowledge it in time. If your handler updates the order successfully but crashes before recording the event as processed, the next delivery can trigger the same business logic again.

That's where an inbox table with a unique constraint on the provider's event ID, combined with transactional processing, becomes important. And if the workflow triggers downstream actions, an outbox pattern helps keep those operations consistent too.

You can't realistically guarantee exactly-once message delivery across distributed services, but you can design the system so repeated delivery doesn't mean repeated business effects.