Idempotency Is the System Design Concept That Actually Saves You

If you ask backend engineers to name system design concepts, you'll get
"scalability," "load balancing," "caching," "sharding" — the big, glamorous
ones. Idempotency rarely makes the list, and I think that's backwards. It's
one of the few concepts that quietly determines whether your system behaves
correctly the moment anything goes wrong, and things going wrong — networks
dropping, clients retrying, workers crashing mid-job — isn't an edge case in
a real system. It's Tuesday.

What idempotency actually means

An operation is idempotent if running it once has the same effect as running
it multiple times. PUT /users/42 { name: "Chika" } is idempotent — do it
once or five times, the user's name ends up "Chika" either way. POST
/orders that creates a new order every time it's called is not — call it
twice and you've got two orders, possibly two charges.

That's the textbook definition. The part that matters in practice is why
you should care: the network doesn't give you exactly-once delivery. A
client sends a request, the server processes it successfully, and the
response gets lost on the way back — timeout, dropped connection, whatever.
From the client's point of view, it looks exactly like the request never
arrived. The only sane thing to do is retry. Which means every operation
your system exposes to retries needs to survive being executed more than
once for the same logical intent.

Where this bites you if you don't design for it

Payment and order creation. The classic example for a reason. A client
submits a payment, the request succeeds server-side, the response times out
before the client sees it, the client retries. Without idempotency
protection, that's a double charge — and it's the kind of bug that doesn't
show up in testing, because your test environment doesn't have a flaky
network. It shows up in production, at scale, as a support ticket queue.

Message queue consumers. Most queue systems (SQS, RabbitMQ, Kafka
consumer groups) give you at-least-once delivery, not exactly-once, as
the default guarantee — a message can be redelivered after a consumer
crashes before acknowledging it, even if the work was actually completed.
If your message handler isn't idempotent, a crash-and-redeliver cycle
silently duplicates whatever side effect that handler causes.

Distributed writes across services. Anytime one service calls another
and the caller can't be certain whether the callee actually completed the
operation before the connection dropped, you have the same problem in
miniature. Retrying blindly assumes non-idempotent operations are safe to
repeat, which is usually the wrong assumption.

How to actually build it in

Idempotency keys. The client generates a unique key per logical
operation (a UUID, typically) and sends it with the request. The server
checks: have I seen this key before? If yes, return the stored result of the
original operation instead of re-executing it. If no, execute it and store
the result against that key. Stripe's API is the reference example most
engineers have actually used as a client — send the same Idempotency-Key
header twice, get the same charge result twice, not two charges.

async function createPayment(req, res) {
  const { idempotencyKey, amount, userId } = req.body;

  const existing = await IdempotencyRecord.findOne({ key: idempotencyKey });
  if (existing) {
    return res.status(existing.statusCode).json(existing.response);
  }

  const payment = await processPayment(userId, amount);

  await IdempotencyRecord.create({
    key: idempotencyKey,
    statusCode: 201,
    response: payment,
  });

  return res.status(201).json(payment);
}

Natural idempotency via upserts. Sometimes you don't need a separate
key at all — the operation itself can be made idempotent by its own
structure. Instead of "insert a new record," do "upsert keyed on a natural
unique identifier" (updateOne({ orderId }, { $set: data }, { upsert: true
})). Run it once or five times, you get the same end state, because the
write is keyed on something that already uniquely identifies the logical
entity rather than creating a new one each call.

Set-based state transitions instead of increments. SET status =
'shipped' is idempotent. UPDATE inventory SET quantity = quantity - 1 is
not — run it twice and you've decremented twice. Where possible, prefer
writes that assign an absolute value or a range-conditioned transition over
writes that apply a relative change, precisely because the former survives
duplication and the latter doesn't.

What idempotency doesn't solve

It's not a substitute for genuine distributed transactions when you need
strict atomicity across multiple resources — it solves "this specific
operation is safe to retry," not "these five operations across three
services either all happen or none do." Sagas and compensating transactions
address that broader problem; idempotency is what makes each individual
step in a saga safe to retry when a step fails partway through, which is
usually why the two show up together in the same system.

It also doesn't remove the need to think about concurrency. Two different
clients hitting the same idempotent endpoint at the same time with
different idempotency keys is a race condition idempotency does nothing
for — that's a locking or optimistic-concurrency problem, a separate
concern that idempotency sometimes gets mistaken for solving.

The actual habit worth building

Before writing any endpoint or message handler that causes a side effect —
a write, a charge, a notification, a state change — ask one question: what
happens if this runs twice for the same logical request? If the honest
answer is "something bad," that's the signal to add an idempotency key, an
upsert, or a set-based transition before it ships, not after the first
double-charge ticket comes in. It's a five-minute question at design time
and a much longer one to answer after the fact in production.