When Your API Ghosts You: A Deep Dive Into Idempotency in REST APIs

You know that sinking feeling when a user clicks "Pay" twice because the spinner froze—and your backend quietly double-charges them? That isn't "network unreliability." That's "we didn't design for idempotency."

In distributed systems, retries are not an edge case; they're the default path. Clients will retry. Proxies will retry. Your own code will retry. The only question is whether your system can handle it without corrupting state.

Situation: The "Phantom Double Charge"

You ship a payments API. A customer hits /payments with a POST, the network hiccups, your client never sees the 201, so it retries.

From your user's perspective:

• "I clicked once. Your app broke."

From your database's perspective:

• "I just created two payments, both valid, good luck explaining that."

The bug isn't in the retry. The bug is that the operation wasn't idempotent.

Task: Make Retries Boring

The real job is not "process this payment." The real job is "process this payment in a way that stays correct even if the same request hits us multiple times."

Idempotency means:

• Multiple identical requests have the same effect as a single request.
• After the first success, every duplicate is effectively a no-op in terms of system state.

Boring? Yes. That's the goal.

Action: Treat Requests Like Events, Not Wishes

Here's the simple move most teams skip: give each potentially destructive request a stable identity.

Typical Pattern:

1. Client generates an Idempotency-Key (UUID, payment intent ID, etc.)
2. Sends it with the POST:

   POST /payments HTTP/1.1
   Idempotency-Key: 123e4567-e89b-12d3-a456-426614174000
   Content-Type: application/json

   {"amount": 5000, "currency": "USD"}

1. Server logic:
   • Check if this key has been seen before
   • If not: process the request, store result keyed by Idempotency-Key
   • If yes: return the previously stored response instead of processing again

Why This Works:

• Network retries become safe—the second call is just a cache lookup
• Client logic simplifies to: "retry until success or clear failure"
• No more worrying about double side effects
• Even if your database is eventually consistent, idempotency keys make this deterministic

Quick Implementation Pattern:

# Pseudocode for the idea

def process_payment(request_body, idempotency_key):
    # Check cache first
    cached_response = idempotency_cache.get(idempotency_key)
    if cached_response:
        return cached_response  # Already processed

    # Process only if new
    result = create_payment(request_body)

    # Store for future retries
    idempotency_cache.set(idempotency_key, result)

    return result

A note on HTTP semantics:

• GET, PUT, DELETE are supposed to be idempotent by design
• POST is not, which is why idempotency keys usually show up there
• Some teams use idempotency keys for all mutable operations—even better

Result: Fewer Fires, More Sleep

Once you design around idempotency:

• Payment "double charges" turn into "same payment, same receipt, every time."
• Support tickets about "I only clicked once" quietly vanish
• Retries become a reliability feature, not a reliability bug
• Your team debugs 3 AM incidents with more confidence

Idempotency won't make your API exciting. It will make it trustworthy. And trust is what lets you ship faster without being afraid of your own error logs.

The Deeper Breakdown

If this resonates—if you're into the kind of "real-world backend hygiene"—idempotency, retries, failure modes, the unglamorous stuff that actually keeps systems up—then you might want to check out CoreCraft.

It's a newsletter focused on exactly this: concrete patterns, real trade-offs, code you can use tomorrow. No fluff, no vague "best practices," just battle-tested abstractions for working engineers.

👉 Read more at CoreCraft

---

Have a production idempotency horror story? Drop it in the comments. Let's normalize talking about this stuff.

Comments

[Tyler]

The idempotency key pattern is the right foundation and the cache-lookup
framing makes retries genuinely boring in the best way.

One dimension worth extending: idempotency solves the "did we process this
request" question cleanly. The harder adjacent problem is "what was the
authoritative state at the time we processed it" — especially in systems
where the resource being mutated is itself receiving conflicting updates
from multiple sources simultaneously.

A payment is relatively clean because the idempotency key scopes to a
single client intent. Device state in IoT and distributed sensor systems
is messier — the same resource can receive conflicting updates from
multiple sources within milliseconds, each with a valid idempotency key,
and the question becomes not "did we process this" but "which of these
valid, deduplicated updates reflects ground truth."

At that point idempotency is necessary but not sufficient. You need a
resolution layer underneath it that can take two non-duplicate events
that genuinely conflict — same device, different reported state, 900ms
apart, both signed, both valid — and return one authoritative answer with
a documented basis for the decision rather than silently picking one and
hoping it was right.

The 3am incident risk doesn't go away with idempotency alone. It shifts
from "we processed this twice" to "we processed the wrong one and had no
way to know." The audit trail you get from idempotency keys tells you what
was processed. It doesn't tell you whether the right thing won.

Solid post — the HTTP semantics note is a detail a lot of engineers skip
over and shouldn't.