Shubham

Posted on Nov 14 Originally published at shubhkumar.in

Idempotency Is a Product Feature

#systemdesign #payments #architecture #distributedsystems

Your users do not care that their Wi-Fi dropped between your API receiving a request and your server returning a response.

They care that clicking "Pay" once did not charge them twice.

They care that refreshing the page did not create two orders.

They care that a mobile app retrying after a tunnel ride did not send duplicate invite emails to the whole company.

That is why idempotency is not only distributed-systems hygiene. It is product behavior.

Most teams meet idempotency during payment integration.

Stripe asks for Idempotency-Key.

AWS asks for ClientToken.

You add a UUID header, cache the response, and move on.

But that framing is too small.

Idempotency defines whether your product remains predictable when the network stops being perfect.

And networks are never perfect.

Requests time out.
Mobile apps get killed mid-flight.
Browsers retry.
Load balancers retry.
SDKs retry.
Users double-click.
Queue consumers receive the same message again.
Webhooks arrive multiple times.
Deployments interrupt requests after the database commit but before the response.

None of these are exotic failures.

They are normal operating conditions for systems that communicate over networks.

If your product cannot answer:

"What happens when the same intent arrives twice?"

then your product has a gap.

The failure users see is not:

"Non-idempotent POST request."

They see:

Duplicate charge
Duplicate booking
Two cloud machines running
Ten identical Slack alerts
Account created but onboarding failed
"Something went wrong" despite the action succeeding
Support unable to determine whether an order exists

That last one matters more than engineers often admit.

Ambiguity is product debt.

The most expensive state in distributed systems is not failure.

It is:

"We do not know."

The Request That Disappeared

Consider a checkout flow:

User clicks Pay.
Client sends POST /payments.
Server creates the payment successfully.
Response gets lost due to a network timeout.
Client does not know whether the payment happened.
User clicks Pay again.

What should happen?

Without idempotency, the second request may create a second payment.

// Without idempotency   double charge
const res = await fetch('/payments', {
  method: 'POST',
  body: JSON.stringify({ amount: 4999, currency: 'usd' }),
});

if (!res.ok) {
  // Timeout or network error   user retries
  const retry = await fetch('/payments', {
    method: 'POST',
    body: JSON.stringify({ amount: 4999, currency: 'usd' }),
  });
  // Two charges. Refund workflow. Support ticket. Lost trust.
}

You now have:

Duplicate charge
Refund workflow
Support ticket
Lost trust
Finance reconciliation work
User wondering whether your product is safe

With idempotency:

// With idempotency   one charge
const key = crypto.randomUUID(); // Generated once, on user action

const sendPayment = async (retryKey: string) => {
  const res = await fetch('/payments', {
    method: 'POST',
    headers: { 'Idempotency-Key': retryKey },
    body: JSON.stringify({ amount: 4999, currency: 'usd' }),
  });
  return res.json();
};

// First attempt
await sendPayment(key);

// On timeout, retry with SAME key
await sendPayment(key);
// Server recognizes key, returns original result.
// One payment. One receipt. One coherent story.

Notice what changed.

Not just database correctness.

The user experience changed from:

"Did you charge me twice?"

"The network was flaky, but my payment went through."

That is product quality.

Retries Are Product Behavior

We often treat retries as infrastructure plumbing.

Exponential backoff
Jitter
Timeout tuning
Circuit breakers

All useful.

None answer the real question.

Can the caller retry safely?

A retry policy without idempotency is a machine gun attached to your foot.

If an operation:

creates a resource
sends money
reserves inventory
issues a token
dispatches notifications
triggers third-party side effects

then retries can multiply outcomes.

// Retry without idempotency   dangerous
async function deployCluster(config: ClusterConfig) {
  try {
    return await api.post('/clusters', config);
  } catch (err) {
    if (isTimeout(err)) {
      // Retry   but the first call may have succeeded
      return await api.post('/clusters', config); // Two clusters now running
    }
    throw err;
  }
}

A reliable system needs two promises:

Temporary failure should be recoverable.
Retrying must not create additional user-visible side effects.

Idempotency is the contract that makes both possible.

// Retry with idempotency   safe
async function deployCluster(config: ClusterConfig) {
  const key = getDeploymentKey(config); // Derived from user action, not request

  try {
    return await api.post('/clusters', config, {
      headers: { 'Idempotency-Key': key },
    });
  } catch (err) {
    if (isTimeout(err)) {
      // Retry with same key   server returns existing cluster or processes
      return await api.post('/clusters', config, {
        headers: { 'Idempotency-Key': key },
      });
    }
    throw err;
  }
}

Same Payload Does Not Mean Same Intent

A common implementation hashes the request body:

hash(userId + requestBody)

It feels clever.

It is often wrong.

Imagine a user launches two identical EC2 instances.

Same:

Image
Region
Instance type
Payload

Are they duplicates?

Maybe.

Maybe not.

The payload cannot reliably tell you intent.

// Payload-based dedup   flawed
function generateKey(req: Request) {
  const body = req.body;
  return `${body.userId}-${body.image}-${body.region}-${body.instanceType}`;
}

// User launches two identical instances intentionally
// Both get same key   second one rejected
// User confused: "Why won't my second instance start?"

Instead, let the client generate an explicit idempotency key.

The client is saying:

These retries belong to one logical action.

// Client-controlled key   correct
function onDeployClick() {
  const key = crypto.randomUUID(); // New key per user action
  api.post('/instances', payload, {
    headers: { 'Idempotency-Key': key },
  });
}

// Same key + same caller = same operation (retry)
// New key + identical payload = possibly new operation (intentional)

That subtle distinction protects product semantics.

Good Idempotency Isn't "Ignore Duplicates"

Stripe compares request parameters when the same key is reused.

Suppose:

checkout-123

first means:

Buy Pro Plan

Later the same key is used for:

Buy Enterprise Plan

Returning the old response silently would hide a bug.

Instead:

Reject the request.

async function handleIdempotentRequest(key: string, payload: Payload) {
  const existing = await db.idempotencyKey.findUnique({ where: { key } });

  if (existing) {
    // Same key reused   check if payload matches
    if (existing.requestHash !== hashPayload(payload)) {
      return new Response(
        JSON.stringify({ error: 'Idempotency key reused with different parameters' }),
        { status: 422 }
      );
    }
    // Same key, same payload   return stored response
    return new Response(existing.responseBody, { status: existing.responseStatus });
  }

  // First request   process and store
  const result = await processPayment(payload);

  await db.idempotencyKey.create({
    data: {
      key,
      requestHash: hashPayload(payload),
      responseBody: JSON.stringify(result),
      responseStatus: 200,
      resourceId: result.id,
    },
  });

  return new Response(JSON.stringify(result), { status: 201 });
}

Good idempotency means:

Honor one declared intent consistently.

Idempotency Needs Product Design

Adding an Idempotency-Key header is necessary.

It is not sufficient.

Suppose a payment provider times out.

You do not know whether the charge succeeded.

Do not show:

Payment failed.

Do not encourage another payment attempt.

Instead say:

Your payment is processing. Please do not retry. We'll confirm the status shortly.

That message reflects backend truth.

A healthy product state model looks like:

Processing
Succeeded
Failed
Requires Action
Reconciling
Reversed

// Product state machine for uncertain outcomes
type PaymentState =
  | { status: 'processing'; message: 'Your payment is being processed.' }
  | { status: 'succeeded'; receiptUrl: string }
  | { status: 'failed'; reason: string }
  | { status: 'requires_action'; redirectUrl: string }
  | { status: 'reconciling'; message: 'Checking with payment provider...' };

function getPaymentDisplay(state: PaymentState) {
  switch (state.status) {
    case 'processing':
      return {
        title: 'Processing',
        message: state.message,
        showRetry: false, // Do NOT let user retry
      };
    case 'succeeded':
      return {
        title: 'Done',
        message: 'Payment confirmed.',
        showRetry: false,
      };
    case 'failed':
      return {
        title: 'Failed',
        message: state.reason,
        showRetry: true, // Safe to retry
      };
    case 'reconciling':
      return {
        title: 'Checking',
        message: state.message,
        showRetry: false, // Do NOT let user retry
      };
  }
}

Users can handle uncertainty.

They cannot handle duplicate charges.

The Real Contract

Think of every write endpoint as mapping:

One User Intent
        ↓
One Durable Outcome

Not:

One HTTP Request
        ↓
One Outcome

Because requests can be duplicated everywhere.

User Tap
      ↓
Browser Retry
      ↓
SDK Retry
      ↓
Load Balancer Retry
      ↓
Service Retry
      ↓
Queue Redelivery
      ↓
Webhook Replay

Every layer may duplicate.

Your product should not.

// The contract: intent → outcome, not request → outcome
interface IdempotentEndpoint {
  // Input: user intent, not raw HTTP
  intent: UserIntent;
  // Key: stable across retries of same intent
  key: string;
  // Output: always the same for same key
  outcome: Outcome;
}

// Implementation
async function handleIntent(intent: UserIntent, key: string): Promise<Outcome> {
  const existing = await lookupByKey(key);
  if (existing) return existing.outcome;

  const outcome = await executeIntent(intent);
  await store(key, outcome);
  return outcome;
}

Operation Identity

Every system eventually invents one.

It may be called:

Idempotency Key
Client Token
Request ID
Operation ID
Payment Intent ID
Command ID
Event ID

The name matters less than the lifecycle.

It should be:

Unique enough
Scoped correctly
Persisted
Reused only for retries
Bound to a request fingerprint
Present in logs, traces, support tools, and downstream services

-- Idempotency record lifecycle
CREATE TABLE idempotency_keys (
  id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  tenant_id     UUID NOT NULL,
  key           TEXT NOT NULL,
  request_hash  TEXT NOT NULL,
  status        TEXT NOT NULL DEFAULT 'pending', -- pending | succeeded | failed
  response      JSONB,
  resource_id   UUID,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at    TIMESTAMPTZ NOT NULL,

  UNIQUE (tenant_id, key)
);

-- Auto-cleanup: keys expire after 24 hours
CREATE INDEX idx_idempotency_expires ON idempotency_keys (expires_at)
  WHERE status = 'pending';

Support teams love operation IDs.

Instead of asking:

"Can you explain what happened?"

they search one identifier.

Everyone sees the same story.

A Practical Implementation

1. Client Generates the Key

Generate it when the user begins the operation.

Not every retry.

// Client side   key lifecycle
function onCheckoutClick() {
  // One key per user action
  const key = crypto.randomUUID();
  sessionStorage.setItem('checkout-key', key);

  api.post('/checkout', cart, {
    headers: { 'Idempotency-Key': key },
  });
}

function onRetryClick() {
  // Reuse existing key
  const key = sessionStorage.getItem('checkout-key');
  if (!key) return redirectToCheckout();

  api.post('/checkout', cart, {
    headers: { 'Idempotency-Key': key },
  });
}

2. Store an Idempotency Record

-- Server side   record creation
BEGIN;

-- Check for existing key
SELECT status, response FROM idempotency_keys
WHERE tenant_id = $1 AND key = $2;

-- If found: return stored response
-- If not found: insert and process

INSERT INTO idempotency_keys (tenant_id, key, request_hash, status, expires_at)
VALUES ($1, $2, $3, 'processing', now() + interval '24 hours')
ON CONFLICT (tenant_id, key) DO NOTHING;

-- Process the request
INSERT INTO orders (...) VALUES (...) RETURNING id;

-- Update key with result
UPDATE idempotency_keys
SET status = 'succeeded', response = $4, resource_id = $5
WHERE tenant_id = $1 AND key = $2;

COMMIT;

3. Validate Key Reuse

// Server middleware   idempotency check
async function idempotencyGuard(req: Request, key: string) {
  const existing = await db.query(
    `SELECT status, response, request_hash FROM idempotency_keys
     WHERE tenant_id = $1 AND key = $2`,
    [req.tenantId, key]
  );

  if (existing) {
    if (existing.status === 'succeeded' || existing.status === 'failed') {
      // Finished   return stored response
      return new Response(existing.response, { status: existing.status === 'succeeded' ? 200 : 500 });
    }

    if (existing.status === 'processing') {
      // Still running   return 409 or 202 depending on API contract
      return new Response(JSON.stringify({ status: 'processing' }), { status: 409 });
    }
  }

  // New request   store and continue
  const requestHash = hashPayload(req.body);
  if (existing && existing.requestHash !== requestHash) {
    return new Response(
      JSON.stringify({ error: 'Key reused with different payload' }),
      { status: 422 }
    );
  }

  return null; // Continue to handler
}

4. Make Writes Atomic

Bad:

Save key
↓
Order creation fails

Retry now sees the key.

No order exists.

Also bad:

Create order
↓
Save key fails

Retry creates another order.

Use one database transaction.

// Atomic idempotency + write
async function createOrderWithIdempotency(key: string, order: OrderData) {
  return db.transaction(async (tx) => {
    // 1. Insert key atomically
    const inserted = await tx.query(
      `INSERT INTO idempotency_keys (tenant_id, key, request_hash, status, expires_at)
       VALUES ($1, $2, $3, 'processing', now() + interval '24 hours')
       ON CONFLICT (tenant_id, key) DO NOTHING
       RETURNING id`,
      [order.tenantId, key, hashPayload(order)]
    );

    if (!inserted) {
      // Key exists   fetch and return stored result
      const existing = await tx.query(
        `SELECT status, response FROM idempotency_keys
         WHERE tenant_id = $1 AND key = $2`,
        [order.tenantId, key]
      );
      return JSON.parse(existing.response);
    }

    // 2. Create order in same transaction
    const result = await tx.query(
      `INSERT INTO orders (...) VALUES (...) RETURNING *`,
      [order]
    );

    // 3. Update key with result
    await tx.query(
      `UPDATE idempotency_keys SET status = 'succeeded', response = $3, resource_id = $4 WHERE id = $1`,
      [inserted.id, JSON.stringify(result), result.id]
    );

    return result;
  });
}

5. Handle External Side Effects

Database transactions cannot roll back:

Email
Kafka
Stripe
Slack
Webhooks

Use the Outbox Pattern.

-- Inside the same transaction as the order
INSERT INTO outbox_events (id, aggregate_type, aggregate_id, event_type, payload)
VALUES (
  gen_random_uuid(),
  'order',
  $order_id,
  'order.created',
  jsonb_build_object('orderId', $order_id, 'email', $customer_email)
);

Worker:

// Outbox worker   idempotent side effects
async function processOutboxEvent(event: OutboxEvent) {
  // Check if already processed
  const processed = await db.query(
    `SELECT id FROM processed_events WHERE event_id = $1`,
    [event.id]
  );
  if (processed) return; // Already handled

  switch (event.eventType) {
    case 'order.created':
      await sendOrderConfirmation(event.payload);
      break;
    case 'payment.captured':
      await notifyFinanceTeam(event.payload);
      break;
  }

  // Mark as processed
  await db.query(
    `INSERT INTO processed_events (event_id, processed_at) VALUES ($1, now())`,
    [event.id]
  );
}

If the worker crashes, resume from durable state.

Not from scratch.

Exactly Once Is Usually Marketing

Exactly-once processing rarely exists.

Messages duplicate.

Consumers restart.

Providers timeout.

Networks lie.

What users actually want is:

Effectively once.

Maybe a message is processed three times.

But:

One order exists.
One payment is captured.
One welcome email is sent.

That is what "once" means.

// Effectively-once pattern
async function handleMessage(msg: QueueMessage) {
  const idempotent = await checkIdempotency(msg.id);
  if (idempotent.alreadyProcessed) {
    console.log(`Skipping duplicate message ${msg.id}`);
    return;
  }

  await processMessage(msg);
  await markProcessed(msg.id);
}

The distinction matters for engineering decisions.

You stop chasing the impossible (true exactly-once delivery).

You start building what users actually need (effectively-once outcomes).

Where Teams Miss It

Payments get attention.

Many other operations deserve the same treatment.

Account Creation

Retry should return the existing account.

Not:

Email already exists.

// Bad: duplicate accounts
try {
  await db.users.create({ email, name });
} catch (err) {
  if (err.code === 'DUPLICATE') {
    throw new Error('Email already exists'); // User confused
  }
}

// Good: idempotent creation
const key = `signup-${email}`;
const result = await idempotentUpsert(key, async () => {
  return db.users.create({ email, name });
});
// Returns existing account on retry   no confusion

Infrastructure Provisioning

One click.

One cluster.

Not two.

Notifications

Webhook replay should not send five emails.

Permissions

Retrying an invite should not generate multiple invite links.

Imports

Retrying CSV uploads should not duplicate every customer.

Subscriptions

Upgrade should not apply discounts twice.

AI Agents

This one is becoming critical.

Agents retry.

Tools fail.

Workflows resume.

If agents:

Create tickets
Send emails
Deploy services
Issue refunds

every action needs operation identity.

// Agent action with idempotency
async function agentAction(action: AgentAction) {
  const key = `agent-${action.workflowId}-${action.stepId}`;

  // Agent retries after tool failure   same key, same result
  return idempotentRequest(key, () => executeAction(action));
}

Agent reliability is an idempotency problem in disguise.

What To Measure

Track:

Idempotency key reuse
Duplicate requests prevented
Parameter mismatch rate
Stuck processing operations
Unknown external outcomes
Reconciliation time
Duplicate downstream side effects
Duplicate order support tickets
Retry success rate

-- Idempotency health dashboard
SELECT
  date_trunc('day', created_at) AS day,
  COUNT(*) AS total_keys,
  COUNT(*) FILTER (WHERE status = 'succeeded') AS succeeded,
  COUNT(*) FILTER (WHERE status = 'failed') AS failed,
  COUNT(*) FILTER (WHERE status = 'processing') AS stuck,
  COUNT(*) FILTER (WHERE expires_at < now() AND status = 'processing') AS expired_stuck,
  AVG(EXTRACT(EPOCH FROM (updated_at - created_at))) AS avg_resolution_seconds
FROM idempotency_keys
WHERE created_at > now() - interval '7 days'
GROUP BY 1
ORDER BY 1 DESC;

Metrics reveal product fragility.

The Idempotency Checklist

Before shipping a write endpoint, ask:

What is one logical user intent?
Which retries belong to that intent?
Who generates the operation key?
What scope prevents collisions?
How long should keys live?
What happens when the same key has different payload?
What should callers receive while work is in progress?
What happens after timeout with unknown outcome?
Can support locate operations using the key?
Are downstream systems also idempotent?
What does the UI say during uncertainty?
Can users retry safely?

If these answers are unclear,

the endpoint is not finished.

The Boring Feature Nobody Praises

Nobody opens a support ticket saying:

Loved how your API returned the same result after packet loss.

They notice the opposite.

They notice:

Money disappearing twice.
Duplicate emails.
Dashboard saying failed even though it worked.
Support saying:

We don't know what happened.

Idempotency is invisible when it works.

Like:

Backups
Seatbelts
Autosave

Invisible does not mean optional.

It is trust infrastructure.

Reliable products do not merely survive retries.

They make retries feel normal.

That is the standard worth building toward.

Users can click again.

Networks can fail.

Workers can restart.

Messages can replay.

And the product still tells one coherent story.

DEV Community