What Happens When the Same Payment Arrives Twice?

Imagine a customer is tipping a waiter in a cafe which is in underground where the internet signal is weak.

The tip request reaches the backend successfully, but the 200 OK response never reaches the customer. The browser assumes the request failed and retries it.

Now the backend receives the same tip twice, which can result in the customer being charged twice.

And now the user gets two bank notifications saying bucks 20 was debited from their account. 🤬

Instead of giving a Bucks 20 tip, the user has now given Bucks 40 because the request was processed twice. Frustrated, the user gives the cafe a bad rating for its payment system. This is a classic idempotency issue, where the same request gets processed more than once.

---

First, Let Create a Basic payment system

Create a basic system that takes a user request with the price and quantity and saves it into the DB of the tip-order table.

And now let's create a test case that simulates the same things that we discussed above, a fake user sending multiple requests of the same tip, and at the same time.

import http from 'k6/http';
import { check } from 'k6';
import { Counter } from 'k6/metrics';

// Track response statuses
const createdCounter = new Counter('status_201');
const conflictOrErrorCounter = new Counter('status_errors');

export const options = {
  scenarios: {
    concurrent_blast: {
      executor: 'shared-iterations',
      vus: 50,             // 50 concurrent virtual users
      iterations: 50,      // exactly 50 requests
      maxDuration: '10s',  // execute within 10s
    },
  },
  thresholds: {
    // All 50 requests must return 201 Created
    'status_201': ['count==50'],
    'status_errors': ['count==0'],
  },
};

const URL = 'http://localhost:8080/tip';
const STATIC_KEY = 'test-idempotency-key-001';

export default function () {
  const payload = JSON.stringify({
    name: 'Alice',
    amt: 50,
  });

  const params = {
    headers: {
      'Content-Type': 'application/json',
      'Idempotency-Key': STATIC_KEY,
    },
  };

  const res = http.post(URL, payload, params);

  const is201 = check(res, {
    'status is 201': (r) => r.status === 201,
  });

  if (is201) {
    createdCounter.add(1);
  } else {
    conflictOrErrorCounter.add(1);
  }
}

This is code to run the same kind of test that we discussed above with the help of the K6 lib.

After running the test, everything looks successful. But when you check the database, you find 50 entries for the same tip.

Think about the worst-case scenario. A user wants to send one 20-buck tip, but because of a race condition, they end up sending 50 tips, bucks 1,000 in total.

That's insane. He just wanted to tip $20, and now he's down $1,000.
Now just hire a good lawyer, Better Call Saul. 😭

---

So, Why This Happened?

This happens because when 50 requests reach the backend at almost the same time, the server treats each one as a new request.

It has no way to know that all 50 requests are actually for the same tip. So, it processes each request independently and creates a new database entry every time.

That's why one $20 tip can end up becoming 50 separate $20 tips in the database.

So, how to solve this!!!???

We can solve this by giving each payment a unique Idempotency-Key.

The client sends this same key with every retry of the same payment. When the backend receives a request, it checks the key if it has already processed or is processing that payment, it doesn't create another payment.

Instead, it returns the existing result or a response indicating that the request is already being processed.

var conn *pgxpool.Pool

type OrderRequest struct {
    Name   string `json:"name"`
    Amount int    `json:"amt"`
}

type OrderResponse struct {
    ID     int    `json:"id"`
    Ikey   string `json:"i_key"`
    Name   string `json:"name"`
    Amount int    `json:"amt"`
    Status string `json:"status"`
}

func handleTip(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    ctx := r.Context()
    ikey := r.Header.Get("Idempotency-Key")

    var req OrderRequest
    _ = json.NewDecoder(r.Body).Decode(&req)

    var order OrderResponse

    // 1. Check if the order already exists
    query := `SELECT id, i_key, name, amt, status FROM orders WHERE i_key = $1`
    err := conn.QueryRow(ctx, query, ikey).Scan(&order.ID, &order.Ikey, &order.Name, &order.Amount, &order.Status)

    if err == nil {
        // Found: return existing order
        w.WriteHeader(http.StatusOK)
        _ = json.NewEncoder(w).Encode(order)
        return
    }

    // 2. Not found: create new order
    insertQuery := `INSERT INTO orders (i_key, name, amt, status, created_at)
        VALUES ($1, $2, $3, $4, $5)
        RETURNING id, i_key, name, amt, status`

    _ = conn.QueryRow(ctx, insertQuery, ikey, req.Name, req.Amount, "Completed", time.Now()).
        Scan(&order.ID, &order.Ikey, &order.Name, &order.Amount, &order.Status)

    w.WriteHeader(http.StatusCreated)
    _ = json.NewEncoder(w).Encode(order)
}

func main() {
    ctx := context.Background()
    conn, _ = pgxpool.New(ctx, "postgres://postgres:strongPass%40123@localhost:5432/my_db")
    defer conn.Close()

    http.HandleFunc("/tip", handleTip)
    _ = http.ListenAndServe(":8080", nil)
}

---

Now try to test this new code again. I got this

ERROR: duplicate key value violates unique constraint "idx_idempotency_key"
DETAIL: Key (key)=(9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d) already exists.

Why does this happen?
Imagine two payment requests arrive at the same time with the same idempotency_key. Since this key doesn't exist in the database yet, both requests try to create a new record.

But we have added a UNIQUE constraint on i_key, so the database allows only one record with that key.

As a result, both requests try to insert the same key, but the database accepts only one and rejects the other. This prevents the same payment from being processed twice.

So, what should we do? Just add this to insert a query.

ON CONFLICT (i_key) DO NOTHING

INSERT into orders (i_key, name, amt, status,created_at)
    VALUES($1 ,$2, $3, $4, $5)
    ON CONFLICT (i_key) DO NOTHING
    RETURNING id, i_key, name, amt, status;

It will drop the query of the same conflicting query request during insert.

---

The next issue that can occur is if a second retry arrives while the first request was still talking to the payment gateway, returning an error or retrying immediately would corrupt the state or fail legitimate requests.

Resolution: Establishing request lifecycle states (PROCESSING, COMPLETED, FAILED). Concurrent incoming requests detecting a PROCESSING state are forced to wait, or receive a 409 Conflict / 425 Too Early, instead of executing a duplicate charge, which we already talked about in our previous blog.

Use Caching for faster retrieval of duplicate requests' status, so that it doesn't create lots of load on the backend service.

---

What if the payload gets tampered, A client sends a previously used Idempotency-Key with a completely different tip amount.

So, just hashing the incoming request body and storing the hash alongside the idempotency key to reject mismatched requests.

---

Learnings...

Now we have learned all about the Idempotency issue and issues that come along with it, and how to resolve them. Just use a unique identification key for the same kind of request to identify them, cache it for faster retrieval, and make sure the values that come with the payload are also always the same.