DEV Community: Shahid Yousuf

PostgreSQL Isolation Levels Explained with Python

Shahid Yousuf — Fri, 30 Oct 2026 04:24:23 +0000

Somewhere in almost every backend there's a check that looks harmless. Before an order goes through, the app makes sure the item is in stock. Before a seat is booked, it counts the seats left. Before an expense is approved, it adds up what the team has already spent and makes sure the new amount still fits. Read, check, write: three lines of code that never fail in development.

Now picture a corporate-card product. The design team has a $5,000 monthly budget, and late on a busy afternoon two managers each approve a $3,000 expense within the same second. Each request adds up the month's approvals and gets $0, because neither has saved its row yet. Each sees that $3,000 fits, inserts its expense and returns 200. The books now show $6,000 approved against a $5,000 budget.

No log shows an error, and every test still passes, because tests send one request at a time. Each transaction was correct on its own. The trouble is what PostgreSQL let them do together, and that's set by the transaction's isolation level: the rule for how much one transaction can see of others running at the same time.

PostgreSQL's default, READ COMMITTED, allows the overspend. The obvious upgrade, REPEATABLE READ, doesn't stop it either. It gives each transaction a frozen snapshot of the data, and two transactions that read the same total and insert different rows pass it cleanly.

There are two ways to make the rule hold. Run the check at SERIALIZABLE and retry whenever PostgreSQL stops one of the transactions, or move the rule onto a single row so that requests for the same team take turns.

In this tutorial, you'll learn what each isolation level does in plain words. Then you'll reproduce three concurrency bugs in Python with psycopg 3, fix each one, and load-test the fixes with 400 approvals: a lost update, a report that doesn't add up, and that overspent budget. Everything was tested with PostgreSQL 18.6, Python 3.14.8 and psycopg 3.3.6.

PostgreSQL Isolation Levels in Plain Words

A transaction is a group of statements that succeed or fail together. While yours runs, other transactions run beside it and change the same tables. The isolation level answers one question: while my transaction runs, how much of their work do I see?

Picture the team's numbers in a shared spreadsheet that several managers edit at once:

READ COMMITTED , PostgreSQL's default: every statement looks at the latest saved data. It's like refreshing the spreadsheet before every glance. You never see someone's half-typed edit, but two glances a second apart can show different numbers.
REPEATABLE READ : the transaction takes a snapshot at its first query and reads only that snapshot until it ends. It's like working from a printout. The numbers never shift under you, but you don't see anything saved after you printed. If you try to change a row that someone else changed after your printout, PostgreSQL stops you with an error.
SERIALIZABLE : the same printout, plus a referee. PostgreSQL keeps track of what each transaction read. If two transactions together produce a result that couldn't happen had they run one after the other, it cancels one of them, and your code runs it again.

The SQL standard also defines READ UNCOMMITTED, which would let you see other people's unsaved changes. PostgreSQL accepts the name but runs it as READ COMMITTED, so you can ignore it.

Each level shuts out more of the bugs you're about to meet:

Bug	What goes wrong	READ COMMITTED	REPEATABLE READ	SERIALIZABLE
Lost update	Two requests read the same total, each adds to it, and the second write erases the first	Happens	Stopped with an error	Stopped with an error
Read skew	One transaction reads two numbers from two different moments, so they don't agree	Happens	Stopped	Stopped
Write skew	Two requests check the same data, each decides it's safe, and each writes a different row	Happens	Happens	Stopped with an error

How each level treats the three bugs in this tutorial, per the PostgreSQL 18 isolation chapter. "With an error" means your code has to retry.

Stronger levels aren't free: they stop bugs by raising errors that your code must handle. So you'll watch each bug happen first, then pick the cheapest fix that stops it.

Prerequisites

To follow along, you need:

Docker with Docker Compose. PostgreSQL and Python both run in containers, so you don't install either locally.
Working knowledge of Python and of SQL transactions (BEGIN, COMMIT, ROLLBACK).
One terminal, open in an empty folder.

Step 1: Set Up the Expense Approval Lab

You need a database with a budget in it and a Python image to run the demos. Each script plays two or more managers, each on its own connection, and runs their statements in a fixed order, so you get the same result every time.

Create the project folder:

mkdir isolation-lab && cd isolation-lab

Create the Compose file:

docker-compose.yml

services:
  db:
    image: postgres:18.6
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: app
      POSTGRES_DB: spend
    volumes:
      - ./schema.sql:/docker-entrypoint-initdb.d/schema.sql:ro
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h localhost -U app -d spend"]
      interval: 1s
      timeout: 3s
      retries: 30

  app:
    build: .
    environment:
      DATABASE_URL: postgresql://app:app@db:5432/spend
    volumes:
      - .:/lab
    working_dir: /lab
    depends_on:
      db:
        condition: service_healthy

The db service runs schema.sql on its first start, through the official image's /docker-entrypoint-initdb.d folder, and only reports healthy once the schema has loaded. The app service mounts the project folder, so it runs the scripts you write.

The image installs one library:

Dockerfile

FROM python:3.14.8-slim
WORKDIR /lab
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
ENV PYTHONUNBUFFERED=1

requirements.txt

psycopg[binary]==3.3.6

The schema has two tables. teams holds each team's budget and a running total of what it has spent, and expenses holds one row per approved expense. Amounts are whole dollars, so the output stays easy to read:

schema.sql

CREATE TABLE teams (
    id     text PRIMARY KEY,
    name   text,
    budget integer NOT NULL,
    spent  integer NOT NULL DEFAULT 0
);

CREATE TABLE expenses (
    id          bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    team_id     text REFERENCES teams,
    amount      integer NOT NULL,
    approved_by text NOT NULL,
    created_at  timestamptz DEFAULT now()
);

CREATE INDEX expenses_team_id_idx ON expenses (team_id);

-- The design team has a $5,000 monthly budget.
INSERT INTO teams (id, name, budget) VALUES ('design', 'Design', 5000);

common.py holds the helpers every script shares:

common.py

import os
import time

import psycopg
from psycopg import IsolationLevel

DSN = os.environ.get("DATABASE_URL", "postgresql://app:app@localhost:5432/spend")

LEVELS = {
    "read-committed": IsolationLevel.READ_COMMITTED,
    "repeatable-read": IsolationLevel.REPEATABLE_READ,
    "serializable": IsolationLevel.SERIALIZABLE,
}


def connect(name: str, level: IsolationLevel = IsolationLevel.READ_COMMITTED) -> psycopg.Connection:
    """One manager's session. The first statement opens a transaction at
    `level`, and it stays open until commit() or rollback()."""
    conn = psycopg.connect(DSN, application_name=name)
    conn.isolation_level = level
    return conn


def reset(budget: int = 5_000) -> None:
    """Back to the start: design team, nothing spent."""
    with psycopg.connect(DSN, autocommit=True) as conn:
        conn.execute("TRUNCATE expenses RESTART IDENTITY")
        conn.execute("UPDATE teams SET budget = %s, spent = 0 WHERE id = 'design'", (budget,))


def wait_until_blocked(name: str, timeout: float = 5.0) -> None:
    """Wait until session `name` is stuck waiting for a lock."""
    with psycopg.connect(DSN, autocommit=True) as conn:
        deadline = time.monotonic() + timeout
        while time.monotonic() < deadline:
            row = conn.execute(
                "SELECT wait_event_type FROM pg_stat_activity WHERE application_name = %s",
                (name,),
            ).fetchone()
            if row and row[0] == "Lock":
                return
            time.sleep(0.01)
    raise TimeoutError(f"{name} never blocked")


def log(who: str, message: str) -> None:
    print(f"[{who}] {message}", flush=True)


def usd(amount: int) -> str:
    return f"${amount:,}"

connect() opens one session per manager. psycopg sets the level through conn.isolation_level, and only between transactions: once a statement has run, the transaction is open and its level is fixed until commit() or rollback(). wait_until_blocked() watches pg_stat_activity, PostgreSQL's live list of sessions, until a manager is stuck waiting for a lock. reset() puts the data back, so you can run the scripts in any order.

Start PostgreSQL, waiting until it's healthy, and build the image:

docker compose up -d --wait db
docker compose build app

Check that the schema loaded:

docker compose exec db psql -U app -d spend -c "SELECT id, budget, spent FROM teams"

The output will look similar to:

Output

   id   | budget | spent 
--------+--------+-------
 design |   5000 |     0
(1 row)

One team, a $5,000 budget, nothing spent. Compose prints container status lines before each script's output; add 2>/dev/null to a docker compose run command to hide them.

Step 2: Lose an Update, Then Let PostgreSQL Do the Math

Start with the simplest way to lose money. Say every approval adds its amount to teams.spent. Alice approves $1,200 and Bob approves $800 at the same moment. Each reads the total, adds their amount in Python and writes the result back. lost_update.py stages that read-modify-write:

lost_update.py

"""Lost update: two managers add to the team's running total at the same time.

Usage: python lost_update.py read-committed | repeatable-read | atomic
"""
import sys
import threading

from psycopg.errors import SerializationFailure

from common import LEVELS, connect, log, reset, usd, wait_until_blocked

ALICE_AMOUNT = 1_200
BOB_AMOUNT = 800


def read_modify_write(level_name: str) -> None:
    level = LEVELS[level_name]
    alice = connect("alice", level)
    bob = connect("bob", level)

    # Both managers read the total before either writes.
    a_spent = alice.execute("SELECT spent FROM teams WHERE id = 'design'").fetchone()[0]
    log("alice", f"read spent = {usd(a_spent)}")
    b_spent = bob.execute("SELECT spent FROM teams WHERE id = 'design'").fetchone()[0]
    log("bob", f"read spent = {usd(b_spent)}")

    # Each adds in Python and writes the result back.
    alice.execute("UPDATE teams SET spent = %s WHERE id = 'design'", (a_spent + ALICE_AMOUNT,))
    log("alice", f"set spent = {usd(a_spent + ALICE_AMOUNT)}")
    alice.commit()
    log("alice", "commit")

    try:
        bob.execute("UPDATE teams SET spent = %s WHERE id = 'design'", (b_spent + BOB_AMOUNT,))
        log("bob", f"set spent = {usd(b_spent + BOB_AMOUNT)}")
        bob.commit()
        log("bob", "commit")
    except SerializationFailure as e:
        log("bob", f"UPDATE failed: SQLSTATE {e.sqlstate}: {e.diag.message_primary}")
        bob.rollback()


def atomic() -> None:
    alice = connect("alice")
    bob = connect("bob")

    # Let the database do the adding, on whatever the row holds right now.
    new = alice.execute(
        "UPDATE teams SET spent = spent + %s WHERE id = 'design' RETURNING spent",
        (ALICE_AMOUNT,),
    ).fetchone()[0]
    log("alice", f"add {usd(ALICE_AMOUNT)} -> spent = {usd(new)} (not committed yet)")

    def bob_adds() -> None:
        new = bob.execute(
            "UPDATE teams SET spent = spent + %s WHERE id = 'design' RETURNING spent",
            (BOB_AMOUNT,),
        ).fetchone()[0]
        log("bob", f"add {usd(BOB_AMOUNT)} -> spent = {usd(new)}")
        bob.commit()
        log("bob", "commit")

    t = threading.Thread(target=bob_adds)
    t.start()
    wait_until_blocked("bob")
    log("bob", "waiting for alice's row lock")
    log("alice", "commit")
    alice.commit()
    t.join()


def main() -> None:
    variant = sys.argv[1] if len(sys.argv) > 1 else "read-committed"
    reset()
    print(f"--- lost update, {variant} ---")
    if variant == "atomic":
        atomic()
    else:
        read_modify_write(variant)

    with connect("check") as conn:
        spent = conn.execute("SELECT spent FROM teams WHERE id = 'design'").fetchone()[0]
    print(f"final spent = {usd(spent)} (expected {usd(ALICE_AMOUNT + BOB_AMOUNT)})")


if __name__ == "__main__":
    main()

Run it at READ COMMITTED first:

docker compose run --rm app python lost_update.py read-committed

The output will look similar to:

Output

--- lost update, read-committed ---
[alice] read spent = $0
[bob] read spent = $0
[alice] set spent = $1,200
[alice] commit
[bob] set spent = $800
[bob] commit
final spent = $800 (expected $2,000)

Both managers approved, and the total says $800. Bob added his $800 to the $0 he had read, which was already out of date by the time he wrote, so his write erased Alice's $1,200. That's a lost update. PostgreSQL saw nothing wrong, because SET spent = 800 is a perfectly valid statement.

Now run the same code at REPEATABLE READ:

docker compose run --rm app python lost_update.py repeatable-read

The output will look similar to:

Output

--- lost update, repeatable-read ---
[alice] read spent = $0
[bob] read spent = $0
[alice] set spent = $1,200
[alice] commit
[bob] UPDATE failed: SQLSTATE 40001: could not serialize access due to concurrent update
final spent = $1,200 (expected $2,000)

This time Bob's UPDATE fails with SQLSTATE 40001, a serialization failure. Bob is working from his printout, and the row changed after he printed it, so PostgreSQL refuses to let him overwrite it. The documentation says to retry the whole transaction from the start; on the retry Bob would read $1,200 and write $2,000. Step 5 builds that retry.

You don't need a stronger level for this bug, though. The atomic() variant lets the database do the adding with spent = spent + %s, at plain READ COMMITTED:

docker compose run --rm app python lost_update.py atomic

The output will look similar to:

Output

--- lost update, atomic ---
[alice] add $1,200 -> spent = $1,200 (not committed yet)
[bob] waiting for alice's row lock
[alice] commit
[bob] add $800 -> spent = $2,000
[bob] commit
final spent = $2,000 (expected $2,000)

Bob's UPDATE found the row locked by Alice's uncommitted change and waited. Once she committed, PostgreSQL ran his + 800 against the new value, $1,200, not the $0 from before, as the READ COMMITTED documentation describes.

So the bug was never the isolation level. It was doing the math in Python, where the database can't tell that the number is stale.

Step 3: Read a Consistent Report with REPEATABLE READ

Readers have their own trap. At month-end, finance runs a report that prints the team's running total next to the sum of its expense rows. Every approval writes both in one transaction, so they should always match. read_skew.py runs the report while Carol approves $250 between its two queries:

read_skew.py

"""Read skew: a month-end report reads two numbers that should always agree.

Usage: python read_skew.py read-committed | repeatable-read
"""
import sys

import psycopg

from common import LEVELS, connect, log, reset, usd


def approve(conn: psycopg.Connection, amount: int, approver: str) -> None:
    """Record an expense and add it to the team's total, in ONE transaction."""
    conn.execute(
        "INSERT INTO expenses (team_id, amount, approved_by) VALUES ('design', %s, %s)",
        (amount, approver),
    )
    conn.execute("UPDATE teams SET spent = spent + %s WHERE id = 'design'", (amount,))
    conn.commit()


def main() -> None:
    level_name = sys.argv[1] if len(sys.argv) > 1 else "read-committed"
    reset()
    print(f"--- read skew, report at {level_name} ---")

    carol = connect("carol")
    approve(carol, 1_000, "carol")
    log("carol", "approved $1,000 (earlier this month)")

    report = connect("report", LEVELS[level_name])
    spent = report.execute("SELECT spent FROM teams WHERE id = 'design'").fetchone()[0]
    log("report", f"teams.spent          = {usd(spent)}")

    approve(carol, 250, "carol")
    log("carol", "approved $250 and committed")

    total = report.execute(
        "SELECT coalesce(sum(amount), 0) FROM expenses WHERE team_id = 'design'"
    ).fetchone()[0]
    log("report", f"sum(expenses.amount) = {usd(total)}")
    report.commit()

    verdict = "agree" if spent == total else "DISAGREE"
    print(f"report numbers {verdict}: {usd(spent)} vs {usd(total)}")


if __name__ == "__main__":
    main()

Then run:

docker compose run --rm app python read_skew.py read-committed

The output will look similar to:

Output

--- read skew, report at read-committed ---
[carol] approved $1,000 (earlier this month)
[report] teams.spent          = $1,000
[carol] approved $250 and committed
[report] sum(expenses.amount) = $1,250
report numbers DISAGREE: $1,000 vs $1,250

The report says the books are $250 apart, and they aren't. At READ COMMITTED each statement refreshes its view, so the first query saw the total before Carol's commit and the second saw the sum after it. That's read skew : one transaction reading two different moments.

Run the report at REPEATABLE READ:

docker compose run --rm app python read_skew.py repeatable-read

The output will look similar to:

Output

--- read skew, report at repeatable-read ---
[carol] approved $1,000 (earlier this month)
[report] teams.spent          = $1,000
[carol] approved $250 and committed
[report] sum(expenses.amount) = $1,000
report numbers agree: $1,000 vs $1,000

Now both numbers come from one printout, and Carol's $250 will show up in the next run. A read-only report never conflicts with anyone, so REPEATABLE READ gives it a consistent answer with no retries to handle.

Step 4: Overspend the Budget with Write Skew

Now the bug from the introduction. This time approvals don't touch the running total. Each one sums the team's expenses, checks the budget, and inserts its own row. Alice and Bob each approve $3,000 against the $5,000 budget, and both check before either inserts:

write_skew.py

"""Write skew: two managers each check the budget, then each insert an expense.

Usage: python write_skew.py read-committed | repeatable-read | serializable
"""
import sys

import psycopg
from psycopg.errors import SerializationFailure

from common import LEVELS, connect, log, reset, usd

AMOUNT = 3_000  # each, against a $5,000 budget


def check(conn: psycopg.Connection, who: str) -> bool:
    budget = conn.execute("SELECT budget FROM teams WHERE id = 'design'").fetchone()[0]
    approved = conn.execute(
        "SELECT coalesce(sum(amount), 0) FROM expenses WHERE team_id = 'design'"
    ).fetchone()[0]
    ok = approved + AMOUNT <= budget
    log(who, f"check: approved {usd(approved)} + {usd(AMOUNT)} <= {usd(budget)}? {ok}")
    return ok


def insert(conn: psycopg.Connection, who: str) -> None:
    conn.execute(
        "INSERT INTO expenses (team_id, amount, approved_by) VALUES ('design', %s, %s)",
        (AMOUNT, who),
    )
    log(who, f"insert {usd(AMOUNT)}")


def main() -> None:
    level_name = sys.argv[1] if len(sys.argv) > 1 else "read-committed"
    reset()
    print(f"--- write skew, {level_name} ---")
    alice = connect("alice", LEVELS[level_name])
    bob = connect("bob", LEVELS[level_name])

    # Both checks happen before either insert.
    check(alice, "alice")
    check(bob, "bob")
    insert(alice, "alice")
    insert(bob, "bob")
    alice.commit()
    log("alice", "commit")
    try:
        bob.commit()
        log("bob", "commit")
    except SerializationFailure as e:
        log("bob", f"COMMIT failed: SQLSTATE {e.sqlstate}: {e.diag.message_primary}")
        bob.rollback()
        log("bob", "retry the whole transaction")
        if check(bob, "bob"):
            insert(bob, "bob")
        else:
            log("bob", "rejected: over budget")
        bob.commit()

    with connect("check") as conn:
        total, count = conn.execute(
            "SELECT coalesce(sum(amount), 0), count(*) FROM expenses WHERE team_id = 'design'"
        ).fetchone()
    print(f"approved {count} expense(s) totalling {usd(total)} on a $5,000 budget")


if __name__ == "__main__":
    main()

Run it at READ COMMITTED:

docker compose run --rm app python write_skew.py read-committed

The output will look similar to:

Output

--- write skew, read-committed ---
[alice] check: approved $0 + $3,000 <= $5,000? True
[bob] check: approved $0 + $3,000 <= $5,000? True
[alice] insert $3,000
[bob] insert $3,000
[alice] commit
[bob] commit
approved 2 expense(s) totalling $6,000 on a $5,000 budget

Both checks saw $0, both inserts succeeded, and the team is $1,000 over budget:

View this diagram in the original post

Write skew: both managers read the same total, and each writes a different row, so they never touch the same row.

You might expect REPEATABLE READ to stop this. It doesn't:

docker compose run --rm app python write_skew.py repeatable-read

The output will look similar to:

Output

--- write skew, repeatable-read ---
[alice] check: approved $0 + $3,000 <= $5,000? True
[bob] check: approved $0 + $3,000 <= $5,000? True
[alice] insert $3,000
[bob] insert $3,000
[alice] commit
[bob] commit
approved 2 expense(s) totalling $6,000 on a $5,000 budget

The same $6,000, and no error. This is write skew : two transactions read the same data, each makes a decision that's right for what it read, and each writes a different row. REPEATABLE READ only complains when two transactions change the same row, as Bob's UPDATE did in Step 2. Alice and Bob each inserted a brand-new row, so there was nothing to complain about.

Now try SERIALIZABLE:

docker compose run --rm app python write_skew.py serializable

The output will look similar to:

Output

--- write skew, serializable ---
[alice] check: approved $0 + $3,000 <= $5,000? True
[bob] check: approved $0 + $3,000 <= $5,000? True
[alice] insert $3,000
[bob] insert $3,000
[alice] commit
[bob] COMMIT failed: SQLSTATE 40001: could not serialize access due to read/write dependencies among transactions
[bob] retry the whole transaction
[bob] check: approved $3,000 + $3,000 <= $5,000? False
[bob] rejected: over budget
approved 1 expense(s) totalling $3,000 on a $5,000 budget

Every statement ran, then Bob's COMMIT failed with 40001, and his retry saw Alice's $3,000 and rejected his own expense. That's the referee at work. PostgreSQL noted what each manager read, and saw that Alice's total would have changed if Bob's insert had gone first, and Bob's would have changed if Alice's had gone first:

View this diagram in the original post

Each manager has to come before the other, so no one-at-a-time order explains the result. PostgreSQL cancels one transaction to break the loop.

The error doesn't always arrive at COMMIT. Depending on timing it can come from any statement, so a retry has to cover the whole transaction.

Want this checked in your own codebase?

Write skew passes every single-request test and every level below SERIALIZABLE, so the first sign is usually a reconciliation that doesn't add up. I offer a focused review of concurrency on money paths: which checks read data another request can change, which isolation level each transaction runs at, how 40001 and deadlocks are retried, and a load test like Step 7's, with written findings. Ask for a concurrency review or book a free 20-minute intro call.

Step 5: Retry Serialization Failures in Python

write_skew.py retried once, by hand; a real approval needs a loop. The documentation's rule is to retry the complete transaction, including the logic that decided what to write, which is why PostgreSQL can't retry for you.

approve.py holds the business rule and three safe ways to run it. This step uses the first, approve_serializable():

approve.py

"""Three safe ways to approve an expense against a team budget, and a demo of each.

Usage: python approve.py serializable | for-update | conditional
"""
import random
import sys
import threading
import time
from collections.abc import Callable

import psycopg
from psycopg import IsolationLevel
from psycopg.errors import DeadlockDetected, SerializationFailure

from common import connect, log, reset, usd, wait_until_blocked

MAX_ATTEMPTS = 10
RETRYABLE = (SerializationFailure, DeadlockDetected)  # SQLSTATE 40001, 40P01
VERBOSE = True


def say(who: str, message: str) -> None:
    if VERBOSE:
        log(who, message)


def nothing() -> None:
    pass


def check_and_insert(
    conn: psycopg.Connection, team: str, amount: int, who: str, pause: Callable[[], None]
) -> bool:
    """The business rule: approve only if the expense fits what is left of the budget."""
    budget = conn.execute("SELECT budget FROM teams WHERE id = %s", (team,)).fetchone()[0]
    approved = conn.execute(
        "SELECT coalesce(sum(amount), 0) FROM expenses WHERE team_id = %s", (team,)
    ).fetchone()[0]
    if approved + amount > budget:
        say(who, f"check: {usd(approved)} + {usd(amount)} > {usd(budget)}, rejected")
        return False
    say(who, f"check: {usd(approved)} + {usd(amount)} <= {usd(budget)}, ok")
    pause()
    conn.execute(
        "INSERT INTO expenses (team_id, amount, approved_by) VALUES (%s, %s, %s)",
        (team, amount, who),
    )
    say(who, f"insert {usd(amount)}")
    return True


def approve_naive(conn: psycopg.Connection, team: str, amount: int, who: str) -> bool:
    """Check, then insert, at whatever level `conn` uses. Not safe."""
    ok = check_and_insert(conn, team, amount, who, nothing)
    conn.commit()
    return ok


def approve_serializable(
    conn: psycopg.Connection,
    team: str,
    amount: int,
    who: str,
    pause: Callable[[], None] = nothing,
) -> tuple[bool, int]:
    """Fix 1: run at SERIALIZABLE and retry the whole transaction when PostgreSQL
    aborts it. Returns (approved, attempts)."""
    for attempt in range(1, MAX_ATTEMPTS + 1):
        try:
            ok = check_and_insert(conn, team, amount, who, pause)
            conn.commit()
            say(who, "commit")
            return ok, attempt
        except RETRYABLE as e:
            conn.rollback()
            say(who, f"attempt {attempt} failed: {e.diag.message_primary}")
            if attempt == MAX_ATTEMPTS:
                raise
            time.sleep(random.uniform(0, min(0.1, 0.002 * 2**attempt)))
    raise AssertionError("unreachable")


def approve_for_update(
    conn: psycopg.Connection,
    team: str,
    amount: int,
    who: str,
    pause: Callable[[], None] = nothing,
) -> bool:
    """Fix 2: READ COMMITTED, but lock the team's row first, so approvals
    for the same team take turns."""
    conn.execute("SELECT 1 FROM teams WHERE id = %s FOR UPDATE", (team,))
    say(who, f"locked team {team}")
    ok = check_and_insert(conn, team, amount, who, pause)
    say(who, "commit")
    conn.commit()
    return ok


def approve_conditional(
    conn: psycopg.Connection,
    team: str,
    amount: int,
    who: str,
    pause: Callable[[], None] = nothing,
) -> bool:
    """Fix 3: READ COMMITTED, check and reserve the money in one UPDATE."""
    row = conn.execute(
        "UPDATE teams SET spent = spent + %s"
        " WHERE id = %s AND spent + %s <= budget"
        " RETURNING spent",
        (amount, team, amount),
    ).fetchone()
    if row is None:
        say(who, f"UPDATE matched 0 rows: {usd(amount)} does not fit, rejected")
        conn.rollback()
        return False
    say(who, f"UPDATE reserved {usd(amount)}, spent now {usd(row[0])}")
    conn.execute(
        "INSERT INTO expenses (team_id, amount, approved_by) VALUES (%s, %s, %s)",
        (team, amount, who),
    )
    pause()
    say(who, "commit")
    conn.commit()
    return True


AMOUNT = 3_000  # each, against a $5,000 budget


def demo_serializable() -> None:
    alice = connect("alice", IsolationLevel.SERIALIZABLE)
    bob = connect("bob", IsolationLevel.SERIALIZABLE)
    alice_checked, bob_checked, alice_done = threading.Event(), threading.Event(), threading.Event()

    def alice_pause() -> None:
        alice_checked.set()
        bob_checked.wait()

    def bob_pause() -> None:
        bob_checked.set()
        alice_done.wait()

    def run_alice() -> None:
        approve_serializable(alice, "design", AMOUNT, "alice", alice_pause)
        alice_done.set()

    a = threading.Thread(target=run_alice)
    a.start()
    alice_checked.wait()
    ok, attempts = approve_serializable(bob, "design", AMOUNT, "bob", bob_pause)
    a.join()
    say("bob", f"approved={ok} after {attempts} attempt(s)")


def demo_locking(approve: Callable[..., bool], waiting: str) -> None:
    alice = connect("alice")
    bob = connect("bob")
    bob_thread = threading.Thread(target=approve, args=(bob, "design", AMOUNT, "bob"))

    def alice_pause() -> None:
        bob_thread.start()
        wait_until_blocked("bob")
        say("bob", waiting)

    approve(alice, "design", AMOUNT, "alice", alice_pause)
    bob_thread.join()


def main() -> None:
    variant = sys.argv[1] if len(sys.argv) > 1 else "serializable"
    reset()
    print(f"--- approve, {variant} ---")
    if variant == "serializable":
        demo_serializable()
    elif variant == "for-update":
        demo_locking(approve_for_update, "SELECT ... FOR UPDATE is waiting for alice")
    else:
        demo_locking(approve_conditional, "UPDATE is waiting for alice's row lock")

    with connect("check") as conn:
        total, count = conn.execute(
            "SELECT coalesce(sum(amount), 0), count(*) FROM expenses WHERE team_id = 'design'"
        ).fetchone()
    print(f"approved {count} expense(s) totalling {usd(total)} on a $5,000 budget")


if __name__ == "__main__":
    main()

check_and_insert() is the business rule, and the whole transaction. approve_serializable() calls it once per attempt. When PostgreSQL cancels an attempt, it rolls back, waits a short random time that grows with each attempt, and starts again from the top, re-reading everything. The random wait stops managers who failed together from retrying together. After 10 attempts it gives up and raises. conn.commit() sits inside the try, because that's often where the failure arrives.

The except catches two errors: SerializationFailure (SQLSTATE 40001) and DeadlockDetected (40P01), which PostgreSQL raises when two transactions wait on each other's locks. In psycopg 3.3.6 both subclass OperationalError, so except TransactionRollback catches neither. In a high-concurrency payments ledger I worked on, concurrent updates to the same accounts deadlocked, and the service retried the transaction after a short, growing wait.

Run the demo:

docker compose run --rm app python approve.py serializable

The output will look similar to:

Output

--- approve, serializable ---
[alice] check: $0 + $3,000 <= $5,000, ok
[bob] check: $0 + $3,000 <= $5,000, ok
[alice] insert $3,000
[alice] commit
[bob] attempt 1 failed: could not serialize access due to read/write dependencies among transactions
[bob] check: $3,000 + $3,000 > $5,000, rejected
[bob] commit
[bob] approved=False after 2 attempt(s)
approved 1 expense(s) totalling $3,000 on a $5,000 budget

Bob's second attempt saw $3,000 already approved and rejected cleanly: the caller gets "over budget", not a database error.

Keep side effects out of the retried function

Anything inside check_and_insert() runs once per attempt. If it sends an email, publishes to Kafka or calls the card provider, a retried approval does that twice. Write the side effect into an outbox table in the same transaction, as in the transactional outbox tutorial, and give any external call an idempotency key so a repeat is harmless.

SERIALIZABLE also only protects transactions that run at SERIALIZABLE. A batch job inserting expenses at READ COMMITTED slips past the referee.

Step 6: Guard the Budget with a Row Lock or One UPDATE

Retries aren't free. The other approach makes approvals for the same team take turns, at plain READ COMMITTED, so there's nothing to retry.

approve_for_update() starts with SELECT 1 FROM teams WHERE id = %s FOR UPDATE. The budget rule spans many expense rows, but they all belong to one team row, and locking that row makes every other approval for the team wait in line:

docker compose run --rm app python approve.py for-update

The output will look similar to:

Output

--- approve, for-update ---
[alice] locked team design
[alice] check: $0 + $3,000 <= $5,000, ok
[bob] SELECT ... FOR UPDATE is waiting for alice
[alice] insert $3,000
[alice] commit
[bob] locked team design
[bob] check: $3,000 + $3,000 > $5,000, rejected
[bob] commit
approved 1 expense(s) totalling $3,000 on a $5,000 budget

Bob waited until Alice committed. Because READ COMMITTED refreshes its view for every statement, his sum, which ran after the wait, included her $3,000. The catch is discipline: every code path that adds expenses must take the same lock first.

Keep the row lock at READ COMMITTED

At REPEATABLE READ the printout is taken by the transaction's first query, and here that's the FOR UPDATE itself, before the wait. Bob would still read $0 after Alice committed and approve $6,000 in total, with no error. Use this fix only at READ COMMITTED.

approve_conditional() goes one step further and puts the whole rule in one statement. It keeps the running total in teams.spent and makes the budget check part of the write:

UPDATE teams SET spent = spent + %s
WHERE id = %s AND spent + %s <= budget
RETURNING spent

Then run:

docker compose run --rm app python approve.py conditional

The output will look similar to:

Output

--- approve, conditional ---
[alice] UPDATE reserved $3,000, spent now $3,000
[bob] UPDATE is waiting for alice's row lock
[alice] commit
[bob] UPDATE matched 0 rows: $3,000 does not fit, rejected
approved 1 expense(s) totalling $3,000 on a $5,000 budget

This is Step 2's wait-and-recheck doing real work. Bob's UPDATE waited on Alice's row lock. When she committed, PostgreSQL checked his WHERE clause again against the new total, found that $3,000 + $3,000 is over budget, and updated nothing. It's the same guard as the WHERE status = 'authorized' capture in the outbox tutorial.

Step 7: Load-Test Five Approval Strategies

Two managers in a fixed order show how each fix works, not what happens under real traffic. load_test.py runs 16 workers, each with its own connection, through 400 approvals of $50 to $500 for one team. Together they ask for $108,488 against a $50,000 budget:

load_test.py

"""Many managers at once: 16 workers share 400 random approvals for one team.

Usage: python load_test.py naive-rc | naive-rr | serializable | for-update | conditional | all
"""
import queue
import random
import sys
import threading

from psycopg import IsolationLevel

import approve
from common import connect, reset, usd

WORKERS = 16
APPROVALS = 400
BUDGET = 50_000  # the 400 requests ask for roughly twice this

STRATEGIES = {
    "naive-rc": IsolationLevel.READ_COMMITTED,
    "naive-rr": IsolationLevel.REPEATABLE_READ,
    "serializable": IsolationLevel.SERIALIZABLE,
    "for-update": IsolationLevel.READ_COMMITTED,
    "conditional": IsolationLevel.READ_COMMITTED,
}


def approve_once(strategy: str, conn, amount: int, who: str) -> tuple[bool, int]:
    match strategy:
        case "naive-rc" | "naive-rr":
            return approve.approve_naive(conn, "design", amount, who), 1
        case "serializable":
            return approve.approve_serializable(conn, "design", amount, who)
        case "for-update":
            return approve.approve_for_update(conn, "design", amount, who), 1
        case "conditional":
            return approve.approve_conditional(conn, "design", amount, who), 1
    raise ValueError(strategy)


def run(strategy: str) -> str:
    reset(BUDGET)
    jobs: queue.Queue = queue.Queue()
    rng = random.Random(42)
    for _ in range(APPROVALS):
        jobs.put(rng.randint(50, 500))

    results: list[tuple[bool, int]] = []

    def worker(n: int) -> None:
        conn = connect(f"w{n:02}", STRATEGIES[strategy])
        while True:
            try:
                amount = jobs.get_nowait()
            except queue.Empty:
                break
            results.append(approve_once(strategy, conn, amount, f"w{n:02}"))
        conn.close()

    threads = [threading.Thread(target=worker, args=(n,)) for n in range(WORKERS)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    with connect("check") as conn:
        total = conn.execute(
            "SELECT coalesce(sum(amount), 0) FROM expenses WHERE team_id = 'design'"
        ).fetchone()[0]
    approved = sum(ok for ok, _ in results)
    retries = sum(attempts - 1 for _, attempts in results)
    most = max(attempts for _, attempts in results)
    return (
        f"{strategy:<13} approved={approved:<3} retries={retries:<4} most_attempts={most:<2}"
        f" total={usd(total)} over_budget={usd(max(0, total - BUDGET))}"
    )


def main() -> None:
    approve.VERBOSE = False
    choice = sys.argv[1] if len(sys.argv) > 1 else "all"
    strategies = list(STRATEGIES) if choice == "all" else [choice]
    print(f"--- {WORKERS} workers, {APPROVALS} approvals, {usd(BUDGET)} budget ---")
    for strategy in strategies:
        print(run(strategy), flush=True)


if __name__ == "__main__":
    main()

The two naive strategies run the unprotected check-then-insert, at READ COMMITTED and at REPEATABLE READ. Run all five:

docker compose run --rm app python load_test.py all

The race makes the numbers vary between runs. The output will look similar to:

Output

--- 16 workers, 400 approvals, $50,000 budget ---
naive-rc      approved=191 retries=0    most_attempts=1  total=$51,899 over_budget=$1,899
naive-rr      approved=194 retries=0    most_attempts=1  total=$52,719 over_budget=$2,719
serializable  approved=183 retries=123  most_attempts=7  total=$49,961 over_budget=$0
for-update    approved=185 retries=0    most_attempts=1  total=$49,952 over_budget=$0
conditional   approved=184 retries=0    most_attempts=1  total=$49,952 over_budget=$0

Read the over_budget column first. I ran the full set six times; across all six runs:

Strategy	Over budget by	Approved	Retries per 400	Most attempts for one approval
Check-then-insert, READ COMMITTED	$1,216 to $2,354	189 to 192	0	1
Check-then-insert, REPEATABLE READ	$2,340 to $3,049	192 to 196	0	1
SERIALIZABLE + retry	$0	183 to 185	101 to 123	7 to 8
READ COMMITTED + `FOR UPDATE`	$0	185 every run	0	1
READ COMMITTED + one conditional `UPDATE`	$0	183 to 187	0	1

400 approvals worth $108,488 against a $50,000 budget, 16 workers, PostgreSQL 18.6 in Docker. Six runs per strategy.

Three things stand out. REPEATABLE READ didn't help at all: the printout keeps each check consistent with itself, and does nothing about the manager next to you.

SERIALIZABLE was correct every time, at the cost of about one retry for every four approvals. Some approvals needed 7 or 8 attempts before they got an answer, so a cap of 3 would have failed real requests in every run. Measure attempts at your own busiest moment before you pick a cap.

Both locking fixes were correct with no retries at all. Their cost is that one team's approvals run one at a time.

When you're done, remove the lab and its data:

docker compose down -v

How to Choose an Isolation Level for Each Rule

Ask what shape each rule has, then pick the cheapest fix that covers it:

Shape of the rule	Example	Fix	What it costs
A single update to a number	`spent = spent + amount`	Nothing extra; let the database do the math at READ COMMITTED	Nothing
One row holds the whole rule	`spent + amount <= budget` on `teams`	One conditional `UPDATE ... WHERE ... RETURNING`, READ COMMITTED	A running total you keep in step with the expense rows
Many rows, one parent row	The sum of a team's expenses	`SELECT ... FOR UPDATE` on the parent first, READ COMMITTED	Every write path must take the lock; one team's approvals take turns
Rules spread across many tables or checks	Dozens of checks across a codebase	SERIALIZABLE + whole-transaction retry	Retries, side effects moved out of the transaction, every writer at SERIALIZABLE
Read-only reports	Month-end reconciliation	REPEATABLE READ	Nothing; it never sees later changes

I'd start from the top of the table. A conditional UPDATE is one readable statement, and when the money runs out it simply updates nothing. SERIALIZABLE earns its cost when the rules multiply, because then you don't have to find and lock every one of them by hand.

One more cost to know about: SERIALIZABLE can cancel transactions that never truly conflicted. When a table is small, PostgreSQL often reads all of it, and then an approval for one team can be cancelled because of another team's insert. That costs a retry, never wrong data, but under load it adds to the real conflicts.

Questions Interviewers Ask About Isolation Levels

These come up in backend interviews, and the lab answers each one:

Question	Short answer
What's the difference between a lost update and write skew?	A lost update is two read-modify-writes of the same row, where the second erases the first. Write skew is two transactions reading the same data and writing different rows. REPEATABLE READ catches the first and not the second.
What is snapshot isolation?	It's what PostgreSQL's REPEATABLE READ is: every read comes from one snapshot, and a transaction fails if it changes a row someone else changed after that snapshot. SERIALIZABLE adds tracking of what each transaction read, so it catches write skew too.
Is PostgreSQL's REPEATABLE READ the same as MySQL's?	No. In InnoDB, the consistent-read documentation says the snapshot "applies to SELECT statements within a transaction, not necessarily to DML statements", so Step 2's read-modify-write would overwrite silently there instead of failing with 40001.
What does SERIALIZABLE guarantee?	That the committed transactions have the same effect as running them one at a time in some order. Not the order they arrived in, and not freedom from errors you must retry.

Troubleshooting

Symptom	Cause	Fix
`relation "teams" does not exist`	An old volume skipped the schema script	`docker compose down -v`, then `docker compose up -d --wait db`
`except TransactionRollback` never catches a serialization failure	In psycopg 3.3.6, `SerializationFailure` (40001) and `DeadlockDetected` (40P01) subclass `OperationalError`	Catch `(SerializationFailure, DeadlockDetected)`, or check `e.sqlstate`
`ProgrammingError: can't change 'isolation_level' now`	A statement already opened the transaction	Commit or roll back first, or set the level right after connecting
40001 escapes the retry loop	The failure came from `COMMIT`, outside the `try`	Keep `conn.commit()` inside the `try`, as `approve_serializable()` does
Approvals give up under load	The retry cap is lower than your busiest moment needs	Measure attempts under load and raise the cap, or use a row lock for busy rules

Conclusion

You reproduced a lost update, a report that read two moments at once, and write skew that slipped past both READ COMMITTED and REPEATABLE READ. Then you stopped the overspend three ways, from up to $3,049 over budget per run down to $0.

The choice comes down to cost. The row lock and the conditional UPDATE need no retries but guard one rule each. SERIALIZABLE guards every rule in every transaction that runs at that level, for 101 to 123 retries per 400 requests here and side effects kept out of the transaction.

Next, look for check-then-act code in your own service: any transaction that reads a sum, a count or a MAX and then writes a different row. A sync service I worked on read MAX(seq) and inserted MAX + 1, so two batches could pick the same number. Point a load test like Step 7's at code like that, and read the PostgreSQL chapter on concurrency control. If a retried transaction calls anything outside the database, add an idempotency key and an outbox first.

If your team moves money through concurrent requests and wants the rules checked, I can review the transaction design and build the load tests that prove it. You can see how I approach distributed systems, or book a free 20-minute intro call.

Originally published at shahidyousuf.dev by Shahid Yousuf.

How to Implement the Transactional Outbox Pattern in Python

Shahid Yousuf — Wed, 21 Oct 2026 04:34:51 +0000

A payment service captures a card payment and records it in PostgreSQL. The rest of the business learns about it from an event on Kafka: a notifier emails the receipt, the ledger books the revenue. In the code, saving the row and publishing the event sit two lines apart, and in every test both happen.

In production, the process sometimes dies between those two lines: a deploy, an out-of-memory kill, a timeout. Commit first and die, and the payment is captured but no event leaves: no receipt, nothing in the ledger. Publish first and die before the commit, and every consumer hears about a capture the database rolled back. Neither shows up as an error, only later, when someone reconciles the ledger.

Earlier this year I fixed a version of this in a sync service between two SaaS systems. It created a record through one system's API, then wrote a mapping row in its own database; when the API call succeeded and the transaction rolled back, the retry created the record twice. Same shape: two writes to two places that can't commit together. That's the dual write problem.

You can't make a PostgreSQL commit and a Kafka produce atomic. The transactional outbox pattern stops trying: the service writes the event into an outbox table in the same transaction as the payment, and a separate relay publishes it afterwards. An event leaves if and only if the payment committed. The price is that the relay can publish an event more than once, so delivery becomes at-least-once, and the real work moves to duplicates and ordering. The cost is a table, a small relay process and a processed-events table in every consumer.

In this tutorial, you'll reproduce the dual write, then build an outbox in PostgreSQL, a polling relay and an idempotent Kafka consumer in Python, break each one, and finish with a chaos test of 1,000 payments. The API in front of this has its own retry problem, covered in How to Implement Idempotency Keys in Go with PostgreSQL. Everything was tested with Python 3.14.8, confluent-kafka 2.16.0, psycopg 3.3.6, PostgreSQL 18.6 and Apache Kafka 4.3.1.

Prerequisites

To follow along, you need:

Docker with Docker Compose. The steps were tested on Docker 29.2.1. Python, PostgreSQL and Kafka all run in containers, so you don't install them locally.
Working knowledge of Python and of SQL transactions.
No Kafka experience is needed. Three terms carry the tutorial: a topic is a named log of messages split into partitions; a consumer group is a set of consumers that share a topic's partitions; and an offset is a consumer group's position in a partition, which it commits to Kafka as it goes.
One terminal, open in an empty project folder.

Step 1: Set Up PostgreSQL and Kafka

First you need something to break: one PostgreSQL server with two databases, a single Kafka broker, and a Python image to run the scripts in. The payments database belongs to the payment service and notifier to the consumer that sends receipts; they share a server only to keep the lab small.

Create the project folder:

mkdir -p outbox-lab/sql && cd outbox-lab

Create the Compose file:

compose.yaml

services:
  postgres:
    image: postgres:18.6
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: app
      POSTGRES_DB: payments
    volumes:
      - ./sql:/docker-entrypoint-initdb.d:ro
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "app", "-d", "payments"]
      interval: 2s
      retries: 30

  kafka:
    image: apache/kafka:4.3.1
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
    healthcheck:
      test: ["CMD", "/opt/kafka/bin/kafka-topics.sh", "--bootstrap-server", "localhost:9092", "--list"]
      interval: 5s
      retries: 30

  app:
    build: .
    environment:
      PAYMENTS_DB: postgresql://app:app@postgres/payments
      NOTIFIER_DB: postgresql://app:app@postgres/notifier
      KAFKA: kafka:9092
    volumes:
      - .:/app
    depends_on:
      postgres:
        condition: service_healthy
      kafka:
        condition: service_healthy
    profiles: ["tools"]

The postgres service mounts sql/ into /docker-entrypoint-initdb.d, whose scripts the official image runs on its first start. Kafka runs in KRaft mode, acting as its own controller; Kafka 4.3 only supports KRaft, since ZooKeeper mode was removed. Topic auto-creation is off, so a mistyped topic name fails loudly. The app service sits in the tools profile, so docker compose up leaves it alone; you run each script with docker compose run.

The image only needs the two client libraries:

Dockerfile

FROM python:3.14.8-slim
RUN pip install --no-cache-dir confluent-kafka==2.16.0 "psycopg[binary]==3.3.6"
WORKDIR /app
ENV PYTHONUNBUFFERED=1

The two SQL files create the schemas. Step 3 explains the outbox table and Step 6 the notifier's tables:

sql/01-payments.sql

CREATE TABLE payments (
    id           text PRIMARY KEY,
    amount_cents integer NOT NULL CHECK (amount_cents > 0),
    currency     text NOT NULL,
    status       text NOT NULL,
    updated_at   timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE outbox (
    id           bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    event_id     uuid NOT NULL UNIQUE DEFAULT gen_random_uuid(),
    aggregate_id text NOT NULL,
    event_type   text NOT NULL,
    payload      jsonb NOT NULL,
    created_at   timestamptz NOT NULL DEFAULT now(),
    published_at timestamptz
);

CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

sql/02-notifier.sql

CREATE DATABASE notifier;
\connect notifier

CREATE TABLE processed_events (
    event_id     uuid PRIMARY KEY,
    processed_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE receipts_sent (
    id         bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    payment_id text NOT NULL,
    event_id   uuid NOT NULL,
    sent_at    timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE payment_status (
    payment_id text PRIMARY KEY,
    status     text NOT NULL
);

common.py holds the connection settings and crash_point, which ends the process with os._exit(1), skipping finally blocks the way a SIGKILL would:

common.py

import os
import random

PAYMENTS_DB = os.environ["PAYMENTS_DB"]
NOTIFIER_DB = os.environ["NOTIFIER_DB"]
KAFKA = os.environ["KAFKA"]
TOPIC = "payments"


def crash_point(name: str) -> None:
    """Kill the process on the spot when CRASH_AT names this point, like a deploy or an OOM kill.
    CRASH_RATE makes it happen on only that fraction of passes."""
    if os.environ.get("CRASH_AT") == name and random.random() < float(os.environ.get("CRASH_RATE", "1")):
        print(f"crash at {name}", flush=True)
        os._exit(1)

These switches only stage failures

CRASH_AT and CRASH_RATE (in common.py), HOLD_SECONDS and SINGLE_ACTIVE (in relay.py) and DEDUPE (in notifier.py) exist only to stage the failures in this tutorial. Their defaults are the safe behavior; leave them unset in anything real.

Start PostgreSQL and Kafka, and build the Python image:

docker compose up -d postgres kafka
docker compose build app

Run docker compose ps until both services show (healthy), then create the topic:

docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic payments --partitions 3 --replication-factor 1

Kafka answers Created topic payments. The replication factor is 1 because there's only one broker.

By the end of Step 6, the pieces fit together like this:

View this diagram in the original post

The payment and its event commit together. The relay publishes the event later, and the notifier records each event ID it handles so a repeat does nothing.

Step 2: Reproduce the Dual-Write Problem

Most services start by writing to the database and publishing to Kafka in the same function, so that's the version to break first. naive.py does it in both possible orders:

naive.py

import json
import sys

import psycopg
from confluent_kafka import Producer

from common import KAFKA, PAYMENTS_DB, TOPIC, crash_point

producer = Producer({"bootstrap.servers": KAFKA})


def publish(payment_id: str, event_type: str, data: dict) -> None:
    event = {"type": event_type, "payment_id": payment_id, "data": data}
    producer.produce(TOPIC, key=payment_id, value=json.dumps(event))
    producer.flush(10)


def capture_commit_first(conn: psycopg.Connection, payment_id: str) -> None:
    with conn.transaction():
        conn.execute(
            "UPDATE payments SET status = 'captured', updated_at = now() WHERE id = %s",
            (payment_id,),
        )
    crash_point("after_commit")
    publish(payment_id, "payment.captured", {})


def capture_publish_first(conn: psycopg.Connection, payment_id: str) -> None:
    with conn.transaction():
        conn.execute(
            "UPDATE payments SET status = 'captured', updated_at = now() WHERE id = %s",
            (payment_id,),
        )
        publish(payment_id, "payment.captured", {})
        crash_point("before_commit")


if __name__ == "__main__":
    order, payment_id = sys.argv[1], sys.argv[2]
    with psycopg.connect(PAYMENTS_DB) as conn:
        conn.execute(
            "INSERT INTO payments (id, amount_cents, currency, status) VALUES (%s, 2500, 'USD', 'authorized')",
            (payment_id,),
        )
        conn.commit()
        if order == "commit-first":
            capture_commit_first(conn, payment_id)
        else:
            capture_publish_first(conn, payment_id)
    print(f"{payment_id} captured")

capture_commit_first commits, then publishes. capture_publish_first publishes inside the open transaction, and producer.flush(10) waits for Kafka's acknowledgement, so the event is on the broker before the commit. Run each order with a crash staged between its two writes:

docker compose run --rm -e CRASH_AT=after_commit app python naive.py commit-first pay_001

Output

crash at after_commit

Then run:

docker compose run --rm -e CRASH_AT=before_commit app python naive.py publish-first pay_002

Output

crash at before_commit

Compare the database with the topic:

docker compose exec postgres psql -U app -d payments -c "SELECT id, status FROM payments ORDER BY id;"
docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic payments --from-beginning --formatter-property print.key=true --timeout-ms 5000

Output

   id    |   status
---------+------------
 pay_001 | captured
 pay_002 | authorized

pay_002 {"type": "payment.captured", "payment_id": "pay_002", "data": {}}
Processed a total of 1 messages

The console consumer also prints a hint about the new consumer protocol and a TimeoutException when --timeout-ms runs out; both are harmless. Look at the two payments side by side. pay_001 is captured in the database, but no event left, so no receipt goes out. pay_002 was never captured, yet consumers were told it was: that's a phantom event. A try/except can't fix either one, because a killed process runs no handler.

Why not Kafka transactions?

Kafka's transactions and idempotent producer make writes into Kafka exactly-once. They don't extend to your database. The Kafka design documentation says exactly-once delivery into other systems generally needs those systems' cooperation, with Kafka providing the primitives (it points to Kafka Connect), and that otherwise Kafka guarantees at-least-once delivery by default. A PostgreSQL commit can't join a Kafka transaction, so the gap between the two writes stays.

Before you move on, clear the payments tables and recreate the topic. The phantom event lacks the event_id that the consumer in Step 6 relies on:

docker compose exec postgres psql -U app -d payments -c "TRUNCATE payments, outbox RESTART IDENTITY;"
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic payments
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic payments --partitions 3 --replication-factor 1

Step 3: Write Events with the Transactional Outbox Pattern

Both failures come from asking two systems to agree on one commit. The fix is to ask only one. The service inserts the event into the outbox table from Step 1, in the same transaction as the payment change, so the two commit together or not at all.

Each column of outbox has one job:

Column	Job
`id`	Identity column; the relay publishes in this order
`event_id`	A UUID that consumers use to recognize a repeat
`aggregate_id`	The payment ID; it becomes the Kafka message key
`event_type`, `payload`	What happened, and its data
`created_at`	When it happened; also the basis for the lag alert in Step 9
`published_at`	`NULL` until the relay has confirmed delivery

The partial index outbox_unpublished covers only rows where published_at IS NULL, so the relay's query stays small however large the table grows, as long as autovacuum keeps up. Create the payment service:

payments.py

import sys

import psycopg
from psycopg.types.json import Jsonb

from common import PAYMENTS_DB


def add_event(conn: psycopg.Connection, payment_id: str, event_type: str, data: dict) -> None:
    conn.execute(
        "INSERT INTO outbox (aggregate_id, event_type, payload) VALUES (%s, %s, %s)",
        (payment_id, event_type, Jsonb(data)),
    )


def authorize(conn: psycopg.Connection, payment_id: str, amount_cents: int) -> None:
    with conn.transaction():
        conn.execute(
            "INSERT INTO payments (id, amount_cents, currency, status) VALUES (%s, %s, 'USD', 'authorized')",
            (payment_id, amount_cents),
        )
        add_event(conn, payment_id, "payment.authorized", {"amount_cents": amount_cents, "currency": "USD"})


def capture(conn: psycopg.Connection, payment_id: str) -> None:
    with conn.transaction():
        row = conn.execute(
            """UPDATE payments SET status = 'captured', updated_at = now()
               WHERE id = %s AND status = 'authorized'
               RETURNING amount_cents, currency""",
            (payment_id,),
        ).fetchone()
        if row is None:
            raise ValueError(f"{payment_id} is not authorized")
        add_event(conn, payment_id, "payment.captured", {"amount_cents": row[0], "currency": row[1]})


if __name__ == "__main__":
    command, payment_id = sys.argv[1], sys.argv[2]
    with psycopg.connect(PAYMENTS_DB, autocommit=True) as conn:
        if command == "authorize":
            authorize(conn, payment_id, int(sys.argv[3]))
        else:
            capture(conn, payment_id)
    print(f"{payment_id}: {command} recorded")

add_event runs inside the same conn.transaction() block as the payment write. In capture, the WHERE status = 'authorized' guard makes a second capture match no rows, and the ValueError rolls back the whole block, outbox insert included.

Authorize and capture one payment, and authorize a second:

docker compose run --rm app python payments.py authorize pay_001 2500
docker compose run --rm app python payments.py capture pay_001
docker compose run --rm app python payments.py authorize pay_002 4000

Each command prints a confirmation such as pay_001: authorize recorded. Check the outbox:

docker compose exec postgres psql -U app -d payments -c "SELECT id, aggregate_id, event_type, published_at FROM outbox ORDER BY id;"

You'll see three rows, ids 1 to 3, each with an empty published_at. Nothing has reached Kafka yet, and that's fine: the events are safe in PostgreSQL.

Step 4: Publish the Outbox with a Relay

Those events are durable, but no consumer can see them. Getting them to Kafka is the relay's job: a separate process that loops: lock a batch of unpublished rows, produce them, wait for Kafka's acknowledgement, mark them published, commit. Create it:

relay.py

import json
import os
import time

import psycopg
from confluent_kafka import KafkaException, Producer

from common import KAFKA, PAYMENTS_DB, TOPIC, crash_point

BATCH_SIZE = 100
RELAY_LOCK = 4200  # any bigint; every relay instance must use the same one

FETCH = """
    SELECT id, event_id, aggregate_id, event_type, payload, created_at
    FROM outbox
    WHERE published_at IS NULL
    ORDER BY id
    LIMIT %s
    FOR UPDATE SKIP LOCKED
"""


def publish_batch(conn: psycopg.Connection, producer: Producer) -> int:
    with conn.transaction():
        rows = conn.execute(FETCH, (BATCH_SIZE,)).fetchall()
        if not rows:
            return 0
        time.sleep(float(os.environ.get("HOLD_SECONDS", "0")))

        errors = []

        def on_delivery(err, msg):
            if err is not None:
                errors.append(err)

        for _id, event_id, aggregate_id, event_type, payload, created_at in rows:
            event = {
                "event_id": str(event_id),
                "type": event_type,
                "payment_id": aggregate_id,
                "occurred_at": created_at.isoformat(),
                "data": payload,
            }
            producer.produce(TOPIC, key=aggregate_id, value=json.dumps(event), on_delivery=on_delivery)

        if producer.flush(30) > 0 or errors:
            raise KafkaException(errors[0] if errors else "delivery timed out")
        crash_point("after_produce")

        conn.execute(
            "UPDATE outbox SET published_at = now() WHERE id = ANY(%s)",
            ([r[0] for r in rows],),
        )
    print(f"published {len(rows)}: {', '.join(f'{r[3]} {r[2]}' for r in rows[:5])}", flush=True)
    return len(rows)


def main() -> None:
    producer = Producer({"bootstrap.servers": KAFKA, "enable.idempotence": True})
    with psycopg.connect(PAYMENTS_DB, autocommit=True) as conn:
        if os.environ.get("SINGLE_ACTIVE", "1") == "1":
            while not conn.execute("SELECT pg_try_advisory_lock(%s)", (RELAY_LOCK,)).fetchone()[0]:
                print("standby: another relay is active", flush=True)
                time.sleep(2)
            print("active", flush=True)
        while True:
            if publish_batch(conn, producer) == 0:
                time.sleep(0.5)


if __name__ == "__main__":
    main()

FETCH takes the 100 oldest unpublished rows and locks them with FOR UPDATE SKIP LOCKED, so a second relay skips rows the first one holds instead of waiting, a mode the PostgreSQL documentation reserves for queue-like tables because it gives an inconsistent view of the data.

Each event is produced with the payment ID as its key. The producer's partitioner hashes the key to pick a partition, so every event for one payment lands in the same partition, and Kafka's introduction promises that "any consumer of a given topic-partition will always read that partition's events in exactly the same order as they were written."

In confluent-kafka-python, produce() only queues the message, and the delivery callback runs later, from poll() or flush(). flush(30) returns the number of messages still queued, so the relay treats anything above zero, or any error passed to on_delivery, as a failure. The exception rolls the transaction back, leaving the rows unpublished, and stops the relay, so run it under a supervisor (systemd, a container restart policy) that starts it again. Only after Kafka has acknowledged every message does the UPDATE set published_at.

The producer sets enable.idempotence, which defaults to false in librdkafka, the C library under the Python client. It implies acks=all, and the same reference warns that "retrying may cause reordering unless enable.idempotence is set to true."

There's a trade-off here. The relay holds its transaction and row locks open while it waits for Kafka, up to 30 seconds per batch on a struggling broker. I'd rather accept that than mark rows before Kafka confirms them, which turns a slow broker into lost events.

Why the Relay Doesn't Track the Last ID It Sent

You might reach for a simpler cursor instead: remember the last id you sent and poll WHERE id > last_seen. This script shows how that loses events:

cursor_gap.py

"""Why a relay must not poll with WHERE id > last_seen_id."""
import threading
import time

import psycopg
from psycopg.types.json import Jsonb

from common import PAYMENTS_DB

INSERT = "INSERT INTO outbox (aggregate_id, event_type, payload) VALUES (%s, 'payment.authorized', %s) RETURNING id"


def slow_transaction():
    with psycopg.connect(PAYMENTS_DB) as conn:
        event_id = conn.execute(INSERT, ("pay_slow", Jsonb({}))).fetchone()[0]
        print(f"slow transaction took id {event_id}, commits in 2s")
        time.sleep(2)
    print(f"slow transaction committed id {event_id}")


with psycopg.connect(PAYMENTS_DB, autocommit=True) as conn:
    last_seen = conn.execute("SELECT coalesce(max(id), 0) FROM outbox").fetchone()[0]
    slow = threading.Thread(target=slow_transaction)
    slow.start()
    time.sleep(0.5)
    fast_id = conn.execute(INSERT, ("pay_fast", Jsonb({}))).fetchone()[0]
    print(f"fast transaction committed id {fast_id}")

    seen = [r[0] for r in conn.execute("SELECT id FROM outbox WHERE id > %s ORDER BY id", (last_seen,))]
    print(f"cursor relay reads ids {seen}, moves last_seen to {max(seen)}")
    last_seen = max(seen)
    slow.join()
    later = [r[0] for r in conn.execute("SELECT id FROM outbox WHERE id > %s ORDER BY id", (last_seen,))]
    print(f"next poll reads ids {later}: id {fast_id - 1} is never published")
    conn.execute("DELETE FROM outbox WHERE aggregate_id IN ('pay_slow', 'pay_fast')")

Then run:

docker compose run --rm app python cursor_gap.py

Output

slow transaction took id 4, commits in 2s
fast transaction committed id 5
cursor relay reads ids [5], moves last_seen to 5
slow transaction committed id 4
next poll reads ids []: id 4 is never published

Look at the last line: id 4 committed without error, and the cursor will never see it. An identity value is assigned at insert, but the row becomes visible only at commit, and transactions don't commit in the order they started (sequences aren't gapless either). The cursor moves past the fast transaction's id 5, and id 4 is skipped forever. Polling published_at IS NULL has no cursor, so a late commit is picked up by the next poll. The script deletes its two rows when it finishes.

Hold off on starting the relay. In Step 5 you'll start it set up to fail.

Step 5: Crash the Relay and Watch Events Duplicate

Marking rows only after Kafka confirms them leaves one window open: Kafka has acknowledged a batch, but the UPDATE marking it published hasn't committed. A crash there looks like this:

View this diagram in the original post

Kafka has the batch, PostgreSQL doesn't know it. The restarted relay can only send it again.

Run the relay with a crash staged at exactly that point:

docker compose run --rm -e CRASH_AT=after_produce app python relay.py

Output

active
crash at after_produce

The relay produced all three events and died before it could record that. A GETPID warning may also appear on a fresh broker (see Troubleshooting). Confirm that nothing was marked:

docker compose exec postgres psql -U app -d payments -c "SELECT count(*) AS unpublished FROM outbox WHERE published_at IS NULL;"

The count is still 3: as far as PostgreSQL knows, nothing went out. Now start the relay for real, in the background:

docker compose run -d --name relay app python relay.py

Wait about 10 seconds, then read its log:

docker logs relay

Output

active
published 3: payment.authorized pay_001, payment.captured pay_001, payment.authorized pay_002

It sent all three events again. Count what reached the topic:

docker compose exec kafka /opt/kafka/bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic payments

Output

payments:0:2
payments:1:4
payments:2:0

Partition 0 ends at offset 2 and partition 1 at offset 4: six messages for three events. Every event is on the topic twice, with the same event_id. Leave the relay running.

You might expect enable.idempotence to catch this. It doesn't, and it can't: the broker deduplicates retries by a producer ID and sequence number, and the restarted relay is a new producer. KIP-98, the proposal that introduced the idempotent producer, states the limit:

Since each new instance of a producer is assigned a new, unique, PID, we can only guarantee idempotent production within a single producer session.

A hand-written relay that marks rows in PostgreSQL can't close this window, only make it rarer. Kafka Connect's exactly-once source mode attempts to close it for connectors such as Debezium, and consumers still see repeats when they crash before committing an offset. The pattern's own description warns that the message relay might publish a message more than once. So the outbox gives you at-least-once delivery , and the consumer has to cope.

Step 6: Make the Consumer Idempotent

If the relay can send an event twice, the consumer is where duplicates have to stop. The notifier turns payment.captured into a receipt and keeps a payment_status projection, its own copy of each payment's status. To survive repeats it uses an inbox table, processed_events, keyed by event_id:

notifier.py

import json
import os
import signal
import sys

import psycopg
from confluent_kafka import Consumer, KafkaException

from common import KAFKA, NOTIFIER_DB, TOPIC, crash_point

DEDUPE = os.environ.get("DEDUPE", "1") == "1"


def handle(conn: psycopg.Connection, event: dict) -> str:
    with conn.transaction():
        if DEDUPE:
            claimed = conn.execute(
                "INSERT INTO processed_events (event_id) VALUES (%s) ON CONFLICT DO NOTHING",
                (event["event_id"],),
            ).rowcount
            if not claimed:
                return "duplicate, skipped"
        status = event["type"].removeprefix("payment.")
        conn.execute(
            """INSERT INTO payment_status (payment_id, status) VALUES (%s, %s)
               ON CONFLICT (payment_id) DO UPDATE SET status = EXCLUDED.status""",
            (event["payment_id"], status),
        )
        if event["type"] == "payment.captured":
            conn.execute(
                "INSERT INTO receipts_sent (payment_id, event_id) VALUES (%s, %s)",
                (event["payment_id"], event["event_id"]),
            )
            return "receipt sent"
        return "status updated"


def main() -> None:
    signal.signal(signal.SIGTERM, lambda *_: sys.exit(0))
    consumer = Consumer({
        "bootstrap.servers": KAFKA,
        "group.id": "notifier",
        "enable.auto.commit": False,
        "auto.offset.reset": "earliest",
    })
    consumer.subscribe([TOPIC])
    with psycopg.connect(NOTIFIER_DB, autocommit=True) as conn:
        try:
            while True:
                msg = consumer.poll(1.0)
                if msg is None:
                    continue
                if msg.error():
                    raise KafkaException(msg.error())
                event = json.loads(msg.value())
                outcome = handle(conn, event)
                crash_point("before_offset_commit")
                consumer.commit(message=msg, asynchronous=False)
                print(f"p{msg.partition()}@{msg.offset()} {event['type']} {event['payment_id']}: {outcome}", flush=True)
        finally:
            consumer.close()


if __name__ == "__main__":
    main()

handle runs in one transaction. It claims the event with INSERT ... ON CONFLICT DO NOTHING, and rowcount says whether the event_id was new. A new event updates the projection and, for a capture, records the receipt; a repeat returns early. The claim and the effects commit together. This is the consumer-side twin of the key table in the idempotency keys tutorial. receipts_sent stands in for the email; a real email provider sits outside this transaction, so pass it the event_id as its own idempotency key.

The offset is committed only after the database commit, with enable.auto.commit off and commit(message=msg, asynchronous=False). A committed offset points at the next message to read, so a crash before the commit means the event comes back, and the inbox turns it away.

First, see what happens without the inbox. Start the notifier with DEDUPE=0:

docker compose run -d --name notifier -e DEDUPE=0 app python notifier.py

Wait about 10 seconds, then read its log:

docker logs notifier

Output

p1@0 payment.authorized pay_001: status updated
p1@1 payment.captured pay_001: receipt sent
p1@2 payment.authorized pay_001: status updated
p1@3 payment.captured pay_001: receipt sent
p0@0 payment.authorized pay_002: status updated
p0@1 payment.authorized pay_002: status updated

Each line is partition, offset, event and outcome. Notice receipt sent twice for pay_001. Count the receipts:

docker compose exec postgres psql -U app -d notifier -c "SELECT payment_id, count(*) AS receipts FROM receipts_sent GROUP BY payment_id;"

pay_001 has 2 receipts for one payment. Stop the notifier, empty its tables and rewind the group:

docker stop notifier && docker rm notifier
docker compose exec postgres psql -U app -d notifier -c "TRUNCATE processed_events, receipts_sent, payment_status;"
docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group notifier --reset-offsets --to-earliest --topic payments --execute

The reset prints a new offset of 0 for each partition. Now run it with the inbox on, the default:

docker compose run -d --name notifier app python notifier.py

Wait about 10 seconds, then read its log:

docker logs notifier

Output

p0@0 payment.authorized pay_002: status updated
p0@1 payment.authorized pay_002: duplicate, skipped
p1@0 payment.authorized pay_001: status updated
p1@1 payment.captured pay_001: receipt sent
p1@2 payment.authorized pay_001: duplicate, skipped
p1@3 payment.captured pay_001: duplicate, skipped

This time the second copy of each event says duplicate, skipped. Run the receipt count again: pay_001 now has 1. The projection shows pay_001 as captured and pay_002 as authorized, matching the payments database.

The relay isn't your only source of repeats. Kafka also redelivers whatever a consumer handled but didn't commit. Restart the notifier with a crash staged between the database commit and the offset commit, then make a new payment:

docker stop notifier && docker rm notifier
docker compose run -d --name notifier -e CRASH_AT=before_offset_commit app python notifier.py
docker compose run --rm app python payments.py authorize pay_004 1500
docker compose run --rm app python payments.py capture pay_004

Wait about 10 seconds, then read the log:

docker logs notifier

The log ends with crash at before_offset_commit: the notifier recorded payment.authorized pay_004 in its database and died before telling Kafka. Start a fresh notifier:

docker rm notifier
docker compose run -d --name notifier app python notifier.py

The new consumer waits until Kafka gives up on the dead one. The classic group protocol, still librdkafka's default, has a 45-second session timeout; it took about 20 seconds in my run. Then read the log:

docker logs notifier

Output

p0@2 payment.authorized pay_004: duplicate, skipped
p0@3 payment.captured pay_004: receipt sent

Kafka redelivered offset 2, the inbox recognized it, and the capture went through once. Same inbox, different failure. Leave this notifier running.

Step 7: Keep Per-Payment Order with One Active Relay

One relay is a single point of failure, so the natural next move is to run two, and SKIP LOCKED lets them share the table. To see what that does to order, replace the relay with one that holds each batch for 6 seconds before producing:

docker stop relay && docker rm relay
docker compose run -d --name relayA -e SINGLE_ACTIVE=0 -e HOLD_SECONDS=6 app python relay.py

Now authorize and capture a payment, and start a second relay, inside those 6 seconds. Chaining the commands keeps the timing reliable:

docker compose run --rm app python payments.py authorize pay_005 3000 && docker compose run --rm app python payments.py capture pay_005 && docker compose run -d --name relayB -e SINGLE_ACTIVE=0 app python relay.py

relayA locks payment.authorized pay_005 and sits on it. relayB skips that locked row, finds payment.captured pay_005, and publishes it at once. Wait about 10 seconds, then check the notifier:

docker logs notifier

The last two lines will look similar to these:

Output

p2@0 payment.captured pay_005: receipt sent
p2@1 payment.authorized pay_005: status updated

The capture arrived before the authorization. Compare the two databases:

docker compose exec postgres psql -U app -d payments -c "SELECT id, status FROM payments WHERE id = 'pay_005';"
docker compose exec postgres psql -U app -d notifier -c "SELECT * FROM payment_status WHERE payment_id = 'pay_005';"

The payments database says captured; the notifier says authorized. Kafka kept the order the relays gave it, and they gave it the wrong one. The inbox can't help: these are two different events.

The fix is to let one relay publish and keep the others on standby. relay.py does that by default with pg_try_advisory_lock, which returns false at once if another session holds the lock on RELAY_LOCK. It's a session-level lock, held until released or the session ends and not tied to any transaction, so it outlives each batch and vanishes with the holder's connection.

Session locks need a session

A connection pooler in transaction mode, such as PgBouncer with transaction pooling, hands each transaction to whichever server connection is free. From the documentation's description of session locks, I'd expect such a pooler to break this lock: it can be taken on one server connection and the next batch can run on another. Give the relay a direct or session-pooled connection.

Restart both relays in their default single-active mode, keeping relayA slow:

docker rm -f relayA relayB
docker compose run -d --name relayA -e HOLD_SECONDS=6 app python relay.py
docker compose run -d --name relayB app python relay.py

Wait about 10 seconds, then read relayB's log with docker logs relayB. It prints standby: another relay is active every 2 seconds. Authorize and capture another payment:

docker compose run --rm app python payments.py authorize pay_006 3000
docker compose run --rm app python payments.py capture pay_006

After relayA's two 6-second holds, docker logs notifier shows payment.authorized pay_006 and then payment.captured pay_006. Now kill the active relay and give the standby work:

docker kill relayA
docker compose run --rm app python payments.py authorize pay_007 2000
docker logs relayB

relayB prints active within one 2-second retry and publishes pay_007: PostgreSQL released the lock when the killed process's connection closed.

That's the forgiving failure. A relay cut off by a network partition is noticed only after TCP timeouts or keepalives, and may keep producing meanwhile. I didn't test that case; it's the lease problem from Distributed Locks in Redis with Go, where a lease can't prove the old owner stopped.

I'd run one active relay and shard relays by aggregate_id only when the lag query in Step 9 says one can't keep up.

Remove both relays before the next step:

docker rm -f relayA relayB

Step 8: Run a Chaos Test

So far each failure happened once, on cue. This step runs them all at random, under load. loadgen.py authorizes and captures payments from 8 threads:

loadgen.py

import sys
from concurrent.futures import ThreadPoolExecutor

import psycopg

from common import PAYMENTS_DB
from payments import authorize, capture


def one(n: int) -> None:
    payment_id = f"pay_{n:05d}"
    with psycopg.connect(PAYMENTS_DB, autocommit=True) as conn:
        authorize(conn, payment_id, 100 + n)
        capture(conn, payment_id)


count = int(sys.argv[1])
with ThreadPoolExecutor(max_workers=8) as pool:
    list(pool.map(one, range(count)))
print(f"{count} payments authorized and captured")

verify.py compares the payments database, the topic's end offsets and the notifier's database:

verify.py

import psycopg
from confluent_kafka import Consumer, TopicPartition

from common import KAFKA, NOTIFIER_DB, PAYMENTS_DB, TOPIC

with psycopg.connect(PAYMENTS_DB) as conn:
    captured, events, unpublished = conn.execute(
        """SELECT (SELECT count(*) FROM payments WHERE status = 'captured'),
                  (SELECT count(*) FROM outbox),
                  (SELECT count(*) FROM outbox WHERE published_at IS NULL)"""
    ).fetchone()

probe = Consumer({"bootstrap.servers": KAFKA, "group.id": "verify"})
on_topic = sum(probe.get_watermark_offsets(TopicPartition(TOPIC, p), timeout=10)[1] for p in range(3))
probe.close()

with psycopg.connect(NOTIFIER_DB) as conn:
    processed, receipts, paid_once, status_captured = conn.execute(
        """SELECT (SELECT count(*) FROM processed_events),
                  (SELECT count(*) FROM receipts_sent),
                  (SELECT count(DISTINCT payment_id) FROM receipts_sent),
                  (SELECT count(*) FROM payment_status WHERE status = 'captured')"""
    ).fetchone()

print(f"payments captured:          {captured}")
print(f"outbox events:              {events} ({unpublished} unpublished)")
print(f"messages on the topic:      {on_topic} ({on_topic - events} duplicates)")
print(f"events processed once:      {processed}")
print(f"receipts sent:              {receipts} for {paid_once} payments")
print(f"projection shows captured:  {status_captured}")

Reset everything so every count belongs to this run:

docker stop notifier && docker rm notifier
docker compose exec postgres psql -U app -d payments -c "TRUNCATE payments, outbox RESTART IDENTITY;"
docker compose exec postgres psql -U app -d notifier -c "TRUNCATE processed_events, receipts_sent, payment_status;"
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic payments
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic payments --partitions 3 --replication-factor 1
docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group notifier

Start two relays and a notifier, each in a restart loop. The relays crash after producing on 10% of batches, the notifier before its offset commit on 0.2% of events:

docker compose run -d --name relayA -e CRASH_AT=after_produce -e CRASH_RATE=0.1 app sh -c 'while true; do python relay.py; done'
docker compose run -d --name relayB -e CRASH_AT=after_produce -e CRASH_RATE=0.1 app sh -c 'while true; do python relay.py; done'
docker compose run -d --name notifier -e CRASH_AT=before_offset_commit -e CRASH_RATE=0.002 app sh -c 'while true; do python notifier.py; done'

Send 1,000 payments:

docker compose run --rm app python loadgen.py 1000

In my run, the load took about 2.5 seconds and the pipeline drained within about 40, most of it one crashed notifier's session timing out. Run verify.py until it reports 0 unpublished and 2,000 processed:

docker compose run --rm app python verify.py

Output

payments captured:          1000
outbox events:              2000 (0 unpublished)
messages on the topic:      2201 (201 duplicates)
events processed once:      2000
receipts sent:              1000 for 1000 payments
projection shows captured:  1000

Notice the numbers that matter: 201 duplicates on the topic, yet 2,000 events processed once and 1,000 receipts. In the logs, relayA crashed 3 times and published 37 batches; relayB published nothing, because relayA's restart loop won the lock back each time. The notifier crashed once and skipped 202 duplicates: 201 relay repeats plus 1 redelivery. The crashes are random, so counts vary from run to run; the zeros and the thousands shouldn't.

With a single active relay, you might expect the topic to be in order. replay_order.py reads the whole topic and counts payments whose authorized event appears after their captured one:

replay_order.py

import json

from confluent_kafka import Consumer

from common import KAFKA, TOPIC

consumer = Consumer({"bootstrap.servers": KAFKA, "group.id": "audit", "auto.offset.reset": "earliest", "enable.auto.commit": False})
consumer.subscribe([TOPIC])
captured, stale, idle = set(), set(), 0
while idle < 5:
    msg = consumer.poll(1.0)
    if msg is None:
        idle += 1
        continue
    idle = 0
    event = json.loads(msg.value())
    if event["type"] == "payment.captured":
        captured.add(event["payment_id"])
    elif event["payment_id"] in captured:
        stale.add(event["payment_id"])
consumer.close()
print(f"payments with an 'authorized' arriving after 'captured': {len(stale)}")

Then run:

docker compose run --rm app python replay_order.py

Output

payments with an 'authorized' arriving after 'captured': 98

This was the result that surprised me in the chaos run: 98 payments, with only one relay active at a time. When a relay crashes after producing, its rollback and the release of its lock happen together, and the next relay to take the lock selects the same oldest unpublished rows. The crashed batch is published again whole, straight after its first copy, so a payment whose authorize and capture were both in that batch reads authorized, captured, authorized, captured on the topic. Order on the topic is the order of first delivery only.

The inbox skipped every replayed event by its event_id. Without it, those payments would flip back to authorized until the replayed capture arrived, and each customer would get a second receipt.

Step 9: Monitor and Clean Up the Outbox

You still need to know when the relay falls behind, and the outbox grows forever unless you clean it. ops.sql handles both:

ops.sql

-- How far behind is the relay?
SELECT count(*) AS unpublished,
       coalesce(max(now() - created_at), interval '0') AS oldest_unpublished_age
FROM outbox
WHERE published_at IS NULL;

-- Delete published events older than 7 days, 10,000 rows per run.
DELETE FROM outbox
WHERE id IN (
    SELECT id FROM outbox
    WHERE published_at < now() - interval '7 days'
    ORDER BY id
    LIMIT 10000
);

Every outbox row is minutes old, so the DELETE would find nothing. To see it work, backdate the first 500 published events (test only):

docker compose exec postgres psql -U app -d payments -c "UPDATE outbox SET published_at = now() - interval '8 days' WHERE id <= 500;"

Output

UPDATE 500

Run ops.sql against the payments database. -T stops Compose from allocating a terminal, because input comes from a file:

docker compose exec -T postgres psql -U app -d payments < ops.sql

Output

 unpublished | oldest_unpublished_age 
-------------+------------------------
           0 | 00:00:00
(1 row)

DELETE 500

The relay is fully caught up, and the 500 backdated rows are gone. I'd alert on oldest_unpublished_age rather than the count: a thousand events a second old is a busy minute, while one event ten minutes old means the relay is down or stuck. Deleting in capped batches keeps each run short; schedule it, and size the 7 days to how long you might need to replay or inspect events.

To confirm the relay's query uses the partial index, run it with EXPLAIN:

docker compose exec postgres psql -U app -d payments -c "EXPLAIN SELECT id, event_id, aggregate_id, event_type, payload, created_at FROM outbox WHERE published_at IS NULL ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED;"

Output

                                             QUERY PLAN                                             
----------------------------------------------------------------------------------------------------
 Limit  (cost=77.92..78.31 rows=31 width=134)
   ->  LockRows  (cost=77.92..78.31 rows=31 width=134)
         ->  Sort  (cost=77.92..78.00 rows=31 width=134)
               Sort Key: id
               ->  Bitmap Heap Scan on outbox  (cost=4.44..77.15 rows=31 width=134)
                     Recheck Cond: (published_at IS NULL)
                     ->  Bitmap Index Scan on outbox_unpublished  (cost=0.00..4.43 rows=31 width=0)
(7 rows)

The plan reads outbox_unpublished. Depending on table statistics, PostgreSQL uses it through a Bitmap Index Scan, as here, or a plain Index Scan; your costs will differ. A sequential scan on outbox would mean the index isn't being used.

When you're done, remove the lab:

docker rm -f relayA relayB notifier
docker compose down -v

Polling Relay vs Debezium CDC

Polling isn't the only way to get rows out of the outbox. The main alternative is change data capture (CDC): Debezium's PostgreSQL connector reads the write-ahead log, PostgreSQL's record of every change, through the pgoutput plugin, and its Outbox Event Router turns outbox inserts into Kafka messages. The latest release at the time of writing is Debezium 3.7.0.Final.

	Polling relay (this tutorial)	Debezium with the Outbox Event Router
How it finds events	`SELECT ... FOR UPDATE SKIP LOCKED` every 0.5 s when idle	Reads the write-ahead log via `pgoutput`
What you run	One small Python process	Kafka Connect plus the Debezium connector
Load on the database	Polling queries and an `UPDATE` per batch	No polling queries
Outbox columns	Whatever you choose	Defaults: `id`, `aggregatetype`, `aggregateid`, `type`, `payload`
Message key	You set it (`aggregate_id` here)	`aggregateid` by default
Delivery	At least once	At least once by default; optional exactly-once into Kafka through Kafka Connect (`exactly.once.source.support=enabled` on the workers, `exactly.once.support=required` on the connector)

Neither removes the consumer's inbox. Debezium's exactly-once mode covers writes into Kafka, and its own documentation says it remains unclear whether that implementation is fully correct; a consumer that crashes before committing its offset is sent the message again either way. If you already run Kafka Connect, Debezium spares you the polling and the relay code. If you don't, I'd start with the polling relay: it's about 70 lines you can read in one sitting, and moving to CDC later only changes how events leave the table.

Troubleshooting

Symptom	Cause	Fix
`Failed to acquire idempotence PID from broker ... Coordinator load in progress: retrying` (a `GETPID` warning)	The idempotent producer asked a freshly started broker for a producer ID before its coordinator was ready	Nothing; librdkafka retries on its own
`Assignments can only be reset if the group 'notifier' is inactive, but the current state is Stable`	A consumer is still a member of the group, often a killed one whose session hasn't timed out	Stop consumers with `docker stop`, or wait out the session timeout, then reset
The group keeps a member after `docker stop`, and the next consumer waits	Python runs as the container's first process, which ignores `SIGTERM` unless it installs a handler, so `docker stop` sends `SIGKILL` after 10 seconds; `finally` never runs and `consumer.close()` never tells Kafka the consumer left	Install a `SIGTERM` handler that calls `sys.exit(0)`, as `notifier.py` does. With it, the group shows `Empty` with 0 members after a stop
`Option --property is deprecated ... Use --formatter-property instead`	Kafka 4.3 prints a deprecation warning for `--property` on the console consumer	Use `--formatter-property print.key=true`
A Java service and a Python service put the same key in different partitions	librdkafka's default partitioner is `consistent_random` (CRC32); the Java producer's default hashes keys with murmur2	Set `"partitioner": "murmur2_random"` in every librdkafka producer that shares a topic with Java producers; librdkafka documents it as functionally equivalent to Java's default
Events are never published, with no error	The relay tracks `id > last_seen`, and a slow transaction committed a lower id after the cursor passed it	Poll `published_at IS NULL`, as in Step 4

Conclusion

You built the transactional outbox pattern in Python: the outbox ties each event to its payment's transaction, and publishing is at-least-once whatever the producer's settings. The session advisory lock stops two relays from reordering events, and the inbox absorbs the reordering that replays still cause; together they turned 1,000 payments and 201 duplicates into exactly 1,000 receipts.

For the people deciding whether it's worth it: the outbox costs a table, a small relay process with a standby, an inbox table and an offset rule in every consumer, a cleanup job and one lag alert. In return, no event depends on a process surviving between two writes. Ask any team that publishes from inside a request handler what happens when the process dies between the commit and the publish; often the answer is reconciliation by hand.

Next, read the pattern on microservices.io and AWS Prescriptive Guidance, and cover the request side with idempotency keys in Go and PostgreSQL. Then run Step 8's chaos test against your own consumers.

If your team is moving events between services and wants the delivery guarantees checked, I can review the design and build the tests that check it. You can see how I approach distributed systems, or book a free 20-minute intro call.

Originally published at shahidyousuf.dev by Shahid Yousuf.

How to Migrate from GCP to AWS: GKE to EKS Step by Step

Shahid Yousuf — Fri, 16 Oct 2026 04:03:21 +0000

A GCP to AWS migration usually starts life as a mapping exercise. Someone lists the running services, finds the AWS counterpart of each, writes Terraform for the target, and the plan reads well: GKE becomes EKS, Cloud SQL becomes RDS, Cloud Storage becomes S3. For stateless apps that picture holds. You build the image, deploy it on the new cluster, point traffic at it, and retire the old copy.

The stateful apps break it. Their databases keep taking writes while you copy them, and data encrypted by a key service in the old cloud has to be re-encrypted before the new deployment can read it. Add changing hostnames and "zero downtime" becomes a promise nobody measured, with a bill on cutover night and another weeks later, when the old project still charges for resources nobody listed.

I recently led a move of five applications from several GKE clusters onto existing EKS clusters: a headless CMS, a React single-page app, a Django and Celery service, a FastAPI auth broker and a small Go service.

Two were stateful, with two PostgreSQL databases (under 1 GB and about 3 GB), about 62 GB of objects and about 300,000 encrypted fields held under dozens of per-customer keys. The plan gave users a maintenance window of about 3 to 4 hours. The data sync itself, measured on a full rehearsal, took 1.5 to 2 hours.

Moving stateful apps between clouds is a data project first. When data has to be re-keyed and hostnames change, the honest promise is a short, measured and reversible window rather than zero downtime. You get there by rehearsing the whole sync against the target, running the cutover as a repeat of that rehearsal, and planning the teardown of the old cloud as carefully as the cutover.

In this guide, you'll map GCP services to AWS, move each kind of data, rehearse the sync, re-encrypt HashiCorp Vault data between clusters, run the cutover with a rollback, redirect old URLs and decommission GCP.

Prerequisites

This is a planning guide with real commands rather than a lab you run start to finish. It assumes you're comfortable with:

Kubernetes Deployments and kubectl, on GKE and on Amazon EKS.
PostgreSQL dumps and restores with psql.
The gcloud CLI, the AWS CLI and rclone installed on the machine that drives the cutover.
An AWS target that already exists, with its network, IAM and cluster in place. The AWS security checklist for small teams covers the baseline to get right first.

Step 1: Map GCP Services to AWS for the Migration

Start with an inventory of every component and every contract between the apps: which app calls which, which one issues the tokens another accepts, which bucket each reads and writes. Those contracts are what break quietly after a move. My default for each component was "carry it forward and reuse the existing secret values", so the client only had to flag exceptions, and each exception became a conscious choice.

GCP	AWS	Outcome
GKE (Standard and Autopilot)	EKS	A namespace on existing clusters
Cloud SQL for PostgreSQL	RDS for PostgreSQL	One instance, a database per app
Cloud Storage	S3	Copied with rclone
Mounted service account JSON keys	IRSA	No static cloud keys in the cluster
Cloud DNS	Route 53	New names only; old zone kept
Istio gateway and Google Cloud load balancing	ALB and ACM	TLS at the ALB, then nginx
Container Registry	ECR	Used
Cloud Functions	Lambda	Not used: a container on EKS
Secret Manager	Secrets Manager	Not used: SOPS into Kubernetes Secrets
Cloud KMS (Vault auto-unseal)	AWS KMS	The existing Vault's unseal
Database Migration Service	AWS DMS	Not used, see Step 2

GCP to AWS service mapping as used in this migration.

Two rows need more than a cell. The source apps reached GCP with service account JSON key files mounted from Kubernetes Secrets. On AWS they use IRSA (IAM roles for service accounts), where a pod's Kubernetes service account maps to an IAM role and gets short-lived credentials, so no long-lived cloud key sits in the cluster any more. That was a small security win from the move itself. For a new setup, AWS now recommends EKS Pod Identity "whenever possible".

For buckets, Google's Storage Transfer Service only writes into Cloud Storage, so it can't help here; AWS DataSync can copy from GCS to S3 and runs agentless in Enhanced mode.

Several things were dropped rather than moved: Argo CD, Istio, the Elastic stack, a MinIO server (its data went to S3) and a staging environment. A migration is the cheapest moment to stop running something, because anything you carry across has to be rebuilt, secured and learned again on the target.

A dedicated EKS cluster was considered and rejected. A namespace on the shared cluster reused the platform's Terraform module pattern, its Vault, its Argo Workflows instance and its Prometheus, Grafana and Loki stack. The small Go service, rewritten from a Cloud Function, became an in-cluster Deployment rather than a Lambda function. The GCP to AWS migration case study covers the engagement as a whole.

Step 2: Choose How the Data Moves

Every stateful component needs one of two strategies. A dump and restore stops writes, copies everything and starts the app on the copy. Change data capture (CDC) streams changes from the source database's log into the target, so the final switch takes minutes. AWS DMS lists Google Cloud for PostgreSQL as a supported source.

CDC buys a shorter window at the price of a replication pipeline to build, watch and verify. With databases under 1 GB and about 3 GB, that price wasn't worth paying. A full dump captures everything, deletes included, and the larger one restored in about half an hour. Row-level deltas, copying only rows changed since an earlier load, were rejected as unreliable.

Data	Size	Strategy	Runs during the window?
CMS database	under 1 GB	Full dump and restore	Yes
Django service database	about 3 GB	Full dump and restore	Yes
Object storage	about 62 GB	Bulk copy early, changes only at cutover	Only the delta
Vault-encrypted fields	about 300,000	Decrypt and re-encrypt (Step 4)	Yes

Each kind of data got the cheapest strategy its size allowed.

Export from Cloud SQL

The Cloud SQL instance's own service account writes the export file, and it has no access to your bucket by default. Grant it roles/storage.objectAdmin on the bucket for the export, as the Cloud SQL export guide describes. Without the grant, the export fails with a 412 error.

SA=$(gcloud sql instances describe ${SOURCE_INSTANCE} \
  --format='value(serviceAccountEmailAddress)')
gcloud storage buckets add-iam-policy-binding gs://${BACKUP_BUCKET} \
  --member="serviceAccount:${SA}" --role=roles/storage.objectAdmin

gcloud sql export sql ${SOURCE_INSTANCE} \
  gs://${BACKUP_BUCKET}/${DB_NAME}.sql.gz \
  --database=${DB_NAME} --offload --async

--offload requests a serverless export, which takes the export work off the source instance. --async returns at once, so poll the operation until it finishes:

gcloud sql operations list --instance=${SOURCE_INSTANCE} --limit=1

When the export is done, the output will look similar to this (illustrative):

Output

NAME                                  TYPE    START                          END                            ERROR  STATUS
a1b2c3d4-0000-0000-0000-example00001  EXPORT  2026-01-01T10:00:00.000+00:00  2026-01-01T10:04:12.000+00:00  -      DONE

Then remove the grant and download the file under the same name:

gcloud storage buckets remove-iam-policy-binding gs://${BACKUP_BUCKET} \
  --member="serviceAccount:${SA}" --role=roles/storage.objectAdmin
gcloud storage cp gs://${BACKUP_BUCKET}/${DB_NAME}.sql.gz ./${DB_NAME}.sql.gz

The export guide also notes:

The export sql command does not contain triggers or stored procedures, but does contain views.

If your schema has either, add them with pg_dump or create them on the target first.

Restore into RDS

A production RDS instance sits in private subnets, so you reach it over a VPN into the VPC or from a pod inside the cluster. Here the larger dump went from a laptop over the VPN. Streaming a multi-gigabyte dump into a pod through kubectl exec was too slow.

export PGHOST=${RDS_HOST} PGDATABASE=${DB_NAME} PGUSER=${DB_USER} PGSSLMODE=require
psql -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'
gunzip -c ${DB_NAME}.sql.gz | psql -v ON_ERROR_STOP=0 -q 2> restore.log

ON_ERROR_STOP=0 keeps going past errors on purpose. A Cloud SQL dump refers to roles that don't exist on RDS, so ownership and GRANT statements fail while the data loads. The log will contain lines similar to these (illustrative):

restore.log

ERROR:  role "example_source_role" does not exist
ERROR:  role "example_source_role" does not exist

Filter those out and read whatever is left. Anything else is a real failure:

grep ERROR restore.log | grep -v 'does not exist'

Then search the restored data for anything that still points at the old cloud. The CMS needed its storage driver switched from local files to S3, its project URL changed, and hundreds of hard-coded old-host URLs rewritten in content fields:

UPDATE ${CONTENT_TABLE}
SET body = replace(body, 'old.example.com', 'new.example.com')
WHERE body LIKE '%old.example.com%';

Put these fix-ups in the runbook, because a fresh dump brings the old values back on every refresh.

Copy objects as a delta

rclone copy is additive: it copies new and changed objects and never deletes from the destination. After a bulk copy days earlier, the cutover run only moves the difference. The command assumes rclone remotes named gcs and s3:

rclone copy gcs:${SOURCE_BUCKET} s3:${TARGET_BUCKET} --size-only -v

--size-only compares object sizes and nothing else, which makes the delta fast. That was acceptable here because these buckets are append-only: objects are written once and never rewritten. A six-file sha256 spot check on the CMS assets matched too. If your objects get rewritten in place, use a checksum comparison and budget the time.

Step 3: Rehearse the Full Sync on the Target

The first estimate was 5 to 6 hours of downtime, most of it the re-encryption. Loading production data into the target once cut the plan to a 3 to 4 hour window (allowing up to 5), and a later rehearsal measured the sync itself at 1.5 to 2 hours.

Before the rehearsal, all five apps ran in dev on a restored copy of production data, and the stateful apps passed written checks. For the CMS, 583 of 583 files were present in S3, and reads and writes through IRSA were tested. One check, a row-count comparison between dev and the source, was skipped and recorded as skipped. A list that quietly drops a check reads later as a check that passed.

The rehearsal then ran the whole production sync, export to verification, without freezing GCP: a rehearsal doesn't need a final consistent state, and an offloaded export is a read-only copy. These are single rehearsal measurements:

Sync step	Rehearsal time
Headless CMS: database restore and files	about 15 min
Django service: database restore	about 30 min
Object storage delta	about 15 min
Vault re-key of encrypted fields	about 40 min
Whole sync	1.5 to 2 h

Single rehearsal measurements. The real window was planned around them.

The rehearsal also fixes the order. Dependencies decide it: here the CMS came before the single-page app that reads from it, and the stateless services, which need only a deploy and a health check, came at the end.

Production data in dev can email real users

Before you restore production data anywhere outside production, make sure outbound email, SMS and webhooks point at a sink or are turned off. Otherwise a password reset in dev can reach a live user.

Step 4: Re-Encrypt Vault Data Between Clusters

The Django service encrypted sensitive fields with Vault's transit secrets engine. Transit is encryption as a service: the app sends plaintext to Vault under a named key and stores the ciphertext that comes back, such as vault:v1:XjsPW..., in PostgreSQL. The prefix records the key version; the app knows which key name to use, here one per customer. The key material never leaves Vault, so copying the database copies ciphertext only the old Vault can read.

There are three ways out:

Option	What it needs	Outcome here
Snapshot the old Vault and restore it on the target	Compatible Vault versions on both sides	Rejected over version mismatch risk
Transit key `backup` and `restore`	`exportable` and `allow_plaintext_backup` enabled on every key. Both are one-way settings, so key material becomes exportable in plaintext for good	Not evaluated here; weigh it against your security policy
Decrypt under the old key, encrypt under a new key, update the row	Both Vaults reachable from one job, and a resumable job	Chosen

Three ways to move transit-encrypted data between Vault clusters.

Transit has its own rewrap endpoint, but it re-encrypts under the latest version of a key inside one Vault, so it doesn't help across clusters. The chosen job worked customer by customer. A simplified sketch of one batch:

rekey_batch.py (sketch)

rows = src_db.fetch("SELECT id, detail FROM findings WHERE customer_id = %s", cid)
plain = source_vault.post(f"transit/decrypt/{key}",
                          batch_input=[{"ciphertext": r.detail} for r in rows])
cipher = target_vault.post(f"transit/encrypt/{key}",
                           batch_input=[{"plaintext": p["plaintext"]} for p in plain])
for row, c in zip(rows, cipher):
    dst_db.execute("UPDATE findings SET detail = %s WHERE id = %s::uuid",
                   c["ciphertext"], row.id)

The names are illustrative, and the real job checked each result for errors. About 7,500 fields were left alone on purpose: history rows belonging to records that had already been deleted. If you skip data deliberately, give verification its own count of it, so "not re-keyed" and "missed" never look the same.

The first full run took about 31 minutes, with a median of 16 seconds per customer and a worst case of 574 seconds; the rehearsal budget used 40. Three problems surfaced on the way.

The first was a size limit. Vault rejected hundreds of fields larger than 1 MB, the largest over 20 MB. The TCP listener doc describes max_json_string_value_length, default 1048576:

Defines the maximum allowed length in bytes for a single string value within a JSON payload.

The JSON limits came with the HCSEC-2025-24 fix for CVE-2025-6203. Splitting fields wasn't possible because the app decrypts each field as one ciphertext, so the limits had to rise on the shared Vault:

vault.hcl (listener stanza)

listener "tcp" {
  # ...existing settings...
  max_request_size             = ${MAX_REQUEST_BYTES}
  max_json_string_value_length = ${MAX_FIELD_BYTES}
  max_json_array_element_count = ${MAX_BATCH_ITEMS}
}

Set the string limit above the base64 size of your largest field, and raise max_request_size (32 MB by default) to match, since a batch carries many fields. max_json_array_element_count caps the size of a batch_input array. On a highly available Vault, roll the change one pod at a time: standbys first, then the active node.

Measure your largest encrypted field before the rehearsal

Vault's default request limits reject large fields part way through a run. Query the largest ciphertext per table on the source first, and raise the listener limits on the target before the rehearsal, not during the cutover.

The second problem was the data path. Piping fields through kubectl exec and copying files with kubectl cp both stalled or truncated the large ones.

A colleague and I moved to staging: one phase decrypts a batch in the old cloud and writes it as an object to a bucket, the other reads it, encrypts it under the new key, updates the rows and deletes the object after the commit. That made the job resumable: on cutover day a transient 401 meant re-running it, and it skipped the customers already done.

Those objects are plaintext. Use a private bucket only the two job identities can read, delete each object on commit, add a short lifecycle rule as a backstop, and confirm the prefix is empty at the end.

The index the re-key job couldn't use

An early version guarded each update by comparing a UUID primary key cast to text with a text parameter. PostgreSQL uses a B-tree index when the indexed column itself is in the comparison; an expression such as id::text needs an index on that expression. So the cast turned every update into a scan. Comparing the column as its own type made the job about 23 times faster.

-- Slow: the cast hides the primary key index
UPDATE findings SET detail = $2 WHERE id::text = $1;

-- Fast: compare the indexed column as its own type
UPDATE findings SET detail = $2 WHERE id = $1::uuid;

The third problem came back two weeks later. To get moving, the app reached Vault with a hand-minted, long-lived token. It expired in dev and decryption broke.

The lasting fix was Vault's Kubernetes auth method: the pod logs in with its service account token and gets a Vault token that expires, so the app has to log in again before it does. Cache the client, re-authenticate before the lease runs out, and retry once on a 403. Without that, a short-lived token only moves the outage from weeks away to within the token's lifetime.

Step 5: Run the Cutover in a Fixed Order

By cutover day nothing should be new: every command has run in the rehearsal except the write freeze and the DNS switch.

View this diagram in the original post

The cutover sequence. Nothing on GCP is deleted until the decommission change.

Take the snapshot and lower the TTL (time to live, how long resolvers cache a DNS answer) well before the window, so caches have expired by the time you switch:

aws rds create-db-snapshot \
  --db-instance-identifier ${RDS_INSTANCE} \
  --db-snapshot-identifier pre-cutover-$(date +%F)

The freeze stops every writer. Here the background workers went to zero first; the web pods stayed up but idle because the re-key job decrypted from inside one of them, and they went to zero once it finished. If your re-key needs a pod in the old cluster, plan that exception explicitly. Record the replica counts before you touch anything, because the rollback needs them:

kubectl --context ${GKE_CONTEXT} -n ${NAMESPACE} get deployments \
  -o custom-columns=NAME:.metadata.name,REPLICAS:.spec.replicas > pre-freeze-replicas.txt
kubectl --context ${GKE_CONTEXT} -n ${NAMESPACE} scale deployment ${WORKER} --replicas=0

Then run Step 2's export, restore and rclone copy, then Step 4's re-key job. Scale the remaining GCP pods to zero, start the apps on EKS, and validate before any DNS changes:

☐ Every pod is Ready with 0 restarts
☐ Every health check returns 200
☐ Every table in each restored database matches the source row for row
☐ The re-key verification reports no fields left undone, apart from the ones skipped on purpose
☐ A cross-app login works: the auth broker issues a JWT and the service that checks it accepts it
☐ Sentry receives events and Loki receives logs from the new pods

GCP stays at zero replicas rather than deleted so you can roll back: scale each Deployment back to its count in pre-freeze-replicas.txt and revert any DNS you changed. If the database on RDS is the problem, re-run the restore or restore the snapshot. An RDS snapshot restore always creates a new DB instance with a new endpoint, and it gets the default parameter group and default VPC security group unless you name others.

An autoscaler won't bring a frozen deployment back

Scaling a Deployment to 0 deactivates its HorizontalPodAutoscaler until you set the replicas above zero again. Put the explicit kubectl scale commands, with the right counts, in both the start-up step and the rollback.

After a successful cutover, rotate any secrets you carried forward. Reusing values kept the move simple, but they have now lived in two clouds.

Want a second pair of eyes on your cutover plan?

Most of the risk in this step is decided weeks earlier: the strategy for each piece of data, whether the rehearsal ran on real data, and whether the rollback was tested. I offer a focused review of cloud migration plans that covers the service map, the data sync and its timings, encrypted data, DNS and redirects, rollback and the decommission list, with written findings your team can act on. Ask for a migration plan review or book a free 20-minute intro call.

Step 6: Redirect Old URLs to the New Hostnames

The apps moved to new hostnames, so the old ones had to answer with a 301 Moved Permanently for bookmarks, links in old emails and other apps' configs. The obvious home for the redirect was the internal ALB that already served the apps. A dry run in dev ruled it out: a VPN connector that served the old DNS zone re-advertised the internal ALB's addresses as /32 routes, which beat the broader subnet route but pointed at a connector that couldn't reach those addresses. Traffic to other internal apps on the same load balancer was blackholed.

The redirect went on the public ALB instead, which needs a certificate for each old hostname. Request it in ACM and add its validation record in the old zone, which still lives in Cloud DNS:

aws acm request-certificate --domain-name app.old.example.com --validation-method DNS
aws acm describe-certificate --certificate-arn ${OLD_HOST_CERT_ARN} \
  --query 'Certificate.DomainValidationOptions[0].ResourceRecord'
gcloud dns record-sets create ${VALIDATION_NAME} --zone=${OLD_ZONE} \
  --type=CNAME --ttl=300 --rrdatas=${VALIDATION_VALUE}

With the AWS Load Balancer Controller, a redirect is an Ingress whose backend is an action defined in an annotation:

old-url-redirect.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: old-url-redirect
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/group.name: ${PUBLIC_ALB_GROUP}
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
    alb.ingress.kubernetes.io/certificate-arn: ${OLD_HOST_CERT_ARN}
    alb.ingress.kubernetes.io/actions.redirect-app: >
      {"type": "redirect", "redirectConfig": {"host": "app.new.example.com",
       "port": "443", "protocol": "HTTPS", "statusCode": "HTTP_301"}}
spec:
  ingressClassName: alb
  rules:
    - host: app.old.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: redirect-app
                port:
                  name: use-annotation

listen-ports is merged across every Ingress in an IngressGroup, so a new port here opens a listener for the whole shared ALB. Keep this Ingress to ports the group already serves.

The redirect only sets the host, port and protocol. ALB redirect actions keep the original path and query unless you override them, so deep links land on the matching page. Test the rule before touching DNS by pinning the old hostname to one of the ALB's addresses:

ALB_IP=$(dig +short ${ALB_DNS} | head -1)
curl -sSI --resolve app.old.example.com:443:${ALB_IP} \
  'https://app.old.example.com/some/path?q=1' | grep -iE 'HTTP/|location'

The output will look similar to this (illustrative):

Output

HTTP/2 301
location: https://app.new.example.com/some/path?q=1

Then repoint the old records one at a time, with the lowest-risk hostname as the canary, and repeat the check against live DNS.

Two follow-ups tend to get missed. The ACM doc says it renews a DNS-validated certificate "as long as the certificate is in use and your CNAME record remains in place". The old zone in Cloud DNS is now a dependency for as long as the redirects live, and the decommission list has to say so.

The other is a hostname nobody listed. One dev hostname's DNS record was never repointed, and nobody noticed for more than a month, until the decommission sweep found it. Keep a list of every old hostname and check each one with curl after the switch.

Step 7: Decommission GCP as a Separate Change

Teardown deserves its own reviewed change, after the new setup has run for a while. The migrated apps' GCP resources were gone 5 to 8 weeks after the cutover, over three passes; the old DNS zone stays on purpose. Each pass found a resource type nobody had listed.

Leftover	Why it survived	How to find it
Reserved static IP addresses	Deleting the cluster did not release them	`gcloud compute addresses list`
A 1 TiB Filestore instance	It kept billing after its volume claim was deleted, with nothing stored on it and no file operations in the 30 days checked	`gcloud filestore instances list`
A network endpoint group (NEG)	It blocked deletion of the VPC	`gcloud compute network-endpoint-groups list`

Three leftovers from one teardown, each a resource type nobody had listed.

Kubernetes' documentation says dynamically provisioned volumes default to a Delete reclaim policy that removes the backing storage, and yet this instance survived. Why isn't known: a Retain policy and a cluster deleted before its volumes are both possible. Either way, check the backing resource, not the Kubernetes object.

A teardown checklist that would have caught all of them:

☐ Export the project's billing report by service and SKU, and list each billable resource type
☐ Before deleting a cluster, list its PersistentVolumes with their reclaim policies and the disks or file shares behind them
☐ After deleting it, list addresses, disks, Filestore instances, NEGs, forwarding rules and backend services directly
☐ List Cloud NAT gateways, Cloud Routers and VPC peerings; NAT IP addresses fall back to reserved and keep billing once their gateway is gone
☐ Mark what must stay, such as the DNS zone that validates the redirect certificates
☐ Watch the next billing cycle for anything that didn't drop to zero

Troubleshooting

Symptom	Cause	Fix
`gcloud sql export sql` fails with a 412 error	The instance's service account can't write to the bucket	Grant `roles/storage.objectAdmin` on the bucket for the export, then remove it (Step 2)
Vault returns "JSON string value exceeds allowed length"	A field is larger than `max_json_string_value_length`	Raise the listener limits on the target before the run (Step 4)
`kubectl exec` pipes or `kubectl cp` stall or truncate data	Large payloads through the Kubernetes API connection	Stage batches in a private bucket, or restore from a host on the VPN (Steps 2 and 4)
A Deployment stays at 0 after the cutover	Its HorizontalPodAutoscaler was deactivated at 0 replicas	Scale it up by hand with the recorded count (Step 5)
Vault returns 403 "permission denied" weeks after deploy	A long-lived token expired	Use the Kubernetes auth method and re-authenticate before expiry (Step 4)
VPC deletion is refused	A network endpoint group still references the network	Delete the NEG with `gcloud compute network-endpoint-groups delete`, then retry (Step 7)

Conclusion

The cost of this plan is a maintenance window users hear about in advance, a full rehearsal on production data, and a separate decommission change weeks later. In return, a 5 to 6 hour first estimate became a 3 to 4 hour planned window around a sync measured at 1.5 to 2 hours.

Next, take your own inventory and fill in the mapping table from Step 1, with a column for "dropped". Check whether your transit keys could be made exportable with plaintext backup, and whether your security policy allows it.

Measure your largest encrypted field, and time a full sync into the target before you promise anyone a window. If the move also restructures AWS accounts, read how to move an AWS account to another organization, and the guide to SCPs, RCPs and declarative policies for guarding the new accounts.

If your team is planning a move between clouds, or onto EKS, I can help plan and run it, from the service map to the decommission. You can book a free 20-minute intro call or tell me about your migration.

Originally published at shahidyousuf.dev by Shahid Yousuf.

How to Implement Idempotency Keys in Go with PostgreSQL

Shahid Yousuf — Tue, 13 Oct 2026 03:15:23 +0000

Payment requests travel over networks that drop things. A phone on a train loses signal for a second, or a mobile SDK stops waiting. The request can reach your server, your server can charge the card, and the answer can vanish on the way back. The client sees an error and does the sensible thing: it sends the request again.

Retrying is the right reaction: the client can't tell a request that never arrived from one whose answer got lost. The trouble is on the server, which without an idempotency key can't tell a retry from a new payment. A payment is a POST, and a server that treats every POST as new work charges the card once per attempt.

So the customer taps Pay once, sees one receipt, and finds two charges of 25.00 on their statement. Your logs show two successful payments a second apart, each correct on its own, and the fix is a refund, an apology and possibly a chargeback, a dispute raised through the customer's bank.

The general fix is an idempotency key : a unique value the client generates once per payment and sends with every attempt. The server does the work the first time it sees a key, records the result, and answers every repeat with that same result.

The key needs a home that every copy of your server can reach and that can settle a race between two requests carrying it. If your payments already live in PostgreSQL, a table with a primary key on the idempotency key does both.

In this tutorial, you'll implement idempotency keys in Go with PostgreSQL for a small payments API, break each version on purpose, and load-test the result against a fake card provider that keeps its own ledger. It's a sequel to Distributed Locks in Redis with Go. Everything was tested with Go 1.27.2, pgx v5.11.0 and PostgreSQL 18.6.

Prerequisites

To follow along, you need:

Go 1.25 or later, which pgx v5.11.0, the PostgreSQL driver, requires. The code also uses crypto/rand.Text (Go 1.24) and method patterns such as "POST /payments" in http.HandleFunc, which arrived in Go 1.22.
Docker, to run PostgreSQL locally.
Working knowledge of net/http, context.Context and SQL. No payments experience is needed.
Ports 5432, 8080 and 8090 free, and three terminal windows open in the project directory.

Step 1: Set Up a Fake Card Provider and a Checkout Client

This step builds a fake card provider on port 8090 and a checkout client that retries like a phone. The provider and the API each keep a ledger, because the bug lives in the gap between what the shop believes it charged and what the card was actually charged.

Start PostgreSQL and wait until it reports accepting connections:

docker run -d --name pg -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:18
docker exec pg pg_isready -U postgres

Output

/var/run/postgresql:5432 - accepting connections

The fallback password is for this container only

The API reads DATABASE_URL and the provider reads PROVIDER_DATABASE_URL. When they're unset, both fall back to this container's postgres:postgres URL, so every command here runs as shown. In a real deployment, set the variables from a secret store.

Create one database per ledger:

docker exec pg psql -U postgres -c "CREATE DATABASE shop" -c "CREATE DATABASE provider"

Output

CREATE DATABASE
CREATE DATABASE

Create the Go module and add pgx:

mkdir payments && cd payments
go mod init example.com/payments
go get github.com/jackc/pgx/v5@v5.11.0

Output

go: creating new go.mod: module example.com/payments
go: to add module requirements and sums:
    go mod tidy
go: added github.com/jackc/pgpassfile v1.0.0
go: added github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761
go: added github.com/jackc/pgx/v5 v5.11.0
go: added golang.org/x/text v0.29.0

In the shop's schema, charges is the ledger and idempotency_keys holds one row per key. Its primary key is (customer_id, key), so two customers who send the same key never see each other's payments.

schema.sql

-- The shop's own ledger: one row per payment it believes it took.
CREATE TABLE charges (
    id          bigserial   PRIMARY KEY,
    customer_id text        NOT NULL,
    amount      bigint      NOT NULL CHECK (amount > 0),
    provider_id text        NOT NULL,
    created_at  timestamptz NOT NULL DEFAULT now()
);

-- One row per idempotency key: who claimed it, for which request, and what we answered.
CREATE TABLE idempotency_keys (
    customer_id   text        NOT NULL,
    key           text        NOT NULL,
    request_hash  bytea       NOT NULL,
    status        text        NOT NULL DEFAULT 'started',
    response_code int,
    response_body bytea,
    locked_at     timestamptz NOT NULL DEFAULT now(),
    created_at    timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (customer_id, key)
);

The provider's table is its record of what it actually charged. Its idempotency_key column is UNIQUE, and because a UNIQUE column treats NULLs as distinct by default, a charge sent without a key never conflicts with anything:

provider.sql

-- The fake card processor's own records: what it actually charged.
CREATE TABLE card_charges (
    id              text        PRIMARY KEY DEFAULT 'ch_' || substr(md5(random()::text), 1, 10),
    idempotency_key text        UNIQUE,
    card            text        NOT NULL,
    amount          bigint      NOT NULL,
    created_at      timestamptz NOT NULL DEFAULT now()
);

Load both schemas:

docker exec -i pg psql -U postgres -d shop < schema.sql
docker exec -i pg psql -U postgres -d provider < provider.sql

Output

CREATE TABLE
CREATE TABLE
CREATE TABLE

The fake provider takes one second per charge and declines the test card tok_declined with 402 Payment Required. With an Idempotency-Key, it behaves like Stripe: it takes a PostgreSQL advisory lock (a lock PostgreSQL holds on a number, here a hash of the key, until the transaction ends), answers 409 if another request with the same key is still running, and returns the original charge if the key was already used.

cmd/provider/main.go

// Command provider is a stand-in for a card processor such as Stripe or Adyen.
// It charges slowly, declines one test card, and honors an Idempotency-Key.
package main

import (
    "context"
    "encoding/json"
    "log"
    "net/http"
    "os"
    "time"

    "github.com/jackc/pgx/v5/pgxpool"
)

type chargeRequest struct {
    Amount int64  `json:"amount"`
    Card   string `json:"card"`
}

func main() {
    url := os.Getenv("PROVIDER_DATABASE_URL")
    if url == "" {
        url = "postgres://postgres:postgres@localhost:5432/provider?pool_max_conns=50" // the local Docker container; one connection per charge in flight
    }
    db, err := pgxpool.New(context.Background(), url)
    if err != nil {
        log.Fatal(err)
    }

    http.HandleFunc("POST /charges", func(w http.ResponseWriter, r *http.Request) {
        var req chargeRequest
        if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
            http.Error(w, "bad request", http.StatusBadRequest)
            return
        }
        ctx := r.Context()
        tx, err := db.Begin(ctx)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }
        defer tx.Rollback(ctx)

        // Like Stripe, answer 409 while another request with the same key is running,
        // and return the original charge once it has finished.
        var key *string
        if k := r.Header.Get("Idempotency-Key"); k != "" {
            key = &k
            var locked bool
            tx.QueryRow(ctx, `SELECT pg_try_advisory_xact_lock(hashtext($1))`, k).Scan(&locked)
            if !locked {
                writeJSON(w, http.StatusConflict, map[string]string{"error": "request_in_progress"})
                return
            }
            var id string
            if tx.QueryRow(ctx, `SELECT id FROM card_charges WHERE idempotency_key = $1`, k).Scan(&id) == nil {
                writeJSON(w, http.StatusCreated, map[string]string{"id": id})
                return
            }
        }

        time.Sleep(time.Second) // the card network is slow
        if req.Card == "tok_declined" {
            writeJSON(w, http.StatusPaymentRequired, map[string]string{"error": "card_declined"})
            return
        }

        // Without a key (NULL), nothing conflicts: every call is a new charge.
        var id string
        err = tx.QueryRow(ctx, `
            INSERT INTO card_charges (idempotency_key, card, amount)
            VALUES ($1, $2, $3)
            RETURNING id`, key, req.Card, req.Amount).Scan(&id)
        if err == nil {
            err = tx.Commit(ctx)
        }
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }
        writeJSON(w, http.StatusCreated, map[string]string{"id": id})
    })

    log.Println("provider listening on :8090")
    log.Fatal(http.ListenAndServe(":8090", nil))
}

func writeJSON(w http.ResponseWriter, code int, v any) {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(code)
    json.NewEncoder(w).Encode(v)
}

payments/server.go holds the API's shared pieces. A payment carries an amount in cents and a card token, a string the provider's card form issues so your server never sees a card number. customerID reads a header as a stand-in for authentication; a real API answers 401 when there's no authenticated customer and caps the body with http.MaxBytesReader.

payments/server.go

// Package payments is a small payments API that charges a card through a provider
// and records each payment in PostgreSQL.
package payments

import (
    "bytes"
    "context"
    "encoding/json"
    "errors"
    "fmt"
    "net/http"
    "time"

    "github.com/jackc/pgx/v5/pgxpool"
)

// PaymentRequest is what the checkout page sends. Amount is in cents; Card is a
// token from the provider's card form, never a card number.
type PaymentRequest struct {
    Amount int64  `json:"amount"`
    Card   string `json:"card"`
}

// Payment is what the API returns for a successful charge.
type Payment struct {
    ID         int64  `json:"id"`
    Amount     int64  `json:"amount"`
    ProviderID string `json:"provider_charge_id"`
}

// Server holds the API's dependencies.
type Server struct {
    DB       *pgxpool.Pool
    Provider *Provider
}

// ErrDeclined means the provider refused the card. That is a final answer, not a fault.
var ErrDeclined = errors.New("card declined")

// ErrInProgress means the provider is still working on a charge with the same key.
var ErrInProgress = errors.New("charge in progress at the provider")

// Provider is a client for the card processor.
type Provider struct {
    URL  string
    HTTP *http.Client
}

// Charge asks the provider to take amount from card. A non-empty key is sent as
// the provider's Idempotency-Key, so repeating the call cannot charge twice.
func (p *Provider) Charge(ctx context.Context, amount int64, card, key string) (string, error) {
    body, _ := json.Marshal(PaymentRequest{Amount: amount, Card: card})
    req, err := http.NewRequestWithContext(ctx, http.MethodPost, p.URL+"/charges", bytes.NewReader(body))
    if err != nil {
        return "", err
    }
    req.Header.Set("Content-Type", "application/json")
    if key != "" {
        req.Header.Set("Idempotency-Key", key)
    }
    resp, err := p.HTTP.Do(req)
    if err != nil {
        return "", err
    }
    defer resp.Body.Close()

    switch resp.StatusCode {
    case http.StatusCreated:
        var out struct{ ID string }
        err := json.NewDecoder(resp.Body).Decode(&out)
        return out.ID, err
    case http.StatusPaymentRequired:
        return "", ErrDeclined
    case http.StatusConflict:
        return "", ErrInProgress
    default:
        return "", fmt.Errorf("provider returned %s", resp.Status)
    }
}

// NewProvider returns a client whose timeout bounds every provider call.
func NewProvider(url string) *Provider {
    return &Provider{URL: url, HTTP: &http.Client{Timeout: 5 * time.Second}}
}

// customerID stands in for authentication: a real API takes it from the session or token.
func customerID(r *http.Request) string {
    return r.Header.Get("Customer-ID")
}

func writeJSON(w http.ResponseWriter, code int, v any) {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(code)
    json.NewEncoder(w).Encode(v)
}

func writeError(w http.ResponseWriter, code int, msg string) {
    writeJSON(w, code, map[string]string{"error": msg})
}

func decode(r *http.Request) (PaymentRequest, error) {
    var req PaymentRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        return req, err
    }
    if req.Amount <= 0 || req.Card == "" {
        return req, errors.New("amount and card are required")
    }
    return req, nil
}

The checkout client plays the phone. Its first attempt gives up after 600 ms; later attempts wait 3 seconds, with a 300 ms backoff that doubles, for up to 4 attempts. -key sends one key per payment (the default), one per attempt (new) or none, and -parallel fires identical copies at once.

cmd/pay/main.go

// Command pay is the checkout client: it sends one payment and retries the way a
// mobile app does when the network is slow.
package main

import (
    "bytes"
    "crypto/rand"
    "flag"
    "fmt"
    "io"
    "net/http"
    "strings"
    "sync"
    "time"
)

func main() {
    amount := flag.Int("amount", 2500, "amount in cents")
    card := flag.String("card", "tok_visa", "card token")
    keyMode := flag.String("key", "once", "once: one key per payment; new: a new key per attempt; none: no key")
    keyValue := flag.String("key-value", "", "use this key instead of a random one")
    firstTimeout := flag.Duration("first-timeout", 600*time.Millisecond, "the first attempt gives up after this long (the phone loses signal)")
    timeout := flag.Duration("timeout", 3*time.Second, "how long later attempts wait for an answer")
    attempts := flag.Int("attempts", 4, "attempts before giving up")
    parallel := flag.Int("parallel", 1, "send this many copies at the same moment")
    flag.Parse()

    key := *keyValue
    if key == "" {
        key = rand.Text() // 26 random characters from crypto/rand
    }
    body := fmt.Sprintf(`{"amount": %d, "card": %q}`, *amount, *card)
    send := func(attempt int) (int, string, error) {
        req, _ := http.NewRequest(http.MethodPost, "http://localhost:8080/payments", strings.NewReader(body))
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Customer-ID", "cus_1001")
        switch *keyMode {
        case "once":
            req.Header.Set("Idempotency-Key", key)
        case "new":
            req.Header.Set("Idempotency-Key", rand.Text())
        }
        wait := *timeout
        if attempt == 1 {
            wait = *firstTimeout
        }
        resp, err := (&http.Client{Timeout: wait}).Do(req)
        if err != nil {
            return 0, "", err
        }
        defer resp.Body.Close()
        b, _ := io.ReadAll(resp.Body)
        extra := ""
        if resp.Header.Get("Idempotent-Replayed") == "true" {
            extra = " (replayed)"
        }
        return resp.StatusCode, string(bytes.TrimSpace(b)) + extra, nil
    }

    // One payment, retried with exponential backoff until it gets a final answer.
    pay := func(id int) {
        backoff := 300 * time.Millisecond
        for attempt := 1; attempt <= *attempts; attempt++ {
            code, msg, err := send(attempt)
            prefix := fmt.Sprintf("attempt %d:", attempt)
            if *parallel > 1 {
                prefix = fmt.Sprintf("copy %d %s", id, prefix)
            }
            switch {
            case err != nil:
                fmt.Println(prefix, "no answer:", shorten(err))
            case code == http.StatusConflict || code >= 500:
                fmt.Println(prefix, code, msg)
            default:
                fmt.Println(prefix, code, msg)
                return
            }
            time.Sleep(backoff)
            backoff *= 2
        }
    }

    var wg sync.WaitGroup
    for i := 1; i <= *parallel; i++ {
        wg.Add(1)
        go func() { defer wg.Done(); pay(i) }()
    }
    wg.Wait()
}

func shorten(err error) string {
    if strings.Contains(err.Error(), "Client.Timeout") {
        return "timed out"
    }
    return err.Error()
}

pgx's connection pool needs two modules that go get didn't record, so let Go fill them in before the first build:

go mod tidy

In terminal 1, start the provider and leave it running. It logs provider listening on :8090:

go run ./cmd/provider

Between steps, you'll empty both ledgers with these two commands:

docker exec pg psql -U postgres -d shop -c "TRUNCATE charges, idempotency_keys RESTART IDENTITY"
docker exec pg psql -U postgres -d provider -c "TRUNCATE card_charges"

Step 2: Reproduce the Double Charge

This step builds the handler most payment endpoints start as. Create it:

payments/naive.go

package payments

import (
    "context"
    "errors"
    "net/http"
)

// CreatePaymentNaive charges the card and records the payment. Every call charges again.
func (s *Server) CreatePaymentNaive(w http.ResponseWriter, r *http.Request) {
    req, err := decode(r)
    if err != nil {
        writeError(w, http.StatusBadRequest, err.Error())
        return
    }
    // Finish the payment even if the caller hangs up: a charge must not be half done.
    ctx := context.WithoutCancel(r.Context())

    providerID, err := s.Provider.Charge(ctx, req.Amount, req.Card, "")
    if errors.Is(err, ErrDeclined) {
        writeError(w, http.StatusPaymentRequired, "card declined")
        return
    }
    if err != nil {
        writeError(w, http.StatusBadGateway, "payment provider unavailable")
        return
    }

    var p Payment
    err = s.DB.QueryRow(ctx, `
        INSERT INTO charges (customer_id, amount, provider_id)
        VALUES ($1, $2, $3)
        RETURNING id, amount, provider_id`,
        customerID(r), req.Amount, providerID).Scan(&p.ID, &p.Amount, &p.ProviderID)
    if err != nil {
        writeError(w, http.StatusInternalServerError, "could not record payment")
        return
    }
    writeJSON(w, http.StatusCreated, p)
}

One line matters before you run it. net/http cancels a request's context when the client's connection closes. A call cancelled mid-charge leaves you not knowing whether the card was charged, so context.WithoutCancel keeps the context's values but drops the cancellation, and the server finishes what it started. That's right for a payment, and it's also what makes the retry dangerous: the first attempt keeps going after the client has moved on.

The API command is a teaching rig: one binary that serves a version of the handler picked with -mode. This first version knows only the naive one; Steps 3 and 4 add the others.

cmd/api/main.go (first version)

// Command api serves POST /payments in one of three versions, chosen with -mode.
package main

import (
    "context"
    "flag"
    "log"
    "net/http"
    "os"
    "time"

    "example.com/payments/payments"
    "github.com/jackc/pgx/v5/pgxpool"
)

func main() {
    mode := flag.String("mode", "idempotent", "naive, checkfirst or idempotent")
    flag.Parse()

    db, err := pgxpool.New(context.Background(), env("DATABASE_URL", "postgres://postgres:postgres@localhost:5432/shop"))
    if err != nil {
        log.Fatal(err)
    }
    s := &payments.Server{DB: db, Provider: payments.NewProvider("http://localhost:8090")}

    var h http.HandlerFunc
    switch *mode {
    case "naive":
        h = s.CreatePaymentNaive
    default:
        log.Fatalf("unknown -mode %q", *mode)
    }
    http.HandleFunc("POST /payments", h)

    srv := &http.Server{Addr: ":8080", ReadHeaderTimeout: 5 * time.Second}
    log.Printf("api (%s) listening on :8080", *mode)
    log.Fatal(srv.ListenAndServe())
}

// env returns the variable's value, or a default that suits the local Docker container.
func env(name, fallback string) string {
    if v := os.Getenv(name); v != "" {
        return v
    }
    return fallback
}

In terminal 2, start the API in naive mode:

go run ./cmd/api -mode naive

In terminal 3, pay once with no idempotency key. Expect the first attempt to time out and the retry to succeed:

go run ./cmd/pay -key none

Output

attempt 1: no answer: timed out
attempt 2: 201 {"id":2,"amount":2500,"provider_charge_id":"ch_abd3fed44e"}

From the customer's side, this is one payment: a spinner, then a receipt. Now look at both ledgers:

docker exec pg psql -U postgres -d shop -c "SELECT id, amount, provider_id FROM charges"
docker exec pg psql -U postgres -d provider -c "SELECT id, idempotency_key, amount FROM card_charges"

Output

 id | amount |  provider_id  
----+--------+---------------
  1 |   2500 | ch_d03cc74dad
  2 |   2500 | ch_abd3fed44e

      id       | idempotency_key | amount 
---------------+-----------------+--------
 ch_d03cc74dad |                 |   2500
 ch_abd3fed44e |                 |   2500

The customer saw one success and was charged twice. The receipt gives it away: payment id 2, because row 1 came from the attempt the phone gave up on. Nothing in either ledger marks the rows as the same payment.

The server did exactly what HTTP allows. POST isn't idempotent, meaning sending it twice isn't guaranteed to have the same effect as sending it once, and RFC 9110 says so in as many words:

Of the request methods defined by this specification, PUT, DELETE, and safe request methods are idempotent.

A client SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent, regardless of the method, or some means to detect that the original request was never applied.

RFC 9110, section 9.2.2

The client broke that rule, but refusing to retry would strand customers on every dropped connection. An idempotency key gives it "some means to know".

Step 3: Add an Idempotency Key and Check It First

This step adds the fix most people write first: look the key up, and charge only if it's new. The client already sends a key by default, 26 random characters from rand.Text, generated once per payment and repeated on every attempt. Create the check-first handler:

payments/checkfirst.go

package payments

import (
    "context"
    "encoding/json"
    "errors"
    "net/http"

    "github.com/jackc/pgx/v5"
)

// CreatePaymentCheckFirst looks the key up, and only charges if it has never seen it.
// The lookup and the save are separate steps, so two requests can both miss.
func (s *Server) CreatePaymentCheckFirst(w http.ResponseWriter, r *http.Request) {
    key := r.Header.Get("Idempotency-Key")
    if key == "" {
        writeError(w, http.StatusBadRequest, "Idempotency-Key header is required")
        return
    }
    req, err := decode(r)
    if err != nil {
        writeError(w, http.StatusBadRequest, err.Error())
        return
    }
    ctx := context.WithoutCancel(r.Context())

    // 1. Have we answered this key before? If so, send the same answer again.
    var code int
    var body []byte
    err = s.DB.QueryRow(ctx, `
        SELECT response_code, response_body FROM idempotency_keys
        WHERE customer_id = $1 AND key = $2`, customerID(r), key).Scan(&code, &body)
    if err == nil {
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(code)
        w.Write(body)
        return
    }
    if !errors.Is(err, pgx.ErrNoRows) {
        writeError(w, http.StatusInternalServerError, "could not read idempotency key")
        return
    }

    // 2. New key: charge and record the payment.
    providerID, err := s.Provider.Charge(ctx, req.Amount, req.Card, "")
    if err != nil {
        writeError(w, http.StatusBadGateway, "payment provider unavailable")
        return
    }
    var p Payment
    err = s.DB.QueryRow(ctx, `
        INSERT INTO charges (customer_id, amount, provider_id)
        VALUES ($1, $2, $3)
        RETURNING id, amount, provider_id`,
        customerID(r), req.Amount, providerID).Scan(&p.ID, &p.Amount, &p.ProviderID)
    if err != nil {
        writeError(w, http.StatusInternalServerError, "could not record payment")
        return
    }

    // 3. Remember the answer for next time.
    body, _ = json.Marshal(p)
    _, err = s.DB.Exec(ctx, `
        INSERT INTO idempotency_keys (customer_id, key, request_hash, status, response_code, response_body)
        VALUES ($1, $2, '', 'completed', 201, $3)
        ON CONFLICT DO NOTHING`, customerID(r), key, body)
    if err != nil {
        writeError(w, http.StatusInternalServerError, "could not save idempotency key")
        return
    }
    writeJSON(w, http.StatusCreated, p)
}

It reads well, but the provider's one-second charge sits between step 1, the lookup, and step 3, the save. Add its case to the switch in cmd/api/main.go, under the naive case:

cmd/api/main.go (excerpt)

    case "checkfirst":
        h = s.CreatePaymentCheckFirst

In terminal 2, stop the API with Ctrl+C and start it in check-first mode. Reset both ledgers with the two TRUNCATE commands from Step 1, then pay with the default key behavior:

go run ./cmd/api -mode checkfirst

Then run:

go run ./cmd/pay

Output

attempt 1: no answer: timed out
attempt 2: 201 {"id":2,"amount":2500,"provider_charge_id":"ch_0ee40e9309"}

id 2 again, and both ledgers hold two rows, as in Step 2. The key table holds one row, because the second save hit ON CONFLICT DO NOTHING:

docker exec pg psql -U postgres -d shop -c "SELECT key, status, response_code FROM idempotency_keys"

Output

            key             |  status   | response_code 
----------------------------+-----------+---------------
 G2D7ZUOEGVRORUF253FGPBOFKF | completed |           201
(1 row)

The retry arrived at about 0.9 s (600 ms timeout plus 300 ms backoff), while the first request was at the provider until about 1.0 s and hadn't saved the key, so the lookup found nothing:

View this diagram in the original post

The retry looks the key up while the first request is still at the provider. Both lookups miss, and both requests charge.

This is a check-then-act race : a decision based on a read that's stale by the time you act on it. The key is only checked, never claimed, and ownership has to come from the one component that sees every request in order: the database.

Step 4: Claim the Idempotency Key Atomically

This step makes the first request to arrive own the key before it does any work. Create payments/idempotent.go from the collapsed block; this step explains the claim and the stored answer, Steps 5 and 6 the rest.

payments/idempotent.go (the full file)

payments/idempotent.go

package payments

import (
    "bytes"
    "context"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "errors"
    "net/http"
    "os"
    "strings"
    "time"

    "github.com/jackc/pgx/v5"
)

// Options switch on the protections one at a time, so each step of the tutorial
// can show what the previous version gets wrong.
type Options struct {
    Lease            time.Duration // 0: a started key is never taken over
    PassKeyToCharge  bool          // send our key to the provider as its Idempotency-Key
    CrashAfterCharge bool          // exit after the provider charges, before we record it
}

type keyRecord struct {
    status   string
    hash     []byte
    lockedAt time.Time // when this request claimed the key; proves it still owns it
}

// fingerprint identifies the request a key was first used with. Hashing the decoded
// struct, not the raw bytes, ignores whitespace and field order.
func fingerprint(r *http.Request, req PaymentRequest) []byte {
    canonical, _ := json.Marshal(req)
    sum := sha256.Sum256(append([]byte(r.Method+" "+r.URL.Path+"\n"), canonical...))
    return sum[:]
}

// claim tries to make this request the owner of the key. It returns true, and the
// claim's locked_at, if this request should do the work. Otherwise it returns the
// key's current record.
func (s *Server) claim(ctx context.Context, customer, key string, hash []byte, opt Options) (bool, keyRecord, error) {
    var rec keyRecord
    var err error
    if opt.Lease == 0 {
        err = s.DB.QueryRow(ctx, `
            INSERT INTO idempotency_keys (customer_id, key, request_hash)
            VALUES ($1, $2, $3)
            ON CONFLICT (customer_id, key) DO NOTHING
            RETURNING locked_at`, customer, key, hash).Scan(&rec.lockedAt)
    } else {
        // Also take over a key whose owner started long ago and never finished.
        err = s.DB.QueryRow(ctx, `
            INSERT INTO idempotency_keys (customer_id, key, request_hash)
            VALUES ($1, $2, $3)
            ON CONFLICT (customer_id, key) DO UPDATE SET locked_at = now()
            WHERE idempotency_keys.status = 'started'
              AND idempotency_keys.request_hash = EXCLUDED.request_hash
              AND idempotency_keys.locked_at < now() - make_interval(secs => $4)
            RETURNING locked_at`, customer, key, hash, opt.Lease.Seconds()).Scan(&rec.lockedAt)
    }
    if err == nil {
        return true, rec, nil
    }
    if !errors.Is(err, pgx.ErrNoRows) {
        return false, keyRecord{}, err
    }

    // Someone else holds the key. Read what they have so far.
    err = s.DB.QueryRow(ctx, `
        SELECT status, request_hash FROM idempotency_keys
        WHERE customer_id = $1 AND key = $2`,
        customer, key).Scan(&rec.status, &rec.hash)
    if errors.Is(err, pgx.ErrNoRows) {
        // Released a moment ago: report it as in progress so the client retries.
        rec.status, rec.hash = "started", hash
        return false, rec, nil
    }
    return false, rec, err
}

// CreatePayment charges at most once per Idempotency-Key and replays the first answer.
func (s *Server) CreatePayment(opt Options) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        customer := customerID(r)
        key := idempotencyKey(r)
        if key == "" || len(key) > 255 {
            writeError(w, http.StatusBadRequest, "Idempotency-Key header is required (at most 255 characters)")
            return
        }
        req, err := decode(r)
        if err != nil {
            writeError(w, http.StatusBadRequest, err.Error())
            return
        }
        hash := fingerprint(r, req)
        ctx := context.WithoutCancel(r.Context())

        claimed, rec, err := s.claim(ctx, customer, key, hash, opt)
        switch {
        case err != nil:
            writeError(w, http.StatusInternalServerError, "could not claim idempotency key")
            return
        case claimed:
            // This request owns the key; carry on below.
        case !bytes.Equal(rec.hash, hash):
            writeError(w, http.StatusUnprocessableEntity, "Idempotency-Key was already used with a different request")
            return
        case rec.status == "completed":
            s.replay(ctx, w, customer, key)
            return
        default:
            w.Header().Set("Retry-After", "1")
            writeError(w, http.StatusConflict, "a request with this Idempotency-Key is still in progress")
            return
        }

        chargeKey := ""
        if opt.PassKeyToCharge {
            // Derived from our key, so every retry and takeover sends the same one.
            // Hashed to 64 hex characters to fit providers' length limits.
            sum := sha256.Sum256([]byte(customer + ":" + key))
            chargeKey = hex.EncodeToString(sum[:])
        }
        providerID, err := s.Provider.Charge(ctx, req.Amount, req.Card, chargeKey)

        if errors.Is(err, ErrDeclined) {
            // A decline is a final answer: store it so a retry gets the same one.
            body, _ := json.Marshal(map[string]string{"error": "card declined"})
            err := s.finish(ctx, customer, key, http.StatusPaymentRequired, body, nil)
            if errors.Is(err, errLostKey) {
                s.replay(ctx, w, customer, key)
                return
            }
            if err != nil {
                writeError(w, http.StatusInternalServerError, "could not save result")
                return
            }
            w.Header().Set("Content-Type", "application/json")
            w.WriteHeader(http.StatusPaymentRequired)
            w.Write(body)
            return
        }
        if errors.Is(err, ErrInProgress) {
            // The provider is still charging under this key (an earlier claim we took
            // over is mid-call). Not a fault: keep the key and let the client retry.
            w.Header().Set("Retry-After", "1")
            writeError(w, http.StatusConflict, "a request with this Idempotency-Key is still in progress")
            return
        }
        if err != nil {
            // A fault is not an answer. Release the key so a retry can run the charge
            // again, but only if it is still our claim and not a later takeover's.
            s.DB.Exec(ctx, `
                DELETE FROM idempotency_keys
                WHERE customer_id = $1 AND key = $2
                  AND status = 'started' AND locked_at = $3`, customer, key, rec.lockedAt)
            writeError(w, http.StatusBadGateway, "payment provider unavailable")
            return
        }

        if opt.CrashAfterCharge {
            os.Exit(1) // simulate a deploy or OOM kill at the worst moment
        }

        var p Payment
        record := func(tx pgx.Tx) ([]byte, error) {
            err := tx.QueryRow(ctx, `
                INSERT INTO charges (customer_id, amount, provider_id)
                VALUES ($1, $2, $3)
                RETURNING id, amount, provider_id`,
                customer, req.Amount, providerID).Scan(&p.ID, &p.Amount, &p.ProviderID)
            if err != nil {
                return nil, err
            }
            return json.Marshal(p)
        }
        err = s.finish(ctx, customer, key, http.StatusCreated, nil, record)
        if errors.Is(err, errLostKey) {
            s.replay(ctx, w, customer, key) // a takeover finished first; answer as it did
            return
        }
        if err != nil {
            writeError(w, http.StatusInternalServerError, "could not record payment")
            return
        }
        writeJSON(w, http.StatusCreated, p)
    }
}

// errLostKey means another request finished this key first.
var errLostKey = errors.New("idempotency key already finished")

// finish saves the answer for a key, and the payment row when there is one, in one
// transaction. Only a request that still sees the key as started may finish it.
func (s *Server) finish(ctx context.Context, customer, key string, code int, body []byte, record func(pgx.Tx) ([]byte, error)) error {
    tx, err := s.DB.Begin(ctx)
    if err != nil {
        return err
    }
    defer tx.Rollback(ctx)

    var status string
    err = tx.QueryRow(ctx, `
        SELECT status FROM idempotency_keys
        WHERE customer_id = $1 AND key = $2
        FOR UPDATE`, customer, key).Scan(&status)
    if err != nil {
        return err
    }
    if status != "started" {
        return errLostKey
    }
    if record != nil {
        if body, err = record(tx); err != nil {
            return err
        }
    }
    _, err = tx.Exec(ctx, `
        UPDATE idempotency_keys
        SET status = 'completed', response_code = $3, response_body = $4
        WHERE customer_id = $1 AND key = $2`, customer, key, code, body)
    if err != nil {
        return err
    }
    return tx.Commit(ctx)
}

// replay sends the stored answer for a finished key.
func (s *Server) replay(ctx context.Context, w http.ResponseWriter, customer, key string) {
    var code int
    var body []byte
    err := s.DB.QueryRow(ctx, `
        SELECT response_code, response_body FROM idempotency_keys
        WHERE customer_id = $1 AND key = $2`, customer, key).Scan(&code, &body)
    if err != nil {
        writeError(w, http.StatusInternalServerError, "could not read stored result")
        return
    }
    w.Header().Set("Content-Type", "application/json")
    w.Header().Set("Idempotent-Replayed", "true")
    w.WriteHeader(code)
    w.Write(body)
}

// idempotencyKey reads the header. The IETF draft sends it as a quoted string,
// most providers send it bare; accept both.
func idempotencyKey(r *http.Request) string {
    k := strings.TrimSpace(r.Header.Get("Idempotency-Key"))
    if len(k) >= 2 && k[0] == '"' && k[len(k)-1] == '"' {
        k = k[1 : len(k)-1]
    }
    return k
}

Then replace cmd/api/main.go with its final version, which defaults to the idempotent handler and adds three flags for Step 6:

cmd/api/main.go

// Command api serves POST /payments in one of three versions, chosen with -mode.
package main

import (
    "context"
    "flag"
    "log"
    "net/http"
    "os"
    "time"

    "example.com/payments/payments"
    "github.com/jackc/pgx/v5/pgxpool"
)

func main() {
    mode := flag.String("mode", "idempotent", "naive, checkfirst or idempotent")
    lease := flag.Duration("lease", 0, "take over a started key after this long (0 = never)")
    passKey := flag.Bool("pass-key", false, "send the idempotency key to the provider")
    crash := flag.Bool("crash-after-charge", false, "exit after the provider charges")
    flag.Parse()

    db, err := pgxpool.New(context.Background(), env("DATABASE_URL", "postgres://postgres:postgres@localhost:5432/shop"))
    if err != nil {
        log.Fatal(err)
    }
    s := &payments.Server{DB: db, Provider: payments.NewProvider("http://localhost:8090")}

    var h http.HandlerFunc
    switch *mode {
    case "naive":
        h = s.CreatePaymentNaive
    case "checkfirst":
        h = s.CreatePaymentCheckFirst
    case "idempotent":
        h = s.CreatePayment(payments.Options{Lease: *lease, PassKeyToCharge: *passKey, CrashAfterCharge: *crash})
    default:
        log.Fatalf("unknown -mode %q", *mode)
    }
    http.HandleFunc("POST /payments", h)

    srv := &http.Server{Addr: ":8080", ReadHeaderTimeout: 5 * time.Second}
    log.Printf("api (%s) listening on :8080", *mode)
    log.Fatal(srv.ListenAndServe())
}

// env returns the variable's value, or a default that suits the local Docker container.
func env(name, fallback string) string {
    if v := os.Getenv(name); v != "" {
        return v
    }
    return fallback
}

PostgreSQL's INSERT ... ON CONFLICT does the check and the claim in one statement, arbitrated by the primary key. With DO NOTHING, a conflicting row isn't inserted, and RETURNING reports only what changed:

Only rows that were successfully inserted or updated will be returned.

PostgreSQL 18 documentation, INSERT

So the claim returns the new key's locked_at if this request inserted it, and zero rows if another request got there first. In pgx, zero rows shows up as pgx.ErrNoRows from Scan, the "lost the claim" signal:

payments/idempotent.go (excerpt)

// claim tries to make this request the owner of the key. It returns true, and the
// claim's locked_at, if this request should do the work. Otherwise it returns the
// key's current record.
func (s *Server) claim(ctx context.Context, customer, key string, hash []byte, opt Options) (bool, keyRecord, error) {
    var rec keyRecord
    var err error
    if opt.Lease == 0 {
        err = s.DB.QueryRow(ctx, `
            INSERT INTO idempotency_keys (customer_id, key, request_hash)
            VALUES ($1, $2, $3)
            ON CONFLICT (customer_id, key) DO NOTHING
            RETURNING locked_at`, customer, key, hash).Scan(&rec.lockedAt)
    } else {
        // Also take over a key whose owner started long ago and never finished.
        err = s.DB.QueryRow(ctx, `
            INSERT INTO idempotency_keys (customer_id, key, request_hash)
            VALUES ($1, $2, $3)
            ON CONFLICT (customer_id, key) DO UPDATE SET locked_at = now()
            WHERE idempotency_keys.status = 'started'
              AND idempotency_keys.request_hash = EXCLUDED.request_hash
              AND idempotency_keys.locked_at < now() - make_interval(secs => $4)
            RETURNING locked_at`, customer, key, hash, opt.Lease.Seconds()).Scan(&rec.lockedAt)
    }
    if err == nil {
        return true, rec, nil
    }
    if !errors.Is(err, pgx.ErrNoRows) {
        return false, keyRecord{}, err
    }

    // Someone else holds the key. Read what they have so far.
    err = s.DB.QueryRow(ctx, `
        SELECT status, request_hash FROM idempotency_keys
        WHERE customer_id = $1 AND key = $2`,
        customer, key).Scan(&rec.status, &rec.hash)
    if errors.Is(err, pgx.ErrNoRows) {
        // Released a moment ago: report it as in progress so the client retries.
        rec.status, rec.hash = "started", hash
        return false, rec, nil
    }
    return false, rec, err
}

The else branch adds a lease for Step 6. If the key vanishes between the two queries (released after a fault, as in Step 5), the request reports it as in progress so the client retries. The claim runs in its own implicit transaction, so it commits the moment the statement finishes.

That's deliberate. If the claim and the charge shared one transaction, a retry's INSERT would wait on the unique index for the whole provider call, holding a database connection, and never get its 409. Commit the claim on its own and keep the slow work outside any transaction.

The handler acts on what claim reports:

payments/idempotent.go (excerpt)

// CreatePayment charges at most once per Idempotency-Key and replays the first answer.
func (s *Server) CreatePayment(opt Options) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        customer := customerID(r)
        key := idempotencyKey(r)
        if key == "" || len(key) > 255 {
            writeError(w, http.StatusBadRequest, "Idempotency-Key header is required (at most 255 characters)")
            return
        }
        req, err := decode(r)
        if err != nil {
            writeError(w, http.StatusBadRequest, err.Error())
            return
        }
        hash := fingerprint(r, req)
        ctx := context.WithoutCancel(r.Context())

        claimed, rec, err := s.claim(ctx, customer, key, hash, opt)
        switch {
        case err != nil:
            writeError(w, http.StatusInternalServerError, "could not claim idempotency key")
            return
        case claimed:
            // This request owns the key; carry on below.
        case !bytes.Equal(rec.hash, hash):
            writeError(w, http.StatusUnprocessableEntity, "Idempotency-Key was already used with a different request")
            return
        case rec.status == "completed":
            s.replay(ctx, w, customer, key)
            return
        default:
            w.Header().Set("Retry-After", "1")
            writeError(w, http.StatusConflict, "a request with this Idempotency-Key is still in progress")
            return
        }

What `claim` reports	Response
This request owns the key	Charge the card and record the payment
Key exists with a different request	`422 Unprocessable Content` (Step 5)
Key exists and is `completed`	Replay the stored status and body
Key exists and is `started`	`409 Conflict` with `Retry-After: 1`

Retry-After is a standard HTTP header that tells the client how many seconds to wait. The codes follow the Idempotency-Key header draft from the IETF, the body that publishes internet standards: 400 for a missing key, 422 for a key reused with a different payload, and 409 when "A request is outstanding for this Idempotency-Key". The draft expired in April 2026 without becoming an RFC, so treat it as a convention. It also says a retry of a completed request gets the previous result, success or error.

After a successful charge, finish saves the shop's ledger row and the key's stored answer in one transaction:

payments/idempotent.go (excerpt)

        var p Payment
        record := func(tx pgx.Tx) ([]byte, error) {
            err := tx.QueryRow(ctx, `
                INSERT INTO charges (customer_id, amount, provider_id)
                VALUES ($1, $2, $3)
                RETURNING id, amount, provider_id`,
                customer, req.Amount, providerID).Scan(&p.ID, &p.Amount, &p.ProviderID)
            if err != nil {
                return nil, err
            }
            return json.Marshal(p)
        }
        err = s.finish(ctx, customer, key, http.StatusCreated, nil, record)
        if errors.Is(err, errLostKey) {
            s.replay(ctx, w, customer, key) // a takeover finished first; answer as it did
            return
        }
        if err != nil {
            writeError(w, http.StatusInternalServerError, "could not record payment")
            return
        }
        writeJSON(w, http.StatusCreated, p)

payments/idempotent.go (excerpt)

// errLostKey means another request finished this key first.
var errLostKey = errors.New("idempotency key already finished")

// finish saves the answer for a key, and the payment row when there is one, in one
// transaction. Only a request that still sees the key as started may finish it.
func (s *Server) finish(ctx context.Context, customer, key string, code int, body []byte, record func(pgx.Tx) ([]byte, error)) error {
    tx, err := s.DB.Begin(ctx)
    if err != nil {
        return err
    }
    defer tx.Rollback(ctx)

    var status string
    err = tx.QueryRow(ctx, `
        SELECT status FROM idempotency_keys
        WHERE customer_id = $1 AND key = $2
        FOR UPDATE`, customer, key).Scan(&status)
    if err != nil {
        return err
    }
    if status != "started" {
        return errLostKey
    }
    if record != nil {
        if body, err = record(tx); err != nil {
            return err
        }
    }
    _, err = tx.Exec(ctx, `
        UPDATE idempotency_keys
        SET status = 'completed', response_code = $3, response_body = $4
        WHERE customer_id = $1 AND key = $2`, customer, key, code, body)
    if err != nil {
        return err
    }
    return tx.Commit(ctx)
}

The AWS Builders' Library article on making retries safe says recording the token and the mutations "needs to be an 'all or nothing' or atomic process"; otherwise a crash between the two writes leaves a payment no retry can see. The status check and FOR UPDATE matter in Step 6.

replay returns the stored answer with Idempotent-Replayed: true, a header Stripe uses so a client can tell a replay from fresh work. No standard defines it.

payments/idempotent.go (excerpt)

// replay sends the stored answer for a finished key.
func (s *Server) replay(ctx context.Context, w http.ResponseWriter, customer, key string) {
    var code int
    var body []byte
    err := s.DB.QueryRow(ctx, `
        SELECT response_code, response_body FROM idempotency_keys
        WHERE customer_id = $1 AND key = $2`, customer, key).Scan(&code, &body)
    if err != nil {
        writeError(w, http.StatusInternalServerError, "could not read stored result")
        return
    }
    w.Header().Set("Content-Type", "application/json")
    w.Header().Set("Idempotent-Replayed", "true")
    w.WriteHeader(code)
    w.Write(body)
}

Restart the API in terminal 2 with no flags, reset both ledgers, and pay with a fixed key:

go run ./cmd/api

Then run:

go run ./cmd/pay -key-value order-7731
docker exec pg psql -U postgres -d shop -c "SELECT key, status, response_code, convert_from(response_body, 'UTF8') AS response_body FROM idempotency_keys"

Output

attempt 1: no answer: timed out
attempt 2: 409 {"error":"a request with this Idempotency-Key is still in progress"}
attempt 3: 201 {"id":1,"amount":2500,"provider_charge_id":"ch_b2d0620c57"} (replayed)
    key     |  status   | response_code |                        response_body                        
------------+-----------+---------------+-------------------------------------------------------------
 order-7731 | completed |           201 | {"id":1,"amount":2500,"provider_charge_id":"ch_b2d0620c57"}
(1 row)

The retry at 0.9 s found the key started and got a 409; the third, at about 1.5 s, got the stored receipt, and each ledger holds one row. Now reset the ledgers and send 20 identical copies at the same instant, counting the distinct lines:

go run ./cmd/pay -parallel 20 -first-timeout 3s | sed -E 's/copy [0-9]+ //' | sort | uniq -c

Output

   1 attempt 1: 201 {"id":1,"amount":2500,"provider_charge_id":"ch_66e10aa679"}
  19 attempt 1: 409 {"error":"a request with this Idempotency-Key is still in progress"}
  19 attempt 2: 409 {"error":"a request with this Idempotency-Key is still in progress"}
  19 attempt 3: 409 {"error":"a request with this Idempotency-Key is still in progress"}
  19 attempt 4: 201 {"id":1,"amount":2500,"provider_charge_id":"ch_66e10aa679"} (replayed)

One copy charged; the other 19 got 409s, then its receipt, and each ledger holds one row. The losers needed four attempts because this client ignores Retry-After; a production client should read it.

Step 5: Fingerprint Requests and Store Only Final Answers

The key still trusts the client never to reuse it for a different payment: a bug that sends order-7731 for a 99.00 payment after a 25.00 one would get the 25.00 receipt back. This step closes that hole, then settles which outcomes a key should remember.

A fingerprint is a hash of the request a key was first used with, stored in request_hash and compared on every repeat:

payments/idempotent.go (excerpt)

// fingerprint identifies the request a key was first used with. Hashing the decoded
// struct, not the raw bytes, ignores whitespace and field order.
func fingerprint(r *http.Request, req PaymentRequest) []byte {
    canonical, _ := json.Marshal(req)
    sum := sha256.Sum256(append([]byte(r.Method+" "+r.URL.Path+"\n"), canonical...))
    return sum[:]
}

It hashes the decoded struct, not the raw body: {"amount": 2500} and {"amount":2500} are the same payment, and json.Marshal writes a struct's fields in a fixed order without extra whitespace. Unknown fields never reach the hash, which is fine because the handler ignores them too.

The header comes in two spellings. The IETF draft makes its value a structured-field String, HTTP's standard quoted-string type (Idempotency-Key: "8e03978e-..."), while providers such as Stripe take it bare. idempotencyKey accepts both:

payments/idempotent.go (excerpt)

// idempotencyKey reads the header. The IETF draft sends it as a quoted string,
// most providers send it bare; accept both.
func idempotencyKey(r *http.Request) string {
    k := strings.TrimSpace(r.Header.Get("Idempotency-Key"))
    if len(k) >= 2 && k[0] == '"' && k[len(k)-1] == '"' {
        k = k[1 : len(k)-1]
    }
    return k
}

Reset the ledgers, pay with a fixed key, then reuse it for a different amount, then send it quoted:

go run ./cmd/pay -key-value order-7731
go run ./cmd/pay -key-value order-7731 -amount 9900 -first-timeout 3s
go run ./cmd/pay -key-value '"order-7731"' -first-timeout 3s

Output

attempt 1: no answer: timed out
attempt 2: 409 {"error":"a request with this Idempotency-Key is still in progress"}
attempt 3: 201 {"id":1,"amount":2500,"provider_charge_id":"ch_dd32a4869f"} (replayed)
attempt 1: 422 {"error":"Idempotency-Key was already used with a different request"}
attempt 1: 201 {"id":1,"amount":2500,"provider_charge_id":"ch_dd32a4869f"} (replayed)

The first payment ran once. The 99.00 payment with the same key got a 422, and the client stopped retrying. The quoted key matched the bare one and replayed the original receipt.

A declined card and a provider that's down both end without a charge, and they need opposite treatment. (The ErrInProgress branch in between comes up in Step 6.)

payments/idempotent.go (excerpt)

        providerID, err := s.Provider.Charge(ctx, req.Amount, req.Card, chargeKey)

        if errors.Is(err, ErrDeclined) {
            // A decline is a final answer: store it so a retry gets the same one.
            body, _ := json.Marshal(map[string]string{"error": "card declined"})
            err := s.finish(ctx, customer, key, http.StatusPaymentRequired, body, nil)
            if errors.Is(err, errLostKey) {
                s.replay(ctx, w, customer, key)
                return
            }
            if err != nil {
                writeError(w, http.StatusInternalServerError, "could not save result")
                return
            }
            w.Header().Set("Content-Type", "application/json")
            w.WriteHeader(http.StatusPaymentRequired)
            w.Write(body)
            return
        }
        if errors.Is(err, ErrInProgress) {
            // The provider is still charging under this key (an earlier claim we took
            // over is mid-call). Not a fault: keep the key and let the client retry.
            w.Header().Set("Retry-After", "1")
            writeError(w, http.StatusConflict, "a request with this Idempotency-Key is still in progress")
            return
        }
        if err != nil {
            // A fault is not an answer. Release the key so a retry can run the charge
            // again, but only if it is still our claim and not a later takeover's.
            s.DB.Exec(ctx, `
                DELETE FROM idempotency_keys
                WHERE customer_id = $1 AND key = $2
                  AND status = 'started' AND locked_at = $3`, customer, key, rec.lockedAt)
            writeError(w, http.StatusBadGateway, "payment provider unavailable")
            return
        }

A decline is a final answer from the card's bank, so the handler stores the 402 and replays it:

go run ./cmd/pay -card tok_declined -key-value order-7732

Output

attempt 1: no answer: timed out
attempt 2: 409 {"error":"a request with this Idempotency-Key is still in progress"}
attempt 3: 402 {"error":"card declined"} (replayed)

A provider fault is different: it means "I don't know". The handler deletes its started claim so a later attempt can run the charge again (Step 6 explains the locked_at condition). Stop the provider in terminal 1 with Ctrl+C, pay with a new key, and look for the key:

go run ./cmd/pay -key-value order-7733 -attempts 2
docker exec pg psql -U postgres -d shop -c "SELECT key, status FROM idempotency_keys WHERE key = 'order-7733'"

Output

attempt 1: 502 {"error":"payment provider unavailable"}
attempt 2: 502 {"error":"payment provider unavailable"}
 key | status 
-----+--------
(0 rows)

The key was released. Start the provider again in terminal 1 and retry the same key:

go run ./cmd/pay -key-value order-7733 -first-timeout 3s

Output

attempt 1: 201 {"id":2,"amount":2500,"provider_charge_id":"ch_738dbd0b2f"}

A fresh charge, not a replay, and id 2 because the order-7731 payment is row 1. Reasonable systems disagree on faults:

Policy	Who does it	Upside	Risk
Store every result, including 500s	Stripe API v1	A retry never re-runs a half-finished request	A brief outage becomes a permanent failure for that key, and the client must resolve the original outcome before paying again
Re-execute after a failure	Stripe API v2	Retries recover from transient faults	Safe only if a rerun can't repeat a side effect
Store final answers, release on faults	This tutorial	Declines stay declined; outages heal	Same as above

I'd rather release on a fault, because the customer still wants to pay once the provider recovers. (Validation failures get a 400 before the claim and are never stored, as with Stripe.) But a 502 can hide a charge that went through moments before the connection broke, and a rerun would charge again. Releasing the key is only safe once the provider itself refuses to charge twice, which is the next step's job.

Never answer a 5xx with a new key

A 5xx doesn't tell the client whether the charge happened. Stripe's advice is plain: "Don't retry with a new idempotency key until you've resolved the original outcome." Retry with the same key, or look the payment up first; a new key turns an unknown outcome into a possible second charge.

Step 6: Survive a Crash with a Lease and a Provider Key

This step kills the server after the provider charged the card and before the shop recorded it, as a deploy or an out-of-memory kill might. The -crash-after-charge flag exits at that point:

payments/idempotent.go (excerpt)

        if opt.CrashAfterCharge {
            os.Exit(1) // simulate a deploy or OOM kill at the worst moment
        }

Reset the ledgers. In terminal 2, start the API with the crash flag; in terminal 3, pay once with a fixed key and a patient client:

go run ./cmd/api -crash-after-charge

Then run:

go run ./cmd/pay -key-value order-8001 -first-timeout 3s -attempts 1
docker exec pg psql -U postgres -d shop -c "SELECT key, status, locked_at FROM idempotency_keys"

Output

attempt 1: no answer: Post "http://localhost:8080/payments": EOF
    key     | status  |           locked_at           
------------+---------+-------------------------------
 order-8001 | started | 2026-10-12 11:39:00.616311+00

The API is gone and its claim is still started. Restart the API without flags and retry:

go run ./cmd/api

Then run:

go run ./cmd/pay -key-value order-8001 -first-timeout 3s -attempts 2

Output

attempt 1: 409 {"error":"a request with this Idempotency-Key is still in progress"}
attempt 2: 409 {"error":"a request with this Idempotency-Key is still in progress"}

Nothing will ever finish that key. The claim needs a lease , ownership that lapses after a set time unless the owner finishes first. locked_at records when the claim was taken, and with -lease set, claim runs its other branch:

payments/idempotent.go (excerpt)

    } else {
        // Also take over a key whose owner started long ago and never finished.
        err = s.DB.QueryRow(ctx, `
            INSERT INTO idempotency_keys (customer_id, key, request_hash)
            VALUES ($1, $2, $3)
            ON CONFLICT (customer_id, key) DO UPDATE SET locked_at = now()
            WHERE idempotency_keys.status = 'started'
              AND idempotency_keys.request_hash = EXCLUDED.request_hash
              AND idempotency_keys.locked_at < now() - make_interval(secs => $4)
            RETURNING locked_at`, customer, key, hash, opt.Lease.Seconds()).Scan(&rec.lockedAt)
    }

The DO UPDATE takes over only a started key with the same fingerprint whose claim is older than the lease; otherwise RETURNING produces zero rows and the request loses the claim as before. The lease is one of the switches in Options:

payments/idempotent.go (excerpt)

// Options switch on the protections one at a time, so each step of the tutorial
// can show what the previous version gets wrong.
type Options struct {
    Lease            time.Duration // 0: a started key is never taken over
    PassKeyToCharge  bool          // send our key to the provider as its Idempotency-Key
    CrashAfterCharge bool          // exit after the provider charges, before we record it
}

Restart the API with -lease 10s, wait until 10 seconds have passed since the crash, retry, and run the ledger queries from Step 2:

go run ./cmd/api -lease 10s

Then run:

go run ./cmd/pay -key-value order-8001 -first-timeout 3s

Output

attempt 1: 201 {"id":1,"amount":2500,"provider_charge_id":"ch_d711b185ca"}
 id | amount |  provider_id  
----+--------+---------------
  1 |   2500 | ch_d711b185ca

      id       | idempotency_key | amount 
---------------+-----------------+--------
 ch_c3f90b8aca |                 |   2500
 ch_d711b185ca |                 |   2500

The card was charged twice, and the shop's ledger shows one payment: ch_c3f90b8aca, made by the crashed request, exists only at the provider. That's worse than Step 2, where the ledgers at least agreed. Only the provider knows whether the crashed request charged, so it has to be asked in a way it can deduplicate. With -pass-key, the handler sends the provider its own Idempotency-Key, derived from the customer and the client's key:

payments/idempotent.go (excerpt)

        chargeKey := ""
        if opt.PassKeyToCharge {
            // Derived from our key, so every retry and takeover sends the same one.
            // Hashed to 64 hex characters to fit providers' length limits.
            sum := sha256.Sum256([]byte(customer + ":" + key))
            chargeKey = hex.EncodeToString(sum[:])
        }
        providerID, err := s.Provider.Charge(ctx, req.Amount, req.Card, chargeKey)

The customer is part of the input because your table scopes keys per customer, while the provider has one key space for your whole account. The SHA-256 hash turns any input into 64 hex characters, which fits Stripe's 255-character limit and Adyen's 64. Square (45 characters), PayPal (38) and Razorpay payouts (36) need something shorter, such as the base64url encoding of the hash's first 24 bytes, which is 32 characters.

Reset the ledgers and repeat the crash with -pass-key on every API start, from terminals 2 and 3 as before:

go run ./cmd/api -crash-after-charge -pass-key
go run ./cmd/pay -key-value order-8001 -first-timeout 3s -attempts 1
go run ./cmd/api -lease 10s -pass-key
go run ./cmd/pay -key-value order-8001 -first-timeout 3s

The last command runs once 10 seconds have passed since the crash:

Output

attempt 1: no answer: Post "http://localhost:8080/payments": EOF
attempt 1: 201 {"id":1,"amount":2500,"provider_charge_id":"ch_038668bc4c"}
 id | amount |  provider_id  
----+--------+---------------
  1 |   2500 | ch_038668bc4c

      id       |                         idempotency_key                          | amount 
---------------+------------------------------------------------------------------+--------
 ch_038668bc4c | 9115b225b4ef86fab14f1e43fc342407f9b1f20d5358ab1da88ef89ac96979ca |   2500

One charge in each ledger, with the same ID: the provider recognized the derived key and returned the crashed request's charge instead of making a second one.

A lease can also run out while its owner is still alive. Reset the ledgers, start the API with -lease 500ms and no -pass-key, pay with the default client, and run the ledger queries:

go run ./cmd/api -lease 500ms

Then run:

go run ./cmd/pay

Output

attempt 1: no answer: timed out
attempt 2: 201 {"id":1,"amount":2500,"provider_charge_id":"ch_47558d25dd"} (replayed)
 id | amount |  provider_id  
----+--------+---------------
  1 |   2500 | ch_47558d25dd

      id       | idempotency_key | amount 
---------------+-----------------+--------
 ch_47558d25dd |                 |   2500
 ch_f5009699df |                 |   2500

The retry at 0.9 s took over a claim older than 500 ms while the original was still at the provider, so both charged. The shop recorded one payment because the original completed the key first and the takeover's finish replayed it. If both finish at the same instant, SELECT ... FOR UPDATE makes the second wait on the row lock. Under Read Committed, PostgreSQL's default isolation level where each statement sees what was committed before it began, it then reads the newest version and backs off.

Now run the same test with -lease 500ms -pass-key:

Output

attempt 1: no answer: timed out
attempt 2: 409 {"error":"a request with this Idempotency-Key is still in progress"}
attempt 3: 201 {"id":1,"amount":2500,"provider_charge_id":"ch_7c98ee1cee"} (replayed)
 id | amount |  provider_id  
----+--------+---------------
  1 |   2500 | ch_7c98ee1cee

      id       |                         idempotency_key                          | amount 
---------------+------------------------------------------------------------------+--------
 ch_7c98ee1cee | 08079351b5a3a1c086fbc082c533c78e84569f2e49e00eef98447e607c5f4076 |   2500

The takeover reached the provider with the same derived key while the original was still charging, and the provider answered 409, as Stripe does for a concurrent request with the same key. The API passed it back without releasing the key (the ErrInProgress branch), and attempt 3 got the original's receipt: one charge, one payment.

One trap remains. Had the original's provider call failed after the takeover, Step 5's release would have deleted the takeover's live claim, which is why the DELETE also matches locked_at = $3, the value claim returned.

This is the lesson of the Redis distributed locks tutorial from the other side: a lease can't prove the previous owner has stopped, so each place where an effect happens checks for itself. There, PostgreSQL refused writes carrying a stale fencing token, a number that grows with every lock grant. Here, the provider refuses a second charge for the same key, and locked_at does a smaller version of the same job for the release: a stale owner's DELETE names a locked_at that no longer exists, so it removes nothing. Set the lease well above the slowest path through the handler, here the provider client's 5-second timeout (the load test uses 30 seconds), and treat it as a way to keep takeovers rare, never as the guarantee.

Card processors offer the same mechanism under different names and rules:

Provider	Where the key goes	Retention	Limit
Stripe (API v1)	`Idempotency-Key` header	At least 24 hours	255 characters
Adyen	`idempotency-key` header on POST	7 to 14 days	64 characters
PayPal	`PayPal-Request-Id` header	Varies by API	38 characters
Square	`idempotency_key` in the request body	See the API's docs	1 to 45 characters
Razorpay (payouts)	`X-Payout-Idempotency` header	See the API's docs	4 to 36 characters

Want this checked on your own payment path?

None of the failures in this step show up in a test that counts only successful responses: you find them by comparing your ledger with the provider's. I offer a focused review of payment and money-movement APIs that covers where retries enter, how keys are claimed and stored, what reaches the provider, and a duplicate-request load test like the one in Step 7, with written findings your team can act on. Ask for a payment API review or book a free 20-minute intro call.

Step 7: Load-Test the Finished API

This step runs the finished configuration, a 30-second lease plus the provider key, against a burst of duplicates. load.sh starts its own provider and API, so stop terminals 1 and 2 first, then build:

go build -o bin/ ./cmd/...

The script sends 30 payments with distinct amounts, 1001 to 1030, each as 5 identical copies at the same instant, and counts the responses:

load.sh

#!/usr/bin/env bash
# load.sh : 30 payments, each sent as 5 identical copies at once, against the final API.
cd "$(dirname "$0")"
docker exec pg psql -U postgres -d shop -qc "TRUNCATE charges, idempotency_keys RESTART IDENTITY"
docker exec pg psql -U postgres -d provider -qc "TRUNCATE card_charges"
pkill -f bin/provider; pkill -f bin/api; sleep 0.3
./bin/provider >/tmp/provider.log 2>&1 &
./bin/api -lease 30s -pass-key >/tmp/api.log 2>&1 &
until curl -s -o /dev/null localhost:8080 && curl -s -o /dev/null localhost:8090; do sleep 0.1; done
rm -f /tmp/pay-*.log
pids=()
for i in $(seq 1 30); do
  ./bin/pay -parallel 5 -first-timeout 3s -amount $((1000 + i)) >/tmp/pay-$i.log 2>&1 &
  pids+=($!)
done
wait "${pids[@]}"
cat /tmp/pay-*.log | sed -E 's/copy [0-9]+ attempt [0-9]+: //; s/\{"id".*\}/{payment}/' | sort | uniq -c
pkill -f bin/provider; pkill -f bin/api

Then run:

bash load.sh

Output

  30 201 {payment}
 120 201 {payment} (replayed)
 360 409 {"error":"a request with this Idempotency-Key is still in progress"}

Of 150 copies, one per group of 5 charged (30 originals); the other 120 each got three 409s while the winner was at the provider (360), then its receipt (120 replays). The run took about 3 seconds. Now count what was charged:

docker exec pg psql -U postgres -d shop -c "SELECT count(*) AS payments, count(DISTINCT amount) AS distinct_amounts FROM charges"
docker exec pg psql -U postgres -d provider -c "SELECT count(*) AS card_charges FROM card_charges"
docker exec pg psql -U postgres -d shop -c "SELECT status, response_code, count(*) FROM idempotency_keys GROUP BY 1, 2"

Output

 payments | distinct_amounts 
----------+------------------
       30 |               30
(1 row)

 card_charges 
--------------
           30
(1 row)

  status   | response_code | count 
-----------+---------------+-------
 completed |           201 |    30
(1 row)

30 payments in the shop's ledger, each a distinct amount, 30 charges at the provider, and 30 keys completed with a 201. The ledgers agree, and that's the check that matters: a test counting only 201s would have passed in Step 2 too.

To check for data races as well, rebuild with Go's race detector and run the script again:

go build -race -o bin/ ./cmd/...
bash load.sh

The race-enabled run gave the same counts, took about 5 seconds, and the logs contained no DATA RACE report.

Step 8: Expire Old Keys

The key table gains a row per payment and never shrinks. Keep each key as long as a retry can plausibly arrive, then delete it; Stripe keeps keys for at least 24 hours, and so does this tutorial. To test without waiting a day, backdate 12 of the load test's 30 keys (test only):

docker exec pg psql -U postgres -d shop -c "UPDATE idempotency_keys SET created_at = now() - interval '25 hours' WHERE key IN (SELECT key FROM idempotency_keys ORDER BY key LIMIT 12)"

Output

UPDATE 12

The cleanup itself is one statement:

docker exec pg psql -U postgres -d shop -c "DELETE FROM idempotency_keys WHERE created_at < now() - interval '24 hours'"
docker exec pg psql -U postgres -d shop -c "SELECT count(*) FROM idempotency_keys"

Output

DELETE 12
 count 
-------
    18
(1 row)

The 12 old keys are gone and 18 remain. Run the DELETE from cron or a scheduled job, and at real volume add an index on created_at so it doesn't scan the whole table.

The IETF draft says the expiry policy SHOULD be documented, because after the window a retry with an old key is a new payment. Choose a window longer than any client's retry schedule, including apps that queue a payment offline.

When you're done, remove the container:

docker rm -f pg

Troubleshooting

Symptom	Cause	Fix
`missing go.sum entry for module providing package github.com/jackc/puddle/v2`	`go get` didn't record the modules pgx's connection pool imports	Run `go mod tidy`, as in Step 1
Duplicate charges even though the API checks keys	The client generates a new key per attempt, so every retry looks new (`-key new` reproduces it: two charges with two different provider keys)	Generate one key per operation, persist it with the pending payment, and send it on every attempt, as Stripe advises
Retries never return a 409; duplicates hang, then get a replay	The claim and the work share one transaction, so a duplicate waits on the uncommitted key row	Commit the claim in its own statement before the slow work, as in Step 4
A retry keeps failing long after the provider recovered	A fault was stored as the key's final answer	Release the key on faults, as in Step 5. Never switch to a new key until the original outcome is known
One customer receives another customer's receipt	Keys are unique globally rather than per customer	Scope keys per customer, as the `(customer_id, key)` primary key does, and include the customer in the key you send to the provider
Every retry gets 409 after a crash or deploy	A `started` key whose owner died, with no lease	Add a lease (Step 6)
Header rejected or never matched	The client sends the draft's quoted form and the server compares it raw	Strip one pair of surrounding quotes, as `idempotencyKey` does
`422` on a retry	The key was reused with a different payload, or the client changed the body between attempts	Keep the body fixed for a key; a new payment needs a new key
Card charged but no payment in the shop's ledger	A takeover re-ran a charge the provider had already made	Pass a derived key to the provider (Step 6)

Conclusion

You built idempotency keys in Go on PostgreSQL and broke each version before replacing it. A check-first lookup still charged twice. An atomic claim fixed the race, a fingerprint caught a reused key, and a lease recovered from a crash, until its own double charge needed a provider key. The final load test sent 150 copies of 30 payments, and both ledgers recorded exactly 30.

Retries are normal on a real network. Handling them costs one table, one extra write per payment, a scheduled cleanup and a provider key on every charge; without them, duplicates show up as refunds, disputes and support tickets.

Next, read Brandur Leach's Implementing Stripe-like Idempotency Keys in Postgres, which extends this design to several side effects with atomic phases and recovery points, and the AWS article on making retries safe with idempotent APIs. Then point Step 7's load test at your own payment path and compare the two ledgers.

If your team is building payment flows, webhooks or any path where a retry can move money twice, I can help design and build it, including the load test that proves the ledgers agree. You can see how I approach distributed systems, book a free 20-minute intro call, or tell me about your system.

Originally published at shahidyousuf.dev by Shahid Yousuf.

Distributed Locks in Redis with Go

Shahid Yousuf — Sat, 10 Oct 2026 09:00:41 +0000

Most backend services run as several identical copies behind a load balancer, so they can take more traffic and survive a crashed server. Each request reads some data, decides what to do and writes the result back. Inside one process, you'd guard that read-decide-write with a mutex. Across servers there's no shared memory to put a mutex in, so the guard has to live somewhere every copy can reach, such as a database row or a Redis distributed lock.

The gap shows up the first time two requests touch the same record at once. A user taps "Pay" twice on a slow connection, or a mobile client retries a payment the moment the first request times out. Two requests for the same wallet land on two different servers within the same millisecond. Both read a balance of 100, both decide 10 is affordable, and both write back 90. The user spent 20, you recorded 10, and nothing in the logs looks wrong.

The standard answer is a distributed lock : one shared place every server can reach that gives one request at a time exclusive access to a record. A database row lock, etcd or ZooKeeper can play that role; Redis is a popular choice because it's fast and often already in the stack. The naive version, one SetNX ("set if not exists": create the key only if nobody holds it) before the work and one Del (delete the key) after, breaks quietly: a crash leaves a lock that never expires, or a slow request deletes someone else's.

A lock kept in Redis is a lease, a key that expires after a set time whether or not you've finished. Take it with SET NX PX (set the key only if it doesn't exist, with an expiry in milliseconds) and a random token, release it only if the token matches, and keep the work shorter than the TTL (time to live, how long the key survives before Redis deletes it). When money is involved, also make the database refuse a write computed from a stale read; no lease can prove you still hold it. If you own the roadmap, the correctness work is a version column and a conditional UPDATE in the database you already run.

In this tutorial, you'll build a distributed lock on Redis and implement it in Go. You'll reproduce the race, fix it by hand, break the fix with a short TTL, switch to the bsm/redislock library, add fencing tokens that PostgreSQL checks, and load-test the result. Everything ran against Redis 8.10.2 and PostgreSQL 18, with go-redis v9.23.0 and bsm/redislock v0.10.0.

Prerequisites

To follow along, you need:

Go 1.26 or later, which go-redis v9.23.0 requires. With the default GOTOOLCHAIN=auto, go get downloads a newer toolchain if yours is older.
Docker, to run Redis and PostgreSQL locally.
Working knowledge of goroutines, context.Context and SQL. No prior Redis experience is needed.
Ports 6379 and 5432 free, or adjust the -p mappings and addresses to match.

Step 1: Reproduce the Race Without a Lock

This step reproduces the double payment on your machine, so every later fix has a baseline. One wallet balance lives in Redis, and 50 goroutines, each a "Pay" request on a different server, try to spend 10 from it at once. Predict how many will believe they can afford it.

Start Redis in a container and check that it answers:

docker run -d --name redis -p 6379:6379 redis:8.10.2


docker exec redis redis-cli ping

Output

PONG

Create a module and add go-redis, bsm/redislock and pgx, the PostgreSQL driver for Step 5. The second output line only appears if your local Go is older than 1.26:

mkdir walletlock && cd walletlock
go mod init example.com/walletlock
go get github.com/redis/go-redis/v9@v9.23.0 github.com/bsm/redislock@v0.10.0 github.com/jackc/pgx/v5@v5.11.0

Output

go: creating new go.mod: module example.com/walletlock
go: github.com/redis/go-redis/v9@v9.23.0 requires go >= 1.26.0; switching to go1.26.9
go: upgraded go 1.24.4 => 1.26.0
go: added github.com/bsm/redislock v0.10.0
go: added github.com/jackc/pgx/v5 v5.11.0
go: added github.com/redis/go-redis/v9 v9.23.0

Spend is a read-modify-write: it reads the balance, checks it and writes the new value back, so its decision rests on a value that can change before the write lands. The sleep stands in for the real work in between.

wallet/wallet.go

// Package wallet holds a demo balance in Redis and spends from it the naive way.
package wallet

import (
    "context"
    "fmt"
    "os"
    "sync"
    "time"

    "github.com/redis/go-redis/v9"
)

const BalanceKey = "wallet:42:balance"

// Connect returns a client for REDIS_ADDR, or localhost:6379 by default.
func Connect() *redis.Client {
    addr := os.Getenv("REDIS_ADDR")
    if addr == "" {
        addr = "localhost:6379"
    }
    return redis.NewClient(&redis.Options{
        Addr: addr,
        // Let context deadlines cut socket reads short. Without this,
        // go-redis waits out its own 5-second ReadTimeout instead.
        ContextTimeoutEnabled: true,
    })
}

// Spend is a read-modify-write: read the balance, check it, write it back.
// The sleep stands in for real work between the read and the write
// (validation, a fraud check, a call to another service).
func Spend(ctx context.Context, rdb *redis.Client, amount int64, work time.Duration) (bool, error) {
    bal, err := rdb.Get(ctx, BalanceKey).Int64()
    if err != nil {
        return false, err
    }
    if bal < amount {
        return false, nil
    }
    time.Sleep(work)
    return true, rdb.Set(ctx, BalanceKey, bal-amount, 0).Err()
}

// Hammer starts n goroutines at the same moment and counts how many
// calls to fn reported success.
func Hammer(n int, fn func() (bool, error)) (accepted int) {
    var wg sync.WaitGroup
    var mu sync.Mutex
    start := make(chan struct{})
    for range n {
        wg.Add(1)
        go func() {
            defer wg.Done()
            <-start
            ok, err := fn()
            if err != nil {
                fmt.Println("error:", err)
                return
            }
            if ok {
                mu.Lock()
                accepted++
                mu.Unlock()
            }
        }()
    }
    close(start)
    wg.Wait()
    return accepted
}

// Check compares the final balance with what the accepted spends imply.
func Check(initial, amount int64, accepted int, balance int64) {
    want := initial - amount*int64(accepted)
    status := "OK"
    if balance != want || balance < 0 {
        status = "BROKEN"
    }
    fmt.Printf("accepted=%d balance=%d expected=%d %s\n", accepted, balance, want, status)
}

Hammer releases every goroutine at once by closing a channel they all wait on. Check is the yardstick for every later step: 10 spends of 10 from 100 must leave 0.

The first program spends with no lock at all:

cmd/step1-race/main.go

package main

import (
    "context"
    "log"
    "time"

    "example.com/walletlock/wallet"
)

func main() {
    ctx := context.Background()
    rdb := wallet.Connect()
    defer rdb.Close()

    if err := rdb.Set(ctx, wallet.BalanceKey, 100, 0).Err(); err != nil {
        log.Fatal(err)
    }

    // 50 requests try to spend 10 each from a balance of 100.
    accepted := wallet.Hammer(50, func() (bool, error) {
        return wallet.Spend(ctx, rdb, 10, 2*time.Millisecond)
    })

    balance, _ := rdb.Get(ctx, wallet.BalanceKey).Int64()
    wallet.Check(100, 10, accepted, balance)
}

go run ./cmd/step1-race

Output

accepted=50 balance=80 expected=-400 BROKEN
accepted=50 balance=70 expected=-400 BROKEN
accepted=50 balance=60 expected=-400 BROKEN

Three of five runs are shown. All 50 requests were accepted, so 500 was spent from 100, yet the balance only dropped to 60 to 80: the ledger shows 50 approved payments and a wallet charged for two to four. Each goroutine read 100, decided it could pay and wrote its own new balance over the others. That failure is a lost update , and the rest of this tutorial fixes it.

Go's race detector doesn't catch it:

go run -race ./cmd/step1-race

Output

accepted=50 balance=70 expected=-400 BROKEN

There's no WARNING: DATA RACE and the exit code is 0. The race detector watches memory inside one Go process, and this shared state lives in Redis. The reliable test is the one Check does: compare the final balance with what the accepted operations imply.

Step 2: Write a Correct Lock by Hand

This step builds the lock: a way for one request to announce "I'm working on wallet 42, wait your turn" where every server can see it. Picture a sign on a door: whoever hangs "busy" first gets the room. In Redis the sign is a key, and it needs two properties: it disappears by itself if the holder crashes, and only the holder may take it down.

Open a Redis shell, take the lock as caller A, try it as caller B, and check A's remaining time:

docker exec -it redis redis-cli

redis-cli

127.0.0.1:6379> SET lock:wallet:42 token-a NX PX 30000
OK
127.0.0.1:6379> SET lock:wallet:42 token-b NX PX 30000
(nil)
127.0.0.1:6379> PTTL lock:wallet:42
(integer) 29884

NX sets the key only if it doesn't exist, so B gets (nil). PX 30000 attaches a 30-second expiry in the same command, which makes the lock the lease from the introduction: if the holder crashes, Redis removes the key. The value is a token unique to this caller, as on the official Redis distributed locks page.

Both must go in one command. The common SETNX then EXPIRE is two, and a crash between them leaves a lock that never expires. The SETNX page still documents an older SETNX-based lock only because existing code links to it, and points you to SET with a Lua release instead; the SET page even discourages the single-command lock in favor of Redlock, covered in the safety section.

The token matters on release. If A finishes late, its lease may have passed to B, and deleting the key by name would remove B's lock:

Using just DEL is not safe as a client may remove another client's lock.

Redis documentation, Distributed locks

Redis runs a Lua script atomically, so a script can compare the token and delete the key with nothing in between. Try a release with the wrong token, then the right one:

redis-cli

127.0.0.1:6379> EVAL 'if redis.call("GET", KEYS[1]) == ARGV[1] then return redis.call("DEL", KEYS[1]) end return 0' 1 lock:wallet:42 token-b
(integer) 0
127.0.0.1:6379> EVAL 'if redis.call("GET", KEYS[1]) == ARGV[1] then return redis.call("DEL", KEYS[1]) end return 0' 1 lock:wallet:42 token-a
(integer) 1

B's release returns 0 and leaves A's lock alone; A's returns 1. The key goes in KEYS, as the scripting docs require for every key a script touches. Exit the shell with exit.

In Go, Acquire hangs the sign, retrying until it succeeds or the context ends, and Release is the token-checked compare-and-delete:

lock/lock.go

// Package lock is a single-node Redis lease: SET NX PX plus a random token.
package lock

import (
    "context"
    "crypto/rand"
    "errors"
    "fmt"
    mrand "math/rand/v2"
    "time"

    "github.com/redis/go-redis/v9"
)

var (
    ErrNotHeld = errors.New("lock not held (expired or taken by someone else)")
    ErrBadTTL  = errors.New("lock TTL must be at least 1ms")
)

// Delete the key only if it still holds our token.
var releaseScript = redis.NewScript(`
if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
end
return 0`)

type Lock struct {
    rdb   *redis.Client
    Key   string
    Token string
    Fence int64 // set by AcquireFenced, zero otherwise
}

// Acquire retries SET key token NX PX ttl until it wins or ctx ends.
func Acquire(ctx context.Context, rdb *redis.Client, key string, ttl time.Duration) (*Lock, error) {
    // A zero or negative TTL would create a lock that never expires.
    if ttl < time.Millisecond {
        return nil, ErrBadTTL
    }
    token := rand.Text() // 26 random base32 characters
    for {
        ok, err := rdb.SetNX(ctx, key, token, ttl).Result()
        if err != nil {
            return nil, err
        }
        if ok {
            return &Lock{rdb: rdb, Key: key, Token: token}, nil
        }
        // Busy: wait 5 to 15 ms, randomized so waiters don't retry in lockstep.
        wait := 5*time.Millisecond + mrand.N(10*time.Millisecond)
        select {
        case <-ctx.Done():
            return nil, fmt.Errorf("acquire %s: %w", key, ctx.Err())
        case <-time.After(wait):
        }
    }
}

// Release deletes the lock only if we still own it.
func (l *Lock) Release(ctx context.Context) error {
    n, err := releaseScript.Run(ctx, l.rdb, []string{l.Key}, l.Token).Int()
    if err != nil {
        return err
    }
    if n == 0 {
        return ErrNotHeld
    }
    return nil
}

rand.Text (Go 1.24 and later) returns at least 128 bits of randomness. releaseScript.Run falls back from EVALSHA (run a script Redis has cached, by its hash) to EVAL (send the whole script) when a restart or failover has emptied the volatile script cache. While the lock is busy, Acquire waits a random 5 to 15 ms; without that randomness, waiters retry in lockstep, and Step 4 measures the cost.

A zero TTL means no expiry

In go-redis, SetNX with a TTL of 0 or redis.KeepTTL sends no expiry, so a new lock never expires. An unset config value can do this silently, which is why Acquire rejects anything under 1 ms.

Wrap the spend in the lock and run it:

cmd/step2-lock/main.go

package main

import (
    "context"
    "log"
    "time"

    "example.com/walletlock/lock"
    "example.com/walletlock/wallet"
)

const lockKey = "lock:wallet:42"

func main() {
    ctx := context.Background()
    rdb := wallet.Connect()
    defer rdb.Close()

    if err := rdb.Set(ctx, wallet.BalanceKey, 100, 0).Err(); err != nil {
        log.Fatal(err)
    }

    accepted := wallet.Hammer(50, func() (bool, error) {
        // Give up waiting for the lock after 5 seconds.
        actx, cancel := context.WithTimeout(ctx, 5*time.Second)
        defer cancel()

        l, err := lock.Acquire(actx, rdb, lockKey, 2*time.Second)
        if err != nil {
            return false, err
        }
        defer func() {
            if err := l.Release(ctx); err != nil {
                log.Println("release:", err)
            }
        }()
        return wallet.Spend(ctx, rdb, 10, 2*time.Millisecond)
    })

    balance, _ := rdb.Get(ctx, wallet.BalanceKey).Int64()
    wallet.Check(100, 10, accepted, balance)
}

go run ./cmd/step2-lock

Output

accepted=10 balance=0 expected=0 OK
accepted=10 balance=0 expected=0 OK
accepted=10 balance=0 expected=0 OK

Exactly 10 spends were accepted and the balance was 0 on all five runs (three shown). The lock turned the burst into a queue, so the double tap now ends as the user expects: the second request waits, reads 90 and pays from that. Confirm no lock was left behind:

docker exec -it redis redis-cli EXISTS lock:wallet:42

Output

(integer) 0

On Redis 8.4 and later, DELEX does the compare-and-delete natively and the Redis docs recommend it there; go-redis v9.23.0 exposes it as DelExArgs, marked experimental:

Lua script (any Redis)

n, err := releaseScript.Run(ctx, l.rdb, []string{l.Key}, l.Token).Int()

Works on any Redis that allows scripting, so it's the portable default.

DELEX (Redis 8.4 and later)

// releaseDelEx is the Redis 8.4+ alternative to the Lua release script.
// go-redis marks DelExArgs as experimental, so its signature may change.
func releaseDelEx(ctx context.Context, rdb *redis.Client, l *lock.Lock) error {
    n, err := rdb.DelExArgs(ctx, l.Key, redis.DelExArgs{Mode: "IFEQ", MatchValue: l.Token}).Result()
    if err != nil {
        return err
    }
    if n == 0 {
        return lock.ErrNotHeld
    }
    return nil
}

Calling it twice on the same lock prints:

first release:  <nil>
second release: lock not held (expired or taken by someone else)

Against Redis 7.4, the same code fails with ERR unknown command 'delex'. Redis Cloud and Redis Software support it; Valkey 9.0 and later uses its own DELIFEQ key value, so check your provider before switching.

Step 3: Handle Work That Outlives the TTL

This step breaks the lock by letting the work outlast the lease, then fixes it. A slow query or a long garbage-collection pause can stretch work past the TTL, and the lease expires regardless: the sign falls off while you're still inside. The fix is a watchdog , a background goroutine that keeps refreshing the lease while the work runs.

Add a token-checked refresh to lock/lock.go, plus a Watch that refreshes every third of the TTL and cancels a context the moment a refresh fails:

lock/lock.go (add to the file)

// Extend the TTL only if the key still holds our token.
var refreshScript = redis.NewScript(`
if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("PEXPIRE", KEYS[1], ARGV[2])
end
return 0`)

// Refresh resets the TTL only if we still own the lock.
func (l *Lock) Refresh(ctx context.Context, ttl time.Duration) error {
    n, err := refreshScript.Run(ctx, l.rdb, []string{l.Key}, l.Token, ttl.Milliseconds()).Int()
    if err != nil {
        return err
    }
    if n == 0 {
        return ErrNotHeld
    }
    return nil
}

// Watch refreshes the lock every ttl/3 and cancels the returned context
// as soon as a refresh fails. Do the protected work with that context.
func (l *Lock) Watch(parent context.Context, ttl time.Duration) (context.Context, context.CancelFunc) {
    ctx, cancel := context.WithCancelCause(parent)
    go func() {
        t := time.NewTicker(ttl / 3)
        defer t.Stop()
        for {
            select {
            case <-ctx.Done():
                return
            case <-t.C:
                // Bound each refresh: a hung call must not outlast the lease.
                rctx, rcancel := context.WithTimeout(ctx, ttl/3)
                err := l.Refresh(rctx, ttl)
                rcancel()
                if err != nil {
                    cancel(fmt.Errorf("lock lost: %w", err))
                    return
                }
            }
        }
    }()
    return ctx, func() { cancel(context.Canceled) }
}

Each refresh gets its own TTL/3 deadline, or a refresh against an unresponsive Redis waits out go-redis's default 5-second ReadTimeout. That deadline only works because Connect sets ContextTimeoutEnabled: when I paused the Redis container, a call with a 100 ms deadline took 5 s to fail on a default client and 100 ms with the option on.

Pass the returned context into the protected work so the next call fails once the watchdog sees the lock is gone. It checks only every TTL/3, which narrows the window without closing it.

This program runs 5 callers whose 300 ms of work is three times their 100 ms TTL. Without the watchdog, expect them to overlap:

cmd/step3-ttl/main.go

package main

import (
    "context"
    "errors"
    "flag"
    "log"
    "sync/atomic"
    "time"

    "example.com/walletlock/lock"
    "example.com/walletlock/wallet"
)

const lockKey = "lock:wallet:42"

func main() {
    ttl := flag.Duration("ttl", 100*time.Millisecond, "lock TTL")
    work := flag.Duration("work", 300*time.Millisecond, "time spent while holding the lock")
    n := flag.Int("n", 5, "concurrent callers")
    watchdog := flag.Bool("watchdog", false, "refresh the lock every ttl/3")
    flag.Parse()

    ctx := context.Background()
    rdb := wallet.Connect()
    defer rdb.Close()
    if err := rdb.Set(ctx, wallet.BalanceKey, 100, 0).Err(); err != nil {
        log.Fatal(err)
    }

    var staleReleases atomic.Int64
    start := time.Now()
    accepted := wallet.Hammer(*n, func() (bool, error) {
        actx, cancel := context.WithTimeout(ctx, 10*time.Second)
        defer cancel()
        l, err := lock.Acquire(actx, rdb, lockKey, *ttl)
        if err != nil {
            return false, err
        }
        defer func() {
            if err := l.Release(ctx); errors.Is(err, lock.ErrNotHeld) {
                staleReleases.Add(1)
            }
        }()

        workCtx := ctx
        if *watchdog {
            var stop context.CancelFunc
            workCtx, stop = l.Watch(ctx, *ttl)
            defer stop()
        }
        ok, err := wallet.Spend(workCtx, rdb, 10, *work)
        if err == nil && workCtx.Err() != nil {
            log.Println(context.Cause(workCtx))
        }
        return ok, err
    })

    balance, _ := rdb.Get(ctx, wallet.BalanceKey).Int64()
    wallet.Check(100, 10, accepted, balance)
    log.Printf("refused stale releases=%d elapsed=%s", staleReleases.Load(), time.Since(start).Round(10*time.Millisecond))
}

Run it without the watchdog, then with it:

go run ./cmd/step3-ttl
go run ./cmd/step3-ttl -watchdog

Output

accepted=5 balance=80 expected=50 BROKEN
2026/10/09 17:17:57 refused stale releases=5 elapsed=740ms

accepted=5 balance=50 expected=50 OK
2026/10/09 17:18:02 refused stale releases=0 elapsed=1.52s

One run of each is shown; all five runs of each gave the same counts.

Without the watchdog, the callers overlapped and the balance lost three of five debits. All 5 releases were refused, because each lock had expired or changed hands, and Step 2's token check kept each caller from deleting someone else's. On call, this looks like lock not held errors and a balance that doesn't match the payments.

With the watchdog, the callers ran one after another (about 1.5 s for 5 jobs of 300 ms) and the balance is correct. But the watchdog runs alongside the work, which is also its weakness:

A watchdog can't save a paused process

The watchdog is a goroutine in the same process, so a stop-the-world pause, a VM migration or a frozen container stops it too. The lease expires and your code resumes believing it still holds the lock. Redis also doesn't use a monotonic clock for TTL expiry, so a wall-clock jump can expire a lock early. Steps 5 and 6 make the database catch what slips through.

Step 4: Use bsm/redislock with Jittered Retries

This step swaps the hand-written lock for a library and fixes how callers wait. In a real service I'd rather not maintain lock code, so it uses bsm/redislock, a small single-node library on go-redis that you can read in one sitting.

A library also reopens a question the hand-written lock answered quietly: how a caller waits. Exponential backoff waits longer after each failed try, on the same schedule for every caller. Picture 50 people turned away at once and told to come back in 16 ms three times, then 32, then 64, then every 128 ms. Each time, they all return together, one gets in, and the rest are sent away again.

Several of bsm/redislock's v0.10.0 defaults behave differently from what you might assume:

Behavior in v0.10.0	What it means for you
Retries are off by default	Without a `RetryStrategy`, `Obtain` tries once and fails if the lock is busy
No deadline on `ctx` means the deadline becomes the TTL	A caller never waits longer than the TTL unless you pass your own deadline
No watchdog	Call `Lock.Refresh` yourself; the README's example refreshes every TTL/3, like Step 3
A wait cut short by the context returns `errors.Join(ErrNotObtained, ctx.Err())`	Compare with `errors.Is`, never `==`
`ExponentialBackoff(min, max)` has no jitter; it and `LimitRetry` keep state	Add jitter yourself. Strategies are safe to share across goroutines (since v0.7.1), but counts carry over, so a shared `LimitRetry` becomes a global limit; build one per `Obtain`

This program obtains the lock 50 times with exponential backoff between 16 ms and 128 ms, plus an optional wrapper that adds jitter , a random spread on each wait:

cmd/step4-redislock/main.go

package main

import (
    "context"
    "errors"
    "flag"
    "log"
    "math/rand/v2"
    "sync/atomic"
    "time"

    "github.com/bsm/redislock"

    "example.com/walletlock/wallet"
)

// fullJitter waits a random time in [1ms, d+1ms), where d is the inner
// strategy's backoff, so callers that failed together don't retry together.
// The 1ms floor matters: redislock treats a backoff under 1ns as "stop
// retrying", so a bare rand.N(d) that returned 0 would end retries silently.
type fullJitter struct{ inner redislock.RetryStrategy }

func (j fullJitter) NextBackoff() time.Duration {
    d := j.inner.NextBackoff()
    if d <= 0 {
        return d // 0 means stop retrying
    }
    return time.Millisecond + rand.N(d)
}

func main() {
    jitter := flag.Bool("jitter", false, "randomize each backoff")
    flag.Parse()

    ctx := context.Background()
    rdb := wallet.Connect()
    defer rdb.Close()
    locker := redislock.New(rdb)
    if err := rdb.Set(ctx, wallet.BalanceKey, 100, 0).Err(); err != nil {
        log.Fatal(err)
    }

    var timedOut, equalsMatched atomic.Int64
    start := time.Now()
    accepted := wallet.Hammer(50, func() (bool, error) {
        // A new strategy per Obtain: the exponential one counts attempts.
        var retry redislock.RetryStrategy = redislock.ExponentialBackoff(16*time.Millisecond, 128*time.Millisecond)
        if *jitter {
            retry = fullJitter{retry}
        }

        actx, cancel := context.WithTimeout(ctx, 5*time.Second)
        defer cancel()
        l, err := locker.Obtain(actx, "lock:wallet:42", 2*time.Second, &redislock.Options{RetryStrategy: retry})
        if errors.Is(err, redislock.ErrNotObtained) {
            timedOut.Add(1)
            if err == redislock.ErrNotObtained {
                equalsMatched.Add(1)
            }
            return false, nil
        } else if err != nil {
            return false, err
        }
        defer l.Release(ctx)

        return wallet.Spend(ctx, rdb, 10, 2*time.Millisecond)
    })

    balance, _ := rdb.Get(ctx, wallet.BalanceKey).Int64()
    wallet.Check(100, 10, accepted, balance)
    log.Printf("gave up=%d (err == ErrNotObtained matched %d) elapsed=%s",
        timedOut.Load(), equalsMatched.Load(), time.Since(start).Round(10*time.Millisecond))
}

Run it without jitter, then with it:

go run ./cmd/step4-redislock
go run ./cmd/step4-redislock -jitter

Output

accepted=10 balance=0 expected=0 OK
2026/10/09 17:18:25 gave up=5 (err == ErrNotObtained matched 0) elapsed=5s
accepted=10 balance=0 expected=0 OK
2026/10/09 17:18:28 gave up=0 (err == ErrNotObtained matched 0) elapsed=2.91s

accepted=10 balance=0 expected=0 OK
2026/10/09 17:18:37 gave up=0 (err == ErrNotObtained matched 0) elapsed=130ms
accepted=10 balance=0 expected=0 OK
2026/10/09 17:18:38 gave up=0 (err == ErrNotObtained matched 0) elapsed=180ms

Two of five runs of each are shown; counts and timings vary.

Both versions keep the balance correct, so the difference is availability. Without jitter, 0 to 5 of the 50 callers gave up after waiting 5 seconds. They failed together, backed off on the same schedule and woke together, so each round let through roughly one caller, and some never got a turn before the 5-second deadline. This is a thundering herd : for the user, a "payment failed" screen after a five-second wait.

Full jitter spreads the wake-ups: each caller waits a random time up to its backoff, and all 50 finished in 130 to 180 ms on every run.

The == count shows a separate bug. Every caller that gave up returned an error that errors.Is(err, redislock.ErrNotObtained) matched and == did not: v0.10.0 joins the context error onto ErrNotObtained whenever a retry wait is cut short by the context deadline, including the default one it derives from the TTL. The README's example passes no retry strategy, so its == works there; add a strategy and it silently stops matching.

Step 5: Add Fencing Tokens So PostgreSQL Rejects Stale Writers

This step makes the database reject a write from a holder that lost its lease without knowing. Say the first tap's request (A) takes the lock, reads 100 and stalls in a garbage-collection pause longer than its lease. The retry (B) takes the lock, pays and finishes. Then A wakes up, still sure it holds the lock.

Think of a deli counter that serves only a ticket higher than the last one it served: ticket 33, back after 34, gets turned away. Martin Kleppmann's answer to the paused holder works the same way. A fencing token is a number that increases with every acquisition and travels with every write; the storage refuses any token lower than the highest it has accepted. With the balance in PostgreSQL, the check happens in the statement that changes it:

View this diagram in the original post

Client A wakes up holding a lease that expired. Its write carries token 33, PostgreSQL has already accepted 34, so the write changes nothing.

Start PostgreSQL with a throwaway password for this container only, and wait until pg_isready reports accepting connections:

export POSTGRES_PASSWORD=your_local_password
docker run -d --name postgres -e POSTGRES_PASSWORD="$POSTGRES_PASSWORD" -p 5432:5432 postgres:18
export DATABASE_URL="postgres://postgres:${POSTGRES_PASSWORD}@localhost:5432/postgres"
docker exec postgres pg_isready -U postgres

Output

/var/run/postgresql:5432 - accepting connections

Create the table. fence records the highest token that has written, and the CHECK constraint is a last guard against a negative balance. Then load it:

schema.sql

CREATE TABLE wallets (
    id      bigint PRIMARY KEY,
    balance bigint NOT NULL CHECK (balance >= 0),
    fence   bigint NOT NULL DEFAULT 0  -- highest fencing token that has written
);

INSERT INTO wallets (id, balance) VALUES (42, 100);

docker exec -i postgres psql -U postgres < schema.sql
docker exec -it postgres psql -U postgres -c 'SELECT * FROM wallets'

Output

CREATE TABLE
INSERT 0 1
 id | balance | fence 
----+---------+-------
 42 |     100 |     0
(1 row)

Redis issues the tokens with INCR, which adds one to a counter and returns the new value. The script takes the lock and increments the counter atomically, so only a caller that got the lock gets a token, and it rejects a TTL under 1 ms (Redis refuses PX 0). Both keys share the hash tag {wallet:42}, because Redis Cluster requires every key a script declares to hash to the same slot and hashes only the text inside the first braces.

lock/fenced.go

package lock

import (
    "context"
    "crypto/rand"
    "errors"
    "fmt"
    mrand "math/rand/v2"
    "time"

    "github.com/redis/go-redis/v9"
)

// Take the lock and, only if that worked, bump the fencing counter.
// Both keys share the {wallet:42} hash tag, so they live in the same
// Redis Cluster slot and the script can touch both.
var acquireFencedScript = redis.NewScript(`
if redis.call("SET", KEYS[1], ARGV[1], "NX", "PX", ARGV[2]) then
  return redis.call("INCR", KEYS[2])
end
return false`)

// AcquireFenced works like Acquire but also returns a fencing token:
// a number that grows by one every time anyone takes this lock.
func AcquireFenced(ctx context.Context, rdb *redis.Client, key, fenceKey string, ttl time.Duration) (*Lock, error) {
    // Under 1ms, PX would be 0, which Redis rejects as an invalid expire time.
    if ttl < time.Millisecond {
        return nil, ErrBadTTL
    }
    token := rand.Text()
    for {
        fence, err := acquireFencedScript.Run(ctx, rdb, []string{key, fenceKey}, token, ttl.Milliseconds()).Int64()
        if err == nil {
            return &Lock{rdb: rdb, Key: key, Token: token, Fence: fence}, nil
        }
        if !errors.Is(err, redis.Nil) {
            return nil, err
        }
        wait := 5*time.Millisecond + mrand.N(10*time.Millisecond)
        select {
        case <-ctx.Done():
            return nil, fmt.Errorf("acquire %s: %w", key, ctx.Err())
        case <-time.After(wait):
        }
    }
}

The next program replays the diagram with A pausing 250 ms against a 100 ms TTL, and seeds the counter so the tokens are 33 and 34. With the check, expect A's write to bounce; without it, expect Step 1's bug:

cmd/step5-fencing/main.go

package main

import (
    "context"
    "flag"
    "fmt"
    "log"
    "os"
    "time"

    "github.com/jackc/pgx/v5/pgxpool"

    "example.com/walletlock/lock"
    "example.com/walletlock/wallet"
)

const (
    lockKey  = "lock:{wallet:42}"
    fenceKey = "fence:{wallet:42}"
    ttl      = 100 * time.Millisecond
)

// The write succeeds only if nobody with a newer token has written since.
const fencedUpdate = `
UPDATE wallets SET balance = $1, fence = $2
WHERE id = 42 AND fence < $2`

const unfencedUpdate = `
UPDATE wallets SET balance = $1, fence = $2
WHERE id = 42`

func main() {
    fenced := flag.Bool("fenced", true, "check the fencing token in the UPDATE")
    flag.Parse()

    ctx := context.Background()
    db, err := pgxpool.New(ctx, os.Getenv("DATABASE_URL"))
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()
    rdb := wallet.Connect()
    defer rdb.Close()

    // Fresh state: balance 100, and a fencing counter that hands out 33 next.
    db.Exec(ctx, `UPDATE wallets SET balance = 100, fence = 0 WHERE id = 42`)
    rdb.Set(ctx, fenceKey, 32, 0)
    rdb.Del(ctx, lockKey)

    update := unfencedUpdate
    if *fenced {
        update = fencedUpdate
    }
    spend := func(name string, l *lock.Lock, read int64) {
        tag, err := db.Exec(ctx, update, read-10, l.Fence)
        if err != nil {
            log.Fatal(err)
        }
        if tag.RowsAffected() == 0 {
            fmt.Printf("%s: write with fence %d REJECTED\n", name, l.Fence)
            return
        }
        fmt.Printf("%s: wrote balance %d with fence %d\n", name, read-10, l.Fence)
    }
    readBalance := func() (b int64) {
        db.QueryRow(ctx, `SELECT balance FROM wallets WHERE id = 42`).Scan(&b)
        return b
    }

    a, err := lock.AcquireFenced(ctx, rdb, lockKey, fenceKey, ttl)
    if err != nil {
        log.Fatal(err)
    }
    aRead := readBalance()
    fmt.Printf("A: got the lock, fence %d, read balance %d\n", a.Fence, aRead)

    fmt.Println("A: paused for 250ms (GC pause, slow network, stopped VM)")
    time.Sleep(250 * time.Millisecond)

    b, err := lock.AcquireFenced(ctx, rdb, lockKey, fenceKey, ttl)
    if err != nil {
        log.Fatal(err)
    }
    bRead := readBalance()
    fmt.Printf("B: got the lock, fence %d, read balance %d\n", b.Fence, bRead)
    spend("B", b, bRead)
    b.Release(ctx)

    fmt.Println("A: wakes up and carries on")
    spend("A", a, aRead)
    if err := a.Release(ctx); err != nil {
        fmt.Println("A: release:", err)
    }

    var balance, fence int64
    db.QueryRow(ctx, `SELECT balance, fence FROM wallets WHERE id = 42`).Scan(&balance, &fence)
    fmt.Printf("final: balance=%d fence=%d (two spends of 10 were attempted)\n", balance, fence)
}

Run it with the fencing check, then without:

go run ./cmd/step5-fencing
go run ./cmd/step5-fencing -fenced=false

Output

A: got the lock, fence 33, read balance 100
A: paused for 250ms (GC pause, slow network, stopped VM)
B: got the lock, fence 34, read balance 100
B: wrote balance 90 with fence 34
A: wakes up and carries on
A: write with fence 33 REJECTED
A: release: lock not held (expired or taken by someone else)
final: balance=90 fence=34 (two spends of 10 were attempted)

A: got the lock, fence 33, read balance 100
A: paused for 250ms (GC pause, slow network, stopped VM)
B: got the lock, fence 34, read balance 100
B: wrote balance 90 with fence 34
A: wakes up and carries on
A: wrote balance 90 with fence 33
A: release: lock not held (expired or taken by someone else)
final: balance=90 fence=33 (two spends of 10 were attempted)

With the check, A's stale write bounces and only B's payment counts. Without it, both believe they spent 10 and the balance shows one debit: two approved payments, one charge, the Step 1 bug with a lock in place. A's code couldn't have caught it, because A held a valid lease when it read. Only the database, which sees every write in order, could refuse.

The counter lives in Redis, so a failover can lose recent increments and reissue a token. The strict fence < $2 limits the damage: a repeated token can win at most once, and any token at or below the stored fence is refused until the counter passes max(fence). Because the fence key has no TTL, keep allkeys-* eviction policies off any Redis that holds locks.

Fencing also doesn't stop two overlapping holders with different tokens from both writing, which is what Step 6 deals with.

Step 6: Verify Under Load by Checking the Balance

This step load-tests the whole path with the lock deliberately broken and checks the final balance. It runs the fenced lock, the read from PostgreSQL and the conditional write with 50 callers. With a 1 ms TTL the lock barely holds, so any protection left comes from the database. The -same-version flag adds one more condition to the UPDATE.

cmd/step6-verify/main.go

package main

import (
    "context"
    "flag"
    "log"
    "os"
    "sync/atomic"
    "time"

    "github.com/jackc/pgx/v5/pgxpool"

    "example.com/walletlock/lock"
    "example.com/walletlock/wallet"
)

func main() {
    ttl := flag.Duration("ttl", 2*time.Second, "lock TTL")
    sameVersion := flag.Bool("same-version", false, "also require the fence we read to be unchanged")
    flag.Parse()

    ctx := context.Background()
    db, err := pgxpool.New(ctx, os.Getenv("DATABASE_URL"))
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()
    rdb := wallet.Connect()
    defer rdb.Close()
    db.Exec(ctx, `UPDATE wallets SET balance = 100, fence = 0 WHERE id = 42`)
    rdb.Del(ctx, "fence:{wallet:42}", "lock:{wallet:42}")

    update := `UPDATE wallets SET balance = $1, fence = $2
               WHERE id = 42 AND fence < $2`
    if *sameVersion {
        update += ` AND fence = $3`
    }

    var rejected atomic.Int64
    accepted := wallet.Hammer(50, func() (bool, error) {
        actx, cancel := context.WithTimeout(ctx, 5*time.Second)
        defer cancel()
        l, err := lock.AcquireFenced(actx, rdb, "lock:{wallet:42}", "fence:{wallet:42}", *ttl)
        if err != nil {
            return false, err
        }
        defer l.Release(ctx)

        var balance, seen int64
        err = db.QueryRow(ctx, `SELECT balance, fence FROM wallets WHERE id = 42`).Scan(&balance, &seen)
        if err != nil || balance < 10 {
            return false, err
        }
        time.Sleep(2 * time.Millisecond)

        args := []any{balance - 10, l.Fence}
        if *sameVersion {
            args = append(args, seen)
        }
        tag, err := db.Exec(ctx, update, args...)
        if err != nil {
            return false, err
        }
        if tag.RowsAffected() == 0 {
            rejected.Add(1)
            return false, nil
        }
        return true, nil
    })

    var balance int64
    db.QueryRow(ctx, `SELECT balance FROM wallets WHERE id = 42`).Scan(&balance)
    wallet.Check(100, 10, accepted, balance)
    log.Printf("writes rejected by the database=%d", rejected.Load())
}

Run it with a healthy 2-second TTL, then with a 1 ms TTL:

go run ./cmd/step6-verify
go run ./cmd/step6-verify -ttl 1ms

Output

accepted=10 balance=0 expected=0 OK
2026/10/09 17:18:39 writes rejected by the database=0

accepted=17 balance=0 expected=-70 BROKEN
2026/10/09 17:18:40 writes rejected by the database=2
accepted=20 balance=0 expected=-100 BROKEN
2026/10/09 17:18:40 writes rejected by the database=1

One 2-second run and two of five 1 ms runs are shown. All five 2-second runs were correct. With a 1 ms TTL, fencing rejected 1 or 2 writes per run and still let 17 to 20 spends through: the token did its job and the balance is still wrong.

The reason is the read. Fencing orders writes but says nothing about what the writer read. With overlapping leases, A (token 33) and B (token 34) can both read 100; A writes 90 first, which is legal, then B writes 90 with a higher token, also legal. B's new balance came from a stale value, and its token couldn't show it.

The fix makes the write depend on what was read. Salvatore Sanfilippo, the creator of Redis, described it in his reply to Kleppmann:

When starting to work with a shared resource, we set its state to "<token>", then we operate the read-modify-write only if the token is still the same when we write.

Salvatore Sanfilippo, Is Redlock safe?

In SQL, the write also requires the fence value it read to be unchanged. This is check-and-set : read a version, do the work, and write only if the version is still the one you read. Because every successful write raises fence, a version never repeats, so a value that changed and changed back can't fool the check:

UPDATE wallets SET balance = $1, fence = $2
WHERE id = 42 AND fence < $2 AND fence = $3

Run the load test with that condition:

go run ./cmd/step6-verify -ttl 1ms -same-version
go run ./cmd/step6-verify -same-version

Output

accepted=10 balance=0 expected=0 OK
2026/10/09 17:18:40 writes rejected by the database=9
accepted=10 balance=0 expected=0 OK
2026/10/09 17:18:40 writes rejected by the database=11

accepted=10 balance=0 expected=0 OK
2026/10/09 17:18:41 writes rejected by the database=0

Two of five 1 ms runs and one of five 2-second runs are shown; all ten runs came out correct.

Once the version check is in place, the database keeps the balance correct on its own, and the lock's job is to make conflicts rare. With a healthy lease no write was rejected; with a broken one, 9 to 11 of the 50 callers were turned away, each an error the caller can retry, not a wrong balance.

When you're done, remove the containers:

docker rm -f redis postgres

Is a Redis Distributed Lock Safe?

It depends on what happens when the lock fails. Kleppmann separates an efficiency lock, whose failure means duplicate work, from a correctness lock, whose failure corrupts data. The wallet needs the second kind.

Single-node Redis is a good efficiency lock and an unreliable correctness lock; the Redis docs call a single instance viable "in applications where a race condition from time to time is acceptable". Replication is asynchronous, so a primary can acknowledge your lock and crash before a replica gets it, letting a second client take the lock on the promoted replica: the docs' "SAFETY VIOLATION!". A restart without appendfsync always can lose the key too, and WAIT doesn't make Redis strongly consistent.

Redlock, described on the same Redis locks page, takes the lock on a majority of independent servers to survive losing one. In February 2016, Martin Kleppmann, author of Designing Data-Intensive Applications, published a critique, and Sanfilippo, who designed Redlock, replied the next day. Read both in full; this summary drops most of the nuance.

Kleppmann judged Redlock against an asynchronous model, where network delays, process pauses and clock errors can be arbitrarily long. Redlock, he argued, assumes they stay small relative to the TTL, and it can't generate fencing tokens, so he called it "neither fish nor fowl" and recommended a consensus system such as ZooKeeper, or at least a database with reasonable transactional guarantees.
Sanfilippo replied that Redlock needs only a semi-synchronous model, where processes measure time at roughly the same rate, which holds as long as nobody steps the system clock (a slewing NTP daemon adjusts it gradually). He argued its random token supports check-and-set, the idea Step 6 applies, and agreed Redis should use a monotonic clock for expiry; the docs still say it doesn't.

You don't have to settle the argument to build something safe. Once the storage enforces the invariant, a lock that occasionally fails costs retries, so the choice that matters is where that enforcement lives:

Approach	Use it when	Cost or risk
Single Redis node, `SET NX PX` plus token	Efficiency, or correctness with a database guard behind it	Loses locks on failover of a single primary, even with replicas, or on a restart without `appendfsync always`
Redlock, for example redsync v4.18.0	You want fewer lost locks and can run an odd number of independent primaries (the docs use 5)	Timing assumptions, no fencing tokens, more servers to operate. redsync defaults to an 8 s expiry, 32 tries and a random 50 to 250 ms delay; its README says the algorithm hasn't been formally analyzed
Majority across keys, rueidislock (rueidis v1.0.78)	You use the rueidis client and want automatic extension	5 s key validity, extended every 2.5 s, 2 of 3 keys by default (`KeyMajority: 1` for one instance); its context is canceled when the lock is lost; no fencing tokens
Atomic Lua script, or `SET ... IFEQ` on Redis 8.4+, on the value itself	The balance lives in Redis	No lock needed, but the logic must fit in one script, and acknowledged writes can be lost on failover, so the balance is only as durable as Redis replication and persistence
`SELECT ... FOR UPDATE`, a version column, a unique constraint or an idempotency key	The balance lives in PostgreSQL	Usually the right first answer; row locks need a consistent lock order to avoid deadlocks
etcd `concurrency.Mutex`	You need a lock across services and no database can arbitrate	Another distributed system to run; a paused holder can still write late, so keep a check in storage too

If the balance is already in PostgreSQL, start there. SELECT ... FOR UPDATE serializes the read-modify-write inside a transaction; when one transaction locks several rows, lock them in a consistent order to avoid deadlocks. If the operation fits in one statement, SET balance = balance - 10 WHERE balance >= 10 needs no explicit lock: under READ COMMITTED, a second concurrent UPDATE waits for the first and then re-checks its WHERE clause against the updated row.

Troubleshooting

Symptom	Cause	Fix
`Bind for 0.0.0.0:6379 failed: port is already allocated`	Another Redis or container already uses the port	Map a different host port, such as `-p 6380:6379`, and set `REDIS_ADDR=localhost:6380`
A lock key with `PTTL` of `-1` that never goes away	The lock was set with TTL 0 or `redis.KeepTTL` (go-redis sends no expiry), or with `SETNX` then `EXPIRE`	Always pass a non-zero TTL in the same `SET` command; delete the stuck key by hand
`lock not held` on release	The work took longer than the TTL, so the lease expired or passed to another caller	Raise the TTL, add a watchdog, and alert on this error: your lease expired mid-work, so another caller may have overlapped with you
Callers time out even though the work is short	Retries without jitter wake in lockstep	Randomize each backoff, as in Step 4
Busy-lock errors logged as Redis failures	`err == redislock.ErrNotObtained` fails on a joined error	Use `errors.Is(err, redislock.ErrNotObtained)`
`ERR unknown command 'delex'` (uppercase in redis-cli)	Redis older than 8.4, or a service without it	Use the Lua release script, or `DELIFEQ` on Valkey 9.0 and later
`FATAL: the database system is starting up`	PostgreSQL was still initializing when the program connected	Wait until `docker exec postgres pg_isready -U postgres` reports `accepting connections`
Balance wrong even with fencing	A newer holder read stale data before an older holder wrote	Also require the version you read to be unchanged, as in Step 6

Conclusion

You built a distributed lock on Redis in Go and load-tested it until it broke. A healthy lease kept the balance right; a lease far too short let 17 to 20 spends through where 10 should have, even with fencing, until the write also checked the version it read. A lease holds only while its timing assumptions do: the pause stays shorter than the TTL and the clocks behave. The database sees every write in order and can check each one, so the final check belongs there.

For a team deciding how much to invest, treat a Redis lock as a performance tool and put correctness in the database: a version column, a conditional UPDATE and a load test that checks the final balance. Ask for that load test in code review, and ask what the final balance was.

Next, read the Redis distributed locks page and Kleppmann's How to do distributed locking back to back, then point the Step 6 load test at your own write path with a 1 ms TTL. If the balance survives that, your lock is free to do its real job of keeping contention down.

If your team is building a path where a lock, a reservation or an event moves money between services, I can help design and build it, including the load test that proves the balance holds. You can see how I approach distributed systems, book a free 20-minute intro call, or tell me about your system.

GitHub Actions OIDC Trust Policy for AWS

Shahid Yousuf — Fri, 09 Oct 2026 06:02:53 +0000

Plenty of AWS accounts have an IAM user that exists only so a GitHub Actions workflow can push an image or run a deploy. Its access key sits in a repository secret and never expires. GitHub Actions OIDC is the better answer.

The trust policy is where this goes wrong, and it got harder this year. Create a repository in your GitHub organization today and its Actions jobs present a different identity to AWS than the repository next to it. The older one says repo:example-org/api:ref:refs/heads/main, while the new one says repo:example-org@1234567/api@456789:ref:refs/heads/main. A trust policy written only for the first format denies the second, and the error tells you nothing about why.

I designed around that trap when I built GitHub OIDC roles for CI image pushes in a multi-account AWS estate, trusted by every repository across two GitHub organizations, so the long-lived access keys those pipelines use can be retired. A GitHub Actions OIDC AWS trust policy that survives this needs three things: accept both subject formats, pin the organization by its numeric ID, and prove the policy against every repository before it ships. After that you switch workflows over additively and retire keys only after watching them go quiet. Done in that order, every step can be rolled back by reverting one workflow file while the old keys still work. More of the governance work is in my AWS multi-account governance case study.

How GitHub OIDC reaches AWS

OIDC (OpenID Connect, a standard for one system to vouch for an identity to another) replaces the stored key. GitHub signs a short-lived token describing the job, and AWS trades it for temporary credentials if the role's trust policy accepts what the token says.

View this diagram in the original post

STS issues credentials only when the token's claims satisfy the role's trust policy.

The job needs id-token: write permission to request the token. The aws-actions/configure-aws-credentials action (currently v6) does the exchange. Its default audience is sts.amazonaws.com, and its default session is one hour, adjustable from 15 minutes to 12 hours. GitHub's guide to OIDC in AWS covers the provider setup. If you lead a team, ask how many IAM users exist only so CI can reach AWS; each is a candidate for OIDC.

Two subject formats, side by side

Trust policies match on the sub claim. GitHub's OIDC reference builds it from the repository plus one context: an environment, a pull request, a branch or a tag.

Job context	Name-only format	Immutable format
Job references an environment	`repo:ORG/REPO:environment:NAME`	`repo:ORG@ORG-ID/REPO@REPO-ID:environment:NAME`
Pull request event, no environment	`repo:ORG/REPO:pull_request`	`repo:ORG@ORG-ID/REPO@REPO-ID:pull_request`
Branch, no environment, not a pull request	`repo:ORG/REPO:ref:refs/heads/BRANCH`	`repo:ORG@ORG-ID/REPO@REPO-ID:ref:refs/heads/BRANCH`
Tag, no environment, not a pull request	`repo:ORG/REPO:ref:refs/tags/TAG`	`repo:ORG@ORG-ID/REPO@REPO-ID:ref:refs/tags/TAG`

Per GitHub's reference, the pull request, branch and tag forms apply only when the job references no environment.

The immutable format is the default for repositories created after July 15, 2026. Older repositories keep the name-only format unless the organization or repository opts in, and the immutable format isn't available on GitHub Enterprise Server.

A rename flips the format

Repository renames and transfers after July 15, 2026 also move to the immutable format. A name-only policy stops matching a repository the day someone renames it.

IDs matter because the OIDC specification requires a subject to be "locally unique and never reassigned", and a recycled name breaks that. GitHub separates name and ID with @, which cannot appear in a GitHub name, after researchers showed an earlier - delimiter was squattable¹. AWS's condition key reference agrees:

AWS IAM documentation: use immutable identifiers, not names

A name that is freed by renaming or deletion can be claimed by a different account. Policies that rely solely on mutable name-based claims (such as repository or actor) could grant access to unintended identities.

If your organization has created or renamed a repository since July 15, 2026, check which format each repository now uses. Then ask whether your AWS roles accept both before the next new one needs to deploy.

Writing the GitHub OIDC trust policy

When you create or update a role that trusts GitHub, IAM checks that token.actions.githubusercontent.com:sub is present and is not solely a wildcard or null, and fails the request otherwise. That rules out trusting by organization ID alone.

You cannot trust by organization ID alone

A policy with only repository_owner_id is rejected. Existing roles are not re-evaluated until someone edits their trust policy, so an old, loose role keeps working until its next edit fails.

The sub condition lists both formats for your organization, and a separate repository_owner_id condition pins the organization by its numeric ID. AWS evaluates multiple values for one key as a logical OR and separate keys or operators as a logical AND, so the token must match one of the sub patterns and carry the right owner ID and the right audience. The slash in each pattern matters: repo:example-org/* doesn't match repo:example-orgX/..., because the character after example-org must be /. The four variants run from widest to narrowest. Account 123456789012, example-org and the IDs are placeholders.

Org-wide

trust-policy-org-wide.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:repository_owner_id": "1234567"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": [
            "repo:example-org/*",
            "repo:example-org@1234567/*"
          ]
        }
      }
    }
  ]
}

One repository

trust-policy-one-repo.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:repository_owner_id": "1234567",
          "token.actions.githubusercontent.com:repository_id": "456789"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": [
            "repo:example-org/example-repo:*",
            "repo:example-org@1234567/example-repo@456789:*"
          ]
        }
      }
    }
  ]
}

One environment

trust-policy-environment.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:repository_owner_id": "1234567",
          "token.actions.githubusercontent.com:repository_id": "456789",
          "token.actions.githubusercontent.com:sub": [
            "repo:example-org/example-repo:environment:production",
            "repo:example-org@1234567/example-repo@456789:environment:production"
          ]
        }
      }
    }
  ]
}

Branch main

trust-policy-ref-main.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:repository_owner_id": "1234567",
          "token.actions.githubusercontent.com:repository_id": "456789",
          "token.actions.githubusercontent.com:sub": [
            "repo:example-org/example-repo:ref:refs/heads/main",
            "repo:example-org@1234567/example-repo@456789:ref:refs/heads/main"
          ]
        }
      }
    }
  ]
}

For the image-push roles I built, I chose org-wide. Each environment account got one OIDC provider and one narrowly scoped role, which can do only the push and pull actions the existing CI key's IAM user uses, only against its own account's registries, with the session capped at four hours. The worst case is a compromised workflow in any repository overwriting an image tag that production pulls. Tag immutability on the registries, or one role per registry, narrows that further.

I rejected a named repository allow-list, because an allow-list is only as complete as the inventory behind it. One built from default-branch workflows misses repositories that run only on other branches, every new repository needs a policy change before it can push, and a name-only list denies every new-format repository. I'd rather accept breadth in the trust policy and keep the permission policy narrow.

AWS warns that sub must be limited to your organization or repository, and strongly recommends protection rules wherever you use GitHub environments. For a role that can change production, I go further: scope it by environment or ref, behind protection rules. Org-wide trust is a deliberate exception I make only when the permission policy is narrow.

I also left GitHub's subject customization (include_claim_keys) alone. A custom template changes the sub for every repository that adopts it, so every trust policy those repositories reach has to change in step with it, and the default format needs no such coordination. Whatever width you choose, ask for a one-line note per role: who can assume it, and the worst case if a workflow is compromised.

Prove it against every repository

A trust policy that looks right is still only a claim, so before shipping I checked it against every repository in both organizations. The first step reads each repository's subject template. The REST endpoint returns use_default, include_claim_keys, use_immutable_subject and sub_claim_prefix.

audit-oidc-subjects.sh

#!/usr/bin/env bash
set -euo pipefail

for org in example-org example-org-two; do
  gh repo list "$org" --limit 1000 --json nameWithOwner --jq '.[].nameWithOwner' |
    while read -r repo; do
      gh api "repos/${repo}/actions/oidc/customization/sub" \
        --jq "[\"${repo}\", .use_default, (.use_immutable_subject // false), (.include_claim_keys // [] | join(\",\"))] | @tsv" ||
        printf '%s\tERROR\n' "$repo"
    done
done

gh needs a token with the repo scope, and --limit 1000 caps the list silently, so raise it for larger organizations. Any repository with a custom template or the immutable flag is one your policy must handle. When I ran this check across both organizations, several recently created repositories already used the new format.

The second step is a simulation. I generated the sub each repository would send for a fixed set of trigger shapes and matched it against the policy's patterns locally. This was a local pattern-match simulation, not the IAM policy simulator. Every real repository was allowed.

A minimal version of the local check

fnmatchcase treats * and ? the way StringLike does. It also treats [ specially, which StringLike does not, so this is a sanity check rather than an IAM evaluation.

simulate_trust.py

from fnmatch import fnmatchcase

OWNER_ID = "1234567"
PATTERNS = ["repo:example-org/*", "repo:example-org@1234567/*"]

def allowed(sub: str, owner_id: str) -> bool:
    return owner_id == OWNER_ID and any(fnmatchcase(sub, p) for p in PATTERNS)

cases = {
    "repo:example-org/api:ref:refs/heads/main": ("1234567", True),
    "repo:example-org@1234567/api@456789:pull_request": ("1234567", True),
    "repo:example-orgx/api:ref:refs/heads/main": ("1234567", False),
    "repo:other-org/api:ref:refs/heads/main": ("1234567", False),
    "repo:example-org-fork/api:pull_request": ("1234567", False),
    "repo:someone@999/example-org@1234567:ref:refs/heads/main": ("1234567", False),
}

for sub, (owner_id, expected) in cases.items():
    assert allowed(sub, owner_id) is expected, sub
print("all cases match")

Allows alone prove half the policy, so the third step is a set of negative controls: subjects that should never get credentials.

Negative control	What it imitates	Result
Look-alike organization name	Your org name plus one character	Denied
Foreign organization	Any other GitHub organization	Denied
Fork-style owner	An organization named after yours with a suffix, such as `example-org-fork`	Denied
Embedded `@` identity	Another owner whose `sub` carries your `org@ID` after its own name	Denied

Each was denied by the sub patterns alone, and the owner-ID condition is a second, independent check. The last two steps cover permissions. I pulled the existing key's last 90 days of API calls and confirmed every one fit inside the new role's allowed actions, and IAM Access Analyzer policy validation returned no errors. Before a trust policy ships, ask for that evidence: every repository simulated, negative cases denied, the existing key's real usage covered.

Switch workflows over additively

The role sits alongside the keys, so you can switch one workflow at a time while the old keys still work as a fallback. For a typical image-push workflow the diff is small.

Before: access keys

.github/workflows/push-image.yml

name: push-image
on:
  push:
    branches: [main]

jobs:
  push:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      - run: aws sts get-caller-identity

After: OIDC

.github/workflows/push-image.yml

name: push-image
on:
  push:
    branches: [main]

permissions:
  id-token: write
  contents: read

jobs:
  push:
    runs-on: ubuntu-latest
    environment: dev
    steps:
      - uses: actions/checkout@v6
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: ${{ vars.AWS_ROLE_ARN }}
          role-duration-seconds: 3600
          aws-region: us-east-1
      - run: aws sts get-caller-identity

Three details in that diff matter:

contents: read stays. A permissions block sets every scope you do not list to none, and actions/checkout needs read access.
The role ARN lives in a GitHub environment variable, one per environment, which keeps dev, staging and prod roles apart.
environment: dev changes the sub. The job now sends repo:...:environment:dev, not ref:refs/heads/main. The org-wide pattern accepts both. A branch-scoped policy would deny it.

role-duration-seconds must not exceed the role's maximum session duration. I show tags for readability; pin both actions to a full commit SHA in real workflows, as GitHub's own example does.

Roll out in environment order: dev, then staging, then prod. For each one, verify three ways: the run succeeds, CloudTrail, in the Region the workflow sets as aws-region, shows an AssumeRoleWithWebIdentity event whose userIdentity.userName is the actual sub, and the role's last activity in IAM updates. If anything fails, revert the workflow change, because the keys still work. Ask for one environment at a time, a named rollback and CloudTrail proof, not just a green build.

Retire the keys, then close the door

Retirement runs per environment. These two commands check when a key was last used and switch it off without deleting it.

check-and-deactivate-key.sh

aws iam get-access-key-last-used --access-key-id "${ACCESS_KEY_ID}"

aws iam update-access-key \
  --user-name "${CI_USER_NAME}" \
  --access-key-id "${ACCESS_KEY_ID}" \
  --status Inactive

The sequence for each environment:

Confirm every workflow in that environment uses the role.
Watch GetAccessKeyLastUsed through a quiet period, for example seven days, with no key use.
Deactivate the key, then wait one release cycle.
Delete the GitHub secrets that held it.
Delete the key, then the IAM user.

Deactivate before you delete

An inactive key can be reactivated in seconds if a forgotten job surfaces. A deleted key cannot. Also look for keys created outside your infrastructure-as-code state: a terraform destroy of the user fails while such a key still exists, because IAM will not delete a user that has an access key.

The last step of the method is a guardrail so new keys don't creep back: an SCP (a service control policy, an organization-level policy that can only deny) that blocks IAM user and key creation for everyone except your landing-zone automation. Stage it before enforcing it.

scp-deny-iam-user-credentials.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyLongLivedIamUserCredentials",
      "Effect": "Deny",
      "Action": [
        "iam:CreateUser",
        "iam:CreateAccessKey",
        "iam:CreateLoginProfile"
      ],
      "Resource": "*",
      "Condition": {
        "ArnNotLike": {
          "aws:PrincipalArn": [
            "arn:aws:iam::*:role/example-landing-zone-automation",
            "arn:aws:iam::*:role/example-break-glass-admin"
          ]
        }
      }
    }
  ]
}

With multiple values under a negated operator, AWS evaluates them as a logical NOR: the deny applies unless the caller matches one of the exempt roles. SCPs do not affect the management account or service-linked roles, so they are not a complete fence. Denying CreateAccessKey also blocks key rotation for any IAM user still in service, so retire those first or exempt them deliberately.

Before attaching, search CloudTrail for the last 90 days of CreateUser, CreateAccessKey and CreateLoginProfile calls. Every caller you find either moves off keys first or goes on the exemption list. For AWS Control Tower, the exemptions include AWSControlTowerExecution. Attach to a small OU first. If you are weighing an SCP against the newer policy types, I compare them in SCP vs RCP vs declarative policies.

"Keys retired" should mean deleted after a watched quiet period, not deactivated and forgotten. Ask for the guardrail to land last, on a test OU first.

Migration checklist

Track the method per environment. Close the ticket only when every box is ticked.

☐ Inventory every IAM user whose keys live in GitHub secrets
☐ Audit every repository's OIDC subject template with the gh api script
☐ One OIDC provider per account; one role per environment, permissions copied from the existing key's real usage
☐ Trust policy accepts both sub formats and pins repository_owner_id
☐ Local simulation of every repository, plus negative controls
☐ Access Analyzer validation clean
☐ Workflows switched dev, staging, prod, each verified in CloudTrail
☐ Quiet period observed per key, then deactivate, wait, delete secrets, delete key and user
☐ Credential-creation SCP staged on a test OU, then enforced

What to do next

Run the audit script against your own organization; it's read-only. If any repository reports use_immutable_subject as true and your trust policies only list names, you have found a repository that will be denied the first time a workflow there tries to assume one of those roles.

Then check your CI roles' sub conditions against the two-format table in Two subject formats, side by side. If you want to talk through the rollout order before touching a pipeline, book a free intro call.

Boost Security Labs, "Sleeper Squats: How a Hyphen (Almost) Unraveled GitHub's Immutable OIDC Subject Claim". ↩

SCP vs RCP vs Declarative Policies in AWS

Shahid Yousuf — Fri, 09 Oct 2026 06:02:48 +0000

Once an AWS organization has more than a handful of accounts, guardrails stop being something each account team sets up for itself. A typical request: S3 Block Public Access switched on in every account, and kept on. AWS Organizations has three policy types that look like they could do that, a service control policy (SCP), a resource control policy (RCP) and a declarative-style policy. Only one of them can actually hold the setting, and picking the wrong one either leaves a gap or blocks your own fix.

The obvious control is an SCP denying s3:PutBucketPublicAccessBlock. Check CloudTrail before you write it: in a real estate most calls to that API are Terraform turning protection on. No condition key separates enabling Block Public Access from disabling it, so the deny stops the safe change along with the dangerous one.

The three types answer different questions. SCPs limit what your own principals can do, RCPs limit how anyone can reach your resources, and declarative policies hold a setting at a fixed value, so the useful question for any guardrail is whether you need to deny a verb or assert a state. The examples come from governing a real multi-account estate, written up as a multi-account AWS governance case study. Every policy is generic, with placeholders for Regions and AWS's documentation example account 123456789012 for every account ID.

SCP vs RCP vs declarative policy at a glance

All three are authorization or management policies in AWS Organizations, attached to the root, an OU (organizational unit, a folder of accounts) or an account. Strictly, AWS calls the EC2 type a declarative policy (DECLARATIVE_POLICY_EC2) and the S3 one an Amazon S3 policy (S3_POLICY). Both use the same syntax and hold a setting, so this guide groups them as "declarative".

	SCP	RCP	Declarative (EC2) / S3 policy
What it limits	IAM users and roles in member accounts, including the member root user	Requests to your resources, from any principal	A service setting, held at a fixed value
Grants access?	Never	Never	Not an access policy; it sets state
Management account	Not affected	Not affected	Not stated in AWS docs; test before relying on it
Service-linked roles	Not affected	Not affected	Governed
Principals outside your org	Not affected	Affected	Not applicable
Report-only mode	None	None	Some attributes have one (Allowed AMIs `audit_mode`, VPC Encryption Controls `attempt_monitor`)
`DescribeEffectivePolicy`	Not supported	Not supported	Supported (EC2 and S3 types)
Size and per-target limit	10,240 characters, 10	5,120 characters, 5	10,000 characters, 10

Sources: the AWS Organizations user guide pages for SCPs, RCPs and declarative policies.

The service-linked roles row matters most: AWS services acting on your behalf slip past SCPs and RCPs, but not past a declarative policy, which the service enforces in its control plane. And none of the three grants anything. If someone proposes an SCP "to give the team access", send the plan back before it starts.

Deny a verb or assert a state

I run every proposed guardrail through three questions:

Who is acting? If the control is about your own users and roles doing something, it is an SCP.
What resource is being reached? If the control is about any caller touching your data (including outsiders), it is an RCP.
What setting must hold? If the control is "this must be on", look for a declarative attribute first.

View this diagram in the original post

Start from the end state you want, not the API you want to block.

Block Public Access is the clean example. The Organizations S3 policy asserts "on" for every account in scope, and in my test Terraform's bucket-level calls kept succeeding (more on that in the declarative section). It also avoids an ordering trap. If you deny the account-level Block Public Access API before the value is set, you've blocked your own fix until you detach the SCP or add an exemption, while with the S3 policy Organizations sets the value for you.

When a team asks for a guardrail, ask them to describe the state they want rather than the action they want to block. The answer usually picks the tool.

SCPs for your own principals

An SCP is a ceiling on IAM permissions in member accounts, and that includes member accounts acting as delegated administrators. An action must be allowed at every level from the root down to the account, and a deny at any level wins. That's why the default FullAWSAccess SCP must stay attached or be replaced by your own allow list.

A baseline that has held up well for me covers five things: no leaving the organization, no closing accounts, no turning off the main security services, no new unencrypted EBS volumes, and no root user minting access keys, IAM users or passwords. Treat the action lists below as a starting point: each service has more off switches (for example guardduty:UpdateDetector and cloudtrail:UpdateTrail).

baseline-scp.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyLeaveOrgAndCloseAccount",
      "Effect": "Deny",
      "Action": ["organizations:LeaveOrganization", "account:CloseAccount"],
      "Resource": "*"
    },
    {
      "Sid": "ProtectSecurityServices",
      "Effect": "Deny",
      "Action": [
        "cloudtrail:StopLogging",
        "cloudtrail:DeleteTrail",
        "guardduty:DeleteDetector",
        "securityhub:DisableSecurityHub",
        "access-analyzer:DeleteAnalyzer"
      ],
      "Resource": "*",
      "Condition": {
        "ArnNotLike": {
          "aws:PrincipalArn": "arn:aws:iam::*:role/AWSControlTowerExecution"
        },
        "StringNotEquals": {
          "aws:PrincipalAccount": "123456789012"
        }
      }
    },
    {
      "Sid": "DenyUnencryptedEbs",
      "Effect": "Deny",
      "Action": ["ec2:CreateVolume", "ec2:RunInstances"],
      "Resource": "arn:aws:ec2:*:*:volume/*",
      "Condition": {
        "Bool": { "ec2:Encrypted": "false" }
      }
    },
    {
      "Sid": "ProtectEbsDefaultEncryption",
      "Effect": "Deny",
      "Action": "ec2:DisableEbsEncryptionByDefault",
      "Resource": "*"
    },
    {
      "Sid": "DenyRootCredentials",
      "Effect": "Deny",
      "Action": ["iam:CreateAccessKey", "iam:CreateUser", "iam:CreateLoginProfile"],
      "Resource": "*",
      "Condition": {
        "ArnLike": { "aws:PrincipalArn": "arn:aws:iam::*:root" }
      }
    }
  ]
}

The two conditions in the security statement are ANDed, so the deny skips both the Control Tower role and the security tooling account (123456789012 here). It exempts the whole account because that account is the delegated administrator for these services and manages them for every member. If you can, narrow it to the specific roles there. EBS encryption-by-default has no declarative attribute, so protecting it stays an SCP job.

Turn on EBS encryption-by-default first

Enable EBS encryption-by-default in every account and Region before you attach the unencrypted-volume deny. Otherwise every volume create that does not ask for encryption fails.

I keep the Region restriction in a separate SCP. Region rules and data-protection rules change at different speeds, and a separate policy keeps the blast radius of each edit small. The pattern follows AWS's deny-by-requested-Region example. The NotAction list here is a short sample of global services; extend it with every global service you use before attaching.

region-scp.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyOutsideApprovedRegions",
      "Effect": "Deny",
      "NotAction": [
        "iam:*",
        "organizations:*",
        "sts:*",
        "support:*",
        "route53:*",
        "cloudfront:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["${APPROVED_REGION_1}", "${APPROVED_REGION_2}"]
        },
        "ArnNotLike": {
          "aws:PrincipalArn": "arn:aws:iam::*:role/AWSControlTowerExecution"
        }
      }
    }
  ]
}

Sweep every service before a Region lock

A pre-check that only lists EC2 resources misses S3 buckets in non-approved Regions. S3 bucket operations resolve to the bucket's Region, so once the SCP lands, such a bucket can no longer be managed. Inventory every service in every Region first.

If you use Control Tower, know that registering an OU attaches only its own aws-guardrails-* SCP. Your baseline needs its own attachment, and you shouldn't edit or detach Control Tower's SCPs. If you're moving accounts between organizations, the order of operations is in how to move an AWS account to another organization.

I handle exceptions with placement, not conditions: put the account that needs one in an OU without the policy, or attach the policy to its siblings individually. That's easier to audit than a tangle of exemption conditions. When you plan the work, budget for a baseline SCP, a separate Region SCP and a pre-attach inventory across every service. The inventory is the part teams skip.

RCPs for the data perimeter

An RCP is evaluated on the resource side and, unlike an SCP, applies to principals outside your organization too. That makes it the right place for data-perimeter rules: "nobody reaches these resources without TLS", whoever they are.

Beyond the table's exceptions, RCPs also skip AWS-managed KMS keys and kms:RetireGrant. They cover a growing list of services, including S3, STS, KMS, SQS, Secrets Manager, DynamoDB, ECR, Cognito, CloudWatch Logs and OpenSearch Serverless. RDS is not on it, so an RCP will not protect an RDS or Aurora database. The TLS-only policy here is AWS's published example, with a Sid added.

tls-only-rcp.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EnforceSecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": [
        "s3:*",
        "sts:*",
        "kms:*",
        "sqs:*",
        "secretsmanager:*",
        "cognito-identity:*",
        "cognito-idp:*",
        "logs:*",
        "dynamodb:*",
        "ecr:*",
        "aoss:*"
      ],
      "Resource": "*",
      "Condition": {
        "BoolIfExists": { "aws:SecureTransport": "false" }
      }
    }
  ]
}

Pilot it on a non-production OU first. S3 website endpoints serve HTTP only, so a CloudFront distribution whose origin is an S3 website endpoint talks to it over HTTP, and under a TLS-only RCP that origin breaks. Find those distributions before you attach.

An RCP is how you protect data from callers you don't control. Before anyone assumes it protects a given data store, ask which services it covers.

AWS declarative policies vs SCP

A declarative policy does not block an API. It tells the service what the setting is, and the service enforces it in its control plane. Three properties follow:

It governs service-linked roles, which SCPs cannot.
It stays in force as the service adds new APIs, so there is no deny list to keep current.
New accounts inherit it the moment they join the OU.

s3-policy.json

{
  "s3_attributes": {
    "public_access_block_configuration": {
      "@@assign": "all"
    }
  }
}

EC2

ec2-declarative.json

{
  "ec2_attributes": {
    "image_block_public_access": {
      "state": { "@@assign": "block_new_sharing" }
    },
    "snapshot_block_public_access": {
      "state": { "@@assign": "block_new_sharing" }
    },
    "serial_console_access": {
      "status": { "@@assign": "disabled" }
    }
  }
}

The S3 policy makes public buckets private on attach

Setting all turns on all four Block Public Access settings and overrides the account's own. A bucket serving public content today stops serving it the moment the policy applies. Find those buckets first, then attach the policy to the other accounts individually, or move the account that needs public access to an OU without it.

The S3 policy has a single attribute, public_access_block_configuration, and it is all or none. AWS's guidance is to disable the S3 policy type if you want to manage Block Public Access per account. It shipped in November 2025, so guides written before then don't mention it. On detach, the setting rolls back to its pre-attach value, so know that value before you count on detach as a rollback.

In my test, with the S3 organization policy in force, a bucket-level PutBucketPublicAccessBlock call still succeeded. Public bucket policies and public ACLs were denied, because S3 applies the most restrictive combination of settings. That was a single test, so verify it yourself before you depend on it.

The EC2 declarative attributes also cover VPC Block Public Access, Allowed AMIs, IMDS defaults and VPC Encryption Controls. An EC2 declarative policy can replace SCP statements that block AMI and snapshot sharing, but prove the new coverage is a strict superset of the old before you remove anything.

IMDSv2 enforcement can stop launches

AWS warns that turning on http_tokens_enforced while launches still allow IMDSv1 (http_tokens optional, from a launch template, launch call or AMI default) causes launch failures. Set http_tokens to required and find IMDSv1 callers first. I left that attribute out deliberately.

For snapshots I chose block_new_sharing over block_all_sharing. block_new_sharing leaves anything already public as it is and stops new public shares, so it can't cut off a consumer you haven't found yet. block_all_sharing makes existing public snapshots private the moment it applies, so inventory them first. Neither mode affects sharing with specific accounts.

For Block Public Access and AMI or snapshot sharing, I'd choose an S3 or EC2 declarative policy over an SCP. It covers more principals and needs no maintenance as APIs change.

Quotas changed in 2026

AWS raised the SCP quotas on 15 May 2026. Current values from the Organizations quotas page, which does not list the S3 policy type separately:

Policy type	Max size	Max attached per root, OU or account
SCP	10,240 characters	10
RCP	5,120 characters	5 (`RCPFullAWSAccess` uses one)
Declarative (EC2)	10,000 characters	10

Inherited policies do not count toward a target's limit. A policy type must be enabled on the root before you can use it. Enabling SCPs auto-attaches FullAWSAccess; enabling the S3 policy or EC2 declarative policy type attaches nothing. RCP slots are the scarce ones, so plan RCPs as a few broad policies rather than one per rule.

Rolling out without breaking production

Testing is where these tools are weakest. There is no report-only mode for SCPs or RCPs. The IAM policy simulator evaluates SCPs that already apply but does not support RCPs, and it will not simulate a draft you have not attached. AWS's advice is to test on a test OU or account, use CloudTrail and last-accessed data, then move up the tree. Checking coverage afterwards is awkward too: DescribeEffectivePolicy does not support SCPs or RCPs, so to confirm an account is covered, walk its parent chain with ListParents and call ListPoliciesForTarget at each level.

validate-before-create.sh

aws accessanalyzer validate-policy \
  --policy-type SERVICE_CONTROL_POLICY \
  --policy-document file://baseline-scp.json

IAM Access Analyzer's validate-policy catches invented action names. For example, there is no s3:DeleteBucketPublicAccessBlock; deleting maps to the Put permission. It also recommends ArnLike over StringLike on ARN condition keys, which is why validation is the first line of the checklist I run for every attachment.

Comparing deployed policy JSON

AWS returns policy documents with keys and arrays in varying order. A plain string diff reports drift that is not there. Sort keys and arrays recursively on both sides before comparing.

☐ Validate the JSON with Access Analyzer before creating the policy
☐ Sweep every service in every Region for resources the policy would strand
☐ Set any value you are about to protect (encryption-by-default, Block Public Access) first
☐ Find every bucket that is public today before attaching the S3 policy
☐ For EC2 declarative policies, generate the Organizations account status report first
☐ Attach to a canary OU with non-production accounts
☐ Prove a fast detach works, and know what the setting reverts to
☐ Walk each account's parent chain to confirm coverage
☐ Check CloudTrail for new access denied errors the next day
☐ Move up one OU at a time

When one of these lands in a change request, look for the canary OU, the detach plan and the next-day CloudTrail check. If any of them is missing, the rollout isn't ready.

What to do next

List every guardrail you have and the state each one is meant to guarantee. Anything that reads "this setting must be on" moves to a declarative policy, anything about outsiders reaching your data belongs in an RCP, and the rest is SCP work.

Then run the checklist above on one policy and one low-risk OU. If you want a second pair of eyes on an estate before you start, book a free intro call; the multi-account governance case study shows the kind of estate this came from.

Move an AWS Account to Another Organization

Shahid Yousuf — Fri, 09 Oct 2026 06:02:44 +0000

When you need to move an AWS account to another organization, the move itself takes minutes. Getting the account governed afterwards is the real work. In a multi-account AWS governance engagement, I worked on exactly this: production accounts that had to end up under the destination's AWS Control Tower, reachable by the same people, and inside the destination's own guardrails.

Done casually, a move looks finished long before it is. The account shows up in the new organization, the Control Tower dashboard says enrolled, and people sign in through the new portal. Meanwhile the old organization can still administer the account, and your own guardrail policies may not reach it at all. Nothing on the screen tells you either of those things.

The right route is to invite the account from the destination organization's management account. When it accepts, AWS moves it across directly: it never leaves its old organization by itself and never runs standalone in between. The older leave-then-join route is the one that produces this error, and you don't need to fix it:

The member account is missing one or more of the prerequisites required to operate as a standalone account. To add what is missing, sign-in to the member account using the AWS Organizations console, then select to leave the organization...

What follows is the order of operations that works with Control Tower, and the three checks that decide whether a moved account is actually governed: sign-on, your own guardrail policies, and the old organization's admin role. On those moves, the client's cloud admin sent and accepted the invitations and ran the enrollment. I planned the sequence, prepared the landing OU, sign-on and baseline policies, and verified each account afterwards.

Why leave-then-join gets refused

The classic way to move an account is leave, then join: the account leaves its organization as a member, runs standalone for a while, then accepts an invite from the new one. To leave, it must be able to stand alone, and that is where it gets stuck.

In accounts created inside an organization, the payment method is usually missing. That was true here, but adding a card wasn't enough: Leave still refused, and AWS's message doesn't say what else it wants. No public API reads whether a payment instrument exists (billing, invoicing, billingconductor, account, ce and budgets all come up empty), but the console does: Billing, then Payment preferences, in the member account lists payment methods. The Leave dialog itself only said:

You can't leave the organization yet

AWS documents a "Complete the account sign-up steps" link for this message. With the invite path, you can skip it. If your migration plan depends on someone typing a payment card into every account by hand, send it back; there's a better path.

A refused removal is safe, but it is not a dry run

The 400 changes nothing. But if the prerequisites are met, that same call removes a production account from its organization on the spot. Do not use it to "test" readiness.

Invite from the destination instead

On 19 November 2025 AWS announced direct account transfers. The destination management account invites the account, the account accepts, and it moves straight across without leaving its old organization first or running standalone in between. In practice the invitation went through, and on acceptance the account left its old organization automatically, with no separate leave step.

The API reference makes this easy to miss. The InviteAccountToOrganization reference still lists ALREADY_IN_AN_ORGANIZATION as a failure reason, though read closely, that list sits under a caveat:

Some of the reasons in the following list might not be applicable to this specific API or operation.

The account migration guide describes the direct transfer, and the API reference's error list is shared boilerplate. When the two disagree, I go with the user guide.

	Leave, then join	Invite from destination
Standalone prerequisites	Required before leaving	AWS's docs say none needed
Payment card per account	Typed in by hand	Not mentioned in AWS's docs. Untested here: the one invited account already had a card.
Billing gap	A standalone period outside consolidated billing	None
Who acts	Member account leaves, then accepts an invite	Destination invites, member accepts
Outcome here	Refused, even with a card added	Worked

The two ways to move an account, as they played out here.

The commands themselves are short. Invite from the destination's management account, then accept in the member account:

Run in the destination management account

aws organizations invite-account-to-organization \
  --target Id=123456789012,Type=ACCOUNT

Run in the member account

aws organizations accept-handshake --handshake-id h-examplehandshakeid111

Read the guide's prerequisites first: rules on the age of the account and the organization, a Seller of Record that must match, and the fact that the account lands in the root of the new organization. Per the invitations page, invitations expire after 15 days and the new organization's policies apply as soon as the account joins. If your team plans to leave, then join, ask them why; the invite path skips per-account billing setup and the standalone gap.

How to move an AWS account to another organization, step by step

With AWS Control Tower in the destination, "moved" really means "moved and enrolled in the right organizational unit (OU)". The sequence:

Register the landing OU with Control Tower while it is still empty. For an empty OU it took about two minutes here; AWS's guidance is to plan for ten or more. Control Tower attaches its own aws-guardrails-* SCP (a service control policy: an org-level policy that can only deny, never grant). That is expected, not drift.
Associate security monitoring with the OU before anything arrives.
Invite from the destination; accept in the member account.
Create the AWSControlTowerExecution role in the account.
Enroll the account through Control Tower into the OU. Until enrollment, the account sits in the root under whatever policies are attached there, so enroll it the same day.

View this diagram in the original post

The full move, from invitation to the cleanup that makes the account fully yours.

The execution role trusts the destination's management account. This is the shape from the enrollment prerequisites; replace 111122223333 with your destination management account ID:

trust.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

The role also needs the AdministratorAccess managed policy, or enrollment fails:

Run in the member account

aws iam create-role --role-name AWSControlTowerExecution \
  --assume-role-policy-document file://trust.json
aws iam attach-role-policy --role-name AWSControlTowerExecution \
  --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

I don't use aws organizations move-account for step 5. With auto-enrollment off, a manual move into a Control Tower OU does not enroll the account and causes inheritance drift. If you want moves to enroll on their own, Control Tower offers automatic enrollment on landing zone 3.1 and later.

I also wouldn't move every account at once. Order them by access risk, and hold back any account whose permission sets don't yet exist in the destination. Whoever owns the migration should confirm the landing OU and its security monitoring are in place, and checked, before the first account arrives.

Sign-on after the move

For users, the account disappeared from the old access portal and appeared in the new one with the same groups, with no access requests to re-file. This works only because the destination's directory already had the same groups, and permission sets were assigned to them on arrival. Nothing in Identity Center carries over by itself. The destination's IAM Identity Center creates the new sign-on roles (AWSReservedSSO_*) when permission sets are assigned, as Identity Center does.

If a group doesn't exist in the destination's directory, its members lose that path into the account, so create groups and permission sets in the destination before you move anything. Engineers will also hit a confusing error: a CLI profile still pointing at the old portal fails with ForbiddenException ... No access. It reads like a missing permission, but it's just the old portal. Point the profile at the new start URL and sign in again.

Root email addresses come across unchanged. One of ours pointed at a mailbox on a legacy domain, so check each one. Budget an identity prep step before the move, and tell engineers ahead of time that their CLI profiles will need a new start URL.

Enrolled is not the same as governed

Run a parity check against the organization's own guardrails after each move. On these moves, the accounts were enrolled, recording AWS Config and reporting to the security services, yet outside every one of the organization's own baseline policies, the SCPs and configuration policies (an EC2 declarative policy and an S3 policy, which set a service's configuration directly), for most of a week.

A dashboard showing "enrolled" means Control Tower's own controls apply, nothing more. Your organization's policies apply to an OU only if they are attached to it or inherited from above it. Here the baseline was attached OU by OU, not at the root, so a brand-new OU inherited none of it. Attaching policies that are safe everywhere at the root closes this class of gap; the trade-off is a wider blast radius for every change to them.

There is no effective-policy API for SCPs, so the check walks the parent chain. Configuration policies do have one:

policy-parity.sh

#!/usr/bin/env bash
# Run with management account credentials.
# List the SCPs that reach an account: its own, each parent OU's, and the root's.
set -euo pipefail
target="${ACCOUNT_ID:?set ACCOUNT_ID}"

while true; do
  echo "== ${target}"
  aws organizations list-policies-for-target \
    --target-id "${target}" --filter SERVICE_CONTROL_POLICY \
    --query 'Policies[].Name' --output text
  read -r parent_id parent_type < <(aws organizations list-parents \
    --child-id "${target}" --query 'Parents[0].[Id,Type]' --output text)
  target="${parent_id}"
  if [["${parent_type}" == "ROOT"]]; then
    echo "== ${target} (root)"
    aws organizations list-policies-for-target \
      --target-id "${target}" --filter SERVICE_CONTROL_POLICY \
      --query 'Policies[].Name' --output text
    break
  fi
done

# Configuration policies have an effective-policy API.
# Repeat with --policy-type DECLARATIVE_POLICY_EC2.
aws organizations describe-effective-policy \
  --policy-type S3_POLICY --target-id "${ACCOUNT_ID}"

Two details keep a check like this honest. Control Tower's list-enabled-baselines returns an OU as an ARN, while Organizations uses the bare ou- ID, so compare like with like. And list the policy types your organization has enabled (aws organizations list-roots --query 'Roots[0].PolicyTypes') instead of hard-coding them, or a check can silently skip one.

Attach missing policies one at a time, smallest blast radius first, and take care with one type in particular:

An S3 policy changes live state on attach

An SCP only denies future calls. An Organizations S3 policy overrides the account's Block Public Access settings the moment you attach it. A bucket that is public today stops being public. Detaching it restores the previous settings. If one account in the OU has public content that must stay public until its owner decides, attach the S3 policy to the other accounts individually and move it to the OU later.

The same check found a long-governed account missing baseline policies too, so the gap is not unique to new OUs. Make the parity check a runbook step after every move, and don't accept a dashboard's "enrolled" as proof that your own policies apply.

The old organization's admin role

Accounts created inside an organization get OrganizationAccountAccessRole, an admin role the management account can assume. Invited accounts do not get one in the new organization.

The old one is the problem. It survives the move, still trusts the old organization's management account, and still has admin rights, so whoever controls that old management account can still administer the account you just moved. The cleanup comes after the move because, until AWSControlTowerExecution and Identity Center access exist, the old role is the only admin path. Do it the same day.

Inspect who the role trusts

aws iam get-role --role-name OrganizationAccountAccessRole \
  --query 'Role.AssumeRolePolicyDocument'

It isn't the only role to check. Old-org StackSets execution roles can also trust the old management account:

Find every role that trusts the old management account

aws iam list-roles \
  --query "Roles[?contains(to_string(AssumeRolePolicyDocument), '${OLD_MGMT_ACCOUNT_ID}')].RoleName"

Replace your access path before you remove the old one

Verify both paths first: the human path (you can sign in through the new organization's Identity Center) and the automation path (AWSControlTowerExecution is assumable from the new management account). Then delete the old role, or retrust it to the new management account.

An account is not migrated until the old organization can no longer administer it. Make that an explicit exit criterion for every move, with someone named to verify it.

Running the next move

Treat each account as its own change: one ticket, both checklists, and a named person for the exit check. Put the baseline attach and the old-role cleanup inside the enrollment runbook, so they happen on move day rather than in a follow-up. If the payment question matters for your budget, invite one account that has no card on file before you plan the rest around either answer.

Before the move:

☐ Read the account migration prerequisites: account and organization age, matching Seller of Record
☐ No SCP or IAM policy in the source organization blocks the migration
☐ Resource policies using aws:PrincipalOrgID found and updated (they break silently after the move)
☐ Account deregistered as a delegated administrator, if it is one
☐ Organization-level billing history exported (AWS deletes it on removal)
☐ No existing AWS Config recorder or delivery channel left in the account (remove before enrolling)
☐ Groups and permission sets exist in the destination's Identity Center
☐ Landing OU registered with Control Tower while empty
☐ Security monitoring associated with the landing OU
☐ Accounts ordered by access risk; any account missing identity prep held back
☐ Root email addresses reviewed

After the move:

☐ Invitation accepted within its 15-day window
☐ AWSControlTowerExecution created with AdministratorAccess and the account enrolled into the OU (not move-account)
☐ Users can sign in through the new portal; CLI profiles updated
☐ Parity check: the organization's baseline SCPs and configuration policies reach the account
☐ Sanctioned public content checked before any S3 policy is attached
☐ Old OrganizationAccountAccessRole, and any other role trusting the old management account, deleted or retrusted

Paste both lists into the ticket for each account, and don't close it until every box is ticked. If you're planning a move like this and want a second pair of eyes on the order of operations, book an intro call. The full engagement it came from is in the AWS governance case study.

AWS Security Checklist for Small Teams

Shahid Yousuf — Thu, 08 Oct 2026 11:30:13 +0000

A small team's AWS setup usually grows by accident. There are one or two accounts, a handful of IAM users created years ago, an access key in the CI system that nobody remembers creating, and the root user still signs in with the founder's personal email. Nothing is on fire, so the AWS security checklist everyone means to work through keeps losing to the roadmap.

The trouble with doing it casually is that most AWS security controls fail quietly. A service shows as enabled while it evaluates nothing, or a guardrail covers EC2 but not the S3 bucket in another Region. You find out at the worst moment: an unexpected invoice, a leaked key, an auditor's question nobody can answer.

This checklist is the short, ordered version. Lock identity down this week, turn on detection this month, and add organization-wide guardrails as you grow. Verify every control after you enable it, because "enabled" and "actually evaluating" are different states. Most of the high-value items cost nothing, and the paid detection services each have a 30-day trial, so you can measure the bill before you commit to it.

AWS already publishes the AWS Startup Security Baseline (SSB), a list of account controls ACCT.01 to ACCT.17 and workload controls WKLD.01 to WKLD.15. It's good, but it isn't ordered by urgency, and it doesn't tell you how to prove a control is working or what it will cost. This guide takes the SSB controls that matter most early on, puts them in order, adds the multi-account guardrails the SSB leaves out, and gives each a verify step and a cost. Where I hit a trap governing a multi-account estate, written up as a multi-account AWS governance case study, I say so.

Prerequisites

To follow along, you need:

The AWS CLI version 2, configured with credentials for the account you're securing.
Administrator access to a standalone account, or to the management account if you already use AWS Organizations (the service that groups accounts under one payer and one set of policies).
A shared mailbox or distribution list for security and billing mail, such as aws-security@your_domain. Personal inboxes leave with people.

Commands use placeholders such as ${ACCOUNT_ID} or your-region. Replace them with your own values.

How to use this checklist

The 19 controls fall into three tiers. The first tier is free and mostly about identity, because a leaked key or a taken-over root user hands over everything. The second turns on detection and data protection. The third only makes sense once you run more than one account.

View this diagram in the original post

Work top to bottom, and verify each control before moving on.

The effort column is my rough judgement for a small team that knows its own systems. Treat it as an order of magnitude, not an estimate.

#	Control	Risk it closes	Rough effort	Cost
This week
1	Root MFA, no root keys, group email	Takeover through the one user you can't restrict	Minutes	Free
2	Security alternate contact	AWS security notices reach nobody	Minutes	Free
3	People sign in through IAM Identity Center	Passwords and keys tied to individuals	Hours	Free
4	Find and retire long-lived access keys	A leaked key works until someone deletes it	Hours	Free
5	CI through OIDC, not stored keys	Keys sitting in CI secrets	Hours per pipeline	Free
6	Budget and Cost Anomaly Detection	Abuse found on the invoice	Minutes	Free
This month
7	Multi-Region CloudTrail trail	No record of who did what	An hour	First management copy free, S3 storage billed
8	GuardDuty in every Region	Nobody watching for threats	An hour	Usage-based, 30-day trial
9	IAM Access Analyzer, external access	Resources shared outside the account	Minutes	Free
10	S3 Block Public Access, account level	Public buckets	An hour with inventory	Free
11	EBS encryption by default	Unencrypted volumes	Minutes per Region	Free
12	IMDSv2 as the account default	Instances on the older metadata service	Hours	Free
13	Secrets out of code and env files	Credentials in repositories	Varies	From free
14	Security Hub CSPM, standards chosen on purpose	No continuous configuration checks	Hours	Per check, plus AWS Config
As you grow
15	Centralized root access	Root credentials in member accounts	An hour	No extra charge
16	Baseline SCP	Someone switches detection off	A day with testing	No extra charge
17	Region SCP	Resources in Regions nobody watches	Days, mostly inventory	No extra charge
18	RDS snapshot sharing check	Database snapshots shared publicly	An hour	Free
19	Declarative policies	Settings drifting back	Hours	No extra charge

AWS Organizations is offered at no additional charge, so the policies in the last tier add nothing to the bill.

Most items below follow the same shape: what to do, why it matters, how to do it, how to verify it, and what it costs.

Do this week: identity and account basics

1. Lock down the root user

The root user can do anything in the account, including things no IAM policy can stop, so it's the one identity you can't afford to lose. AWS now requires MFA on the root user for all account types, with registration due within 35 days of first sign-in. Delete any root access keys, and move the root email to a group address so recovery doesn't depend on one person's inbox.

Console

Sign in as the root user. On the right of the navigation bar, choose your account name, then Security credentials. In the Multi-factor authentication (MFA) section, choose Assign MFA device (AWS steps). Delete any keys in the Access keys section on the same page. To change the email, choose your account name, then Account ; next to Account details , choose Actions , Update email address and password (AWS steps).

CLI

aws iam get-account-summary \
  --query 'SummaryMap.{MFA:AccountMFAEnabled,RootKeys:AccountAccessKeysPresent}'

The output will look similar to:

Output

{
    "MFA": 1,
    "RootKeys": 0
}

AccountMFAEnabled should be 1 and AccountAccessKeysPresent should be 0. Anything else means the root user still needs work. Cost: free.

2. Set the security alternate contact

AWS sends security notices to the security alternate contact. If it's empty or points at someone who left, the notice goes nowhere. See updating alternate contacts.

aws account put-alternate-contact \
  --alternate-contact-type=SECURITY \
  --email-address=aws-security@your_domain \
  --name="Security team" \
  --phone-number="${PHONE_NUMBER}" \
  --title="Security contact"
aws account get-alternate-contact --alternate-contact-type=SECURITY

The second command should show your group address. From a management account, add --account-id ${ACCOUNT_ID} to set the contact on a member account; that needs trusted access for AWS Account Management enabled in Organizations first. Cost: free.

3. Give people IAM Identity Center, not IAM users

IAM users carry a password and often an access key that never expires. IAM Identity Center gives each person one sign-in, short-lived credentials and permission sets you manage centrally, which is what the IAM best practices recommend for human access. Set it up, move each person across, then remove their IAM user. Verify with the credential report from the next item: no human IAM user should have a password. Cost: free.

4. Find and retire long-lived access keys

Access keys don't expire. A key that leaks into a repository, a laptop backup or a CI log keeps working until someone deletes it. Start with the credential report, a CSV of every IAM user and the state of their credentials.

aws iam generate-credential-report
aws iam get-credential-report --query Content --output text \
  | base64 --decode > credential-report.csv

Run generate-credential-report until it returns "State": "COMPLETE", then download; the second command decodes the base64 content to a file. AWS regenerates the report at most once every four hours. For each user, read access_key_1_last_rotated and access_key_1_last_used_date (and the access_key_2_ pair). Old or idle keys are your first candidates.

Retire keys in two moves: deactivate first, delete later. A deactivated key can be switched back on in seconds if something breaks; a deleted one can't. See managing access keys.

aws iam update-access-key --user-name ${USER_NAME} \
  --access-key-id ${ACCESS_KEY_ID} --status Inactive

Recent use is not the full inventory

Before you deactivate a key, find every workflow that references it, not only the ones that ran recently. A release job or a quarterly report can sit idle for months and then fail on the day it matters. The same applies to roles: IAM's last-used data for a role only covers the trailing 400 days.

Cost: free.

5. Move CI to OIDC instead of stored keys

The most common long-lived key is the one in your CI system's secrets. With OIDC federation, the CI job presents a signed token and assumes an IAM role for the length of the run, so there's no key to leak. I cover the full trust policy in GitHub Actions OIDC trust policy for AWS.

Three things matter when you plan this across many repositories. Keep a sub condition, because IAM rejects a GitHub trust policy without one, and add the repository_owner_id claim to pin your organization by its numeric ID. Accept both subject formats GitHub now issues, because newer repositories use the immutable one. And run OIDC alongside the old key: retire the key only after each environment has had a green run on OIDC.

Verify by watching the old key's last-used date in the credential report. When it stops moving, the pipeline no longer needs it. Cost: free.

6. Turn on a budget and Cost Anomaly Detection

Abuse of a compromised account, such as crypto-mining instances, often shows up on the bill first. AWS Budgets are free to monitor, and the first two action-enabled budgets are free too. Cost Anomaly Detection watches spending patterns rather than a fixed ceiling.

Console

In Billing and Cost Management , open Budgets , create a monthly cost budget a little above your normal spend, and add email alerts at 80% and 100% of actual spend to your group address. Then open Cost Anomaly Detection and review the monitor and its alert subscription.

CLI

aws budgets create-budget --account-id ${ACCOUNT_ID} \
  --budget '{"BudgetName":"monthly-cost","BudgetLimit":{"Amount":"500","Unit":"USD"},"TimeUnit":"MONTHLY","BudgetType":"COST"}' \
  --notifications-with-subscribers '[
    {"Notification":{"NotificationType":"ACTUAL","ComparisonOperator":"GREATER_THAN","Threshold":80,"ThresholdType":"PERCENTAGE"},
     "Subscribers":[{"SubscriptionType":"EMAIL","Address":"aws-billing@your_domain"}]},
    {"Notification":{"NotificationType":"ACTUAL","ComparisonOperator":"GREATER_THAN","Threshold":100,"ThresholdType":"PERCENTAGE"},
     "Subscribers":[{"SubscriptionType":"EMAIL","Address":"aws-billing@your_domain"}]}
  ]'

Replace 500 with a figure that fits your account.

The default anomaly alert is set for a bigger bill than yours

If you started using Cost Explorer on or after 27 March 2023, AWS created a default anomaly monitor for you. Its alert only fires on anomalies above $100 and above 40%, according to the Cost Anomaly Detection FAQ. On a small bill that may never trigger. Lower the threshold to something that would actually surprise you.

Cost: free.

Do this month: detection and data protection

7. Record every Region with CloudTrail

CloudTrail is your record of who did what, and when; without it, an incident investigation has nothing to read. A multi-Region trail also covers the Regions you don't use. With Organizations, create an organization trail from the management account so every member account is covered.

First enable trusted access for CloudTrail (organizations only) and create a bucket for the logs:

aws organizations enable-aws-service-access \
  --service-principal cloudtrail.amazonaws.com
aws s3 mb s3://${TRAIL_BUCKET} --region your-region
aws s3api put-bucket-policy --bucket ${TRAIL_BUCKET} \
  --policy file://trail-bucket-policy.json

The bucket policy has to let CloudTrail write to the bucket. Copy it from the Amazon S3 bucket policy for CloudTrail, using the organization variant if you're creating an org trail. Then create the trail and start it, following the organization trail CLI steps:

aws cloudtrail create-trail --name org-trail \
  --s3-bucket-name ${TRAIL_BUCKET} \
  --is-multi-region-trail \
  --is-organization-trail
aws cloudtrail start-logging --name org-trail
aws cloudtrail get-trail-status --name org-trail --query IsLogging

Drop --is-organization-trail (and the trusted-access command) for a standalone account. The output will look similar to:

Output

true

Per CloudTrail pricing, the first copy of management events is free; you pay for the S3 storage. If you later enroll accounts into AWS Control Tower while they still have their own account-level trails, you can end up paying for the same events twice, so remove the redundant trails first.

8. Turn on GuardDuty in every Region

GuardDuty is AWS's threat detection service: it analyzes CloudTrail events, VPC flow logs and DNS logs and raises findings for suspicious activity. It's Regional, and a Region where workloads can run but GuardDuty is off is a Region nobody is watching.

aws guardduty create-detector --enable --region your-region

Route findings to email through an EventBridge rule and an SNS topic. Start with the event pattern:

guardduty-findings-rule.json

{
  "source": ["aws.guardduty"],
  "detail-type": ["GuardDuty Finding"]
}

Then create the topic, subscribe your group address, and point the rule at the topic:

TOPIC_ARN=$(aws sns create-topic --name guardduty-findings \
  --query TopicArn --output text)
aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol email \
  --notification-endpoint aws-security@your_domain
aws events put-rule --name guardduty-findings \
  --event-pattern file://guardduty-findings-rule.json
aws events put-targets --rule guardduty-findings \
  --targets "Id"="sns","Arn"="$TOPIC_ARN"

The topic's access policy must allow events.amazonaws.com to call sns:Publish, or EventBridge drops the findings without telling you. Confirm the subscription email, too. EventBridge rules are Regional, so repeat this in each Region. With a GuardDuty delegated administrator, create the rules in that account, one per Region, because it receives member findings in each Region.

To verify the whole path, generate sample findings and confirm one reaches the inbox:

DETECTOR_ID=$(aws guardduty list-detectors --query 'DetectorIds[0]' --output text)
aws guardduty create-sample-findings --detector-id ${DETECTOR_ID}

GuardDuty keeps findings for 90 days, so export them to S3 if you want a longer record. You don't need to turn on VPC flow logs for GuardDuty, which reads that data independently; enable them later if you want them for your own forensics.

Cost is usage-based. In the first pricing tier in us-east-1, GuardDuty pricing lists $4.00 per million CloudTrail events and $1.00 per GB of flow and DNS logs analyzed. New detectors also turn on most protection plans (S3, EKS, Malware, RDS, Lambda), each billed separately; review them on the trial's usage page and turn off what you don't run. The 30-day trial lets you measure first.

9. Run IAM Access Analyzer for external access

IAM Access Analyzer reads your resource policies and reports anything reachable from outside your account or organization. External access analysis is free. Analyzers are Regional, so create one per Region.

aws accessanalyzer create-analyzer --analyzer-name external-access \
  --type ACCOUNT --region your-region
aws accessanalyzer list-findings-v2 --analyzer-arn ${ANALYZER_ARN}

Use --type ORGANIZATION from the management or delegated administrator account to cover every member account. Organization analyzer findings are visible only in the account that owns the analyzer (the management account or the delegated administrator). Don't create a duplicate analyzer in a member account because the list looked empty from there.

The finding I once reasoned my way past was a queue policy that allowed any principal ("Principal": "*") and relied on an aws:SourceArn condition naming an S3 bucket. S3 bucket ARNs carry no account ID, so that condition doesn't pin access to your account; if the bucket is ever deleted, anyone can create one with the same name. Access Analyzer was right to call it public. If isPublic disagrees with your reading of a policy, trust the analyzer. AWS's confused deputy guidance recommends pairing it with aws:SourceAccount:

Condition block that pins the source account

"Condition": {
  "ArnLike": { "aws:SourceArn": "arn:aws:s3:::example-bucket" },
  "StringEquals": { "aws:SourceAccount": "123456789012" }
}

Optional: the unused-access analyzer

A second analyzer type reports roles, users, keys and permissions nobody has used in a set period. It isn't Regional.

aws accessanalyzer create-analyzer --analyzer-name unused-access \
  --type ACCOUNT_UNUSED_ACCESS \
  --configuration '{"unusedAccess":{"unusedAccessAge":90}}'

This one is paid: $0.20 per IAM role or user per month. AWS's pricing example puts 70 principals at $14 a month; at thousands of roles it becomes hundreds of dollars a month. Setup details are in creating an unused access analyzer.

10. Block public S3 access at the account level

Bucket-level settings protect one bucket. The account-level Block Public Access setting protects every bucket, including the next one somebody creates.

Find your intentionally public buckets first

The account setting overrides bucket policies and ACLs that grant public access. If you serve a static site or public downloads straight from S3, turning this on breaks them. List those buckets first and move them behind CloudFront, or accept that this account can't use the setting yet.

Console

In the S3 console, choose Account and organization settings in the navigation pane. Under Block Public Access settings for this account , choose Edit , turn on all four settings and choose Save changes , then type confirm and choose Confirm.

CLI

aws s3control put-public-access-block --account-id ${ACCOUNT_ID} \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

Verify with aws s3control get-public-access-block --account-id ${ACCOUNT_ID}. The output will look similar to:

Output

{
    "PublicAccessBlockConfiguration": {
        "BlockPublicAcls": true,
        "IgnorePublicAcls": true,
        "BlockPublicPolicy": true,
        "RestrictPublicBuckets": true
    }
}

Cost: free. To hold this setting across an organization, use an Organizations S3 policy (a declarative policy type) rather than an SCP deny (item 19).

11. Encrypt new EBS volumes by default

EBS encryption by default makes every new volume and snapshot copy encrypted without anyone remembering a flag. It's a per-Region setting, and existing volumes stay as they are.

Console

In the EC2 console, select the Region, choose Settings in the navigation pane, then the Data protection and security tab. In the EBS encryption section, choose Manage , select Enable , then choose Update EBS encryption.

CLI

aws ec2 enable-ebs-encryption-by-default --region your-region
aws ec2 get-ebs-encryption-by-default --region your-region \
  --query EbsEncryptionByDefault

The output will look similar to:

Output

true

Repeat for every Region you use. Cost: free.

12. Make IMDSv2 the account default

The instance metadata service hands an EC2 instance its role credentials, and IMDSv2 requires a session token for every request. You can set it as the account default for new instances, per Region:

aws ec2 modify-instance-metadata-defaults --region your-region \
  --http-tokens required --http-put-response-hop-limit -1
aws ec2 get-instance-metadata-defaults --region your-region \
  --query AccountLevel.HttpTokens

The output will look similar to:

Output

"required"

Set the hop limit to 2 only if containers on the instance need the instance role, because going into a container counts as an extra network hop. To keep containers away from the instance's credentials, set it to 1 on those instances. -1 means no preference: at launch it becomes 2 for AMIs marked ImdsSupport: v2.0, such as Amazon Linux 2023, and 1 otherwise. Existing instances are unchanged. Cost: free.

Enforcement makes launches fail

Adding --http-tokens-enforced enabled turns the default into a hard rule: any launch that explicitly asks for IMDSv1 (a launch template or AMI with HttpTokens: optional) fails, and IMDSv1 can't be re-enabled on existing instances. Update your templates first, then turn on enforcement.

While you're on EC2: use Session Manager instead of opening SSH to the internet. The SSB lists it as WKLD.06. Instances no longer need port 22 open, and every session can be logged.

13. Move secrets out of code and environment files

Database passwords and API tokens in a repository or a committed .env file outlive every rotation policy you write. Move them to Secrets Manager at $0.40 per secret per month, or to Parameter Store SecureString parameters, where standard parameters are free. I'd use Secrets Manager where you want automatic rotation. To check a repository's history for secrets already committed, run a scanner such as gitleaks (gitleaks detect in the repository).

14. Turn on Security Hub CSPM with standards you chose

Security Hub CSPM (cloud security posture management) runs continuous checks against your configuration using AWS Config, and gathers findings from GuardDuty and Access Analyzer in one place. It needs the AWS Config recorder on in each Region, with IAM and other global resources recorded in one Region only, as the Security Hub prerequisites describe. Two settings then decide whether it's useful.

The first is the standards. Turning CSPM on with default standards enables AWS Foundational Security Best Practices and CIS v1.2.0, even though CIS v1.4.0, v3.0.0 and v5.0.0 are available. Decline the defaults and pick the versions you actually want to be measured against:

aws securityhub enable-security-hub --no-enable-default-standards
aws securityhub describe-standards --query 'Standards[].StandardsArn'
aws securityhub batch-enable-standards \
  --standards-subscription-requests StandardsArn=${STANDARDS_ARN}

Pass each ARN you want from the describe-standards output. The second setting is the home Region.

Match the home Region to where IAM is recorded

If you use central configuration or cross-Region aggregation, and your home Region differs from the Region recording global resources, the IAM controls sit at WARNING or DISABLED and never evaluate. On the estate I worked on, that is exactly where they sat until the home Region was moved to match.

Verify that the IAM. controls show PASSED or FAILED; DISABLED or WARNING means they aren't running.

CSPM costs $0.0010 per check for the first 100,000 checks, and the first 10,000 finding ingestions a month are free, per CSPM pricing. AWS Config is billed separately at $0.003 per configuration item recorded (Config pricing). There's a 30-day trial, and the console shows estimated costs during the trial.

AWS also sells a newer unified Security Hub, priced differently: the Essentials plan is $3.75 per resource unit, where one unit is one EC2 instance, 12 Lambda functions, 18 ECR images or 125 IAM users and roles (unified pricing). Make sure you're signing up for the product you meant.

Do as you grow: multi-account guardrails

One account means one blast radius: a mistake in a test stack can reach production, and everyone who needs test access gets a door into prod. Once you have more than a couple of people shipping, move to AWS Organizations with at least separate production and non-production accounts, keep the management account for billing and policies only, and apply the guardrails below.

15. Turn on centralized root access

Your security tooling will flag "root MFA missing" on member accounts. Accounts created in Organizations now start with no root credentials, but older and invited accounts often still have a password, and anyone who controls that root email inbox can reset it. The fix is centralized root access, which removes member root credentials and lets the management account run privileged root tasks when one is needed.

aws organizations enable-aws-service-access --service-principal iam.amazonaws.com
aws iam enable-organizations-root-credentials-management
aws iam enable-organizations-root-sessions

Once member root credentials are removed, AWS evaluates those accounts as not applicable for root MFA rules, so the Security Hub root MFA control (IAM.9) stops failing for them.

16. Attach a baseline SCP

A service control policy (SCP) caps what principals in member accounts can do, whatever their IAM permissions say. A baseline SCP stops anyone, including an attacker with an admin role, from leaving the organization, switching off detection or acting as root. The SCP guide covers the mechanics, and AWS's deny-changes-to-security-services examples go further than this one.

baseline-scp.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyLeaveOrg",
      "Effect": "Deny",
      "Action": "organizations:LeaveOrganization",
      "Resource": "*"
    },
    {
      "Sid": "ProtectDetection",
      "Effect": "Deny",
      "Action": [
        "cloudtrail:StopLogging",
        "cloudtrail:DeleteTrail",
        "cloudtrail:UpdateTrail",
        "cloudtrail:PutEventSelectors",
        "guardduty:DeleteDetector",
        "guardduty:UpdateDetector",
        "guardduty:DisassociateFromMasterAccount",
        "securityhub:DisableSecurityHub",
        "securityhub:DisassociateFromAdministratorAccount",
        "access-analyzer:DeleteAnalyzer"
      ],
      "Resource": "*",
      "Condition": {
        "ArnNotLike": {
          "aws:PrincipalArn": ["arn:aws:iam::*:role/your-security-admin-role"]
        }
      }
    },
    {
      "Sid": "DenyRootUser",
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "ArnLike": { "aws:PrincipalArn": "arn:aws:iam::*:root" },
        "Null": { "aws:AssumedRoot": "true" }
      }
    }
  ]
}

The Null condition exempts privileged root sessions started from the management account (item 15), so centralized root tasks keep working. The root statement follows AWS's own example.

Replace your-security-admin-role with the name of the role your security tooling uses. If you use Control Tower, add arn:aws:iam::*:role/AWSControlTowerExecution to the same ArnNotLike list. Attach the policy to a test OU (organizational unit, a folder of accounts) first, watch CloudTrail for AccessDenied errors from legitimate work, then widen it.

Control Tower has one more catch: registering an OU attaches its own guardrails, not your baseline SCP, so compare attached policies with a governed OU. I cover that gap in how to move an AWS account to another organization.

17. Restrict Regions with an SCP

A Region-deny SCP limits accounts to an allowed list, so resources can't appear in Regions nobody watches. Start from the Region-controls/Deny-access-to-AWS-based-on-the-requested-AWS-region.json example in the aws-samples SCP examples repository, the IAM Region-deny example, or the Control Tower Region deny control.

Inventory S3 and every Regional service, not only EC2

S3 operations resolve to the bucket's own Region. If a bucket lives outside your allowed list, the policy makes it unmanageable. On the estate I worked on, a pre-check that only looked at EC2 missed buckets in another Region. The policy was detached within minutes and only a sandbox was affected, but the pre-check now sweeps S3 bucket locations and the other Regional services before any attach, not EC2 alone.

If one team needs an extra Region, attach a variant policy to its OU rather than widening a shared one, which would loosen it for production too.

18. Check RDS snapshot sharing

Resource control policies (RCPs) set an organization-wide data perimeter for supported services, but they don't cover RDS, so check database snapshot sharing directly:

for id in $(aws rds describe-db-snapshots --snapshot-type manual \
    --query 'DBSnapshots[].DBSnapshotIdentifier' --output text); do
  aws rds describe-db-snapshot-attributes --db-snapshot-identifier "$id" \
    --query "DBSnapshotAttributesResult.DBSnapshotAttributes[?AttributeName=='restore'].AttributeValues[]" \
    --output text | grep -qw all && echo "PUBLIC: $id"
done

A restore attribute value of all means the snapshot is public. For Aurora, run the same check with describe-db-cluster-snapshots and describe-db-cluster-snapshot-attributes. Repeat per Region.

19. Hold settings in place with declarative policies

Some controls are states, not actions: S3 Block Public Access on, IMDSv2 required, AMIs and snapshots never shared publicly. Declarative policies hold those settings at a fixed value across the organization, enforced by the service itself. SCP vs RCP vs declarative policies compares all three policy types, RCPs included.

Verify the controls actually work

When a check fails, suspect the check first. On the estate I worked on, I logged more than a dozen cases where a "failure" turned out to be a bug in the tooling, not drift in the account. The habits that came out of it:

Sort before you diff. AWS returns policy JSON with keys and arrays in no fixed order. Sort both recursively before comparing a deployed policy with the one in your repository, or every comparison reports drift.
Normalize identifiers. Some APIs return full ARNs where others return bare IDs. aws controltower list-enabled-baselines, for example, returns ARNs. Compare like with like.
Ask the right account. Before declaring a service absent, query the delegated administrator account. Organization-wide findings often live only there.
Treat errors as unknown, not clean. I've seen a rate-limited code search report "no matches" because every throttled call came back empty. A sweep must tell "checked and clean" apart from "the check failed".
Include a control case. Run every check against one resource you know is compliant and one you know isn't. If both pass, the check is broken.

A small helper for the first habit:

normalize_policy.py

import json

def normalize(value):
    """Recursively sort dict keys and list items so two policies compare equal."""
    if isinstance(value, dict):
        return {key: normalize(value[key]) for key in sorted(value)}
    if isinstance(value, list):
        items = [normalize(item) for item in value]
        return sorted(items, key=lambda item: json.dumps(item, sort_keys=True))
    return value

def same_policy(deployed: str, expected: str) -> bool:
    return normalize(json.loads(deployed)) == normalize(json.loads(expected))

What it costs

You can't price this without your own usage numbers, so price from the model and let the trials measure it. Every figure below comes from AWS's pricing pages; check them for your Region.

Service	What you pay for	Free allowance or trial
CloudTrail	Additional event copies, S3 storage	First copy of management events free
GuardDuty	$4.00 per million CloudTrail events, $1.00 per GB flow and DNS logs (first tier, us-east-1), plus protection plans	30-day trial
Access Analyzer	Unused access: $0.20 per role or user per month	External access analysis free
Security Hub CSPM	$0.0010 per check (first 100,000)	10,000 finding ingestions a month free; 30-day trial
AWS Config	$0.003 per configuration item recorded	No free tier
Secrets Manager	$0.40 per secret per month	Parameter Store standard parameters free
AWS Budgets	Action-enabled budgets beyond the first two	Monitoring free

Pricing models only. Your total depends on activity, account count and Regions.

The only worked totals I'd quote are AWS's own: 70 principals in Access Analyzer's unused-access analysis cost $14 a month, and the AWS Config pricing page walks through its own example. For everything else, start the GuardDuty and Security Hub CSPM trials in the same week and take their real usage to whoever approves the spend.

Troubleshooting

Symptom	Likely cause	Fix
Security Hub IAM controls stuck at WARNING or DISABLED	Home Region differs from the Region recording global resources	Align the home Region with the AWS Config global-resource Region (item 14)
Access Analyzer shows no findings in a member account	Organization analyzer findings are visible only in the account that owns the analyzer	Look there; don't create a duplicate analyzer (item 9)
No GuardDuty emails after sample findings	SNS topic policy doesn't allow EventBridge to publish, or the subscription isn't confirmed	Fix the topic policy and confirm the subscription (item 8)
Duplicate CloudTrail charges after Control Tower enrollment	Old account-level trails still running alongside the organization trail	Remove the redundant trails (item 7)
EC2 launches fail after an IMDS change	Enforcement on, with templates still asking for IMDSv1	Update templates to require IMDSv2, then re-enable enforcement (item 12)
`AccessDenied` on an S3 bucket after a Region SCP	Bucket lives in a Region outside the allow list	Detach or adjust the policy, then inventory S3 in every account (item 17)
Public website bucket stopped working	Account-level Block Public Access overrides the bucket policy	Serve it through CloudFront, or exclude that account (item 10)
No anomaly alerts ever arrive	Default monitor threshold of $100 and 40% is above your spend pattern	Lower the alert threshold (item 6)

The full checklist

Copy this into your tracker; it adds three related tasks to the 19 controls. Tick an item only after its verify step passes.

☐ Root user: MFA on, no access keys, group email address
☐ Security alternate contact set to a group address
☐ People sign in through IAM Identity Center; no human IAM users with passwords
☐ Credential report reviewed; old access keys deactivated, then deleted
☐ CI pipelines on OIDC; old CI keys' last-used dates have stopped moving
☐ Monthly budget with alerts; anomaly alert threshold lowered
☐ Multi-Region (or organization) CloudTrail trail logging
☐ GuardDuty on in every allowed Region; sample finding reached the inbox; unused protection plans off
☐ Access Analyzer external access analyzer in every Region; findings reviewed
☐ S3 Block Public Access on at the account level
☐ EBS encryption by default on in every Region
☐ IMDSv2 required as the account default; enforcement planned
☐ SSH closed to the internet; Session Manager in use
☐ Secrets moved to Secrets Manager or Parameter Store; repository history scanned
☐ Security Hub CSPM on, standards chosen, IAM controls showing PASSED or FAILED
☐ AWS Backup plan running; one restore tested
☐ Organizations with separate prod and non-prod accounts
☐ Centralized root access on
☐ Baseline SCP tested on an OU, then attached; every Control Tower OU carries it
☐ Region SCP attached after a full S3 and Regional-service inventory
☐ No RDS or Aurora snapshots shared publicly
☐ Declarative policies holding S3, IMDS and public-sharing settings

Conclusion

You now have a working order for the controls that matter most from the AWS Startup Security Baseline, plus the multi-account guardrails it leaves out: identity this week, detection this month, guardrails as you add accounts. The first tier costs nothing; the second has a real but measurable bill. The decision to bring to whoever owns the budget: approve the free items now, and approve the GuardDuty and Security Hub CSPM trials with a date to review their usage.

Whatever you turn on, run its verify step. A control that is enabled but not evaluating looks exactly like one that works, right up until you need it. If you're moving to more than one account, read SCP vs RCP vs declarative policies before writing your first guardrail, and GitHub Actions OIDC trust policy for AWS before retiring your CI keys.

If you'd rather have someone walk your accounts against this list, I do short AWS security and governance reviews: written findings, prioritized fixes and the commands to verify each one. Consulting starts at $75 an hour (see rates). The first step is a free 20-minute intro call, or you can request a quote for a security review directly.

🚀 Embracing the Inevitable: Change as the Sole Constant in Software

Shahid Yousuf — Sat, 11 Nov 2023 15:08:51 +0000

In the exhilarating realm of software development, one fundamental truth prevails – change is the only constant. As technology hurtles forward, the evolution of design patterns and principles has become the compass guiding developers through the dynamic landscape of innovation

In the ever-changing landscape of software design, the evolution of patterns and principles has been pivotal. Over the years, these design foundations have adapted to meet the demands of dynamic development. Here's a glimpse into this transformative journey:

Object-Oriented Design to SOLID Principles: From the foundational principles of Object-Oriented Design (OOD), the industry embraced SOLID principles (Single Responsibility, Open/Closed, Liskov Substitution, Interface Segregation, Dependency Inversion). This shift refined the approach to building maintainable and scalable software by emphasizing modularity and flexibility.
Evolution of Gang of Four (GoF) Design Patterns: The timeless Gang of Four design patterns (like Singleton, Factory, Observer) laid the groundwork for effective software design. As development needs evolved, so did these patterns. Modern applications often incorporate variations and adaptations, showcasing the dynamic nature of design patterns to address contemporary challenges.
Introduction of Domain-Driven Design (DDD): DDD emerged as a paradigm shift, placing a strong focus on the domain itself. By aligning design with the business domain, DDD provides a strategic approach to modeling complex systems. This design evolution fosters a shared understanding between developers and domain experts, enhancing the effectiveness of software solutions.
Event-Driven Architecture (EDA): With the rise of scalable and distributed systems, Event-Driven Architecture gained prominence. Design patterns like Event Sourcing and CQRS (Command Query Responsibility Segregation) allow systems to handle events efficiently, enabling adaptability to changing business requirements and ensuring robustness in complex workflows.
Microservices and Design Patterns: The shift towards microservices architecture introduced new design challenges and opportunities. Patterns like API Gateway, Service Mesh, and Circuit Breaker became crucial for managing the intricacies of microservices. These patterns enable the construction of resilient, scalable, and independently deployable services, aligning with the need for agile development.
Responsive UI with Frontend Design Patterns: As user interfaces evolved, so did frontend design patterns. The advent of patterns like Redux for state management, and Component-Based Architecture for building reusable UI elements, showcased a shift towards more responsive and modular frontend design, accommodating the demands of modern user experiences.

In conclusion, the evolution of design patterns and principles mirrors the industry's commitment to refining approaches for building robust and adaptable software. Embracing new paradigms and adapting existing patterns ensures that software design remains responsive to the ever-changing demands of technology. 🌐