DEV Community: 晖莫

The Runbook That Sent Me to a Dashboard That No Longer Exists

晖莫 — Sun, 04 Oct 2026 13:00:02 +0000

The alert fired at 02:14. Payments queue depth climbing, no consumers draining it. The runbook told me to check the "Payments Worker Health" dashboard in Grafana. It did not exist. Someone had renamed it to svc-payments-consumer; the runbook still pointed at the old name.

Fine. Annoying, but survivable. Two steps later it told me to restart the consumer with payments-worker restart --queue=main. The binary is now ledger-consumer. That command has not existed for months.

I typed it anyway. command not found. I sat at 02:19 with a paging incident and a document that had lied to me twice. Then I closed the runbook and stopped reading it.

The cost is not the two wrong steps

Here is what happened next. Everything else in the runbook — checking broker backlog before restarting anything, the DLQ inspection, the note that a stuck consumer is usually a broker problem — I skipped. I went to memory and guesswork.

I restarted the consumer. That was the wrong move. The broker had partitioned and the consumer was holding a lock it could not release. Restarting made it worse. The runbook had a section warning about exactly that. I never got to it.

That is the real failure mode. It is not that step four is stale. It is that a responder who has been lied to twice treats the remaining eight steps as suspect, including the correct ones. A bad runbook destroys the value of the good parts.

Why they rot

Runbooks get written once, right after an incident, by the person with the most context. That person already understood the system, so they wrote an index into their own head. "Check the Payments Worker Health dashboard" made sense to them, not to a stranger.

Then the system moved. Dashboards get renamed during cleanup. Commands get consolidated when a repo splits. Nobody deletes the old line, because deleting it is not part of any ticket, and whoever would notice never needs the runbook.

No bug, no alerting on documentation drift. It drifts until an incident finds out.

What makes a runbook survive

A step survives if a tired person can run it and see, without judgment, whether it worked. That means executable and verifiable:

# Is the consumer actually draining? Expect lag to fall over ~30s.
ledger-consumer status --queue payments --format json | jq '.lag_seconds'

# Broker-side backlog. If this is climbing while consumer lag is flat,
# the consumer is not the problem. Do not restart it.
rabbitmqctl list_queues name messages_ready messages_unacknowledged | grep payments

The commands can be copied and run, and each has an expected result next to it. "Expect lag to fall" tells the responder what done looks like. No interpretation required.

A link beats a description. Do not name a dashboard — paste the URL. Do not describe where the log lives — paste the query. A link that 404s is a signal you can act on. A name that quietly points at a renamed thing is a trap.

Every runbook needs a named owner and a review date at the top. Not a team. A person:

owner: @dana
reviewed: 2026-08-14
review_by: 2027-02-14

The date converts an invisible problem into a calendar item, and tells a responder how much to trust the page. A runbook reviewed four months ago is different from one reviewed three years ago.

"Contact X" is not a step

"Page the payments team" is not a step. It is an admission that the runbook has nothing to say, and it fails at the moment it matters: the named person is asleep, on a plane, or already in the incident channel asking what you tried.

Replace it with what you would ask them to do. If the answer is "ask Dana to run the replay tool," then the step is how to run the replay tool, what a good run looks like, and what to do if it fails. If Dana is genuinely the only one who can do it safely, write the threshold that makes it Dana's problem and what to do meanwhile. Escalation is a fallback, not a step.

Fix it during the incident, not after

Runbooks rot because updating them is a separate chore with its own ticket, and separate chores lose. Do not make it separate. Make the runbook part of the fix.

My rule now: the incident is not closed until the runbook the responder actually used has been corrected. Not in a follow-up. Before the postmortem, before the channel goes quiet. If you hit a stale step, fix it in the same session while the context is still in your head.

Concretely, the closing checklist becomes:

Every step you ran: did it work as written?
Every step that failed: corrected now, with a command that exists.
Any new step you improvised: added.
Owner and review date bumped.

Anyone who fought an incident has already debugged the documentation. That knowledge lasts about an hour. Spend it in the same session, on the same page. Otherwise the next responder at 02:14 gets the same broken dashboard, the same dead command, and one less reason to trust the rest of it.

I write about production failures in Postgres, queues, and distributed systems.

Subscribe by email · RSS · Bluesky

Configuration Drift Is a Production Incident With a Long Fuse

晖莫 — Sat, 03 Oct 2026 13:00:02 +0000

The checkout pods came back up after a routine node rotation and started returning 502s on every request that touched the payments path. Same image tag, same application version, same deploy log as the week before. The pods were healthy. The CPU was flat. The payment client was timing out against a provider that answered every health check we threw at it.

We restarted pods. We rolled back a deploy that turned out to be unrelated. We checked the provider status page twice. The answer was an environment variable that only existed on the nodes we had just replaced.

The change nobody wrote down

Months before that morning, the payment provider had a slow afternoon. Requests were hanging on a client timeout longer than our load balancer was willing to wait, so every slow response turned into a 502 at the edge. A deploy through the pipeline meant review, a build, and a canary. An operator with console access ran one command against the running deployment instead, raising the client timeout, and watched the error rate fall.

It worked. The graph went green. Everyone moved on to the next page. The variable lived in the running environment and nowhere else.

That is how drift accumulates. It is rarely one dramatic edit. It is a kubectl set env here, a console slider there, an env var added to a single task definition because staging had the wrong value, a cron schedule someone patched directly on the box because the repo version had a typo in it. Each change is small, each is justified in the moment, and each is invisible to anyone who was not in the room.

The bill arrives later, and it arrives somewhere else.

Why the incident makes no sense

Drift failures are confusing because every signal you normally trust is silent. The code did not change, so the deploy log is clean. The image tag did not change, so the registry tells you nothing. Nobody touched the config repo, so the diff you would normally read is empty.

What changed is the ground under the code. A node rotation, a scale-up, an instance refresh, a region failover — anything that builds new instances from the repo's version of the truth instead of the running environment's version. The old instances still carry the hotfix. The new ones do not. For a while you run a mixed fleet, which is the worst version of the problem, because two pods behind the same service will answer the same request differently.

The symptom usually looks like a hard failure with a soft explanation. A timeout that comes back from the dead. A feature flag that turns itself off. A connection pool that quietly shrinks to its original size. If you catch yourself saying "it works on the old pods," you are looking at drift.

Compare the running config to the repo

The only comparison that finds drift is running state against committed state. Documentation describes intent at the time someone wrote it, and that someone may have left the company. The repo is the source. Everything else is hearsay.

So make the running environment describe itself, and diff that description against the repo:

# Dump the live environment and diff it against the committed one.
kubectl exec deploy/payments -- env | sort > /tmp/live.env
grep -E '^[A-Z_]+=' config/payments.env | sort > /tmp/repo.env
diff -u /tmp/repo.env /tmp/live.env

Run it on a schedule, not while you are on fire. Anything present in /tmp/live.env and missing from /tmp/repo.env is a finding, and so is any value that differs. The same shape works for database parameters, feature-flag defaults, and infrastructure a console manages.

Here is the file that diff compares against, and the value that leaked out of it:

# config/payments.yaml — what the service is supposed to run with.
payment:
  clientTimeoutMs: 8000
  retries: 3
  circuitBreaker:
    enabled: true

The console edit set clientTimeoutMs to 20000. The repo still says 8000. Every pod created from the repo gets 8000, and the provider is still slow.

The rule that stops it

If it is not in the repo, it does not exist. That is the whole rule. A setting that lives only inside a running process is not configuration. It is an undocumented mutation with an expiry date, and the expiry date is the next restart.

The escape hatch is real, so write it down before you need it. Emergencies happen, and sometimes changing a value by hand at 2 a.m. is the correct call. The hatch has a price: every manual change gets a commit opened against the repo in the same shift, linked in the incident channel, with the reason in the message. The incident does not close until that commit merges. No commit, no fix.

Then drift becomes visible in the one place people actually look. Reviewers see the change. The next operator inherits the reason, not just the value. And the next node rotation deploys an environment that matches the working one, because the working one was never a secret.

I write about production failures in Postgres, queues, and distributed systems.

Subscribe by email · RSS · Bluesky

Your edge rate limit is not protecting the database

晖莫 — Fri, 02 Oct 2026 13:00:03 +0000

Our search endpoint's p99 fell off a cliff on a Tuesday afternoon. The edge dashboard was calm: request rate per IP flat, 429s near zero, WAF quiet. Postgres was logging FATAL: remaining connection slots are reserved, and every service on that database was stuck waiting for a pool connection. I had spent the morning tuning the wrong layer.

Request rate and query cost are different units

The edge counts requests per IP per second. That is a scalar. One request equals one unit, no matter what the request does.

The database counts concurrent queries and what each one costs. A request that returns a memoized ID is nearly free. A request that scans a large table with LIKE '%term%', or runs a report with several joins and an unindexed ORDER BY, holds a backend process, memory, and often a lock for seconds. A thousand cheap requests and a dozen expensive ones are the same number to the edge and completely different numbers to Postgres. Your limit can be perfectly enforced and still irrelevant, because the thing you capped is not the thing that ran out.

The mismatch is worse than it looks. The edge limit is per second. The database ceiling is concurrent connections, a fixed integer set at boot, minus the superuser reserve. Once those slots are taken, no request gets one until another finishes. That is a queue with a hard wall, and the wall is reached by duration multiplied by concurrency, not by rate.

Per-IP limits do nothing against one valid token

A per-IP rule assumes the load is spread across many addresses. The incidents I have been paged for came from one client with one valid bearer token. A mobile app retrying a failed export. An internal job fanning a report out across every tenant. A customer's integration polling a list endpoint with a filter that defeats the index.

One IP. One identity. Well under any sane per-IP threshold. Each request burning hundreds of milliseconds of database time. The edge sees a polite client. The pool sees a stampede.

Caching is the real protection for expensive reads

If a query is expensive and its inputs repeat, do not throttle it. Do not run it. A cache protects the database because it removes the work instead of declining it.

The catch is the key. A cache keyed on the URL with a good global hit ratio can still miss on every request to the one endpoint that matters, because each request carries a different combination of sort, page, and date range. Key on the normalized query and put the filter set in the key. Bound the cardinality, or you have built a slower database with worse durability.

For reads that genuinely cannot be cached, the fallback is a cheaper query: an index that matches the filter, a materialized view refreshed on a schedule, a precomputed count. Measure before guessing. Run EXPLAIN (ANALYZE, BUFFERS) with real parameters and look at actual rows and actual time. Those two numbers decide whether the answer is a cache, an index, or an admission limit.

Put the limit where the resource is

The limit belongs next to the pool, not at the edge. A semaphore in front of the connection pool caps concurrency at the exact resource that runs out.

import asyncio

DB_CONCURRENCY = 32
db_gate = asyncio.Semaphore(DB_CONCURRENCY)

async def fetch_report(pool, query, params):
    async with db_gate:                 # the real ceiling: concurrent queries
        async with pool.acquire() as conn:
            await conn.execute("SET LOCAL statement_timeout = '2s'")
            return await conn.fetch(query, params)

Two things are doing the work here. The semaphore is a ceiling on concurrent queries, not on requests per second, so it fails closed exactly when the database would. statement_timeout means a query that goes wrong dies inside Postgres instead of holding a backend forever. Set it at the transaction level so it does not leak into the next checkout of that connection.

For anything heavier, add a cost-based admission check: ask the planner for an estimate, and if the projected cost exceeds a budget, push the job to a queue or reject it with a clear error rather than starting work you cannot finish.

Then instrument the resource, not the edge. Watch pg_stat_activity for state, wait event, and query start time. A long-running query is load. A session sitting in idle in transaction is a leak, and no rate limit will fix it.

The edge limit is still worth having

None of this makes the edge rule useless. It absorbs the dumb traffic: credential stuffing, scrapers, a retry loop that never authenticates. It is cheap, it is global, and it keeps noise away from your application. Keep it.

Just stop treating it as the last line of defence. The edge limit protects the edge. The semaphore, the statement timeout, and the cache protect the database. Count the thing that runs out.

I write about production failures in Postgres, queues, and distributed systems.

Subscribe by email · RSS · Bluesky

The Retry You Added to the Write Path Made Duplicates Worse

晖莫 — Thu, 01 Oct 2026 13:00:04 +0000

The duplicate charges showed up in support tickets, both the same shape: one order, two rows in payments, same amount, same card, a few hundred milliseconds apart. The gateway logged two POSTs with different request ids and no shared key. Our retry wrapper had done exactly what we told it to.

The retry policy came from a shared HTTP client that had only ever wrapped reads. max_attempts=3, exponential backoff, retry on timeout, retry on connection errors, retry on any 5xx. Someone reused the call site for a write, and nothing in that client objected. The policy was correct for GETs and wrong for POSTs, and the code could not tell the difference.

A timeout is not a failure

A ReadTimeout means the client stopped waiting. It does not mean the server stopped working. The usual sequence is: TCP connect succeeded, request line, headers and body went out, the server started processing, the response came back late or never. The socket gave up on our side. On their side the row was already committed.

On a read path that mistake costs one extra query, so nobody notices. On a write path it costs one extra order. Same exception, same handler, different blast radius.

Before the request left, and after

These are different failures and they need different handling. Name the exception rather than catching Exception.

A ConnectTimeout or ConnectionRefusedError on a fresh connection happens before anything reached the server. Nothing was written. Retrying is reasonable, with one caveat: make sure no bytes of the request body were sent, because a ConnectionError raised after a partial send is not the same event.

A ReadTimeout, or RemoteDisconnected after the body was sent, is the unknown case. The server may have processed the request completely. It may have processed it and then failed while writing the response. There is no local signal that separates "never arrived" from "arrived and committed." The two situations collapse into one exception class, which is exactly why a blanket decorator is unsafe.

Retrying a 500 from a peer

A 500 is not evidence that nothing happened. It is evidence that something broke after the request arrived. The handler ran. The write may have committed and the error may have come from a post-commit step: sending a receipt, updating a search index, serializing a response. Retrying that POST inserts the row a second time.

A 502 or 503 from a proxy is more ambiguous. The upstream may never have been reached, or it may have received the request and answered too slowly for the proxy. Unless you own that layer and know its behavior, treat it as unknown too.

The only condition under which retrying a write is safe is idempotency: the server can distinguish a repeat from a new intent. That requires a key that travels with the original request, not one generated when the retry fires.

Make the key part of the request

# wrong: a fresh key on every attempt, so the retry looks like a new charge
def charge(order_id, amount):
    for attempt in range(3):
        key = str(uuid4())
        try:
            return gateway.post("/charges", json={"amount": amount},
                                headers={"Idempotency-Key": key})
        except (ReadTimeout, ConnectionError):
            continue

# right: the key comes from the intent and is stable across attempts
def charge(order_id, amount):
    key = "charge:%s" % order_id
    for attempt in range(3):
        try:
            return gateway.post("/charges", json={"amount": amount},
                                headers={"Idempotency-Key": key})
        except ConnectTimeout:
            continue                      # nothing was sent
        except ReadTimeout:
            raise UnknownWrite(order_id)  # arrived, outcome unknown

The second version makes the retry a repeat of the same request. The gateway sees charge:4471 again, finds its stored result, and returns that instead of taking the money twice. If the endpoint does not implement the key, no client-side change makes the retry safe.

Treat a timed-out write as unknown

The honest state after a ReadTimeout on a POST is unknown. Not failed. Code that models it as failed retries, and the retry is what creates the duplicate.

What to do instead:

Retry only writes you have made idempotent, and derive the key from the operation: order id, job id, a value you already persist. Never generate it inside the retry loop.
Classify before you retry. Retry a connect failure that provably happened before the body was sent. Do not retry a response timeout, a 500 from a peer that received the request, or an ambiguous 502 unless the operation is idempotent.
On an unknown write, stop and reconcile. Read the resource back by the key you sent, or query the server's idempotency record, before doing anything else.
Store the key on the write path so a retry after a process restart uses the same value. A key built in memory does not survive a crash.
If the endpoint has no idempotency key and you cannot add one, the only safe retry is one where no request was sent. Everything else needs a reconciliation job or a human.

To see how often this bites you, measure the timeout and 5xx rates on write endpoints separately from read endpoints, and count duplicate-key violations or duplicate rows per endpoint. A write endpoint with a nonzero timeout rate and no idempotency key is already producing duplicates. They show up as support tickets before they show up in your dashboards.

I write about production failures in Postgres, queues, and distributed systems.

Subscribe by email · RSS · Bluesky

ORDER BY Without a Tiebreaker Is a Flaky Test Generator

晖莫 — Wed, 30 Sep 2026 13:00:04 +0000

I had a test that asserted orders.first().id == 9917. It passed locally for months. Then it failed in CI, twice in a row, then passed again on a rerun. Nobody had touched the query.

The query was this:

SELECT id, created_at, total_cents
FROM orders
WHERE customer_id = $1
ORDER BY created_at DESC
LIMIT 1;

The test seeded three orders for one customer inside a single transaction. All three got created_at = now(). now() is transaction_timestamp() — it does not advance during a transaction. So three rows shared the exact same timestamp to the microsecond.

ORDER BY created_at DESC gives one value. Three rows tie on it. Postgres picks whichever it happens to read first. That order depends on the plan, the physical row order, the visibility map, and whether it chose an index scan or a seq scan. Local Postgres and CI Postgres do not make the same choice.

The standard does not promise you anything here

SQL does not define which row comes back when the sort key ties. The ORDER BY clause establishes a partial order, not a total one. Two rows with equal keys are "equal" for sorting purposes, and the engine may emit them in any order. No standard text says otherwise.

This is not a Postgres bug. It is Postgres behaving correctly. The bug is in the query, and then in the test that trusted it.

LIMIT 1 makes it worse. Without the limit you would see the instability — the row order would visibly shuffle between runs. LIMIT 1 hides it and turns it into a coin flip that lands on heads in your terminal.

Pagination is where this stops being a test problem

I hit this again in a feed endpoint. Keyset pagination over created_at alone:

SELECT id, created_at, body
FROM posts
WHERE created_at < $1
ORDER BY created_at DESC
LIMIT 20;

A backfill wrote a chunk of posts with the same created_at, all from one job running now() in one transaction. The first page ended at created_at = '2026-09-14 10:03:22.481913+00'. The next page asked for everything strictly before that value. Every post sharing that timestamp got skipped. Users saw a gap in the feed and I spent an afternoon blaming the client.

Flip the comparison to <= and you get the mirror failure: the boundary rows repeat on every page. Offset pagination has the same shape. OFFSET 20 counts rows in an order the database never promised, so a row inserted or reordered between requests can push another row across the page boundary.

Add a unique column to the sort key

The fix is one line and it is not optional. Every ORDER BY that feeds a LIMIT, an OFFSET, or a cursor needs a unique tiebreaker. The primary key is right there.

ORDER BY created_at DESC, id DESC

Now the sort key is unique. (created_at, id) is a total order over the table. Ties cannot exist, so the engine has no freedom left to exercise. LIMIT 1 returns exactly one row, deterministically.

Pagination needs the same treatment in the predicate, not just the sort:

SELECT id, created_at, body
FROM posts
WHERE (created_at, id) < ($1, $2)
ORDER BY created_at DESC, id DESC
LIMIT 20;

Row-value comparison does the lexicographic work in one expression. Pass the last row's created_at and id as the cursor. Nothing is skipped, nothing repeats.

For the index, an index on (created_at DESC, id DESC) matches the sort directly. Check with EXPLAIN (ANALYZE, BUFFERS) — if you see a Sort node above the scan, the index is not matching your order and you are paying for it on every page.

Make it a rule

A tiebreaker is not a stylistic preference. It belongs in review. Two habits make it stick for me:

If I write ORDER BY, I write the unique column in the same keystroke. ORDER BY created_at DESC alone looks unfinished now.
Any test that reads .first() or .last() from a query must sort on a unique key, or it is not testing the query — it is testing the plan.

I also stopped seeding multiple rows with now() when the test cares about their order. clock_timestamp() advances; now() does not. Better yet, set explicit timestamps in the fixture so the data has the order the test claims to assert.

The failure mode is quiet. Nothing errors, no constraint fires, no log line appears. You get a wrong row, a missing row, or a green test that owes you a red one later. Add the tiebreaker.

I write about production failures in Postgres, queues, and distributed systems.

Subscribe by email · RSS · Bluesky

Blue-green deploys do not save you from a bad migration

晖莫 — Tue, 29 Sep 2026 13:00:05 +0000

The deploy went green in ninety seconds. Then the error rate on blue hit 100%, and every request touching /orders returned column "orders.status_v2" does not exist. Two healthy app stacks, no working rollback.

Blue-green only ever protected one thing

Blue-green gives you two copies of your application code and a router that flips between them. It does not give you two copies of your database, because you cannot cheaply fork a live Postgres cluster with continuous writes and merge the forks later. Both environments point at the same host, the same schema, the same tables.

So the moment migrate finishes, blue runs new code against a new schema and green runs old code against that same new schema. The swap is atomic. The schema change is not.

Here is the migration that took us down:

ALTER TABLE orders DROP COLUMN status;
ALTER TABLE orders ADD COLUMN status_v2 text NOT NULL DEFAULT 'pending';

Order matters. Green still selects status. Between the DROP and the moment green stops serving, every green query fails. Green keeps serving for as long as the router takes to drain, plus every in-flight request, plus every retry. The rollback plan said "flip back to blue." Blue was already broken by a migration that ran before the flip.

Rollback means two different things

Code rollback is a pointer change: flip the router, and the old artifact still exists. It works only while the data underneath still fits.

Schema rollback is not symmetric. The reverse of DROP COLUMN is ADD COLUMN, and the values are gone. You cannot un-drop a column, un-truncate a table, or un-run an UPDATE that rewrote rows in place. The database has one state, it moves forward, and it has no previous release to point at.

There is a third, worse case: the migration fails halfway. Postgres runs most DDL in a transaction, so a failed ALTER TABLE inside BEGIN rolls back cleanly. But a file with several statements, some CONCURRENTLY, or explicit commits between steps leaves a schema that is neither the old shape nor the new one. That is the state nobody rehearsed.

Reversible schema, irreversible data

The distinction I should have drawn before that deploy.

A schema change is reversible when old code can still run against the new schema. Adding a nullable column, an index, a table — all reversible, because old code ignores what it does not know about.

A data change is irreversible when the migration rewrites or removes values. Dropping a column, a lossy type cast, a backfill of computed values — the new shape is not derived from the old one, so no reverse migration reconstructs it.

A reversible schema change often accompanies an irreversible data change. That is the trap. ADD COLUMN status_v2 alone is safe. ADD COLUMN plus a backfill plus a DROP COLUMN in one file is a one-way door.

Expand, contract, flag

The safe path has three releases, not two.

Release one: expand. Add the new column as nullable, and dual-write in application code. Old code reads the old column and keeps working. New code writes both. Nothing breaks if you roll back, because both shapes exist.

def write_order(order):
    db.execute(
        "UPDATE orders SET status = %s, status_v2 = %s WHERE id = %s",
        (order.legacy_status, order.status, order.id),
    )

Release two: backfill in batches, outside the deploy, with a query you can stop. Backfill must be restartable. Run it in chunks, log the last processed id, and make a rerun a no-op.

Release three: switch reads to the new column behind a feature flag. The flag is the rollback. If reads break, flip the flag, not the deploy. Only after the flag has been on for a full traffic cycle do you contract — drop the old column in a boring release where the code no longer references it.

Do the flag before the drop, not after. A flag is reversible in seconds; a DROP COLUMN is not.

When the migration is already half-applied

Stop the deploy. Do not roll back the app, and do not run a second migration to "fix" the first. Both make the schema harder to reason about.

Find out what actually landed. In Postgres, read the real state instead of trusting your migration table:

SELECT column_name, data_type, is_nullable
FROM information_schema.columns
WHERE table_name = 'orders'
ORDER BY ordinal_position;

Then check whether the migration is holding a lock. pg_stat_activity and pg_locks show a blocked ACCESS EXCLUSIVE waiting on a long-running read. Kill the blocker if it is a stale report query, or your next deploy deadlocks.

Then pick a direction and commit. If the old shape still exists, disable the new code path with the flag and leave the schema alone — you are back to release one. If the old shape is gone, the only path is forward: ship code matching the current schema, patch the failing queries today, and accept that you are deploying under pressure. Restore from a backup only if you can afford to lose everything written since the migration, and measure that window first.

The lesson I keep: the database is not a deployment target. It is a shared, single-state dependency, and every migration is a change to production data. Rehearse the migration on a copy with real row counts, and time how long the lock is held.

I write about production failures in Postgres, queues, and distributed systems.

Subscribe by email · RSS · Bluesky

The Dashboard Nobody Trusts: A Metric Without an Owner

晖莫 — Mon, 28 Sep 2026 12:59:52 +0000

The checkout dashboard has a panel called payment_gateway_queue_depth. It has been at 0 for eleven months. Nobody noticed, because nobody looks at it. It was added at 2am during an incident, and the person who added it left the team in March.

That is the normal story. A panel appears when we are scared, it answers one question once, and then it stays on the dashboard forever. Within a year it is flat, wrong, or so noisy that people scroll past it.

Does knowing it change what you do at 3am?

This is the only test that matters. Take a panel and ask: if this number were alarming right now, what would I do differently? If the answer is "nothing, but it would be interesting," the panel is decoration.

I use a blunter version with my team. Pick the metric. Now finish this sentence: "If this spikes, I will ..." If you cannot fill in the blank with a command, a rollback, or a page to another team, the metric is not operational. It is trivia with a graph attached.

The same test deletes alerts. An alert that fires and gets ignored is worse than no alert, because it trains everyone to treat the pager as background noise. If an alert has fired more than a handful of times and nobody acted on any of them, it is a deletion candidate. Either fix the threshold so it only fires when action is needed, or delete the rule and keep the graph for humans to read during incidents.

Collected because it was easy

Most dead metrics got there the same way: the exporter emitted it for free, so we scraped it. Request rate per endpoint, response size percentiles, cache hit ratio by region — all collected, none attached to a decision.

There is a real difference between a metric with a decision attached and one collected because it was easy. The first one has a threshold, an owner, and a runbook. The second one has a dashboard slot.

Service-level indicators sit on the decision side. They describe what the user experiences: successful checkouts per minute, search latency under 500ms, payment authorization success rate. Vanity throughput graphs sit on the other side. Total requests served tells you the system is busy. It does not tell you whether anyone got what they came for.

I have watched a service serve 40% more requests than the week before while its success rate dropped, and the throughput graph looked like a win. The SLI panel next to it was the one that mattered.

When a service has no SLI, that is the panel to build. Not another throughput chart.

Attach an owner and a line of intent

Every panel gets two pieces of metadata, in the dashboard config itself, not in a wiki page nobody reads.

panels:
  - title: payment_gateway_queue_depth
    owner: payments-team
    intent: "If depth > 500 for 10m, page payments on-call and check gateway health."
    query: sum by (gateway) (payment_gateway_queue_depth)
    alert: PaymentGatewayQueueBacklog
    last_reviewed: 2026-09-14

The owner field is a real team, not a person, because people rotate. The intent line is the "what would we do about this" sentence, written down. If nobody can write that line, the panel does not get created.

last_reviewed is the part that keeps the whole thing alive. Once a quarter, walk the dashboard and read those dates. Anything older than two quarters gets either a fresh intent line from its owner or a delete commit. A stale panel is a claim that someone is watching. That claim is false, and it is worse than an empty dashboard because it makes us believe we have coverage.

The graph that is flat at 0 is the dangerous one. It looks healthy. It looks calm. It has not received a single data point since the exporter changed its metric name in the spring.

Measure the decay instead of guessing

Do not estimate how many panels are dead. Measure it. For each panel, query the underlying series over the last 30 days and count series with zero variance. For each alert rule, count firings and correlate them with the incident channel in the same window. Alerts that fire with no corresponding human action are your deletion list.

Then delete. Not archive — delete, so the next person who opens the dashboard sees only things that have an owner and a reason.

A dashboard is a set of promises that someone is watching. Keep the promises you can keep, and remove the rest. The panel that nobody owns does not monitor anything. It just makes the silence look intentional.

I write about production failures in Postgres, queues, and distributed systems.

Subscribe by email · RSS · Bluesky

The Unique Index That Makes Your Endpoint Retry-Safe

晖莫 — Sun, 27 Sep 2026 12:59:53 +0000

The payment form double-submitted on a flaky connection. Two POSTs hit /payments with the same idempotency key. One row landed in the table. The client got 200 OK both times. I asked why the second insert did not fail, and got three different answers from three engineers.

The real answer was in a migration from before any of us joined.

CREATE UNIQUE INDEX payments_idempotency_key_idx
    ON payments (idempotency_key);

That index, and nothing in the Python, is what made the endpoint safe to retry.

The constraint is the retry policy

The handler looked like this.

def create_payment(req):
    try:
        cur.execute(
            """INSERT INTO payments (idempotency_key, account_id, amount_cents)
               VALUES (%s, %s, %s)
               ON CONFLICT DO NOTHING""",
            (req.key, req.account_id, req.amount_cents),
        )
        db.commit()
    except Exception:
        db.rollback()
    return {"status": "accepted"}

Note the bare ON CONFLICT DO NOTHING. It names no target and no index. That clause compiles whether or not a unique constraint exists. When one exists, the duplicate is absorbed. When none exists, the same code inserts a second row and returns success. The endpoint's retry safety lives entirely in the schema, not in the code.

The except Exception makes it worse. It catches the genuine constraint violation just as quietly as it catches a dropped connection, so an absorbed duplicate and a rolled-back write produce the same response.

Why "it happens to work" is the problem

Nothing in that function announces an idempotency contract. The index looks like ordinary data hygiene, the kind someone adds during a cleanup ticket. So it gets treated as removable.

Two changes break it. Someone drops the index to cut write amplification on a hot table, and the bare ON CONFLICT DO NOTHING starts silently inserting duplicates. Or someone rewrites the branch to DO UPDATE so the response can carry the stored row, and a retry now overwrites an amount that already settled.

Neither change fails a test, because the test sends one request and asserts one row. That assertion passes with the index, without it, and with DO UPDATE. The suite was measuring the wrong thing the whole time.

Absorbed is not the same as correct

An absorbed duplicate means the database kept one row. It says nothing about what the client received. If the handler builds a fresh UUID per call, the retry gets a different resource id for the same charge, and any client that reconciles by id now holds two. If the response is just {"status": "accepted"}, the client learns nothing about which write won.

A correct replay returns the same answer twice: same status, same body, same id. That is the property clients actually depend on, and it is the property you should assert.

def test_retry_returns_same_body(client, db):
    body = {"idempotency_key": "k-1", "account_id": 7, "amount_cents": 500}
    first = client.post("/payments", json=body)
    second = client.post("/payments", json=body)
    assert first.status_code == second.status_code == 200
    assert first.json() == second.json()
    assert db.scalar(
        "SELECT count(*) FROM payments WHERE idempotency_key = 'k-1'"
    ) == 1

Run that against the handler above and the row assertion passes while the body assertion fails if you generate the id per call. That failure is the point. It tells you the guarantee was only ever half there.

Finding the accidental ones

Grep the migrations for UNIQUE and CREATE UNIQUE INDEX. Grep the handlers for ON CONFLICT and for IntegrityError. Every place that catches a unique violation is declaring a retry contract it never wrote down.

Then ask one question per handler: does any test send the same request twice? If not, the guarantee is unverified and undefended. Write the two-request test first. If it passes, you have pinned the behaviour to the schema and to a test at the same time. If it fails, you just found a latent double-write before a customer did.

Name the constraint in a comment next to the conflict clause. Not for documentation's sake, but so the next person weighing write throughput against dropping that index can see what it is holding up. A unique index that is load-bearing should read like one.

I write about production failures in Postgres, queues, and distributed systems.

Subscribe by email · RSS · Bluesky

Your Test Suite Is Slow Because It Tests Postgres

晖莫 — Sat, 26 Sep 2026 13:00:00 +0000

I quit running the test suite on my laptop. Not deliberately. I just stopped typing the command, because by the time it finished I had lost the thread of whatever I was fixing. Standup would end, I would push, and wait for CI to tell me what I broke. That is the real cost of a slow suite: not the wall-clock time, but the gap between writing a line and knowing whether it works.

Ours was slow for a boring reason. Every test opened its own connection, applied the migrations, and talked to a real Postgres instance. A test that checked whether an email address was normalized still paid for a connection, because the fixture chain handed it a session it never used.

Find out what the tests are actually waiting on

The number to measure is not total runtime. It is per-test setup cost. Time each fixture, or wrap connection setup in a timer and log it, then sort the results. What you want is the ratio between time spent arranging infrastructure and time spent running the code under test. If setup dominates, the suite is measuring Postgres, not your logic. pytest --durations=20 names the slowest tests, but add your own fixture timing too, because the fixture is where the money goes.

Then read each test and ask one question: does this test fail if the SQL is wrong? Not "does it touch a table" — does it assert something only a database can tell you.

Which tests actually need a database

A test needs Postgres if it exercises one of three things:

a query — the WHERE clause, a JOIN, an index assumption, an ORDER BY that depends on collation
a constraint — a unique index, a foreign key, a check constraint, a NOT NULL
a transaction boundary — rollback on error, isolation, locking, a deferred constraint

Everything else is using the database for convenience. If a test loads a row through a repository, calls a function on it, and asserts on the return value, the database was an expensive way to construct an object. That test wants a plain object.

The line is not "unit versus integration" by file layout. A test that calls UserRepository.get(id) against a real connection is an integration test even when it lives in tests/unit/ and never starts an HTTP client. Naming a directory does not change what the process does.

Shared schema, transaction per test

The fix that buys back most of the time is not deleting the database. It is applying the schema once and isolating each test in a transaction that rolls back.

@pytest.fixture(scope="session")
def engine():
    engine = create_engine(TEST_DATABASE_URL)
    Base.metadata.create_all(engine)  # migrations run once per session
    yield engine
    engine.dispose()

@pytest.fixture
def session(engine):
    connection = engine.connect()
    transaction = connection.begin()
    session = Session(bind=connection, join_transaction_mode="create_savepoint")
    yield session
    session.close()
    transaction.rollback()  # nothing survives the test
    connection.close()

The schema is built once. Each test gets a connection and a transaction, and rollback is the cleanup. No truncate loop between tests, no migrations re-run per test, no ordering dependencies leaking through leftover rows. Tests that need to observe a real commit opt out with an explicit marker, so the expensive path is visible in the file instead of hidden in a fixture.

One caveat: code that opens its own connection will not see the test's uncommitted rows. Make the session injectable, or move those tests into the small set that runs against a real server. Do not paper over it with a shared connection.

The tests that genuinely need Postgres

Keep them. Do not mock the query planner. A small set of tests that hit real Postgres and assert on real SQL behavior is worth more than a pile of tests that pretend, but it should be a set you can name, not the default for everything in the suite.

Mark them (@pytest.mark.postgres), run them on every push in CI, and keep the fast tier on every save locally. The default command runs the fast tests; the database tests are one flag away. When they fail, the failure means something about the SQL, which is exactly the signal you wanted from the database in the first place.

Calling something a unit test does not make it fast, and importing a repository does not make it an integration test. What matters is whether a real database is the only thing that can fail.

I write about production failures in Postgres, queues, and distributed systems.

Subscribe by email · RSS · Bluesky

Cache invalidation is a domain problem, not a TTL setting

晖莫 — Fri, 25 Sep 2026 12:59:57 +0000

Support pasted a screenshot at 9:40am: a customer's invoice page showed a balance of $0.00, and a second tab showed $412.60. Same account, same minute. No deploy that morning. The database was right. Our cache was wrong, and nothing in the system was responsible for telling it so.

We had a TTL. That is the part that stings. Every key expired in five minutes, which felt careful at the time, and for two years it was fine because nobody reconciled two tabs at once.

A TTL is a bet, not a correctness mechanism

A TTL says: I believe this answer stays true for N seconds. That is a guess about your write patterns, disguised as a config value. It has no idea that invoices.balance.8842 was just invalidated by a payment webhook. When a TTL-driven cache is wrong, it is not broken — it is behaving exactly as designed and answering a question about the past.

The honest framing: a cache is a stored copy of a decision you already made. The database decided the balance was $412.60. Our cache held a copy of the decision that said $0.00, made before the payment landed. The only interesting question is which writes make that copy wrong, and which code paths know about those writes.

If you want to know how much staleness you actually tolerate, measure it. Write the timestamp of the cached write into the value, and log now - write_time on every read of a key that later turns out stale. That distribution tells you what your TTL is really costing. Do not guess the number; sample it.

The write path is the only thing that knows

Reads cannot invalidate. A read has no idea whether another writer is mid-transaction. The write path — the one place that changes the underlying decision — is the only code that knows the old answer is now garbage.

We ran cache-aside: the reader did GET, missed, queried Postgres, then SETEX. It looked clean. The bug was ownership. Nobody owned invoices.balance.{id}. The reader created it. The payment service, which actually changed the balance, had never heard of the key. So the payment service wrote to Postgres and the cache kept serving its stale copy until the TTL swept it away.

Write-through inverts that. The writer updates the store and the cache in one place, so the key has an owner:

def record_payment(account_id, amount, conn):
    with conn.transaction():
        conn.execute(
            "UPDATE invoices SET balance = balance - %s WHERE account_id = %s",
            (amount, account_id),
        )
        row = conn.execute(
            "SELECT balance FROM invoices WHERE account_id = %s", (account_id,)
        ).fetchone()
    # The writer owns the key. Reads never create it.
    redis.set(f"invoices.balance.{account_id}", row["balance"], ex=300)
    redis.publish("invoice.changed", str(account_id))

The ex=300 is now a safety net for a missed write, not the correctness story. That is the difference. When invalidation is owned by the writer, the TTL becomes a backstop you hope never fires.

Be careful with delete-on-write too. DEL then re-populate races: a reader can miss, read the pre-commit row, and SETEX the old value back on top of your fresh write. Delete outside the transaction and re-populate from the committed row, or version the key.

Expiry is a coordinated attack on your database

The other failure nobody plans for: everyone's key expires at the same moment. A popular key with a 300-second TTL, written at 10:00:00 during a batch job, expires at 10:05:00 for every reader at once. All of them miss, all of them hit Postgres, and the database that was comfortably handling 200 reads per second suddenly takes the full load.

Three things helped. Jitter the TTL so keys written together do not die together. Collapse concurrent misses onto a single in-flight fetch so one request rebuilds and the rest wait on it. And for the hottest keys, refresh in the background before expiry so the miss never happens in the request path. Test this deliberately: expire your top key by hand during peak traffic and watch the database.

Make invalidation an event, not a cron

The fix that finally held was deleting the sweeper job. We had a cron that walked recent invoices and refreshed their cache entries every five minutes. It was a second source of truth about when data changed, and it was always slightly behind the writes it was chasing.

Instead, the payment service publishes invoice.changed with the account id. A subscriber deletes the balance key. The domain already had a name for this moment — a payment was recorded — so the event existed before the cache did. The cache became a consumer of a fact the business cared about, not a timer with an opinion.

That reframing is the whole job. Stop asking what TTL to pick. Ask which writes make this answer wrong, and who in your code knows they happened. If the answer is "a cron job, eventually," you have a staleness budget, not a cache.

What the comments added

@compoundlabs found the hole in the record_payment snippet above, and it is worth fixing rather than hand-waving: redis.set runs after with conn.transaction() closes, so the commit and the invalidation are two separate failure domains. A crash in between leaves the old balance in the cache, and the only thing that saves you is ex=300 — which means that window was still a staleness budget, just a short one. I presented the TTL as a backstop while quietly depending on it.

The fix is a transactional outbox. Write the intent to invalidate in the same transaction that changes the balance, and let something else publish it:

def record_payment(account_id, amount, conn):
    with conn.transaction():
        conn.execute(
            "UPDATE invoices SET balance = balance - %s WHERE account_id = %s",
            (amount, account_id),
        )
        conn.execute(
            "INSERT INTO outbox (topic, payload) VALUES (%s, %s)",
            ("invoice.changed", json.dumps({"account_id": account_id})),
        )

A relay tails outbox, publishes invoice.changed, then marks the row sent. The intent to invalidate now commits atomically with the fact that made it necessary, so there is no window where the database moved and nothing was told.

Delete, do not set. Outbox delivery is at-least-once, so the consumer has to be idempotent. Deleting a key is; SETting a recomputed value is not, because a delayed duplicate can put an older balance back on top of a newer one. That is the strongest argument for invalidating instead of writing — and as a bonus, a consumer that only deletes does not care about duplicates or out-of-order delivery at all. The moment you cache a value derived from the event, you owe each key a version and an "ignore anything older than what I already applied" rule.

The window does not close, it moves. Reads during relay lag still see the old value, so the thing to alert on is relay lag and rows piling up in outbox — not cache hit rate. Keep the TTL as the backstop for a relay that is simply down, but stop describing it as the mechanism.

If you cannot add a table to that database, make the read safe instead: version the cached value from a column the write already bumps, and have the reader refuse a value older than the row it can see. That is an outbox in disguise, and it needs no relay.

I write about production failures in Postgres, queues, and distributed systems.

Subscribe by email · RSS · Bluesky

Zero-Downtime Migrations Need Two Deploys, Not a Clever Script

晖莫 — Thu, 24 Sep 2026 12:59:59 +0000

Our checkout page started timing out at 14:02. Not slowly — completely. Every request that touched the orders table hung, connection pool filled, and the whole app went down behind it. The migration I had just run took 40 milliseconds. That was the part that confused me for a while.

ALTER TABLE orders ADD COLUMN currency text NOT NULL DEFAULT 'usd'; is fast. It also takes an ACCESS EXCLUSIVE lock on orders. That lock does not care how fast the statement runs. It cares that every read and write already in flight has to finish first, and everything arriving after it queues behind. A 40ms statement plus a 3-second queue of waiting queries is a 3-second outage on that table. Multiply by the retry storm and the pool is gone.

The lock is the outage. Statement speed is almost irrelevant.

The shape that actually works

The broken mental model is "one migration, made atomic and careful." The real constraint is different: during a rollout, old instances and new instances are both live. Both run against the same schema for some window. A migration that only works for the new code breaks the old code, and a migration that only works for the old code breaks the new code. Neither version of a single-script plan survives that window.

So the change has to be split. Expand, then contract. Both halves are boring on their own, and that is the point.

Expand: add, write both, backfill, read new

Expand is the additive half. Add the new column or table without breaking anyone, then move traffic over one step at a time.

-- Step 1: nullable, no default rewrite, no long lock.
ALTER TABLE orders ADD COLUMN currency text;

-- Step 2: a NOT VALID constraint takes only a brief lock and
-- does not scan the table. Existing rows are not checked yet.
ALTER TABLE orders
  ADD CONSTRAINT orders_currency_not_null
  CHECK (currency IS NOT NULL) NOT VALID;

Then the code deploys in stages. First a release that writes both the old and the new field, still reading the old one. Then a backfill, in batches, in the background. Then a release that reads the new field and keeps dual-writing. Then a release that stops writing the old field.

The backfill is where I have caused the second-worst incidents. A single UPDATE orders SET currency = 'usd' WHERE currency IS NULL on a large table rewrites every row, generates a huge amount of WAL, and holds row locks for the duration. On a primary with streaming replicas, that WAL is what your replicas have to replay. Push too much and replication lag climbs. Replicas fall behind the primary, and read-your-own-write breaks for users hitting them.

Batch it, sleep between batches, and watch lag as the loop runs, not after.

while True:
    rows = db.execute("""
        UPDATE orders SET currency = 'usd'
        WHERE id IN (
            SELECT id FROM orders
            WHERE currency IS NULL
            ORDER BY id LIMIT 1000
            FOR UPDATE SKIP LOCKED
        )
        RETURNING id
    """)
    if not rows:
        break
    time.sleep(0.05)
    if replica_lag_seconds() > 5:
        time.sleep(2)

Measure the batch size against your own table. What matters is that you check replication lag inside the loop and slow down or stop when it grows.

Lock timeouts are the safety net

Before any DDL, set a lock timeout in the same transaction:

BEGIN;
SET lock_timeout = '3s';
SET statement_timeout = '15s';
ALTER TABLE orders ADD COLUMN currency text;
COMMIT;

If the ALTER cannot get its lock within 3 seconds, it fails instead of blocking every query behind it. A failed migration is a retry. A queued ACCESS EXCLUSIVE lock is an outage. I would rather rerun a migration than explain a 14:02 incident again.

This does not make the migration safe by itself. It makes the failure mode small.

Contract: drop last, after nothing reads it

Dropping is the easy half to get wrong, because it feels finished the moment the new code ships. It is not. Drop the old column only after you have confirmed no running code reads it and no query in your logs still references it. In a rolling deploy with long-lived instances, "the new release is out" and "old instances are gone" are different facts.

Break the work across two deploys at minimum, usually more. The clever single script is the wrong shape because it assumes one instant where the schema and all running code shift together. That instant does not exist in a rolling deploy, and every design that pretends it does is a lock waiting for traffic.

Check the lock mode of any statement you plan to run, keep the transaction short, and put a timeout on it. Then do the other half next week.

I write about production failures in Postgres, queues, and distributed systems.

Subscribe by email · RSS · Bluesky

Your /health Endpoint Is Lying Because It Never Touches a Dependency

晖莫 — Wed, 23 Sep 2026 12:59:56 +0000

Errors spiked at 14:07 and the graphs made no sense. Success rate dropped to about 70%, not zero. Some requests succeeded, some returned 500 with a Postgres connection timeout, and a few completed normally. Our dashboard showed three healthy instances the entire time. Every probe had passed. The load balancer was still routing traffic to an app that had been unable to talk to its database for twenty minutes.

That is the failure mode. A health check that only proves the process exists keeps a broken instance in rotation, and the outage shows up as random errors instead of a clean removal.

Liveness and readiness answer different questions

Liveness asks: should the supervisor restart this process? Readiness asks: should the load balancer send it traffic? If one endpoint answers both, you get one of two bad outcomes.

When the check is shallow, it tells the supervisor "no restart needed" — correct, the process really is fine. Then it tells the load balancer the same thing, and that answer is wrong. The binary answer cannot be right for both readers.

When the check is deep and shared, a dead database makes every instance unready, so every instance gets pulled, and the load balancer has nowhere to send traffic. That is a cascading failure dressed up as a health check.

Keep them separate. This is a Kubernetes example but the split applies to any orchestrator or proxy.

livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2

/livez returns 200 if the event loop can serve an HTTP response. Nothing else. It never queries the database, never calls a downstream service, and never allocates a connection pool.

/readyz checks the things that must work for a request to succeed. It is allowed to fail, and failing means removal from rotation, not a restart.

The first time I split these apart, the restart storms stopped. The app had been OOM-restarting because the deep check ran on every liveness tick and opened database connections the collector never released. A probe that does real work has real cost, and paying that cost every ten seconds on every instance adds up.

Timeouts are the whole check

A readiness probe with no timeout is worse than no probe. If the database call hangs, the probe hangs, the orchestrator waits, and traffic keeps flowing to the stuck instance for the entire timeout window.

Give the check a hard deadline shorter than the probe's own timeout, and fail closed:

import asyncpg

async def readyz(pool):
    try:
        async with pool.acquire(timeout=0.5) as conn:
            await conn.fetchval("SELECT 1", timeout=0.5)
        return 200, {"status": "ready"}
    except (asyncpg.PostgresError, TimeoutError) as exc:
        return 503, {"status": "unready", "reason": type(exc).__name__}

Measure what your database actually returns under normal load — p99 of SELECT 1 on a warm pool — then set the probe deadline above that and far below your request timeout. Do not guess the number. Log the probe duration and read the histogram.

Cache the result too. If the load balancer hits /readyz from ten addresses per second, do not run ten queries per second. Cache the verdict for one or two seconds. The cache TTL becomes your detection latency, so keep it short, and document it.

Return 503, not 200 with a JSON body that says "status": "degraded". Load balancers read status codes. A 200 with bad news inside is a 200.

Degraded is not down

The hardest case is a dependency that is slow, not dead. The database accepts connections but queries take four seconds instead of forty milliseconds. Marking the instance unready here removes capacity exactly when the system needs all of it, and the next instance inherits the same slow database.

Shed load instead of failing. Keep /readyz at 200, and expose a second signal the application itself uses:

func (a *App) admit(ctx context.Context) error {
    if a.db.Latency() > a.degradedThreshold {
        if atomic.LoadInt64(&a.inflight) > a.maxInflight/2 {
            return ErrShedLoad
        }
    }
    return nil
}

Return 429 or 503 with Retry-After on the shed requests. The instance stays in rotation, serves what it can, and stops piling work onto a dependency that is already behind. Readiness stays green. The error rate is visible and bounded instead of total.

The process being up is not a health signal

An HTTP 200 on /health tells you a socket accepted a connection and a handler returned. It says nothing about the database, the cache, the queue, or the disk. It is a liveness signal at best, and most teams already get liveness for free from the orchestrator's process supervisor.

Before you trust the next green dashboard, do this: kill the network path from one instance to its database, then watch. If traffic keeps arriving, your health check is lying. If every instance drops out at once, it is lying in the other direction.

I write about production failures in Postgres, queues, and distributed systems.

Subscribe by email · RSS · Bluesky