DEV Community: Libme

Is Your CDN Actually Caching? Audit Hit Ratio and Asset Fan-Out Before You Migrate Buckets

Libme — Mon, 24 Aug 2026 22:04:53 +0000

Before you move a bucket to escape egress fees, measure two things: your byte-weighted cache hit ratio and your requests per unique object. A low hit ratio caused by unstable cache keys is a bug you can fix in an afternoon; a low hit ratio caused by genuinely one-off assets is a structural fact no CDN can fix, and that's the case where zero-egress storage earns the migration. Most teams never separate the two and end up paying a migration to hide a caching bug.

A commenter on an earlier post about storage egress put it well: audit how many unique assets you serve versus how many come from cache, before you touch anything.

Why doesn't my egress bill match my download volume?

Because two meters are running, and they bill different bytes.

CDN data-out: every byte delivered to a browser, cached or not.
Origin egress: only the bytes your CDN had to fetch from the bucket — your cache misses.

If you serve 5 TB to users at a 95% hit ratio, your bucket only emitted about 250 GB. That's the number a storage migration changes. The other 4.75 TB is billed by whoever runs your edge, and moving the bucket doesn't touch it.

This is why the AWS-native path confuses people: S3-to-CloudFront transfer is free, so putting CloudFront in front of S3 relabels the egress bill as CloudFront data-out rather than removing it. Cloudflare doesn't meter CDN bandwidth per GB, though as of mid-2026 its terms restrict serving large volumes of non-HTML content on non-enterprise plans.

Takeaway: a storage migration only moves the origin-egress meter, so measure cache misses — not total downloads — before estimating the savings.

How do I measure cache hit ratio without buying an analytics add-on?

Start with two curls. The first fills the cache, the second should hit it:

BASE=$(printf 'https://%s' "cdn.example.invalid")   # your edge hostname
for i in 1 2; do
  curl -sSI "$BASE/u/9f3ab2/avatar.webp" \
    | grep -iE '^(cf-cache-status|x-cache|age|cache-control|vary):'
done

If the second response still says Miss from cloudfront or cf-cache-status: MISS, your cache key is unstable and no TTL tuning will help. An Age header that resets to 0 on every request is the same symptom with a friendlier face.

Two curls tell you about one object. For the real number, parse the logs. CloudFront access logs are tab-separated with a #Fields: header line, so map columns instead of hardcoding positions:

#!/usr/bin/env python3
"""Byte-weighted hit ratio and asset fan-out from CloudFront access logs."""
import gzip, sys
from collections import Counter

HIT = {"Hit", "RefreshHit", "OriginShieldHit"}
requests_by_obj, bytes_by_obj = Counter(), Counter()
hit_bytes = miss_bytes = 0

for path in sys.argv[1:]:
    opener = gzip.open if path.endswith(".gz") else open
    with opener(path, "rt") as fh:
        fields = None
        for line in fh:
            if line.startswith("#Fields:"):
                fields = line.split()[1:]
                continue
            if line.startswith("#") or not fields:
                continue
            row = dict(zip(fields, line.rstrip("\n").split("\t")))
            size = int(row.get("sc-bytes") or 0)
            requests_by_obj[row["cs-uri-stem"]] += 1
            bytes_by_obj[row["cs-uri-stem"]] += size
            if row.get("x-edge-result-type") in HIT:
                hit_bytes += size
            else:
                miss_bytes += size

total_bytes = hit_bytes + miss_bytes
if not total_bytes:
    sys.exit("no log rows parsed - check the file paths")

uniq = len(requests_by_obj)
top = bytes_by_obj.most_common(50)
print(f"byte-weighted hit ratio : {hit_bytes / total_bytes:.1%}")
print(f"origin egress (misses)  : {miss_bytes / 1e9:.2f} GB")
print(f"unique objects          : {uniq:,}")
print(f"requests per object     : {sum(requests_by_obj.values()) / uniq:.1f}")
print(f"top 50 objects          : {sum(b for _, b in top) / total_bytes:.1%} of bytes")

For Cloudflare Logpush (newline-delimited JSON), the same numbers stream out of jq plus awk:

zcat logs/*.log.gz \
  | jq -r '[.CacheCacheStatus, .EdgeResponseBytes, .ClientRequestURI] | @tsv' \
  | awk -F'\t' '
      { bytes += $2; if ($1 == "hit") hits += $2
        if (!($3 in seen)) { seen[$3] = 1; uniq++ } }
      END { printf "hit ratio : %.1f%%\nunique URIs: %d\nreq/URI   : %.1f\n",
                   100 * hits / bytes, uniq, NR / uniq }'

Two caveats. sc-bytes is bytes sent to the viewer, so ranged and aborted downloads make it an approximation of origin fetch bytes — good enough to size a decision, not to reconcile an invoice. Counting only hit as a hit is deliberately conservative: revalidated still costs an origin round trip.

Takeaway: byte-weighted hit ratio is the only version of the metric that maps to money — a 99% request hit ratio means nothing if the 1% misses are your video files.

What makes a hit ratio quietly collapse?

Rarely short TTLs, which is what everyone checks first. It's almost always that the cache key isn't stable across requests for the same bytes:

Presigned URLs. An S3 presigned URL carries X-Amz-Signature and X-Amz-Date in the query string, and both change on every generation. If query parameters are in your cache key, every request is a unique object to the edge and your hit ratio is structurally zero. This is the big one.
Cache-busting parameters added by an image component or analytics wrapper (?v=<timestamp>, ?_=<random>).
Forwarded cookies. Include a session cookie in the key and your hit ratio equals your per-user re-download rate.
Vary: User-Agent or Vary: Accept from an image-transform layer, fragmenting one object into many variants — as do multiple hostnames for the same asset.

Check your cache policy's query-string, header, and cookie inclusion lists, then re-run the two-curl test with a stripped URL to confirm which one it is.

Presigned URLs can't be made cacheable — you can only stop using them on the hot path. What works is a path-based token with a quantized window, so every request inside that window produces byte-identical URLs:

import hashlib, hmac, time

WINDOW = 3600  # seconds; also your worst-case revocation lag

def asset_path(key: str, secret: bytes) -> str:
    slot = int(time.time()) // WINDOW
    mac = hmac.new(secret, f"{key}:{slot}".encode(), hashlib.sha256)
    return f"/a/{slot}/{mac.hexdigest()[:16]}/{key}"

Your edge (a worker, Lambda@Edge, or your own origin) recomputes the HMAC for the current and previous slot and rejects anything else. The drawback is real: revocation is coarse, so a link stays valid up to one full window after you cut access. Fine for avatars and exports; for sensitive objects, keep presigned URLs and accept that those bytes are always origin egress.

Takeaway: if assets are served through rotating signatures, your hit ratio isn't low — it's zero by construction, and that's a URL design problem, not a storage problem.

What do the two numbers actually tell me to do?

Requests per unique object	Byte-weighted hit ratio	Diagnosis	Fix first
High (20+)	High (90%+)	Working as intended	Nothing — origin egress is already small
High (20+)	Low (under 60%)	Unstable cache key or too-short TTLs	Cacheability; a migration would hide a bug
Low (1–3)	Low, and it can't go higher	Genuine long tail: one-off generated files	Storage with no egress charge
Low overall, bytes concentrated in a few objects	Mixed	A handful of large files dominate	Long TTLs on those; leave the rest alone

The third row is the one people miss. AI-generated assets — a rendered PDF, a one-off image, an export requested once and never again — have a fan-out near 1. Caching them is nearly pointless: you pay the origin fetch either way, and the object is evicted before a second request that never comes. If you want the managed version of this, Cloudflare R2 is the one that removes the egress meter entirely instead of discounting it, so long-tail traffic stops being a variable cost. Backblaze B2 gets most of the way there with free egress up to a multiple of what you store — and free to Cloudflare's network via the Bandwidth Alliance — until your read-to-store ratio drifts past that multiple.

Takeaway: fan-out near 1 is the only signal that reliably justifies migrating for egress, because it's the one thing better caching cannot improve.

How do I turn this into a before/after number?

Take the miss bytes the script printed — that's the monthly origin egress you actually pay for. Price that figure two ways: per GB above whatever allowance applies (S3, and B2 past its included multiple), or zero (R2, and B2 within the allowance). Then add operations, because with fan-out near 1 every delivered object is roughly one origin GET, and on R2 the Class A writes on the upload side are the pricier meter. Storage is usually the rounding error, which is why ranking providers by per-GB storage rate ranks them wrong.

Run the script against a month of logs before and after any cacheability fix. In my experience the fix lands first and the number moves enough that the migration case either becomes obvious or evaporates.

Takeaway: the migration decision is one subtraction — miss bytes times the egress rate, minus the operations you'd pay at the new provider.

FAQ

How do I check if my CDN is caching my S3 files?
Request the same URL twice with curl -sSI and read the x-cache (CloudFront) or cf-cache-status (Cloudflare) header. The first response will be a miss; if the second is also a miss, something in your cache key — query strings, cookies, or a Vary header — differs between the two requests.

Does putting a CDN in front of S3 eliminate egress fees?
No, it moves them. S3-to-CloudFront transfer is free, so the bucket's egress drops to near zero, but CloudFront then bills data-out to viewers. The saving comes from cache hits reducing origin fetches, not from the CDN being free.

What's a good cache hit ratio for user-uploaded assets?
For shared assets like avatars and thumbnails, aim for 90%+ byte-weighted. For one-off generated files with a fan-out near 1, even 30% may be the ceiling — there the ratio isn't the problem to solve, and egress pricing models are.

Bottom line

Run the log script before you run a migration. If requests per unique object is high and the hit ratio isn't, you have a cache-key bug — fix the presigned URLs or the stray query strings and the egress bill drops without changing providers. If fan-out is genuinely near 1, caching has nothing left to give, and zero-egress storage like R2 (or B2 within its free-egress multiple) is the honest answer. Either way, the two numbers cost you an afternoon and tell you which of those two projects you're actually signed up for.

Why Your OAuth Integration Randomly Returns invalid_grant (and How to Stop Two Workers From Racing)

Libme — Mon, 24 Aug 2026 08:14:41 +0000

If your third-party OAuth integration works for days and then dies with {"error":"invalid_grant"}, the cause is usually not clock skew, a wrong client secret, or an expired consent. It is two of your own processes calling the token endpoint with the same refresh token at the same time. When the provider rotates refresh tokens, the second call presents a token that was already consumed, and many providers treat that as replay and revoke the whole token family — which is why the user has to click "Reconnect" instead of just retrying.

The fix is to make refresh a single-flight operation per integration, and to stop treating "the access token expired" as something every worker discovers independently.

What does `invalid_grant` actually mean?

invalid_grant is the most overloaded error in OAuth 2.0. The spec assigns it to any grant that is "invalid, expired, revoked, does not match the redirect URI, or was issued to another client," so providers pile several unrelated conditions into one string. In practice, on a refresh call it means one of:

The refresh token expired (idle-expiry, common on providers that expire unused tokens).
The user or an admin revoked the app's access.
The client credentials do not match the token.
The refresh token was already used once and the provider rotates.

Only the last one comes and goes. If your error rate is bursty, correlates with traffic spikes or with cron minute boundaries, and affects healthy accounts that just worked, you are looking at rotation plus concurrency. Refresh-token rotation is a recommended practice in the OAuth 2.0 Security Best Current Practice and it is the default in OAuth 2.1 drafts, so assume any provider you integrate with today may rotate — check the response body rather than the docs, because the behavior sometimes differs per app registration.

The tell is simple: if the token response includes a refresh_token field with a value different from the one you sent, that provider rotates, and every refresh is a state mutation you have to serialize.

Why does this only blow up in production?

Locally you run one process. In production you run a web dyno, three queue workers, and a scheduled job — all sharing one row in your integrations table.

The race is boring and unavoidable without a lock:

Worker A reads the token row, sees expires_at is in the past, POSTs to the token endpoint.
Worker B reads the same row 40 ms later, sees the same stale expires_at, POSTs the same refresh token.
The provider rotates: A gets a new pair, B's request presents an already-redeemed token.
B gets invalid_grant. Depending on the provider, the family is now revoked and A's brand-new token is dead too.

That last step is what makes this worth fixing properly instead of retrying. A retry loop makes it worse: it turns one replayed token into five, which looks exactly like the attack that reuse detection exists to catch.

The window is not "when the token expires" — it is "when several workers first notice."

How do you serialize refresh across processes?

If you already run Postgres, advisory locks are the cheapest correct answer: no extra infrastructure, no lease expiry to reason about, and the lock is released automatically when the transaction ends, including on a crashed connection.

create table oauth_tokens (
  integration_id     text primary key,
  access_token       text        not null,
  refresh_token      text        not null,
  prev_refresh_token text,
  expires_at         timestamptz not null,
  rotated_at         timestamptz not null default now()
);

import hashlib
from datetime import datetime, timedelta, timezone

import httpx

REFRESH_SKEW = timedelta(seconds=120)
TOKEN_URL = "https://provider.example.com/oauth/token"


def _lock_key(integration_id: str) -> int:
    """Stable 64-bit signed key for pg_advisory_xact_lock."""
    digest = hashlib.sha256(f"oauth:{integration_id}".encode()).digest()
    return int.from_bytes(digest[:8], "big", signed=True)


def _read(conn, integration_id):
    return conn.execute(
        "SELECT access_token, refresh_token, expires_at "
        "FROM oauth_tokens WHERE integration_id = %s",
        (integration_id,),
    ).fetchone()


def get_access_token(conn, integration_id: str) -> str:
    now = datetime.now(timezone.utc)
    access, refresh, expires_at = _read(conn, integration_id)
    if expires_at - REFRESH_SKEW > now:
        return access  # fast path: no lock, no network call

    with conn.transaction():
        conn.execute("SET LOCAL lock_timeout = '15s'")
        conn.execute("SELECT pg_advisory_xact_lock(%s)", (_lock_key(integration_id),))

        # Re-read inside the lock: someone may have refreshed while we waited.
        access, refresh, expires_at = _read(conn, integration_id)
        if expires_at - REFRESH_SKEW > now:
            return access

        resp = httpx.post(
            TOKEN_URL,
            data={
                "grant_type": "refresh_token",
                "refresh_token": refresh,
                "client_id": CLIENT_ID,
                "client_secret": CLIENT_SECRET,
            },
            timeout=10.0,
        )
        resp.raise_for_status()
        body = resp.json()

        conn.execute(
            """
            UPDATE oauth_tokens
               SET prev_refresh_token = refresh_token,
                   access_token       = %s,
                   refresh_token      = %s,
                   expires_at         = %s,
                   rotated_at         = now()
             WHERE integration_id = %s
            """,
            (
                body["access_token"],
                body.get("refresh_token", refresh),
                now + timedelta(seconds=body["expires_in"]),
                integration_id,
            ),
        )
        return body["access_token"]

Three details carry most of the value. The fast path avoids taking a lock on every API call, so the hot path stays a single indexed read. The re-read inside the lock is the part people skip — without it, every worker that queued behind the lock still performs its own refresh the moment it acquires one. And body.get("refresh_token", refresh) keeps working on providers that do not rotate, so the same code path handles both.

The honest drawback: this holds a database transaction open across an HTTP call. With a 10-second client timeout and a 15-second lock_timeout, worst case you pin one pooled connection for ~25 seconds. If your pool is small or you refresh thousands of integrations concurrently, move the refresh into a dedicated worker instead of doing it inline.

Should you refresh on 401 or before expiry?

Refresh proactively. Reacting to a 401 means every worker discovers expiry at the same moment — you have built a thundering herd on purpose, and you now need the lock to be perfect because it is on the critical path of every request.

The REFRESH_SKEW above (two minutes) means the first worker to come within the skew window refreshes while the current token is still valid, and everyone else keeps using the still-good token instead of blocking. You still keep 401 handling as a fallback for revocation and for providers that expire tokens earlier than advertised, but it should be rare enough that a 401 in your logs is a signal rather than routine noise.

Treat a 401 on a token you refreshed 30 seconds ago as an alert, not a retry.

Which locking strategy fits your stack?

Approach	Safe across processes	Extra infra	Main weakness
In-process mutex / `asyncio.Lock`	No	None	Only protects one process; useless with multiple dynos
Redis `SET key val NX EX 30` lease	Mostly	Redis	Lease can expire mid-refresh; no fencing, so two refreshes are still possible
Postgres `pg_advisory_xact_lock`	Yes	None if you have Postgres	Holds a connection across the HTTP call
`SELECT ... FOR UPDATE` on the token row	Yes	None	Same connection cost, plus it blocks unrelated readers of that row
Dedicated refresh worker (one consumer per integration)	Yes	Queue	More moving parts; needs its own liveness monitoring

If you would rather not own token storage at all, Nango handles OAuth token refresh and rotation for third-party integrations as a managed service, and Merge does the same as part of a unified API layer — both cost real money per connected account and both put a vendor between you and the provider's raw API, which matters the day you need an endpoint they have not mapped.

For most teams running a single Postgres, the advisory lock is the correct amount of machinery.

How do you tell a race apart from a real revocation?

Log the discriminating fields, not the error string. On every refresh attempt, record the integration id, the worker/process id, a hash prefix of the refresh token you sent (never the token), and the rotated_at you read. Then:

Two refreshes for the same integration within seconds, from different workers → race.
One refresh, invalid_grant, and the token was last rotated hours ago → genuine expiry or revocation; re-consent is the only fix.
invalid_grant on a token whose hash prefix matches prev_refresh_token → you replayed a rotated token, usually from a retry or a stale cached copy in process memory.

That last case is why the prev_refresh_token column exists. It costs one text field and turns an unexplainable error into a one-query diagnosis.

FAQ

What causes invalid_grant on refresh token requests?
Most often the refresh token was already used and the provider rotates refresh tokens, so the second use is rejected as replay. The other common causes are user revocation, idle expiry of an unused refresh token, and client credentials that do not match the token.

Does every OAuth provider rotate refresh tokens?
No, but you cannot assume either way. Check whether the token response contains a refresh_token different from the one you sent, and write your storage code so it persists a new one whenever it appears.

Can I just retry after invalid_grant?
No. If rotation caused it, retrying replays an already-redeemed token and can trigger the provider's reuse detection, which revokes the entire token family and forces the user to reconnect. Re-read your stored token first; if it changed, another worker already refreshed and you should use the new one.

Bottom line

If you integrate with any provider that returns a new refresh_token on refresh, treat token refresh as a mutation that must happen exactly once per integration and put a cross-process lock around it — a Postgres advisory lock is enough for most teams and needs no new infrastructure. Refresh proactively inside a skew window rather than reacting to 401s, so workers never discover expiry simultaneously. Keep the previous refresh token in a column purely for diagnosis; it is the difference between "random OAuth errors" and a five-minute root cause. And never retry invalid_grant blindly — that is the one response where a retry can turn a recoverable error into a forced reconnect.

S3 vs R2 vs B2 for User Uploads: When Egress Fees Should Change Your Architecture

Libme — Sun, 23 Aug 2026 21:17:11 +0000

If your app stores files that users repeatedly download — avatars, exports, video, generated PDFs — the storage line on your bill is almost never the problem. Egress is. Amazon S3 charges per gigabyte leaving the network, Cloudflare R2 charges nothing for egress and makes its money on storage plus operations, and Backblaze B2 gives you free egress up to a multiple of what you store and bills beyond that. Pick based on your read-to-store ratio, not on the per-GB storage rate everyone quotes.

That ratio is the whole decision, and most teams never measure it before choosing.

What actually drives an object storage bill?

Four meters run at once, and only the first is the one people compare:

Storage — GB-months sitting at rest.
Egress — bytes leaving the provider's network toward the public internet.
Operations — writes/lists (usually "Class A") and reads (usually "Class B"), billed per million.
Anything in front of it — CDN requests, image transforms, function invocations.

For a backup target, storage dominates and the cheapest per-GB wins. For user uploads served back to browsers, egress dominates and can exceed storage by an order of magnitude — a 50 GB bucket whose files get pulled a hundred times a month is a 5 TB egress bill attached to a rounding-error storage bill.

Here's the shape of each provider as of mid-2026. I'm deliberately describing pricing models rather than quoting rates, because rates move and a stale number in a blog post is worse than no number:

	Amazon S3	Cloudflare R2	Backblaze B2
Egress to internet	Per GB, the dominant cost at scale	None	Free up to a multiple of stored data, per GB above that
Egress to own CDN	Free to CloudFront	N/A (Cloudflare CDN is in front)	Free to Cloudflare via the Bandwidth Alliance
Storage rate	Lowest of the three at list	Middle	Lowest of the three at list
Operations	Billed, cheap	Billed, Class A noticeably pricier than Class B	Billed, with a free daily allowance
S3 API compatibility	The reference	High, with gaps	High, with gaps
Storage classes / lifecycle	Deep (IA, Glacier tiers)	Shallow	Moderate
Regional control	Explicit regions	Location hints, not hard regions	Explicit regions

The uncomfortable part of that table for AWS shops: S3's cheap storage is real, and its egress is what funds it.

Takeaway: compare providers on egress-to-storage ratio for your actual traffic, because that ratio is the only axis where the three differ by more than a small multiple.

When is S3 still the right call?

More often than the "R2 has no egress fees" discourse suggests.

Keep uploads on S3 when your bytes mostly don't leave AWS — a bucket read by ECS tasks, Lambda, Athena, or SageMaker in the same region pays no internet egress at all. Same-region service-to-service traffic is the case S3's pricing is built for, and moving that bucket to another provider means you now pay for the round trip in latency and, on the way in, in cross-cloud transfer.

Keep it on S3 too when you need what only S3 has: lifecycle transitions into archival tiers, Object Lock for compliance retention, bucket-level replication rules, event notifications wired into the rest of AWS, or the long tail of tooling that assumes real S3 semantics. If your compliance story includes "objects are immutable for seven years," S3 is the one with the mature answer and the auditor-friendly paper trail.

Takeaway: if your read traffic terminates inside AWS, egress-free storage saves you nothing and costs you integration.

When do egress fees justify moving?

Run this before you argue about it. Pull a month of CloudFront or S3 access logs and get bytes-out against bytes-stored:

-- Athena over S3 server access logs: monthly bytes served, top prefixes
SELECT
  regexp_extract(key, '^([^/]+)/', 1) AS prefix,
  count(*)                            AS requests,
  sum(bytessent) / 1024.0 / 1024 / 1024 AS gb_sent
FROM s3_access_logs
WHERE operation = 'REST.GET.OBJECT'
  AND parse_datetime(requestdatetime, 'dd/MMM/yyyy:HH:mm:ss Z')
      >= current_date - interval '30' day
GROUP BY 1
ORDER BY gb_sent DESC
LIMIT 20;

Divide monthly GB sent by GB stored. My rough decision line, from doing this on several small products:

Under ~2x — egress is noise. Stay where you are.
2x to ~10x — worth putting a CDN in front and re-measuring. Caching at the edge often deletes the problem more cheaply than a migration does, and S3-to-CloudFront origin fetches don't incur internet egress.
Over ~10x, with a low cache hit ratio (large unique files, signed one-time downloads, ML artifacts) — this is where zero-egress pricing genuinely changes the bill, because a CDN can't cache what's never requested twice.

That last category is the honest answer to "when should I move." If you want zero-egress object storage with an S3-compatible API and a CDN already in the same network path, Cloudflare R2 is the one that removes the bandwidth line from the bill entirely. If you want the cheapest at-rest storage for large archives with generous but bounded free egress, Backblaze B2 is the one that keeps cold data cheap without archival-tier retrieval games.

Takeaway: a CDN in front of S3 beats a migration for cacheable traffic; migration wins when every download is unique.

Why does the AWS SDK break against R2 and B2?

This is the part that eats an afternoon. The AWS SDK for JavaScript v3 started sending flexible checksums (CRC32) by default on requests, and S3-compatible providers that haven't implemented those headers reject them. You get something unhelpful like:

NotImplemented: Header 'x-amz-checksum-crc32' with value '...' not implemented

or a bare 501 on what was, five minutes ago, a working PutObject. Nothing about your credentials or bucket changed — the SDK's defaults did. The fix is to make checksums opt-in:

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

const s3 = new S3Client({
  region: "auto",
  endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
  credentials: {
    accessKeyId: process.env.R2_ACCESS_KEY_ID,
    secretAccessKey: process.env.R2_SECRET_ACCESS_KEY,
  },
  // Required for non-AWS S3-compatible endpoints that reject CRC32 headers
  requestChecksumCalculation: "WHEN_REQUIRED",
  responseChecksumValidation: "WHEN_REQUIRED",
});

await s3.send(new PutObjectCommand({
  Bucket: "uploads",
  Key: "u/42/avatar.png",
  Body: bytes,
  ContentType: "image/png",
}));

Two more compatibility traps in the same family. Presigned PUT URLs generated by a checksum-happy SDK embed a signed header the browser never sends, so the upload fails signature validation at the edge rather than in your code — generate presigned URLs with the same WHEN_REQUIRED client. And CORS is configured per provider, not per SDK: a browser upload that works in curl and fails in the browser is a bucket CORS rule, every time.

Takeaway: "S3-compatible" means the API surface, not the SDK defaults — pin checksum behavior explicitly the moment you point an AWS SDK at a non-AWS endpoint.

What does a low-regret setup look like?

Keep the provider swappable so this decision stays cheap to revisit:

Talk to storage through one module that exposes put, get, presignPut, presignGet, delete. Never let a bucket name or an endpoint URL leak into request handlers.
Store the object key in your database, never a full URL. URLs change when providers do; keys don't.
Serve through your own domain via a CDN from day one. Then a provider change is an origin change, not a link-rot event across every row you've ever written.
Tag or prefix by workload (uploads/, exports/, backups/) so the query above can tell you which workload is generating egress. Aggregate bucket totals hide the one prefix that's actually the bill.

Honest drawbacks, since every tool here has one: R2's operation pricing punishes chatty small-object workloads and its lifecycle/archival story is thin, B2's free egress is bounded by your stored volume so a small bucket with viral traffic still pays, and S3 charges you for the privilege of leaving.

FAQ

Is Cloudflare R2 actually free to download from?
Egress to the internet is not billed, but storage and per-operation charges are. A workload with millions of tiny reads and writes can cost more on R2 than on S3 despite paying nothing for bandwidth.

Can I use the AWS SDK with Cloudflare R2 or Backblaze B2?
Yes — both expose S3-compatible APIs and work with the standard AWS SDKs by overriding the endpoint. On AWS SDK for JavaScript v3 you must also set requestChecksumCalculation and responseChecksumValidation to WHEN_REQUIRED, or uploads fail with a not-implemented checksum header error.

How do I know if S3 egress fees are worth migrating away from?
Divide monthly GB served by GB stored. Under about 2x, egress is noise; above about 10x with poor CDN cacheability, zero-egress providers meaningfully change the bill.

Bottom line

If your objects are read mostly by other AWS services, stay on S3 — you're not paying internet egress and you'd lose lifecycle tiers, Object Lock, and event integration for nothing. If your objects are served to browsers and cache well, put a CDN in front of S3 and re-measure before migrating; that alone resolves most egress complaints. If every download is unique and large enough that caching can't help, R2's zero-egress pricing is the one that removes the line item, and B2 is the better fit when cold storage volume dominates and your egress stays within its free multiple. Whatever you pick, keep object keys — not URLs — in your database, so the next re-measurement is a config change instead of a migration.

Before You Set plan_cache_mode, Write the Regression Test That Proves It Worked

Libme — Sat, 22 Aug 2026 23:23:25 +0000

A single fast query on a single tenant does not prove you fixed a Postgres plan-caching problem. Postgres decides between a custom plan and a cached generic plan by comparing average estimated cost across executions, so a change that rescues your smallest tenant can quietly add planning work to the tenant that generates 95% of your traffic. The test that actually settles it warms one connection past the plan switch, replays low, medium, and high selectivity parameters, and asserts latency, rows, buffers, and plan shape — not just wall-clock on the one case that hurt.

A reader made this point on an earlier post of mine about prepared statements, and it's the part I had underbuilt. Below is the harness I use now.

Why doesn't one fast query prove the fix?

Postgres builds a custom plan (re-planned with your actual parameter values) for the first five executions of a prepared statement, averages their estimated cost, and from the sixth execution onward compares that average against a generic plan built with no knowledge of the values. If the generic plan doesn't look more expensive, it locks in.

Two consequences fall out of that mechanism, and both break naive tests:

The decision is made on an average. Whether a generic plan wins depends on which parameters warmed the statement. Warm it with your rare, highly selective tenant and you may get a different outcome than production, where 95% of executions carry the dominant value.
Forcing custom plans is not free. plan_cache_mode = force_custom_plan means re-planning on every execution. For a query planned in 0.4 ms and executed in 900 ms, that's noise. For a short OLTP query planned in 3 ms and executed in 1.2 ms, you just tripled its cost — for the workload that dominates your CPU.

Takeaway: the fix and the regression are the same change viewed from two different parameter distributions, so the test has to carry both.

How do you make a test session use the cached generic plan?

The hard part is that plan caching is connection-local state. Open a fresh connection, run the query once, and you measure a custom plan — the exact thing that never reproduces the bug. You have to warm past the switch inside the same session, using the parameter distribution production actually sends.

The most direct way to do that is a SQL-level prepared statement, which puts the counters somewhere you can read:

PREPARE q(text) AS
SELECT id, status, created_at
  FROM events
 WHERE tenant_id = $1 AND status = 'pending'
 ORDER BY created_at DESC
 LIMIT 50;

-- warm with the DOMINANT parameter, not the pathological one
EXECUTE q('bigcorp');  -- x6

SELECT name, custom_plans, generic_plans
  FROM pg_prepared_statements WHERE name = 'q';

pg_prepared_statements exposes custom_plans and generic_plans counters in PostgreSQL 14 and later, and they are the only honest answer to "did my warmup take?" If generic_plans is still 0 after warmup, your test is measuring a custom plan and every assertion below it is meaningless.

What tripped me up the first time: the plan cache is keyed to the statement, and EXPLAIN (ANALYZE) SELECT … is a different statement from SELECT …. If you warm the raw query and then inspect with an EXPLAIN-wrapped copy, you are inspecting a freshly planned custom plan and will conclude, wrongly, that everything is fine. Prepare once, then EXPLAIN the EXECUTE.

Takeaway: assert on generic_plans > 0 before you assert on anything else, because a warmup that silently failed produces a green test that proves nothing.

Building fixtures from the real parameter distribution

Three buckets is usually enough, drawn from your actual data rather than invented:

Bucket	How to pick it	What it catches
High selectivity (rare value)	A parameter matching a tiny fraction of rows	The original pathology — generic plan sequential-scans for a 400-row answer
Median	The value nearest the median row count per key	Drift in the middle of the distribution as data grows
Dominant (hot value)	The value your logs show most often, usually also the largest	Regression from the fix — added planning cost on the hot path

Pull them with one query rather than guessing:

SELECT tenant_id, count(*) AS rows
  FROM events
 GROUP BY tenant_id
 ORDER BY rows DESC;

This only works against a database whose statistics resemble production. A uniformly seeded test dataset cannot reproduce a skew bug, because with uniform data the generic plan is genuinely correct. If you can't run against a production-shaped copy, this whole class of test is theater — restore a sanitized dump instead.

Takeaway: fixtures chosen from count(*) GROUP BY are evidence; fixtures chosen because they looked representative are assumptions.

What should the test assert besides latency?

Latency alone is flaky under CI noise. Assert the physical work and the plan shape too — those are stable enough to gate a merge on.

# plan_stability_test.py  (psycopg 3)
import psycopg
from psycopg import sql

DSN = "postgresql:///app"  # same role and database as the app

STMT = """
SELECT id, status, created_at
  FROM events
 WHERE tenant_id = $1 AND status = 'pending'
 ORDER BY created_at DESC
 LIMIT 50
"""

# label, parameter, max_ms, max_shared_blocks, expected plan family
BUCKETS = [
    ("rare",     "acme",     15.0,    400, "Limit[IndexScan(events_tenant_status_created_idx)]"),
    ("median",   "midco",    60.0,   5000, "Limit[IndexScan(events_tenant_status_created_idx)]"),
    ("dominant", "bigcorp", 250.0,  60000, "Limit[IndexScan(events_tenant_status_created_idx)]"),
]

def plan_family(node):
    """Node-type fingerprint: ignores row counts, catches plan-shape changes."""
    label = node["Node Type"].replace(" ", "")
    if "Index Name" in node:
        label += f"({node['Index Name']})"
    kids = ",".join(plan_family(k) for k in node.get("Plans", []))
    return f"{label}[{kids}]" if kids else label

def explain_execute(cur, param):
    q = sql.SQL("EXPLAIN (ANALYZE, BUFFERS, SETTINGS, FORMAT JSON) EXECUTE q({})")
    cur.execute(q.format(sql.Literal(param)))
    return cur.fetchone()[0][0]

def main():
    failures = []
    with psycopg.connect(DSN, autocommit=True) as conn, conn.cursor() as cur:
        cur.execute("PREPARE q(text) AS " + STMT)

        # warm past the switch using the dominant parameter
        for _ in range(6):
            cur.execute(sql.SQL("EXECUTE q({})").format(sql.Literal("bigcorp")))
            cur.fetchall()

        cur.execute("SELECT custom_plans, generic_plans "
                    "FROM pg_prepared_statements WHERE name = 'q'")
        custom, generic = cur.fetchone()
        print(f"warmup: custom_plans={custom} generic_plans={generic}")

        for label, param, max_ms, max_blocks, expected in BUCKETS:
            runs = [explain_execute(cur, param) for _ in range(5)]
            plans = [r["Plan"] for r in runs]
            ms = sorted(r["Execution Time"] for r in runs)[len(runs) // 2]
            blocks = plans[0]["Shared Hit Blocks"] + plans[0]["Shared Read Blocks"]
            family = plan_family(plans[0])

            print(f"{label:9} {ms:8.2f} ms  {blocks:7} blocks  {family}")
            if ms > max_ms:
                failures.append(f"{label}: {ms:.1f}ms > {max_ms}ms")
            if blocks > max_blocks:
                failures.append(f"{label}: {blocks} blocks > {max_blocks}")
            if family != expected:
                failures.append(f"{label}: plan family {family} != {expected}")

    if failures:
        raise SystemExit("PLAN REGRESSION\n  " + "\n  ".join(failures))
    print("ok")

if __name__ == "__main__":
    main()

The fingerprint matters more than it looks. Asserting on raw EXPLAIN text fails every time a row estimate shifts; asserting on node types plus index name fails only when the plan genuinely changes shape — which is exactly the event you want a build to stop for. If EXPLAIN ANALYZE's per-row timing overhead distorts your latency numbers, run the same harness with TIMING OFF and gate on buffers and plan family alone.

Takeaway: buffers and node types are the assertions that survive a noisy CI runner; latency is the one you keep loose.

Which knob do you actually reach for?

Option	What it does	Cost	Reach for it when
`plan_cache_mode = auto` (default)	Five custom plans, then compares	None	Data is roughly uniform across the parameter
`plan_cache_mode = force_custom_plan`	Re-plans every execution	Planning time on every call	Skew is severe and execution dominates planning
Set it per-statement via `SET LOCAL`	Scopes the override to one transaction	Same, but bounded	Only one or two queries are pathological
Disable server-side prepares in the driver	No plan cache at all	Loses parse-time reuse everywhere	You need a fast, blunt rollback

Scope beats globals here. SET LOCAL plan_cache_mode = force_custom_plan inside the transaction that runs the skewed query fixes that query and leaves the rest of your workload on the default — and it shows up in EXPLAIN (SETTINGS) output, so your harness can verify the setting was actually in effect rather than assume it.

For continuous coverage between CI runs, the in-tree auto_explain module logs real plans for slow executions with no extra infrastructure, at the price of log volume you have to manage. If you want plan history retained and compared over time without building that yourself, pganalyze is the managed option that tracks per-query plan changes, with the usual trade-off of another vendor in your data path.

Takeaway: a per-transaction SET LOCAL you can verify is safer than a global GUC you have to remember.

When "same SQL" isn't the same query

The last thing to diff is the session contract, because two connections can run byte-identical SQL against different relations:

SELECT current_user, session_user, current_setting('search_path') AS search_path;

SELECT name, setting FROM pg_settings
 WHERE name IN ('plan_cache_mode','row_security','work_mem','TimeZone',
                'random_page_cost','effective_cache_size','jit','enable_seqscan')
 ORDER BY name;

Three of these bite regularly. A different search_path resolves an unqualified table name to a different schema. Row-level security applies to the app role but is bypassed by the table owner, so a test run as owner sees a table with no policy predicates attached — a different plan by construction. And TimeZone changes what a date_trunc or range predicate actually matches, which changes selectivity.

Run the harness as the application role, against the application database, or you are testing a query that doesn't exist in production.

Takeaway: connect as the app role, or your plan test is measuring a query production never runs.

FAQ

How do I check if Postgres is using a generic or custom plan?
Query pg_prepared_statements for the custom_plans and generic_plans columns, available in PostgreSQL 14 and later. For a statement you prepared with PREPARE, EXPLAIN (ANALYZE) EXECUTE stmt(...) shows the plan being used; a generic plan displays $1 in place of the parameter value.

Does plan_cache_mode = force_custom_plan hurt performance?
Yes, for short queries. It re-plans on every execution, so a query that plans in 3 ms and runs in 1 ms becomes roughly four times more expensive. Use SET LOCAL to scope it to the transaction running the skewed query instead of setting it globally.

Why is my query fast in psql but slow from the application?
psql sends literal values, so the planner uses column statistics for that specific value. Your driver sends the query as a prepared statement with bound parameters, and after five executions Postgres may switch to a generic plan built without knowledge of those values — which is catastrophic when your data is skewed.

Bottom line

If you're about to change a plan-caching setting, spend the extra hour on the harness rather than the extra hour re-reading EXPLAIN output. Warm one connection past the switch with the dominant parameter, replay rare/median/dominant fixtures pulled from a real GROUP BY, and assert on plan family and buffers with a loose latency bound. Verify generic_plans > 0 before trusting a single number below it, and run as the application role so RLS and search_path match production. Teams with one pathological query should reach for SET LOCAL plan_cache_mode; teams that keep rediscovering this should keep the harness in CI, because tenant distributions drift and the bug comes back on its own.

Your Cache Expired and the Database Spiked: Four Fixes for Cache Stampedes

Libme — Sat, 22 Aug 2026 14:52:45 +0000

When a hot cache key expires, every in-flight request misses at the same instant and all of them recompute the same value against the same database. That is a cache stampede, and it shows up as a latency cliff on a fixed interval rather than a gradual degradation. The fixes, in the order I would apply them: jitter your TTLs, add single-flight locking so only one worker recomputes, serve stale while revalidating in the background, and — if you need the last bit of smoothness — recompute probabilistically before expiry.

What does a cache stampede actually look like in the logs?

The tell is periodicity. Not "the database is slow under load" but "the database is fine, then pinned, then fine again, every five minutes." If your hot key TTL is 300 seconds and your p99 graph has teeth exactly 300 seconds apart, you are looking at your own TTL, not a traffic pattern.

On the application side you get a burst of timeouts against one endpoint, and the errors sit downstream of the cache rather than in it. In a Postgres-backed service the burst reads as connection pressure — pool checkout timeouts, or the server refusing connections outright:

FATAL: sorry, too many clients already

In a Go or Node service the same event surfaces as a wall of context-deadline errors that all reference the same query. Redis itself looks healthy throughout: hit rate dips for a second or two, memory is flat, INFO commandstats shows nothing unusual. That combination — healthy cache, spiking origin, fixed interval — is the fingerprint.

Log the cache-miss path with the key name and count misses per key per second. A stampede is not "many misses"; it is many misses on one key inside one recompute window.

Why doesn't adding cache capacity or raising the hit rate fix it?

Because hit rate is the wrong number. Suppose one key serves 500 requests per second and takes 200 ms to recompute. The moment it expires, roughly 100 requests arrive before the first recompute finishes, and each sees an empty slot and starts its own. Your steady-state hit rate can be 99.8% and the incident still happens, because the damage is done by concurrency inside a 200 ms window, not by the volume of misses over an hour.

This is also why a bigger Redis instance, a longer TTL, or a warm-up job on deploy do not help. A longer TTL makes stampedes rarer and worse — more accumulated traffic per expiry, and a colder recompute when it lands. Warm-up jobs are counterproductive if they populate thousands of keys in a loop with identical TTLs, since you have just synchronized all of them to expire in the same second.

The takeaway: a stampede is a concurrency-control bug in your cache-miss path, not a capacity problem.

Fix 1: Jitter every TTL, without exception

This is a one-line change and it dissolves the synchronized-expiry class of the problem entirely. Never write a fixed TTL:

import random

def ttl_with_jitter(base_seconds, spread=0.1):
    # 60s base -> a value uniformly in [54, 66]
    return int(base_seconds * random.uniform(1 - spread, 1 + spread))

Ten percent spread on a 60-second TTL smears a bulk warm-up across a 12-second window instead of one instant. It does nothing for a single very hot key — that key still has one expiry moment, and everyone still piles into it — which is why jitter is necessary but never sufficient.

Fix 2: Single-flight, so only one worker recomputes

Single-flight means the first request to notice the miss takes a short-lived lock and does the work; everyone else either waits for it or serves something else. In Redis the lock is SET key value NX EX seconds:

import json, time, random
import redis

r = redis.Redis(decode_responses=True)

def _store(key, value, fresh_for, keep_stale_for):
    fresh = fresh_for * random.uniform(0.9, 1.1)
    entry = {"value": value, "fresh_until": time.time() + fresh}
    r.set(key, json.dumps(entry), ex=int(fresh + keep_stale_for))

def get_cached(key, compute, fresh_for=60, keep_stale_for=600, lock_ttl=10):
    lock_key = f"lock:{key}"
    raw = r.get(key)

    if raw is not None:
        entry = json.loads(raw)
        if time.time() < entry["fresh_until"]:
            return entry["value"]
        # Stale but usable: one caller refreshes, everyone else serves stale.
        if r.set(lock_key, "1", nx=True, ex=lock_ttl):
            try:
                value = compute()
                _store(key, value, fresh_for, keep_stale_for)
                return value
            finally:
                r.delete(lock_key)
        return entry["value"]

    # Cold miss: nothing to serve, so losers wait briefly for the winner.
    if r.set(lock_key, "1", nx=True, ex=lock_ttl):
        try:
            value = compute()
            _store(key, value, fresh_for, keep_stale_for)
            return value
        finally:
            r.delete(lock_key)

    for _ in range(20):
        time.sleep(0.05)
        raw = r.get(key)
        if raw is not None:
            return json.loads(raw)["value"]
    return compute()

Two details bite people here. The lock needs its own TTL, or a worker that gets OOM-killed mid-recompute leaves the key permanently unrefreshable. And r.delete(lock_key) in the finally block is technically unsafe: if the recompute overruns lock_ttl, the lock has already expired and been taken by someone else, so you just deleted their lock. The correct version stores a random token in the lock and deletes it with a small Lua script that compares the token first. For a 10-second lock around a 200 ms query the exposure is small — but it is why single-flight code that "works fine" in staging occasionally double-computes in production.

In Go, do not write this by hand: golang.org/x/sync/singleflight collapses duplicate calls, with the caveat that it is per-process, so ten pods still produce ten recomputes.

Single-flight turns N concurrent recomputes into one, but the N-1 losers on a cold miss are still waiting.

Fix 3: Stale-while-revalidate, which is the one that removes the cliff

The code above is already stale-while-revalidate: the entry carries a logical freshness timestamp and the Redis TTL is set much longer. Between fresh_until and physical expiry, readers get an instant cached response while exactly one worker refreshes in the background. Nobody blocks, so there is no cliff — the origin sees one query per refresh interval regardless of traffic.

The cost is correctness: you knowingly serve data up to keep_stale_for seconds old when refreshes fail. For a dashboard aggregate or a search facet count, that is free. For anything a user just wrote and expects to see, it is not — those keys need explicit invalidation on write and stale windows measured in seconds.

At the HTTP layer you may not need application code at all. Varnish coalesces concurrent requests for the same object by default and can serve stale content while fetching a fresh copy; nginx does the narrower version with proxy_cache_lock on; plus proxy_cache_use_stale updating;. Both apply only to full HTTP responses keyed by URL, so neither helps the internal fragment caches that usually cause this.

The rule of thumb: if you can name a tolerable staleness window for a key, stale-while-revalidate is strictly better than locking alone.

Fix 4: Probabilistic early recomputation

If you cannot serve stale — say the value is a signed token or a count that must never go backwards — each reader can independently decide to refresh slightly early, with probability rising as expiry approaches. This is the XFetch approach from the 2015 paper on optimal probabilistic cache stampede prevention, and it is about six lines:

import math, random

def should_refresh_early(delta_seconds, ttl_remaining, beta=1.0):
    """delta_seconds = how long the last recompute took."""
    return delta_seconds * beta * -math.log(random.random()) >= ttl_remaining

Store delta alongside the value when you write it. Expensive values start volunteering for refresh earlier; cheap ones wait. Raising beta above 1.0 makes refreshes more eager. I have shipped this, but it is the fix I reach for last: it adds a tunable nobody will remember the meaning of in six months, and it only helps when stale-serving is off the table.

Which fix should you actually apply?

Approach	Origin load at expiry	Reader waits?	Serves stale?	Main cost
Jittered TTL	Unchanged per key	Yes	No	Nothing — always do it
Single-flight lock	1 recompute	Yes, on cold miss	No	Lock-expiry edge cases
Stale-while-revalidate	1 recompute	No	Yes	Bounded staleness
Probabilistic early refresh	1 recompute, before expiry	Rarely	No	An opaque tuning knob

For most services the answer is the middle two together — exactly what the code above does: single-flight on cold misses, stale-while-revalidate on warm ones, jitter on every write.

FAQ

What is a cache stampede?
A cache stampede (also called a thundering herd or dog-piling) happens when a cached value expires and many concurrent requests miss simultaneously, each recomputing the same value against the origin. The result is a burst of identical queries that can saturate the database even though overall hit rate stays high.

How do I prevent a Redis cache stampede?
Use SET lock:<key> <token> NX EX <seconds> so only one worker recomputes a given key, store the entry with a logical freshness timestamp shorter than its Redis TTL so other readers serve slightly stale data instead of blocking, and add ±10% jitter to every TTL so bulk-populated keys never expire in the same second.

Does increasing the TTL fix a cache stampede?
No. A longer TTL makes stampedes less frequent but more severe, because more traffic accumulates behind each expiry and the recompute is colder. The concurrency of the miss is what hurts, not how often it occurs.

Bottom line

If you have a periodic latency cliff and a healthy-looking cache, add jitter today and single-flight this week. If the value tolerates being a few seconds old — most aggregates, counts, and rendered fragments do — go straight to stale-while-revalidate, the only option here where readers never wait on a recompute. Reserve probabilistic early refresh for values that genuinely cannot be served stale, and log misses per key so the next incident takes one query to diagnose instead of an hour.

Rate Limiting Your Own API: Should the Counter Live in Redis, Postgres, or at the Edge?

Libme — Fri, 21 Aug 2026 15:02:57 +0000

If your rate limiter keeps its counter in process memory, your advertised limit is a lie as soon as you run more than one instance — four replicas means roughly four times the traffic gets through. Move the counter somewhere all instances can see: Redis if you need low latency at real volume, Postgres if you already run one and your traffic is modest, the edge if what you're stopping is volumetric abuse rather than per-customer quota. The hard part isn't the algorithm; it's deciding what happens when the counter store itself is down.

The symptom: a 100 req/min limit that lets through 400

The bug report reads like this: a customer on a "100 requests per minute" plan is clearly doing 300-plus and never sees a 429. Nothing looks broken — the limiter is running, it returns X-RateLimit-Remaining, and if you hammer it locally it works perfectly. Then you check how many app instances are running.

Most drop-in middleware defaults to an in-memory store — express-rate-limit, for instance, ships with a memory store by default and its docs are explicit that this doesn't work across multiple processes. Each process keeps its own independent bucket. With four containers behind a load balancer, a caller round-robins across four separate 100-request budgets. Nothing errors. The number is just wrong.

There's a quieter version of the same bug: one instance, but the process restarts on every deploy. Counters reset, and anyone who times their burst around a deploy gets a fresh budget. Same root cause — the counter's lifetime is tied to the process, and the process is not the unit your limit is defined over.

If your rate limit is defined per customer, the counter has to live somewhere that outlives and spans your processes.

Where can the counter actually live?

Location	Accuracy across instances	Added latency per request	Main failure mode	Best for
In-process memory	None (per replica)	~0	Silently multiplies your limit	Single-process tools, local dev
Redis (or Redis-compatible)	Exact, atomic via Lua	Sub-millisecond on the same network	Redis down → fail-open or fail-closed decision	Per-key quotas at real volume
Postgres	Exact, atomic via upsert	A write per request, plus pool contention	Write amplification, autovacuum load	Modest traffic, no new dependency wanted
Edge / CDN / API gateway	Exact enough, but scoped to what the edge can see	None to your origin	Can't see app-level identity or plan tier	Volumetric abuse, IP floods

These aren't mutually exclusive. The sane end state for most teams is two layers: something blunt at the edge that keeps a flood off your origin, and something identity-aware in the app that enforces the quota you actually sell.

Pick the location by what the limit is keyed on: IP-shaped limits belong at the edge, plan-shaped limits belong where your app knows who the caller is.

How do you implement a token bucket in Redis without race conditions?

The naive version — GET, check, SET — is a read-modify-write race, and under exactly the concurrency you're trying to limit, it's wrong. Do the whole thing in one Lua script so it executes atomically on the server:

-- KEYS[1]  = bucket key, e.g. "rl:tenant_42"
-- ARGV[1]  = capacity (max burst)
-- ARGV[2]  = refill rate, tokens per second
-- ARGV[3]  = current time in ms (from the caller)
-- ARGV[4]  = cost of this request, usually 1
local key      = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill   = tonumber(ARGV[2])
local now      = tonumber(ARGV[3])
local cost     = tonumber(ARGV[4])

local state  = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(state[1])
local ts     = tonumber(state[2])

if tokens == nil then
  tokens = capacity
  ts     = now
end

local elapsed = math.max(0, (now - ts) / 1000)
tokens = math.min(capacity, tokens + elapsed * refill)

local allowed = 0
if tokens >= cost then
  tokens  = tokens - cost
  allowed = 1
end

redis.call('HSET', key, 'tokens', tokens, 'ts', now)
-- expire once a full refill would have happened anyway
redis.call('PEXPIRE', key, math.ceil((capacity / refill) * 1000) + 1000)

return { allowed, math.floor(tokens) }

Wiring it up with ioredis, which lets you register the script once and call it like a normal command:

const Redis = require('ioredis');
const redis = new Redis(process.env.REDIS_URL);

redis.defineCommand('takeToken', { numberOfKeys: 1, lua: LUA_SCRIPT });

async function allow(tenantId, { capacity = 100, refillPerSec = 100 / 60 } = {}) {
  const [allowed, remaining] = await redis.takeToken(
    `rl:${tenantId}`, capacity, refillPerSec, Date.now(), 1
  );
  return { allowed: allowed === 1, remaining };
}

Two details matter. The timestamp comes from the application, not from TIME inside the script — that keeps it deterministic and, more usefully, testable with a fake clock. And the PEXPIRE is what keeps this from becoming an unbounded key space: idle tenants evict themselves once enough time has passed that their bucket would be full anyway.

Token bucket, not fixed window, is the default I reach for because it allows a legitimate short burst while still holding the long-run average — which is usually what a customer expects from "100 per minute."

The atomic script isn't an optimization; a rate limiter with a read-modify-write race is a rate limiter that fails precisely under load.

Can you do this in Postgres if you don't want another dependency?

Yes, and for a lot of small teams it's the right call. A fixed-window counter is a single atomic statement:

create table rate_limit_counters (
  bucket_key   text        not null,
  window_start timestamptz not null,
  hits         int         not null default 0,
  primary key (bucket_key, window_start)
);

insert into rate_limit_counters (bucket_key, window_start, hits)
values ($1, date_trunc('minute', now()), 1)
on conflict (bucket_key, window_start)
do update set hits = rate_limit_counters.hits + 1
returning hits;

One round trip, returns the post-increment count, no race. Delete rows older than a couple of windows on a schedule and the table stays tiny.

The honest drawbacks: every request now writes to your primary database. That's WAL traffic, dead tuples, and autovacuum work proportional to request rate, and it takes a connection from the same pool your real queries use — so a traffic spike contends with itself twice. Fixed windows also allow a boundary burst: a caller can spend a full window's budget in the last second of one window and again in the first second of the next, so a "100/min" limit tolerates 200 requests inside one two-second span. You can smooth that with a sliding window (two adjacent counters, weighted by how far you are into the current window), which is worth doing if the burst actually hurts you.

Postgres is a perfectly good rate limit store right up until the limiter's write volume becomes a meaningful fraction of your database's write volume.

What belongs at the edge instead?

The edge sees a request before it costs you anything, which is exactly why it's the right place for blunt limits and the wrong place for nuanced ones. It can't tell that this caller is on the Scale plan with a quota that resets on the 3rd — that's app knowledge. It can tell that one IP is sending 5,000 requests a minute to your login endpoint.

If you want that layer managed, Cloudflare's rate limiting rules will drop volumetric abuse at the edge before it reaches your origin, at the cost of only being able to key on what a request looks like from outside your app. For serverless and edge runtimes where holding a long-lived Redis TCP connection is awkward, Upstash offers Redis over HTTP with a per-request pricing model, which sidesteps the connection-pooling problem those runtimes have. If you already terminate traffic through an API gateway, its built-in throttling usually covers the coarse layer — but check whether its counter is per-node or global, because managed gateways aren't automatically immune to the bug this post opens with.

Use the edge to protect your infrastructure and the app to enforce your contract; trying to make one layer do both is where the design gets bad.

What should the response actually say?

A 429 with an empty body teaches the caller nothing, and a client that doesn't know when to retry will just retry immediately.

Return 429 Too Many Requests with a Retry-After header, in seconds. This one is a real standard and well-supported client-side.
Add RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset if you want callers to self-pace. As of mid-2026 those fields are still an IETF draft rather than a finished RFC, and plenty of APIs use the older X-RateLimit-* spelling — pick one, document it, don't change it.
Decide fail-open vs fail-closed explicitly. If Redis is unreachable, does the request pass or get rejected? For a public API guarding cost, fail-closed; for an internal service where the limiter is a safety belt, fail-open with a loud alert. What you don't want is for that to be an accident of where your try/catch happens to sit.

Write the fail-open decision down in the code as a named constant, because you will only find out what you chose during an incident.

FAQ

Should I rate limit by IP address or by API key?
By API key or account ID whenever the caller is authenticated — IPs are shared by corporate NATs and mobile carriers, so IP limits punish legitimate users in groups. Use IP limits only for unauthenticated endpoints like login and signup, where there's no better identity available.

Do I need Redis to rate limit an API?
No. A single Postgres table with an upsert-and-return counter is atomic and correct, and it's the right choice if your request volume is well below your database's write capacity. Move to Redis when the limiter's writes start showing up in your database's load profile.

What's the difference between 429 and 503 for rate limiting?
Return 429 when this specific caller exceeded their quota — it's about them, and Retry-After tells them when to come back. Return 503 when your service as a whole is shedding load; 429 implies the caller can fix the problem by slowing down, 503 doesn't.

Bottom line

If you're running more than one instance, the in-memory limiter you installed on day one is not enforcing the number you think it is — verify that first, before you tune anything. Small teams with an existing Postgres and moderate traffic should start with the upsert counter and skip the extra dependency entirely. Reach for Redis with an atomic Lua token bucket when per-request database writes stop being free, and add an edge rule on top when the traffic you're fighting is a flood rather than a customer over quota. Whichever you choose, write the 429 response and the store-is-down behavior deliberately, because those two things are what your callers and your on-call actually experience.

Why Your Query Is Fast in psql but Slow From the App: Postgres Plan Caching Explained

Libme — Fri, 21 Aug 2026 03:14:41 +0000

If a query is instant when you paste it into psql but slow when your application runs it, the plan is almost never the problem you think it is. The most common cause is that your driver sent it as a prepared statement, and after a few executions Postgres switched from a plan tuned to your actual parameter values to a generic plan built without them. The fix is usually a one-line setting, not a new index.

I lost the better part of a day to this on a table with heavily skewed data. Same database, same user, same query text, 300x difference in latency depending on who was asking.

What the symptom actually looks like

The tell is that the two environments disagree while everything you can see is identical:

-- in psql: 2.8 ms
SELECT id, status, created_at FROM events
WHERE tenant_id = 'acme' AND status = 'pending'
ORDER BY created_at DESC LIMIT 50;

The application logs the same statement at 800–1200ms. There's an index on (tenant_id, status, created_at DESC). EXPLAIN in psql shows a clean index scan. Nothing in the app is doing anything exotic — one query, 50 rows.

What differs is how the statement reaches the server. In psql you sent literal values, so the planner saw tenant_id = 'acme' and could use the column statistics for that specific value. Your driver sent WHERE tenant_id = $1 with the value bound separately. Postgres plans that once and may reuse the plan.

Takeaway: when psql and the app disagree on speed for the same SQL, stop looking at indexes and start looking at parameter binding.

How does Postgres decide to reuse a plan?

For a prepared statement, Postgres builds a custom plan (re-planned with the actual parameters) for the first executions, tracks their estimated cost, and compares that average against the cost of a generic plan built with no knowledge of the values. From the sixth execution onward, if the generic plan doesn't look more expensive on average, it locks in and stops re-planning. That threshold is a fixed number in the source, not a knob you tune.

This is a good trade when data is uniform. It is destructive when it isn't. If 98% of your rows are tenant_id = 'bigcorp', the generic plan is built around the average selectivity across all tenants, and the planner concludes a sequential scan or a bitmap heap scan is reasonable. For acme — 400 rows out of 40 million — that plan is catastrophic, and it will be reused for the entire life of that connection.

Two more things that only bite the parameterized version:

Partial indexes stop matching. An index defined WHERE status = 'pending' can be matched against a literal status = 'pending' but not against status = $2, because at generic-plan time the planner doesn't know $2 is 'pending'.
LIKE prefix optimization disappears. LIKE 'abc%' can be rewritten into a range scan; LIKE $1 cannot, without the value.

Takeaway: a generic plan is planned for your average row, so any column with skewed values is a landmine.

How do I prove this is what's happening?

Don't guess from application timings. Reproduce it in a single psql session — the behavior is per-connection, and you can drive it by hand:

PREPARE q(text, text) AS
  SELECT id, status, created_at FROM events
  WHERE tenant_id = $1 AND status = $2
  ORDER BY created_at DESC LIMIT 50;

EXPLAIN (ANALYZE, BUFFERS) EXECUTE q('acme', 'pending');  -- run this 6+ times

Run it repeatedly and watch the plan. If execution 1 shows an Index Scan and execution 6 flips to a Seq Scan or a Bitmap Heap Scan, you have your answer — and you've reproduced it without touching the app.

On Postgres 16 and later you can skip the ritual and ask directly:

EXPLAIN (GENERIC_PLAN)
  SELECT id FROM events WHERE tenant_id = $1 AND status = $2
  ORDER BY created_at DESC LIMIT 50;

To catch it in production instead of on your laptop, enable auto_explain and have the slow plans logged as they happen:

LOAD 'auto_explain';
SET auto_explain.log_min_duration = '200ms';
SET auto_explain.log_analyze = on;

Loading it per-session like this is the safe way to test; making it permanent means adding it to shared_preload_libraries and a restart, and log_analyze adds real per-query overhead, so keep the duration threshold high on a busy server. If you want to know which statements to point it at first, pg_stat_statements is the extension that tells you where the time actually goes — it aggregates by normalized query text, so the parameterized version shows up as one row with its own mean and max.

Takeaway: PREPARE plus six EXECUTEs in one psql session reproduces the bug in under a minute, which is faster than any amount of reading application logs.

What do I actually change?

Situation	Fix	Cost of the fix
Skewed column, plan flips to generic	`plan_cache_mode = force_custom_plan`	Re-plans every execution (planning is ~µs–low ms)
Only one or two queries affected	Set `plan_cache_mode` on that session/transaction only	Needs a code path to scope it
Driver prepares everything by default	Lower or disable the driver's prepare threshold	Loses prepared-statement parse savings
PgBouncer in transaction mode	Configure prepared-statement support, or stop using named statements	Version-dependent; see below
Plan is fine, stats are stale	`ANALYZE`, raise `default_statistics_target` on that column	More planning time, better estimates

The blunt instrument, available since Postgres 12:

-- per session, or per role/database via ALTER ROLE ... SET
SET plan_cache_mode = force_custom_plan;

I default to scoping this narrowly. Setting it database-wide fixes the skewed queries and quietly taxes every other prepared statement with re-planning forever. ALTER ROLE app_worker SET plan_cache_mode = 'force_custom_plan' on the specific worker role that runs the tenant-scoped queries is usually the right blast radius.

Where the driver sits matters as much as the server setting:

node-postgres (pg) only creates a named prepared statement when you pass a name in the query config. Plain client.query(text, values) is unnamed — parsed each time, planned with the values, immune to this problem.
The PostgreSQL JDBC driver switches to a server-side prepared statement after a handful of executions of the same statement, controlled by prepareThreshold. Setting it to 0 disables server-side preparation entirely.
ORMs on top of connection poolers are where this gets ugly. If you run PgBouncer in transaction pooling mode, named prepared statements historically broke outright because the server connection underneath you changes between transactions. PgBouncer added support for protocol-level prepared statements in 1.21 (released late 2023), gated behind max_prepared_statements, which defaults to 0 — meaning off unless you turned it on. If you want a pooler where this isn't a running concern, PgBouncer with max_prepared_statements configured explicitly is the setup that keeps prepared statements and transaction pooling working together instead of forcing you to choose.

Takeaway: fix it at the narrowest scope that works — one role or one query beats a database-wide setting you'll forget you set.

What if it isn't the plan cache?

Before you go changing planner settings, rule out the other reasons psql and your app disagree. In rough order of how often I've actually hit them:

You're not measuring the same thing. App timing usually includes connection acquisition, TLS, row serialization, and ORM hydration. A query that's 3ms at the server and 900ms in your log may be spending 890ms turning rows into objects.
Different search_path or role. The app connects as a different role and hits a different schema — often an unindexed copy in a test schema.
Session GUCs. work_mem set per-role changes whether a sort spills to disk. statement_timeout masks the real duration.
Connection acquisition, not query time. If the pool is exhausted, every query looks slow. pg_stat_activity will show sessions waiting, not running.
Actual row counts differ. Your app pages through 50,000 rows; you tested LIMIT 50.

The clean way to separate these: log the server-side duration with log_min_duration_statement and compare it against the duration your application recorded. If the server says 3ms and your app says 900ms, the database is innocent.

Takeaway: confirm the server itself is slow before you tune the server.

FAQ

Why is my query fast in pgAdmin but slow in my application?
Almost always because your application sends the query as a parameterized prepared statement and the GUI tool sends literal values. After roughly five executions Postgres may switch that prepared statement to a generic plan built without your parameter values, which performs badly on columns with skewed data. Reproduce it with PREPARE and repeated EXECUTE in psql.

How do I stop Postgres from using a generic plan?
Set plan_cache_mode = force_custom_plan, available since Postgres 12. Scope it to the session, transaction, or role that runs the affected queries rather than setting it database-wide, since it forces re-planning on every execution.

Does PgBouncer break prepared statements?
In transaction pooling mode it did, because the underlying server connection changes between transactions. PgBouncer 1.21 added support for protocol-level prepared statements via the max_prepared_statements setting, which is disabled by default — you have to set it to a non-zero value explicitly.

Bottom line

If psql is fast and your app is slow on identical SQL, check parameter binding before you add an index. Reproduce it with PREPARE plus six EXECUTEs, and if the plan flips, set plan_cache_mode = force_custom_plan on the narrowest scope that covers the offending queries — a role, ideally, not the whole database. If the plan doesn't flip, the problem is on your side of the wire: measure server-side duration with log_min_duration_statement and compare, because ORM hydration and pool waits both look exactly like a slow query from the application's point of view.

Your Secrets Manager Ends at process.env: Where Secrets Actually Leak at Runtime

Libme — Thu, 20 Aug 2026 12:57:39 +0000

Whichever secrets store you pick — a cloud provider's secret manager, Doppler, 1Password, Vault, or encrypted files in git — almost all of them hand your process the same thing at boot: a flat blob of environment variables. That means the vendor decision governs storage, distribution, and rotation, but it barely touches the path most small-team secret leaks actually take, which is a secret getting printed: into a log line, an error report, a subprocess, or a CI debug step. A commenter on an earlier post made exactly this point, and it's the more useful half of the problem.

This post is about what you do after the vendor choice. It's Node-flavored, but the leak paths are language-agnostic.

Why does the store barely change the leak path?

Every injector converges. doppler run -- node server.js, op run -- node server.js, Vault Agent templating a file, Kubernetes envFrom.secretRef, or a plain .env — by the time your first line of code executes, the secret is a string in process.env, readable by anything in the process and by anything that can serialize an object containing it.

So the threat model shifts. Storage-layer questions ("who can read the secret in the vault?") are answered by the vendor. Runtime questions are yours alone:

Leak path	Who ends up seeing it	Does the vendor help?
`console.log(config)` in a debug session	Anyone with log access, forever	No
Error reporter attaching request/config context	Your SaaS error tracker's operators	No
`child_process.spawn` inheriting the full env	Any script you shell out to, and its logs	No
`/proc/<pid>/environ` on the box	Any process running as the same user	No
`kubectl describe pod` with literal env values	Anyone with read access to the namespace	Partly — `secretKeyRef` shows the reference, not the value
CI job echoing a derived value	Anyone who can read build logs	Partly — masking only catches exact matches

Two of those deserve a note. On Linux, a process's environment is readable at /proc/<pid>/environ by processes running as the same user, so "it's only in memory" is weaker than it sounds on a shared host. And in Kubernetes, env vars sourced with secretKeyRef show up in kubectl describe pod as a reference rather than a value — but env vars set as literals in a manifest are printed in full.

The vendor decides who can fetch the secret; your code decides how many places it can be printed.

How do you stop a secret from ever printing?

Make the secret a type that cannot serialize, instead of a string you promise never to log. In Node, three hooks cover essentially every accidental print: toString, toJSON, and the custom inspect symbol.

// secret.js
const REDACTED = "[redacted]";

export class Secret {
  #value;
  constructor(value, name) {
    this.#value = value;
    this.name = name;
  }
  expose() {
    return this.#value;
  }
  toString() {
    return REDACTED;
  }
  toJSON() {
    return REDACTED;
  }
  [Symbol.for("nodejs.util.inspect.custom")]() {
    return `Secret(${this.name})`;
  }
}

Now console.log(secret) prints Secret(STRIPE_SECRET_KEY), JSON.stringify({ secret }) produces {"secret":"[redacted]"}, and a template literal like `key=${secret}` yields key=[redacted]. The only way to get the real value out is .expose() — a single greppable token that shows up in code review.

Wire it into one env module that parses at boot and fails loudly, then removes the raw values from the ambient environment:

// env.js — the only file allowed to touch process.env
import { z } from "zod";
import { Secret } from "./secret.js";

const schema = z.object({
  DATABASE_URL: z.string().url(),
  STRIPE_SECRET_KEY: z.string().startsWith("sk_"),
  SESSION_SECRET: z.string().min(32),
  LOG_LEVEL: z.string().default("info"),
});

const parsed = schema.safeParse(process.env);
if (!parsed.success) {
  console.error("Invalid environment:", parsed.error.flatten().fieldErrors);
  process.exit(1);
}

const SENSITIVE = ["DATABASE_URL", "STRIPE_SECRET_KEY", "SESSION_SECRET"];

export const env = Object.freeze({
  ...parsed.data,
  ...Object.fromEntries(
    SENSITIVE.map((k) => [k, new Secret(parsed.data[k], k)]),
  ),
});

// Shrink the blast radius: after this point the raw strings are not
// in the ambient environment, so a subprocess or a dump can't pick them up.
for (const key of SENSITIVE) delete process.env[key];

Call sites become explicit:

import { Pool } from "pg";
import { env } from "./env.js";

export const pool = new Pool({ connectionString: env.DATABASE_URL.expose() });

Two honest caveats. First, delete process.env.X only helps if it runs before anything else reads that variable, so this module must be imported first — and you must not delete variables that libraries read from the environment themselves (AWS_* for the AWS SDK, PGPASSWORD for libpq, OTEL_* for OpenTelemetry). Delete only the keys your own code owns. Second, the Secret wrapper stops accidental printing, not a determined .expose() in the wrong place; it converts a whole class of invisible mistakes into a visible one.

A secret that redacts itself on serialization turns "we should be careful with logging" into a property of the type system rather than a team norm.

What about the loggers and error reporters?

Structured loggers can redact by path. With Pino:

import pino from "pino";

export const log = pino({
  redact: {
    paths: [
      "req.headers.authorization",
      "req.headers.cookie",
      "*.password",
      "*.token",
      "*.secret",
    ],
    censor: "[redacted]",
  },
});

The limitation matters more than the feature: path-based redaction only inspects the properties of the logged object. If someone writes log.info(`connecting to ${url}`) with the secret already interpolated into the string, redaction sees a plain message and does nothing. That's precisely why the Secret class above overrides toString — the two mechanisms cover each other's gaps.

For error tracking, the risk isn't usually the framework grabbing your environment; it's you attaching a config object to the scope. Sentry's Node SDK doesn't ship process.env on its own, but setContext("config", config) will happily upload whatever you hand it, and a beforeSend hook is the right place to enforce that:

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  beforeSend(event) {
    delete event.contexts?.config;
    if (event.request?.headers) {
      delete event.request.headers.authorization;
      delete event.request.headers.cookie;
    }
    return event;
  },
});

Redaction that runs at the logging boundary is necessary but not sufficient — anything that stringifies a secret before it reaches the boundary sails straight through.

How do subprocesses and CI leak secrets?

child_process.spawn and exec pass the parent's entire environment to the child by default. Every shell script, every image-processing binary, every migration tool you shell out to receives your full secret set, and if that child prints its environment on error, so does your log. Pass an allowlist instead:

import { spawn } from "node:child_process";

spawn("./scripts/import.sh", [], {
  env: {
    PATH: process.env.PATH,
    NODE_ENV: process.env.NODE_ENV,
    IMPORT_BUCKET: process.env.IMPORT_BUCKET,
  },
});

CI has the same shape with an extra trap. GitHub Actions masks registered secret values in log output, but the mask is a literal string match: base64-encode a secret, slice it, or embed it in a JSON payload you print, and the transformed value is not masked. Treat CI masking as a safety net for typos, not as a control.

Finally, make the invariant testable so it doesn't decay:

import { inspect } from "node:util";
import { env } from "../env.js";

test("env never serializes secret values", () => {
  const dump = JSON.stringify(env) + inspect(env, { depth: 5 });
  expect(dump).not.toMatch(/sk_[A-Za-z0-9]/);
  expect(dump).not.toMatch(/postgres:\/\/[^@\s]*:[^@\s]+@/);
});

Pair it with an ESLint no-restricted-properties rule banning process.env outside env.js, and the runtime hygiene survives new contributors.

If your secret handling isn't covered by a test that fails, it's a convention, and conventions regress at the third hire.

Does any of this change which store you should pick?

Slightly, at the margins. If you want an injector that also scrubs the child process's output, 1Password's CLI is the one that masks known secret values in the stdout and stderr of the command it runs, which catches sloppy prints you didn't anticipate — though it only knows the values it injected, and it puts a vendor CLI in the boot path of every local command. Doppler's doppler run is the most frictionless injector for a mixed team and handles rotation cleanly, but it's still a plain env injection, so everything above still applies. Vault can template secrets to a tmpfs file that your app reads and re-reads, which keeps them out of the environment entirely and supports rotation without a restart — at the cost of running and operating Vault, which is a real job. As of mid-2026, no injector removes the need for the Secret type and the logging rules; they only reduce how much you're gambling on discipline.

Pick the store for rotation and access control, then assume it delivers a plain env blob and design the runtime as if it did.

FAQ

Is it safe to store secrets in environment variables?
It's acceptable for most small teams, but it's not free. Environment variables are visible to child processes, readable at /proc/<pid>/environ by same-user processes on Linux, and easy to serialize by accident. Load them once at startup, wrap them in a type that redacts on serialization, and delete the raw values from process.env afterward.

How do I stop secrets from appearing in logs in Node.js?
Use two layers. Wrap secret values in a class that overrides toString, toJSON, and Symbol.for("nodejs.util.inspect.custom") so interpolation and inspection print [redacted], and configure your structured logger's redaction paths for request headers and common field names. Path-based redaction alone misses secrets already interpolated into a message string.

Do child processes inherit environment variables in Node.js?
Yes. spawn, exec, and fork pass the parent's full process.env to the child unless you set the env option explicitly. Pass an allowlist containing only the variables the child actually needs.

Bottom line

Choosing between a cloud secret manager, Doppler, 1Password, Vault, and SOPS is a real decision, and it's the right one to make for rotation speed and access control. But it ends at your process boundary. Spend the afternoon after that decision on the runtime side: one env module that parses and fails fast, a Secret type that can't serialize, logger redaction, an allowlist for subprocess environments, and one test that fails if a raw value ever shows up in a dump. That's the part that actually determines whether your next incident is a stack trace or a secret in a log line someone screenshots into Slack.

When Should You Move Off Managed Hosting to Kubernetes? The Thresholds That Actually Matter

Libme — Thu, 20 Aug 2026 08:42:41 +0000

If you are trying to decide whether it is finally time to run a cluster, the honest answer is that request volume is almost never the trigger. The triggers are structural: per-tenant isolation you cannot express in your database, workloads you do not trust, a compliance boundary that requires network-level segmentation with an audit trail, or a pile of homegrown scripts that have quietly become a bad orchestrator. Before any of those, you should be able to prove you have run out of headroom on the managed platform you already pay for.

A reader on an earlier post about boring stacks made the sharpest version of this point: the real failure mode that pushes teams to Kubernetes is not high traffic, it is complex multi-tenant stateful workloads and compliance boundaries that a managed platform cannot express. They then asked the question this post exists to answer — what specific metrics and team thresholds should sit in the decision table. Here is the one I use.

Why is "we're getting too much traffic" almost never the real trigger?

Because managed platforms scale vertically and horizontally long past where most teams assume they stop. A single large instance running a well-tuned application server handles a volume of traffic that surprises people who have only ever read about scale, and every managed runtime worth using will run several identical copies of your container behind a load balancer without you learning a new config language.

Before you accept "we need to scale" as an argument, measure your actual headroom. The database is usually the real ceiling, and it is the one people misread most often — an app that falls over at 300 concurrent requests is frequently exhausting Postgres connections, not CPU:

SELECT
  (SELECT setting::int FROM pg_settings WHERE name = 'max_connections') AS max_connections,
  count(*) AS in_use,
  round(
    100.0 * count(*) / (SELECT setting::int FROM pg_settings WHERE name = 'max_connections'),
    1
  ) AS pct_used
FROM pg_stat_activity;

If that comes back at 95% while your instance CPU sits at 20%, Kubernetes solves nothing for you. A connection pooler and a smaller per-process pool size solve it in an afternoon. The same applies to the app tier: if your peak CPU utilization never crosses 40% on the instance size you are paying for, you do not have a scaling problem, you have a latency or concurrency problem hiding behind one.

If you cannot point at a specific resource you have saturated after tuning, you do not have a scaling argument — you have a discomfort.

What are the structural signals that actually justify a cluster?

These are the four I treat as genuine. Each one has a test you can apply today, and each one has a boring-stack workaround you should exhaust first.

Signal	How to test it	Try this first	If that fails
Per-tenant stateful isolation	Can one tenant's data or load reach another's, in a way a customer contract forbids?	Row-level security, or a schema/database per tenant on managed Postgres	Per-tenant namespaces with resource quotas
Untrusted or arbitrary workloads	Are you executing code you did not write — customer plugins, build jobs, notebooks?	Managed sandbox/build services, hard per-job timeouts	Real pod-level isolation and admission control
Compliance boundary	Does an auditor need to see enforced network segmentation and deploy provenance?	Separate managed environments per boundary, VPC peering rules	Network policies, RBAC, signed image admission
Deploy coordination	Count the lines of custom bash that restart, health-check, and roll out services	A managed runtime with health checks and rolling deploys	Declarative orchestration

The multi-tenancy row is where most teams misdiagnose themselves, so it deserves the concrete version. "Multi-tenant stateful workload" usually means shared rows in shared tables, and that is a database problem with a database answer:

ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
ALTER TABLE invoices FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON invoices
  USING (tenant_id = current_setting('app.tenant_id')::uuid);

Then set the tenant once per transaction, from your connection checkout or request middleware:

BEGIN;
SET LOCAL app.tenant_id = '3f1c2a9e-0b47-4f8a-9d21-5c6e8f0a1b23';
SELECT * FROM invoices WHERE status = 'open';
COMMIT;

FORCE ROW LEVEL SECURITY matters because policies are skipped for the table owner by default, which is exactly the role most applications connect as — that omission is the single most common way an RLS setup looks correct and enforces nothing. SET LOCAL scopes the setting to the transaction so a pooled connection cannot leak one tenant's context into the next request.

If that satisfies your isolation requirement, you did not need an orchestrator, you needed twelve lines of SQL. If your requirement is instead "tenant A's batch job must never consume tenant B's CPU," you have a genuine scheduling problem and the case for Kubernetes gets real.

Isolation requirements that live in your data model are database work; isolation requirements that live in the kernel are orchestrator work.

What team size can actually carry a cluster?

The number I use is not a headcount, it is an ownership commitment: someone whose job description includes the cluster, with roughly half a day a week of unglamorous maintenance budgeted, and at least one other person who can do a node upgrade when that person is on vacation. On a team of five engineers that is 10% of your engineering capacity gone, permanently, to something your customers never see.

The recurring work is real and it does not go away with a managed control plane: node group upgrades on the provider's deprecation schedule, ingress controller and cert-manager version bumps, CNI plugin compatibility, RBAC drift, and the periodic afternoon spent discovering that a pod is pending because of a resource request nobody remembers writing. Managed offerings like EKS and GKE remove the control plane from your worry list and genuinely lower the floor, but the worker nodes, networking, and deploy pipeline are still yours.

There is also a middle tier that most decision tables skip entirely. If your only real need is "run these containers, restart them when they die, roll them out without downtime," AWS Fargate gives you container scheduling with no nodes to patch, at the cost of slower cold starts and a weaker local development story. If your workloads are HTTP services that can tolerate scale-to-zero, Google Cloud Run handles request-driven autoscaling and TLS termination without exposing you to any cluster concepts at all, though it constrains you to its request lifecycle. If you want orchestration primitives without the Kubernetes ecosystem surface area, HashiCorp Nomad schedules containers and plain binaries with a config format a new engineer can read in an afternoon, with the tradeoff of a much smaller community and fewer off-the-shelf integrations.

Adopt Kubernetes when you need what only Kubernetes gives you, not when you need what any scheduler gives you.

How do you make the switch without a big-bang migration?

Move one stateless service first, and pick the least important one you have — an internal admin tool, a metrics exporter, a webhook receiver. It should be something whose outage costs you an apology, not revenue. That first workload is how you discover the parts nobody writes tickets for: image pull secrets, DNS resolution inside the cluster, log shipping, and how you actually get a shell when something breaks.

Keep state outside the cluster for as long as you can. Managed Postgres, managed object storage, and managed queues stay exactly where they are; running your own database inside Kubernetes is a separate project with its own operator, backup, and failover story, and taking both on at once is how migrations stall for a quarter.

Set an explicit rollback condition before you start — something like "if the admin tool is not stable on the cluster within three weeks, it goes back." Migrations without a stated failure condition tend to continue on sunk cost alone.

Your first cluster workload should be chosen for how little it matters, not how well it demonstrates the platform.

FAQ

How many services do you need before Kubernetes is worth it?
There is no clean count, but the useful proxy is deploy coordination, not service count. If you are running fewer than roughly five services and your deploys are independent, a managed runtime handles it; once services must be rolled out in a specific order, share service discovery, and are held together by custom scripts you are afraid to edit, the orchestrator is doing work you are currently doing by hand.

Is Kubernetes required for SOC 2 or HIPAA compliance?
No. Neither framework names any orchestrator. What auditors ask for is enforced access control, network segmentation between environments, encryption, and evidence of change management — all of which managed platforms can satisfy with separate environments and provider-level controls. Kubernetes becomes relevant when your segmentation requirements are finer-grained than the boundaries your platform lets you draw.

Can you run a multi-tenant SaaS without Kubernetes?
Yes, and most do. Tenant isolation at the data layer with row-level security or a database per tenant covers the majority of contractual requirements. You need kernel-level isolation only when tenants can trigger workloads that compete for CPU and memory, or when you execute code the tenant supplied.

Bottom line

If you are on a managed platform and shopping for a reason to leave it, measure your headroom first — connection saturation, CPU utilization at peak, and how much of your latency is your own code. Adopt Kubernetes when you hit a structural wall: untrusted workloads, isolation that must be enforced below your application, or an audit boundary your platform cannot draw. If you need scheduling but not the full ecosystem, Fargate, Cloud Run, and Nomad are the honest middle, and each costs you flexibility in exchange for the operational load it removes. And if nobody on your team can name the person who owns node upgrades six months from now, the answer for today is still no.

Where Should a Small Team Store Its Secrets? Comparing .env Files, SOPS, Doppler, 1Password, and Vault

Libme — Wed, 19 Aug 2026 21:42:06 +0000

If your team is under about ten people and already runs on one cloud provider, the cheapest correct answer is your provider's own secret store plus OIDC in CI, and no long-lived keys anywhere. If you're spread across several providers or hand secrets to non-infra teammates, a hosted secrets manager with a CLI injector — Doppler or 1Password — pays for itself the first time a rotation doesn't take an afternoon. Encrypted files in git (SOPS) are the right answer for a narrow case: config that must version alongside code, reviewed in PRs.

What almost never works past the second engineer is passing .env files around in Slack. Here's why, concretely, and what to replace it with.

The failure mode that ends the .env era

The bug that finally forced this decision on me wasn't a breach. It was a deploy that came up healthy and served traffic against the wrong database for twenty minutes.

The shape is always the same:

// db.js — the line that costs you an incident
const url = process.env.DATABASE_URL || "postgres://localhost:5432/app_dev";

A secret gets renamed in one place and not another, or a CI job runs before the env is populated, and process.env.DATABASE_URL is undefined. The || fallback is silent by design. Nothing throws, health checks pass, and the failure only surfaces when someone notices writes going nowhere.

Two fixes, and you want both. First, fail fast at boot so an absent secret is a crash, not a fallback:

// env.js — parse once, at startup, before anything connects
import { z } from "zod";

const schema = z.object({
  DATABASE_URL: z.string().url(),
  STRIPE_SECRET_KEY: z.string().startsWith("sk_"),
  SESSION_SECRET: z.string().min(32),
});

const parsed = schema.safeParse(process.env);
if (!parsed.success) {
  console.error("Invalid environment:", parsed.error.flatten().fieldErrors);
  process.exit(1);
}

export const env = parsed.data;

Import env everywhere instead of touching process.env directly, and add a lint rule banning process.env outside that one file. Now a missing secret is a loud, immediate, unambiguous crash.

Second, remove the drift itself by having exactly one place a secret lives. That's the actual decision below.

Fail-fast env parsing is the highest-leverage twenty lines in this entire post — do it before you pick a vendor, because it turns every secrets bug from a silent wrong answer into a stack trace.

What are the real options, and what does each cost you?

Approach	Best when	Real drawback
`.env` files, shared manually	Solo, one machine, throwaway projects	No rotation story, no audit trail, leaks via Slack/backups
SOPS + age/KMS, committed to git	Config that must version with code and be PR-reviewed	Rotation = commit + redeploy; key management is on you
Cloud-native (AWS Secrets Manager, Parameter Store, GCP Secret Manager, Azure Key Vault)	Already all-in on one cloud	Clumsy for local dev; per-secret and per-API-call billing adds up at high call volume
Doppler	Multi-provider deploys, want a fast CLI + integrations	Another vendor in your boot path; SaaS-first
1Password Secrets Automation	Team already lives in 1Password; humans and machines need the same vault	Service-account model takes a beat to grasp; usage-metered
Vault / OpenBao (self-hosted)	Dynamic short-lived DB creds, strict compliance	Genuine operational burden — unseal, HA, upgrades

A few honest notes on each, since the marketing pages won't give you these.

Cloud-native stores are the default nobody regrets on cost or reliability, and AWS Systems Manager Parameter Store's standard tier in particular is the underrated option — plain-string parameters with KMS encryption, no per-secret monthly charge on the standard tier as of mid-2026 (Secrets Manager bills per secret per month plus API calls, so check your call pattern before assuming it's cheap). The pain is local development: your laptop now needs cloud credentials to boot the app, which is exactly the long-lived key you were trying to eliminate.

Doppler is the one that handles the "same secret, five environments, three deploy targets" problem without a bespoke sync script, and its CLI injects secrets as environment variables for the duration of a process:

doppler run --project api --config dev -- node server.js

Nothing lands on disk, and doppler run composes with whatever your app already expects. The drawback is real: you've added a network dependency to your startup path, and you should understand its cached-fallback behavior before you put it in front of production boots.

1Password Secrets Automation is the right pick when the same credential needs to be readable by a person during an incident and by a machine during a deploy, because it's one vault with one audit log for both. The CLI resolves op:// references at launch:

# .env.template — safe to commit; contains references, not values
DATABASE_URL=op://prod/postgres/url
STRIPE_SECRET_KEY=op://prod/stripe/secret_key

op run --env-file=.env.template -- node server.js

Committing a template of references is the part that quietly fixes onboarding — a new hire clones, runs, and gets the right values without anyone DM'ing them a file.

Vault (or OpenBao, the Linux Foundation fork created after HashiCorp's 2023 license change) earns its complexity on exactly one feature: dynamic secrets. It can mint a Postgres user valid for an hour and revoke it automatically, so a leaked credential expires on its own. If nobody on your team wants to own unseal keys and HA topology, use the managed offering or don't use Vault — a badly-run Vault is worse than Parameter Store.

Vault is a correct answer to a question most small teams don't have yet; the question it answers is "how do I make leaked credentials expire by themselves."

How do you get secrets into CI without storing secrets in CI?

This is the part teams skip, and it's where the highest-value credentials sit. A long-lived cloud access key pasted into repository secrets is the single most valuable thing an attacker can get from your CI — it survives rotation of everything else.

Use OIDC federation instead. GitHub Actions can exchange a short-lived workflow identity token for cloud credentials, so no static key exists to steal:

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy
          aws-region: us-east-1
      - run: ./deploy.sh

The trust policy on that IAM role should pin the repository and the branch or environment — a wildcard subject condition means any workflow in your org can assume it. GitLab CI, CircleCI, and Buildkite all have equivalent OIDC flows.

If your CI still holds a long-lived cloud key in 2026, fixing that beats every other item on this list.

When is an encrypted-file approach (SOPS) actually right?

SOPS encrypts only the values in a YAML/JSON file, leaving keys readable, so a diff still shows which setting changed without revealing anything:

sops --encrypt --age age1ql3z... secrets.dev.yaml > secrets.dev.enc.yaml
sops --decrypt secrets.dev.enc.yaml

That property — reviewable diffs — is the whole argument. It fits GitOps and Kubernetes flows where config already ships through git, and it works offline with no vendor in the boot path.

The cost is that rotation means a commit, a merge, and a deploy, and every old value stays in git history forever. Once a secret has been in a repo, rotating it is the only real remediation — deleting the commit is not, since clones and forks keep the object. Treat SOPS as configuration-that-happens-to-be-sensitive, not as a credential vault.

FAQ

How should a small startup manage secrets across dev, staging, and production?
One store, three scopes, no files on laptops. Use your cloud provider's secret store if you're single-cloud, or Doppler/1Password if you aren't, and inject values into the process at launch instead of writing .env files to disk. Validate every required variable at boot so a missing secret crashes instead of falling back.

Is it safe to commit a .env file if it's encrypted?
Encrypted with SOPS or git-crypt, yes, with two caveats: the decryption key must live outside the repo, and any value that has ever been committed in plaintext must be rotated, because git history and existing clones keep it permanently.

Do I need HashiCorp Vault for a five-person team?
Almost certainly not. Vault's payoff is dynamic, short-lived credentials and fine-grained policy; below that bar, a managed secret store plus OIDC in CI gives you most of the security benefit with none of the unseal-and-HA operational burden.

Bottom line

Single-cloud teams should use their provider's secret store with OIDC in CI and stop there — it's the lowest-cost, lowest-drama option, and it removes the long-lived keys that actually get exploited. Teams deploying across several platforms, or handing credentials to people who don't touch infrastructure, get real time back from Doppler or 1Password's CLI injection. Reach for SOPS when sensitive config genuinely needs to be reviewed in pull requests, and for Vault or OpenBao only when you specifically want credentials that expire on their own. Whatever you choose, the boot-time validation and the CI key removal matter more than the vendor.

Your Postgres Migration Runner Needs a Retry Contract, Not Just a Lock Timeout

Libme — Mon, 17 Aug 2026 15:23:11 +0000

Setting lock_timeout on a migration keeps a blocked ALTER TABLE from freezing your traffic, but on its own it just converts one outage into a flaky deploy. The complete version is five rules: exactly one migrator at a time (advisory lock), a short lock timeout scoped to the transaction, the blocker logged before every retry, jittered backoff so replicas don't wake in lockstep, and a wall-clock budget after which the deploy fails instead of retrying forever. A reader raised this on an earlier post about migration locks, and it's the part most runners get wrong.

Everything here holds for Postgres 12 and up, as of mid-2026.

Why do several replicas try to run the same migration at once?

Most deploy systems run migrations as a pre-start step in the application container. Roll out four replicas and you have four processes racing to apply the same DDL. Usually the migration table's own row lock hides this, and you never notice.

You notice when the DDL is blocked. All four sit in the lock queue behind the same idle-in-transaction session. Then the blocker commits, all four wake, and whichever loses the race hits a duplicate-object error or, worse, a partially applied migration:

ERROR:  column "shipped_at" of relation "orders" already exists

Now the deploy is red for a reason that has nothing to do with the real problem. Worse, if you added deterministic exponential backoff — sleep 1s, 2s, 4s — every replica computes the same delays, so they retry in unison and keep colliding on the same schedule. Backoff without jitter is a synchronized herd.

Takeaway: a blocked migration turns a harmless replica race into a deploy failure, so serialization has to happen before the retry logic, not after.

How do I make sure only one migrator runs at a time?

Postgres advisory locks. They're application-defined locks on an arbitrary bigint key, scoped to the database, with no table involved:

-- session-level: held until unlocked or the connection closes
SELECT pg_try_advisory_lock(8675309);   -- true = you are the migrator
-- ...run migrations...
SELECT pg_advisory_unlock(8675309);

Use pg_try_advisory_lock (returns immediately) rather than pg_advisory_lock (waits forever) so a losing replica can decide what to do rather than hanging. Two reasonable choices: exit 0 and let the winner apply the schema, or wait a bounded time if the replica cannot safely serve traffic against the old schema. Pick one deliberately — silently exiting 0 is a real footgun when the new code needs the new column.

One deployment detail decides which lock scope you can use. If your connection goes through PgBouncer in transaction pooling mode, session-level advisory locks are unsafe, because the connection returns to the pool between transactions while the lock is still attached to it. There, use pg_advisory_xact_lock(8675309) inside the migration transaction, which Postgres releases automatically at commit or rollback.

Some runners already do this for you, and it's worth checking before you build your own:

Runner	Cross-process locking (as of mid-2026)	Watch out for
Flyway	Yes, database-level lock on Postgres	Verify behavior when running through a transaction pooler
Liquibase	Yes, via a `DATABASECHANGELOGLOCK` row	A killed run can leave the lock row set; needs manual release
Rails Active Record	Yes, advisory lock by default	Can be disabled in config; check it wasn't turned off
golang-migrate	Yes, advisory lock on the Postgres driver	Lock is per-database, so shared databases share the lock
Alembic	No built-in lock	Serialization is your job — wrap the runner yourself

If you want serialization without writing a runner, Flyway is the one that takes a database lock for you and fails the second process cleanly instead of letting it race.

Takeaway: check whether your migration tool already serializes runs before adding your own advisory lock — two locking schemes are not safer than one.

Where should lock_timeout actually be set?

Inside the transaction, with SET LOCAL, so it evaporates at commit:

BEGIN;
SET LOCAL lock_timeout = '3s';
SET LOCAL application_name = 'migrator:add_shipped_at';
ALTER TABLE orders ADD COLUMN shipped_at timestamptz;
COMMIT;

The reason to prefer SET LOCAL over a plain SET is pooling: a plain SET sticks to the backend, and a pooled connection can hand that 3-second timeout to unrelated application queries later. The distinctive application_name costs nothing and makes the blocker/waiter pair obvious in pg_stat_activity at 2am.

Two exceptions worth knowing. CREATE INDEX CONCURRENTLY cannot run inside a transaction block, so SET LOCAL doesn't apply — issue a session-level SET lock_timeout on that connection instead. And don't wrap CONCURRENTLY work in a tight statement_timeout: if the timeout kills it, you're left with an INVALID index that you must drop before retrying.

SELECT indexrelid::regclass FROM pg_index WHERE NOT indisvalid;

Takeaway: SET LOCAL lock_timeout bounds lock acquisition for this transaction only, which is exactly the scope you want on a pooled connection.

How should the retry loop back off, and what should it log first?

Full jitter — sleep a uniform random amount between zero and the current exponential cap — is what breaks the synchronized herd. And before each sleep, capture who blocked you, because a retry that succeeds tells you nothing about the query that will block tomorrow's migration too.

There's a catch people get wrong: you cannot call pg_blocking_pids(pg_backend_pid()) from the connection that is currently blocked — it's busy waiting. Sampling it live requires a second observer connection. The cheap approximation is to probe right after the lock_timeout fires, from the same connection, for the oldest transactions holding locks on that table:

SELECT a.pid, a.application_name, a.state,
       now() - a.xact_start AS xact_age,
       left(a.query, 80) AS query
FROM pg_stat_activity a
JOIN pg_locks l ON l.pid = a.pid
WHERE l.relation = 'public.orders'::regclass
  AND a.pid <> pg_backend_pid()
ORDER BY a.xact_start
LIMIT 5;

Occasionally the blocker has already finished and you log nothing. That's an acceptable trade for not running an observer thread. Here is the whole contract in one runner, using psycopg 3:

import random, time
import psycopg

LOCK_KEY = 8675309       # same constant in every replica
BUDGET_SECONDS = 300     # hard stop for the whole migration step
BASE, CAP = 1.0, 20.0

def run_migration(dsn, ddl, table):
    conn = psycopg.connect(dsn, autocommit=True,
                           application_name="migrator")
    with conn.cursor() as cur:
        cur.execute("SELECT pg_try_advisory_lock(%s)", (LOCK_KEY,))
        if not cur.fetchone()[0]:
            print("another migrator holds the lock; nothing to do")
            return 0

    deadline, attempt = time.monotonic() + BUDGET_SECONDS, 0
    try:
        while True:
            attempt += 1
            try:
                with conn.transaction(), conn.cursor() as cur:
                    cur.execute("SET LOCAL lock_timeout = '3s'")
                    cur.execute(ddl)
                print(f"applied on attempt {attempt}")
                return 0
            except psycopg.errors.LockNotAvailable:
                log_blockers(conn, table)
                delay = random.uniform(0, min(CAP, BASE * 2 ** (attempt - 1)))
                if time.monotonic() + delay > deadline:
                    print("lock budget exhausted; failing the deploy")
                    return 1
                print(f"blocked; retry {attempt} in {delay:.1f}s")
                time.sleep(delay)
    finally:
        with conn.cursor() as cur:
            cur.execute("SELECT pg_advisory_unlock(%s)", (LOCK_KEY,))
        conn.close()

Note that only LockNotAvailable (SQLSTATE 55P03) is retried. A syntax error or a constraint violation is not a transient condition, and retrying it just burns the budget.

Takeaway: retry only the lock-timeout error, sleep a random interval rather than a computed one, and log the blocker before every sleep.

When should the deploy just fail?

When the budget runs out. A migration that has been blocked for five minutes is not waiting on a slow query; it's waiting on something structural — an idle-in-transaction connection from a pool with no idle_in_transaction_session_timeout, a BI tool holding a long read, a pg_dump that overlaps your deploy window. Retrying past that point hides the diagnosis you actually need.

Symptom in the retry log	Likely cause	Fix, not a retry
Blocker `state = idle in transaction`, age growing	App or pool leaking an open transaction	Set `idle_in_transaction_session_timeout`
Blocker is a long analytics `SELECT`	Reporting traffic on the primary	Move reads to a replica, or migrate off-peak
Blocker is another `migrator:*`	Advisory lock missing or disabled	Fix serialization first
No blocker found, still timing out	Blocker is short but constant	Raise `lock_timeout` slightly, or use a quieter window

Takeaway: the budget's job is to convert an invisible hang into a red deploy with the blocker's identity attached.

FAQ

Why does my Postgres migration fail with "canceling statement due to lock timeout"?
Because lock_timeout is doing its job: your DDL waited longer than that limit for a lock another session held. That error means your app stayed up. Look at what held the lock — usually an idle-in-transaction connection or a long-running read — rather than raising the timeout.

Can two application replicas run database migrations at the same time?
Yes, unless something stops them. Serialize the runner with pg_try_advisory_lock on a fixed key, or confirm your migration tool takes its own lock. Behind PgBouncer in transaction pooling mode, use pg_advisory_xact_lock instead, since session-level locks outlive the transaction that took them.

Should migration retries use exponential backoff?
Use exponential backoff with full jitter — a random sleep between zero and the current cap. Deterministic delays make every replica wake at the same instant and contend again. Cap total retry time with a wall-clock budget and fail the deploy when it's exhausted.

Bottom line

If your migrations already use lock_timeout and a non-blocking two-step for rewrites, the missing piece is the runner around them. Add one migrator via advisory lock (or verify your tool has one), scope the timeout with SET LOCAL, tag the session with a distinctive application_name, log blockers before each jittered retry, and stop hard at a fixed budget. Teams on Flyway, Liquibase, Rails, or golang-migrate mostly need to verify the lock is on and add the budget; teams on Alembic or a hand-rolled script need the whole contract. The point isn't to make blocked migrations succeed — it's to make them fail fast, once, with the blocker named.

Your Postgres Backups Are Untested Until You Restore One: A Drill for Small Teams

Libme — Mon, 17 Aug 2026 15:20:20 +0000

A backup job that exits 0 tells you a file was written. It does not tell you the file can become a running database with your schema, your extensions, and your roles intact. The only way to know is to restore it on a schedule and time yourself, and the first restore you ever attempt should not be during an outage.

This is the drill I run — roughly monthly, and always after any change to the schema, the extension list, or the Postgres major version. It takes about twenty minutes once it's scripted, and every single time I've introduced it somewhere, the first run failed.

Why does a backup that "succeeded" fail to restore?

Because pg_dump captures the contents of one database, not the environment it lived in. The three things it leaves behind are the three things that break your restore:

Roles are cluster-level, not database-level. pg_dump does not include CREATE ROLE. Restore into a fresh cluster and you get:

pg_restore: error: could not execute query: ERROR:  role "app_user" does not exist
Command was: ALTER TABLE public.orders OWNER TO app_user;

Extensions must exist as installed binaries on the target machine. The dump contains CREATE EXTENSION vector;, but that only works if the target already has the pgvector shared library on disk:

ERROR:  could not open extension control file "/usr/share/postgresql/16/extension/vector.control": No such file or directory

This is the one that bites people who dump from a managed provider with extensions preinstalled and restore into a stock postgres:16 container.

Client and server versions have to line up. pg_dump refuses to dump from a server newer than itself:

pg_dump: error: server version: 16.3; pg_dump version: 15.6
pg_dump: error: aborting because of server version mismatch

The failure mode that actually hurts is silent: your cron box quietly ships an older pg_dump than your upgraded server, the job starts failing, and nobody reads the log because nothing pages on a backup job. Alert on backup failure and staleness, not just on failure.

A backup job's exit code proves a file exists; only a restore proves the file is a database.

What does the drill actually look like?

Five steps, all scriptable, all runnable on a laptop.

Pull the most recent backup artifact — from object storage, not from a copy sitting on the same host as production.
Start a throwaway Postgres of the same major version as production.
Restore into it with errors treated as fatal.
Run assertions: row counts on your three or four most important tables, plus a query that exercises an extension.
Record the wall-clock time and tear it down.

Here's the core of it. Taking the dump:

#!/usr/bin/env bash
set -euo pipefail

pg_dump \
  --format=custom \
  --no-owner \
  --no-privileges \
  --file="backup_$(date -u +%Y%m%dT%H%M%SZ).dump" \
  "$DATABASE_URL"

--no-owner and --no-privileges are what make the dump portable: ownership statements get dropped, so the restore doesn't demand that app_user exists on the target. You then re-apply grants from your migration tooling, where they belong. Keep roles in version control as SQL; do not rely on them surviving in a dump.

The drill itself:

#!/usr/bin/env bash
set -euo pipefail

DUMP="$1"
PGVER="${PGVER:-16}"
CONTAINER="restore-drill-$$"

start=$(date +%s)

docker run -d --name "$CONTAINER" \
  -e POSTGRES_PASSWORD=drill \
  -p 55432:5432 \
  "pgvector/pgvector:pg${PGVER}"

until docker exec "$CONTAINER" pg_isready -U postgres -q; do sleep 1; done

export PGPASSWORD=drill
CONN="postgresql://postgres@localhost:55432/postgres"

pg_restore --dbname="$CONN" --exit-on-error --jobs=4 "$DUMP"

psql "$CONN" -v ON_ERROR_STOP=1 -f drill_assertions.sql

echo "restore completed in $(( $(date +%s) - start ))s"
docker rm -f "$CONTAINER" >/dev/null

Two flags carry most of the weight. --exit-on-error is the important one: by default pg_restore prints errors, keeps going, and exits 0, which means an unattended restore check without it will happily report success on a half-populated database. --jobs=4 parallelizes table data and index builds, and it only works with the custom or directory formats — another reason to stop using plain SQL dumps for anything large.

The assertions file is deliberately boring, and it should fail loudly:

\set ON_ERROR_STOP on

-- structural: does the extension work, not just exist?
SELECT '[1,2,3]'::vector <-> '[3,2,1]'::vector AS distance_check;

-- volumetric: catch a restore that "worked" but landed empty
DO $$
DECLARE n bigint;
BEGIN
  SELECT count(*) INTO n FROM orders;
  IF n < 1000 THEN
    RAISE EXCEPTION 'orders table has only % rows — restore is suspect', n;
  END IF;
END $$;

-- freshness: how much data would we actually have lost?
SELECT now() - max(created_at) AS data_age FROM orders;

That last query is the one worth reading out loud in a team channel. It converts an abstract retention policy into a number: if we restored right now, we would be missing this much. That number is your real RPO, and it is usually worse than whatever the backup docs implied.

Assert on row counts and data age, not on the restore's exit code — an empty database restores perfectly.

Nightly dumps or point-in-time recovery?

pg_dump gives you a consistent snapshot and nothing between snapshots. If it runs at 03:00 and you lose the primary at 17:00, you have lost fourteen hours. Point-in-time recovery closes that gap by shipping the write-ahead log continuously, so you can replay to a chosen moment — including "one second before that DELETE without a WHERE."

Approach	Typical RPO	Restore complexity	Where it fits
`pg_dump` custom format to object storage	Since last dump (hours)	Low — one command	Side projects, small internal apps, portable migrations
Managed provider snapshots + PITR	Seconds to minutes	Low, but provider-shaped	Anything on RDS, Cloud SQL, Supabase, Neon, Crunchy Bridge
pgBackRest or WAL-G to your own bucket	Seconds to minutes	Medium — real config, real ops	Self-hosted Postgres you intend to keep
Physical replica only	Near zero for hardware loss	N/A	Not a backup — replicates your `DROP TABLE` faithfully

That last row is the trap I see most often. A standby protects you from a dead machine; it does not protect you from a bad migration, because the destructive statement replicates in milliseconds. Keep both.

If you self-host and want continuous archiving without writing your own WAL shipping, pgBackRest is the option that handles full/differential/incremental backups, parallel compression, and retention expiry with one config file and one command — at the cost of a genuinely non-trivial setup pass and a config format you will have to read the docs for every time you touch it. WAL-G is the leaner alternative when you want to push straight to S3-compatible storage with minimal moving parts, though you'll find fewer worked examples when something goes wrong. On the managed side, Neon's branching turns a restore drill into creating a branch from a past timestamp and pointing a test connection string at it, which is the lowest-friction version of this whole workflow — the constraint being that it's provider-specific, so you should still keep an independent logical dump if you ever want to leave.

Whatever you pick, store backups in an account or bucket that your application's credentials cannot delete. Ransomware and a bad terraform destroy fail the same way.

A replica is not a backup, and a backup you cannot restore without the original provider is a hostage.

How often should you run the drill?

Monthly is a reasonable floor for a small team. Run it additionally after: a Postgres major version upgrade, adding or removing an extension, changing the backup tool or its flags, and any change to who owns the storage bucket. Wire it into CI on a schedule if you can — a weekly GitHub Actions job that restores yesterday's dump into a service container and runs the assertions gives you a red build instead of a discovery at 2 a.m.

Track exactly two numbers over time: minutes to a usable database (your RTO) and the data_age from the assertions (your RPO). If either number surprises someone on the team, you've found the actual gap. As of mid-2026 I've never seen a first drill where both numbers matched what people assumed.

FAQ

How do I test a Postgres backup without touching production?
Restore the dump into a disposable container on a non-production port, run assertions against it, then destroy the container. Nothing in the drill connects to production except the read that fetched the backup file from object storage.

Why does pg_restore exit 0 even though it printed errors?
Because pg_restore treats most errors as non-fatal by default so it can restore as much as possible. Pass --exit-on-error (and --single-transaction if you want all-or-nothing) whenever a script is checking the result.

Is a read replica enough of a backup for a small app?
No. A replica protects against losing a machine but faithfully replicates destructive SQL such as a bad DELETE or migration. You need point-in-time recovery or periodic dumps to recover from a mistake you made yourself.

Bottom line

If you run a side project or a small internal app, nightly pg_dump --format=custom --no-owner to object storage plus a monthly scripted restore drill is enough, and it's an afternoon of work. Once real customer data is involved, move to continuous archiving — your provider's PITR if you're managed, pgBackRest or WAL-G if you're self-hosted — and keep the logical dump as your escape hatch from the provider. Never count a physical replica as a backup. And measure the drill: the restore you have never timed is the one that takes three hours on the worst possible day.

DEV Community: Libme

Is Your CDN Actually Caching? Audit Hit Ratio and Asset Fan-Out Before You Migrate Buckets

Why doesn't my egress bill match my download volume?

How do I measure cache hit ratio without buying an analytics add-on?

What makes a hit ratio quietly collapse?

What do the two numbers actually tell me to do?

How do I turn this into a before/after number?

FAQ

Bottom line

Related reading

Why Your OAuth Integration Randomly Returns invalid_grant (and How to Stop Two Workers From Racing)

What does invalid_grant actually mean?

Why does this only blow up in production?

How do you serialize refresh across processes?

Should you refresh on 401 or before expiry?

Which locking strategy fits your stack?

How do you tell a race apart from a real revocation?

FAQ

Bottom line

Related reading

S3 vs R2 vs B2 for User Uploads: When Egress Fees Should Change Your Architecture

What actually drives an object storage bill?

When is S3 still the right call?

When do egress fees justify moving?

Why does the AWS SDK break against R2 and B2?

What does a low-regret setup look like?

FAQ

Bottom line

Related reading

Before You Set plan_cache_mode, Write the Regression Test That Proves It Worked

Why doesn't one fast query prove the fix?

How do you make a test session use the cached generic plan?

Building fixtures from the real parameter distribution

What should the test assert besides latency?

Which knob do you actually reach for?

When "same SQL" isn't the same query

FAQ

Bottom line

Related reading

Your Cache Expired and the Database Spiked: Four Fixes for Cache Stampedes

What does a cache stampede actually look like in the logs?

Why doesn't adding cache capacity or raising the hit rate fix it?

Fix 1: Jitter every TTL, without exception

Fix 2: Single-flight, so only one worker recomputes

Fix 3: Stale-while-revalidate, which is the one that removes the cliff

Fix 4: Probabilistic early recomputation

Which fix should you actually apply?

FAQ

Bottom line

Related reading

Rate Limiting Your Own API: Should the Counter Live in Redis, Postgres, or at the Edge?

The symptom: a 100 req/min limit that lets through 400

Where can the counter actually live?

How do you implement a token bucket in Redis without race conditions?

Can you do this in Postgres if you don't want another dependency?

What belongs at the edge instead?

What should the response actually say?

FAQ

Bottom line

Related reading

Why Your Query Is Fast in psql but Slow From the App: Postgres Plan Caching Explained

What the symptom actually looks like

How does Postgres decide to reuse a plan?

How do I prove this is what's happening?

What do I actually change?

What if it isn't the plan cache?

FAQ

Bottom line

Related reading

Your Secrets Manager Ends at process.env: Where Secrets Actually Leak at Runtime

Why does the store barely change the leak path?

How do you stop a secret from ever printing?

What about the loggers and error reporters?

How do subprocesses and CI leak secrets?

Does any of this change which store you should pick?

FAQ

Bottom line

Related reading

When Should You Move Off Managed Hosting to Kubernetes? The Thresholds That Actually Matter

Why is "we're getting too much traffic" almost never the real trigger?

What does `invalid_grant` actually mean?