DEV Community: Libme

Webhook Signature Verification Fails: Your Framework Already Parsed the Body

Libme — Sun, 13 Sep 2026 23:58:29 +0000

If a webhook signature check fails and you are sure the secret is right, the body your code is hashing is almost never the body the sender hashed. Something between the socket and your handler parsed the JSON and re-serialized it, and HMAC does not forgive a single changed byte. The fix is always the same shape: get the raw request bytes before any body parser runs, hash those, and compare with a constant-time function.

This post walks through the failure as it shows up with Stripe and GitHub, why the obvious workaround (JSON.stringify(req.body)) fails in ways that look random, and the raw-body recipe for each framework I have had to do this in.

What does the error actually look like?

With stripe-node the message is explicit, which is why it ends up in so many search queries:

Webhook Error: No signatures found matching the expected signature for payload.
Are you passing the raw request body you received from Stripe?

GitHub gives you nothing that helpful. Your own check just returns false, the delivery shows a 401 in the repository's webhook settings, and you start doubting the secret. A quick way to rule the secret out: copy the raw payload from GitHub's "Recent Deliveries" tab, HMAC it locally with the secret, and compare with the X-Hub-Signature-256 header shown on the same page. If that matches and your server still rejects it, the server is hashing different bytes.

Both senders sign the exact byte sequence they put on the wire. Stripe signs ${timestamp}.${rawBody} with HMAC-SHA256 and sends t=...,v1=... in the Stripe-Signature header. GitHub signs the raw body and sends sha256=<hex> in X-Hub-Signature-256. Neither signs "the JSON document" in any abstract sense.

The signature is over bytes, not over the data those bytes encode, so any step that decodes and re-encodes the body invalidates it.

Why does re-serializing the parsed body not work?

The first thing most people try is JSON.stringify(req.body) and hoping it round-trips. It sometimes does, which is the worst outcome, because it passes in tests and fails in production on the first payload that contains a float, a non-ASCII character, or whitespace the sender happened to emit.

const raw = '{"amount": 1000, "note": "caf\\u00e9", "ratio": 1.0}';
JSON.stringify(JSON.parse(raw));
// '{"amount":1000,"note":"café","ratio":1}'

Three differences from one line: the spaces after colons are gone, é became a literal é, and 1.0 became 1. Any one of them changes the hash. Python's json.dumps has its own defaults (spaces after separators, ensure_ascii=True) that differ from JavaScript's, so a Node sender and a Python receiver will disagree even on the "same" object.

There is no serializer setting that reliably reproduces another system's output; the only byte-exact copy of the payload is the one you received.

How do I get the raw body in each framework?

The pattern is the same everywhere: read the body stream once, as bytes, before anything else consumes it. Where it goes wrong is framework-specific.

Framework	Raw body access	What silently breaks it
Express	`express.raw({ type: "application/json" })` on the webhook route only	Registering `app.use(express.json())` before the webhook route
Next.js App Router	`await request.text()` in the route handler	Calling `request.json()` first; the stream can be read once
Next.js Pages Router	`export const config = { api: { bodyParser: false } }` and read the stream	Forgetting the config export; the default parser runs
FastAPI / Starlette	`await request.body()`	Nothing serious; Starlette caches the body, so this is the easy one
Flask	`request.get_data()`	Using `request.data`, which is empty for form mimetypes
AWS Lambda (API Gateway)	`event.body`, decode with base64 when `event.isBase64Encoded` is true	Hashing the base64 string instead of the decoded bytes

Express deserves the full example because route order is the trap. The raw parser has to be attached to the webhook route and the JSON parser has to come after it, or be scoped so it never sees that path:

import express from "express";
import Stripe from "stripe";

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY);
const app = express();

// Webhook route first, with a raw parser. req.body is a Buffer here.
app.post(
  "/webhooks/stripe",
  express.raw({ type: "application/json" }),
  (req, res) => {
    let event;
    try {
      event = stripe.webhooks.constructEvent(
        req.body,
        req.headers["stripe-signature"],
        process.env.STRIPE_WEBHOOK_SECRET
      );
    } catch (err) {
      return res.status(400).send(`Webhook Error: ${err.message}`);
    }
    // handle event.type here
    res.sendStatus(200);
  }
);

// Everything else gets parsed JSON.
app.use(express.json());

If you cannot reorder middleware (a shared app factory, a framework wrapper you do not control), the verify hook on express.json is the escape hatch. It runs with the original buffer before parsing and lets you stash it:

app.use(
  express.json({
    verify: (req, res, buf) => {
      req.rawBody = buf;
    },
  })
);

Then verify against req.rawBody. This works for every route at once, at the cost of holding a second copy of each body in memory, which matters only if your JSON bodies are large.

For a GitHub-style HMAC where you write the check yourself, the comparison has to be constant-time and length-checked, because timingSafeEqual throws on mismatched lengths rather than returning false:

import { createHmac, timingSafeEqual } from "node:crypto";

export function verifyGithubSignature(rawBody, headerValue, secret) {
  if (!headerValue) return false;
  const expected =
    "sha256=" + createHmac("sha256", secret).update(rawBody).digest("hex");
  const a = Buffer.from(expected);
  const b = Buffer.from(headerValue);
  return a.length === b.length && timingSafeEqual(a, b);
}

And the Python equivalent, where hmac.compare_digest already handles the constant-time part:

import hashlib
import hmac
import os

from fastapi import FastAPI, HTTPException, Request

app = FastAPI()
SECRET = os.environ["GITHUB_WEBHOOK_SECRET"].encode()


@app.post("/webhooks/github")
async def github_webhook(request: Request):
    raw = await request.body()  # bytes, exactly as received
    sig = request.headers.get("x-hub-signature-256", "")
    expected = "sha256=" + hmac.new(SECRET, raw, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, sig):
        raise HTTPException(status_code=401, detail="bad signature")
    payload = await request.json()  # safe to parse now
    ...

Whichever framework you use, the webhook route should be the one place in the app where the body parser is explicitly bypassed, and the code should say so in a comment for the next person who "fixes" the middleware order.

What else changes the bytes between sender and handler?

Once the body parser is out of the way, the remaining causes are infrastructure. Two I have actually hit:

An API gateway with a body mapping template or JSON transformation enabled. Anything that reformats request bodies for downstream services is re-serializing them. The fix is to exempt the webhook path from the transformation, not to try to reverse it.

Lambda behind API Gateway with binary media types configured. The body arrives base64-encoded and isBase64Encoded is true; hashing event.body directly hashes the base64 text. Decode to a Buffer first, then hash.

Ordinary reverse proxies and CDNs do not touch bodies, so nginx, Caddy, or Cloudflare in front of the app are not suspects here. Compression is also not a suspect for inbound webhooks, because the sender does not gzip the request body and the proxy decompresses before your handler sees it anyway.

If the parser is bypassed and verification still fails, look for a component that advertises "transformation" or "mapping" of request bodies; it is doing exactly what it says.

How do I test this without waiting for real events?

For Stripe, the CLI's stripe listen --forward-to localhost:3000/webhooks/stripe gives you a local signing secret and replays real event shapes, which is the fastest loop I know for this class of bug; its limitation is that it only speaks Stripe. For any other sender, expose the local port with ngrok or Cloudflare Tunnel and use the provider's "redeliver" button. If you want a hop that records every delivery and lets you replay it against a rebuilt handler, Hookdeck is the one that sits in front of your endpoint and keeps the raw payload and headers, at the cost of adding an external dependency on the path that has to be up when the payment provider retries.

Whatever you use, write one unit test that feeds a hard-coded raw payload and a signature you computed yourself, so a future middleware change fails in CI rather than on a customer's checkout.

FAQ

Why does Stripe say "No signatures found matching the expected signature for payload" in Express?
Because express.json() parsed the body before constructEvent ran, so the handler hashed a re-serialized object instead of the bytes Stripe signed. Attach express.raw({ type: "application/json" }) to the webhook route and register it before the JSON parser.

Can I verify a webhook signature from a parsed JSON body?
No. Serializers normalize whitespace, unicode escapes, and number formatting, so the output is not byte-identical to what the sender signed. You need the original request bytes.

Should I compare the signature with ===?
No. Use crypto.timingSafeEqual in Node or hmac.compare_digest in Python, and check lengths first in Node because timingSafeEqual throws on different-length inputs.

Bottom line

If you own the app, put the raw-body parser on the webhook route and the JSON parser after it, and treat that ordering as load-bearing. If you cannot control middleware order, use the parser's verify hook to capture the buffer for every request. If the parser is already out of the picture and signatures still fail, the culprit is a gateway or serverless layer re-encoding the body, not your secret. In every case, lock the behavior in with a test that uses a fixed payload and a precomputed signature.

Playwright vs Cypress When Your E2E Tests Pass Locally but Fail in CI

Libme — Sun, 13 Sep 2026 01:01:35 +0000

Tests that pass on your laptop and fail in CI are almost never a tool bug; they are timing assumptions that a slower, colder CI runner exposes. Both Playwright and Cypress can be made stable, but they get there differently: Playwright gives you built-in retries, per-test traces, and free sharding across runners; Cypress gives you retry-able assertions inside the browser and a very good interactive debugger, with parallelization living behind its paid cloud service. If you are choosing today, Playwright is the lower-friction path for a CI-first team, and Cypress is the stronger pick when the people writing tests are mostly debugging them by watching the browser.

What does "flaky in CI only" actually look like?

The two failure texts I see most often are worth recognizing on sight. In Playwright:

Error: locator.click: Test timeout of 30000ms exceeded.
Call log:
  - waiting for getByRole('button', { name: 'Save' })
  -   locator resolved to <button disabled …>Save</button>
  - attempting click action
  -   waiting for element to be visible, enabled and stable
  -   element is not enabled

In Cypress:

CypressError: Timed out retrying after 4000ms: Expected to find element: `[data-test=save-button]`, but never found it.

Locally these never fire because your dev server responds in tens of milliseconds and the button becomes enabled before the test even looks for it. In CI, the app boots cold, the runner has two vCPUs shared with the browser, and the same request takes long enough that the test's assumption ("by the time I click, the form is ready") stops holding. The tool is telling the truth: the element genuinely was not ready inside its default budget.

The dead ends people try first are the same in both tools: adding waitForTimeout(2000) or cy.wait(2000), then bumping the number when it fails again. That trades a flaky test for a slow one and still fails on the one run where CI is slower than your padding. The fix is always to wait for the condition the click depends on, and both tools have a proper primitive for that.

Takeaway: a CI-only failure is a hidden sleep in your test logic, and the fix is naming the condition you were implicitly assuming.

How do Playwright and Cypress wait differently?

Playwright auto-waits before every action: click() will not fire until the element is attached, visible, stable, enabled, and receiving pointer events. Assertions like expect(locator).toBeEnabled() retry until the expect timeout. What it does not do is wait for arbitrary application state, so if your button is enabled before its data is loaded, you have to say what you mean:

import { test, expect } from '@playwright/test';

test('saves the profile', async ({ page }) => {
  const profileLoaded = page.waitForResponse(
    (r) => r.url().includes('/api/profile') && r.status() === 200
  );
  await page.goto('/settings');
  await profileLoaded;

  await page.getByLabel('Display name').fill('Jay');
  await page.getByRole('button', { name: 'Save' }).click();
  await expect(page.getByText('Saved')).toBeVisible();
});

Cypress runs inside the browser and makes commands retry-able rather than actions auto-waiting. cy.get() and the assertion chained to it keep retrying until the default command timeout, and the idiomatic wait for network is an intercept alias:

it('saves the profile', () => {
  cy.intercept('GET', '/api/profile').as('profile');
  cy.visit('/settings');
  cy.wait('@profile');

  cy.findByLabelText('Display name').type('Jay');
  cy.get('[data-test=save-button]').should('be.enabled').click();
  cy.contains('Saved').should('be.visible');
});

Both are correct. The difference that matters in CI is what happens when they are not correct. Playwright's error above came with a call log that said "element is not enabled", which points straight at the cause. Cypress's error tells you the element was never found, and to see why you open the run's screenshot or video, or use Test Replay if you are on Cypress Cloud. In my experience the call log alone resolves a good share of CI-only failures without downloading anything.

One Cypress-specific trap: because commands are queued and run asynchronously, mixing them with plain await or if on a value you have not yet yielded produces tests that pass locally by luck. Playwright's plain async/await model has fewer of these footguns, but it has its own: forgetting an await on an action silently races the next line, and TypeScript will not always warn you.

Takeaway: Playwright waits on the action and tells you which precondition failed; Cypress waits on the query and tells you it eventually gave up.

What do retries and artifacts cost you in each tool?

Retries are a diagnostic tool, not a fix, and both tools support them in a way that keeps the first-run failure visible.

Playwright's config lets you retry only in CI and capture a full trace only when a retry happens, which keeps artifact size small on green runs:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 2 : undefined,
  fullyParallel: true,
  reporter: process.env.CI ? 'blob' : 'html',
  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
});

A test that passes on retry is reported as "flaky" rather than "passed", so you can grep the report for tests that are quietly degrading. The trace file opens in the trace viewer with a DOM snapshot at every step, network log, and console output; it is the single feature that made me stop reproducing CI failures locally.

Cypress configures retries per mode, and since version 13 (as of my last check) does not record video by default, so you opt in:

// cypress.config.js
const { defineConfig } = require('cypress');

module.exports = defineConfig({
  retries: { runMode: 2, openMode: 0 },
  video: true,
  e2e: { baseUrl: 'http://localhost:3000' },
});

The honest limitation on the Cypress side is that the richest artifact, Test Replay, requires recording to Cypress Cloud. Without it you are working from screenshots and video, which show what happened but not the network timeline. On the Playwright side, the honest limitation is that traces on a large suite with trace: 'on' get big fast and slow uploads down, which is why the on-first-retry setting is the one to keep.

Takeaway: turn on retries in CI only, and treat "flaky" in the report as a bug queue, not a pass.

When does parallelization change the decision?

This is where the tools diverge most sharply. Playwright shards a suite across CI machines with a flag and merges the reports afterward, with no service involved:

# .github/workflows/e2e.yml (excerpt)
strategy:
  fail-fast: false
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: npx playwright test --shard=${{ matrix.shard }}/4
  - uses: actions/upload-artifact@v4
    with:
      name: blob-report-${{ matrix.shard }}
      path: blob-report

A follow-up job downloads the blobs and runs npx playwright merge-reports --reporter html ./all-blob-reports. Test isolation comes from a fresh browser context per test, which is cheap, so fullyParallel is usually safe unless tests share a database row.

Cypress runs one spec at a time per machine. Splitting specs across machines is a Cypress Cloud feature (load-balanced by historical duration), and the open-source alternatives are third-party orchestrators you host yourself. For a small suite this does not matter. For a suite that has crossed the point where a single runner takes fifteen minutes, it is often the deciding factor, because you are choosing between a recurring bill and operating another service.

Concern	Playwright	Cypress
Waiting model	Auto-wait on actions, retrying `expect`	Retry-able commands and chained assertions
Default failure detail	Step call log in the error	Screenshot; video and Test Replay optional
Retries	`retries` in config, per-project	`retries` per run/open mode
Trace/time-travel	Trace viewer, local files	Time-travel in interactive runner; Test Replay via Cloud
Parallel across machines	Built-in `--shard`, free	Cypress Cloud or self-hosted orchestrator
Multi-tab / multi-origin	Native contexts and pages	`cy.origin()` for cross-origin; no true multi-tab
Browsers	Chromium, Firefox, WebKit	Chrome family, Firefox, WebKit experimental
Debugging locally	UI mode, `--debug`, trace viewer	Interactive runner with DOM snapshots per command

If your team's bottleneck is CI wall-clock time and you want sharding without a vendor, Playwright is the one that gives you free cross-machine parallelism with a single flag. If your team's bottleneck is people understanding why a test failed while watching it run, Cypress is the one whose interactive runner makes every command's before-and-after DOM state clickable without extra setup.

Takeaway: the moment your suite outgrows one runner, the parallelization model stops being a feature comparison and becomes a budget line.

FAQ

Why do my Playwright tests pass locally but fail in CI?
Because CI is slower and colder, so elements that were already ready on your machine are still loading when the test acts. Replace fixed waits with expect(locator) assertions or page.waitForResponse for the specific request the action depends on, and enable trace: 'on-first-retry' to see the exact step that stalled.

How do I make Cypress tests less flaky in CI?
Set retries: { runMode: 2, openMode: 0 }, wait on intercepted requests with cy.intercept().as() and cy.wait('@alias') instead of cy.wait(ms), and chain .should('be.enabled') before clicking. Treat any test that only passes on retry as a bug to fix, not a green result.

Is Playwright or Cypress better for CI?
Playwright has the lower CI cost because retries, traces, and cross-machine sharding are built in and free. Cypress is competitive on a single runner but its parallelization and richest failure artifacts depend on Cypress Cloud.

Bottom line

If you are starting a new suite and it will run primarily in CI, pick Playwright: auto-waiting reduces the class of timing bugs, the trace viewer answers "why did this fail" without a local repro, and sharding is a flag rather than a subscription. If your team already has a Cypress suite and mostly debugs by watching the interactive runner, stay and fix the flake with intercept aliases and CI-only retries; the tool is not the problem. Migrate only when a single runner's wall-clock time becomes the thing blocking merges, because that is the one gap the free tier of Cypress does not close.

Your Nightly Job Ran Twice on the DST Switch: Making Scheduled Jobs Timezone-Safe

Libme — Fri, 11 Sep 2026 00:26:28 +0000

If your scheduler stores a wall-clock time in a local timezone, a job at 01:30 runs twice on the fall-back date and a job at 02:30 runs zero times on the spring-forward date. Setting TZ=America/New_York on your cron does not fix this — it is what causes it. The fix is to separate when the trigger fires from what logical period the job covers, and to make the job claim its logical date before doing work.

I lost a morning to this: a billing summary that emailed customers twice on November 1, once at 01:30 EDT and once at 01:30 EST, 60 minutes apart, from a scheduler nobody had touched in months.

What actually happens at the transition

Two local wall-clock times are broken, not one. In America/New_York, DST for 2026 starts Sunday March 8 and ends Sunday November 1 (EU zones switch on different dates — March 29 and October 25 in 2026, still observed as of mid-2026).

Fall back: the clock goes 01:59 → 01:00. Every wall-clock time in the 01:00–01:59 range happens twice, one hour apart in real time. A cron entry for 30 1 * * * fires on both.
Spring forward: the clock goes 01:59 → 03:00. Wall-clock times in 02:00–02:59 never occur. A cron entry for 30 2 * * * fires once, twice, or not at all depending on which scheduler you use — none of them agree.

Python's zoneinfo makes the ambiguity visible through PEP 495's fold flag:

from datetime import datetime, timezone
from zoneinfo import ZoneInfo

ny = ZoneInfo("America/New_York")
first  = datetime(2026, 11, 1, 1, 30, tzinfo=ny)           # fold=0
second = datetime(2026, 11, 1, 1, 30, fold=1, tzinfo=ny)

print(first.astimezone(timezone.utc))   # 2026-11-01 05:30:00+00:00
print(second.astimezone(timezone.utc))  # 2026-11-01 06:30:00+00:00
print(first == second)                  # True  <- the trap

Those are two different instants an hour apart, and Python reports them as equal, because same-zone comparison ignores fold by design. Any dedupe check written as if run_at == last_run_at: skip silently passes through the duplicate. Convert to UTC before comparing and the check works.

The nonexistent time is quieter. datetime(2026, 3, 8, 2, 30, tzinfo=ny) constructs happily with a -05:00 offset, but round-tripping it through UTC lands on 03:30 EDT — the value you built never existed on any clock.

Takeaway: a local wall-clock time is not a unique instant twice a year, and your language will not raise an error when you treat it like one.

Why "just set the scheduler's timezone" isn't enough

Scheduler-native timezone support is real and worth using, but it only decides when to fire. It gives you nothing about the far more expensive question: did this job already run for this business day?

Platform	Local timezone support	Behavior you still have to handle
Linux `cron` (`CRON_TZ=`)	Yes	Ambiguous hour can fire twice; gap behavior varies by cron implementation
systemd timers	Yes (`OnCalendar` + `Timezone=`)	`Persistent=true` fires missed runs at boot — a second execution
Kubernetes `CronJob`	Yes, `spec.timeZone` (stable since 1.27)	Docs state a schedule may create two Jobs or none for one slot
GitHub Actions `schedule`	No — UTC only	Delivery is best-effort and can be delayed under load
Amazon EventBridge Scheduler	Yes, IANA names	Retries and flexible time windows can re-invoke the target
Google Cloud Scheduler	Yes, `timeZone`	At-least-once delivery; the target must be idempotent

Notice the pattern in the right column: every one of these is at-least-once. DST is not a special case, it is just the twice-a-year reminder that your job needs to survive being invoked more than once for the same period.

Takeaway: scheduler timezone settings fix the wall clock, not the duplicate; none of these platforms promise exactly-once.

The two-clock rule

Every scheduled job has two clocks, and mixing them is the actual bug:

The trigger clock — a real instant, always UTC, best-effort, may fire early, late, or twice.
The logical clock — the business period the run covers ("the report for 2026-11-01"), which advances exactly once per period no matter what the trigger does.

Once you name the logical date, the fix writes itself. Tick frequently in UTC, compute whether the local target time has passed for the current logical date, then claim that date before working:

from datetime import datetime, date, time, timedelta, timezone
from zoneinfo import ZoneInfo

UTC = timezone.utc

def local_target_utc(day: date, wall: time, tz: ZoneInfo) -> datetime:
    """UTC instant for `wall` clock time on `day` in `tz`.
    Nonexistent (spring forward) -> the instant the clock jumps to.
    Ambiguous (fall back) -> the first, earlier occurrence."""
    first = datetime.combine(day, wall).replace(tzinfo=tz, fold=0)
    resolved = first.astimezone(UTC).astimezone(tz)
    if resolved.time() != wall:          # the wall time was skipped
        return resolved.astimezone(UTC)
    return first.astimezone(UTC)

def is_due(now_utc: datetime, tz: ZoneInfo, wall: time) -> date | None:
    logical = now_utc.astimezone(tz).date()
    return logical if now_utc >= local_target_utc(logical, wall, tz) else None

Verified against the 2026 transitions in America/New_York: a 02:30 target on March 8 resolves to 07:30 UTC (03:30 EDT, the moment the clock lands), and a 01:30 target on November 1 resolves to 05:30 UTC — the first 01:30, deterministically, not whichever one the scheduler happens to hit.

The claim is a unique constraint, not a lock you have to think about:

create table job_runs (
  job_name     text        not null,
  logical_date date        not null,
  started_at   timestamptz not null default now(),
  primary key (job_name, logical_date)
);

cur.execute(
    "insert into job_runs (job_name, logical_date) values (%s, %s) "
    "on conflict do nothing",
    ("billing_summary", logical),
)
if cur.rowcount == 0:
    return  # already ran for this logical date

Run this every 15 minutes in UTC from any scheduler on the table above and the DST question disappears: extra triggers hit the conflict and return, a missed trigger is picked up by the next tick, and a retried invocation is free. One caveat worth handling on first deploy — a brand-new job whose target time has already passed today will fire immediately, so seed the table with today's row if you don't want that.

Takeaway: a unique key on (job, logical_date) turns at-least-once delivery into exactly-once work, and DST stops being a scheduling problem.

Which approach for which job?

Approach	Survives gap/ambiguity	Survives duplicate trigger	Complexity
Local-timezone cron entry	No	No	Lowest
UTC cron + convert inside the job	Yes	No	Low
Frequent UTC tick + claim logical date	Yes	Yes	Medium
Durable execution engine	Yes	Yes	Highest

For jobs where a duplicate run is merely wasteful (cache warmers, syncs that overwrite), UTC cron plus in-job conversion is enough. For anything that sends, charges, or emits an external side effect, use the claim. If you want the managed version of the whole model, Temporal gives you schedules with a timezone plus workflow-ID deduplication so a repeated trigger resolves to the same durable execution rather than a second one — at the cost of running a server and rewriting the job as a workflow. If your jobs are already a DAG of data transformations, Airflow's data-interval model bakes the logical clock in as a first-class concept, though it brings a scheduler, metadata database, and upgrade treadmill with it.

How do you test this before November?

Don't wait for the transition. Enumerate it — step a fake clock through the window in UTC and assert the run count:

def test_one_run_per_logical_date():
    ny, claimed = ZoneInfo("America/New_York"), set()
    t = datetime(2026, 10, 31, 12, 0, tzinfo=UTC)
    while t < datetime(2026, 11, 2, 12, 0, tzinfo=UTC):
        logical = is_due(t, ny, time(1, 30))
        if logical:
            claimed.add(logical)   # set stands in for the unique constraint
        t += timedelta(minutes=15)
    assert len(claimed) == 3       # Oct 31, Nov 1, Nov 2 — one each

Point the same test at March 8 and at a European zone. The scheduling logic must be a pure function of (now_utc, tz, wall_time) for this to be testable at all — which is the real reason to keep it out of the cron expression.

Takeaway: if you cannot simulate the DST window in a unit test, your scheduling logic lives in the wrong layer.

FAQ

Does cron run a job twice during daylight saving time?
Yes, if the cron entry uses a local timezone and the scheduled time falls in the repeated hour — typically 01:00–01:59 on the fall-back date. Times in the skipped hour on the spring-forward date may not run at all. A cron entry in UTC fires exactly once but drifts by an hour in local terms.

Should I store timestamps in UTC or local time?
Store instants in UTC (timestamptz in Postgres) and store the user's IANA timezone name, like Europe/Berlin, in a separate column. Never store a fixed offset such as +01:00 — it is wrong for half the year and cannot be corrected later.

How do I run a job at 9am in each user's local timezone?
Tick every 15 minutes in UTC, resolve 09:00 in each user's IANA zone to a UTC instant for that user's current local date, and process users whose target has passed and who have no row yet for that logical date. Do not precompute a year of trigger timestamps — timezone rules change by government decree, and the tzdata update will invalidate them.

Bottom line

If a duplicate run is harmless, put the schedule in UTC and convert to local time inside the job — two lines of zoneinfo and you are done. If a duplicate run sends an email, charges a card, or posts to a partner API, add the logical-date claim; it costs one table and one insert, and it retires the DST question along with retries, missed triggers, and overlapping runs. Keep the scheduler dumb and the job smart: any platform can fire a UTC tick, but only your code knows what period the work belongs to. Test the March and November windows now, while it is cheap.

Postgres Autovacuum Isn't Keeping Up: Diagnosing Bloat, Long Transactions, and Wraparound Warnings

Libme — Tue, 08 Sep 2026 18:45:17 +0000

If your table keeps growing while the row count stays flat, autovacuum is probably running fine and still removing nothing — because something is holding an old snapshot open. Check pg_stat_activity and pg_replication_slots before you touch a single autovacuum setting. Only after you've ruled out blockers does tuning thresholds and cost limits make any difference.

I lost most of a day to this once. A table with a steady ~2 million rows had grown well past what its data should occupy, sequential scans were creeping, and pg_stat_user_tables showed last_autovacuum updating every few minutes. Autovacuum was doing its job on schedule and accomplishing nothing, because a reporting connection had been sitting idle in transaction since the previous deploy.

Why "autovacuum ran" and "dead rows were removed" are different things

Postgres uses MVCC: an UPDATE writes a new row version and leaves the old one in place, and a DELETE just marks the old one dead. Vacuum reclaims those dead versions — but only the ones no live snapshot could still need. The cutoff is the oldest transaction anywhere in the system that might still look backwards.

That means vacuum's effectiveness is capped by the oldest snapshot on the instance, not by how often it runs. A vacuum that runs every minute against a table protected by a two-hour-old transaction will remove nothing for two hours, and it will report success every time.

Since Postgres 16, VACUUM VERBOSE and the autovacuum log line tell you this directly — the output includes how many dead tuples were left behind because they weren't yet removable. That line is the fastest honest signal you have.

Takeaway: autovacuum frequency is irrelevant if the removable cutoff isn't advancing.

How do I tell if autovacuum is falling behind?

Start with the tables themselves:

SELECT relname,
       n_live_tup,
       n_dead_tup,
       round(n_dead_tup::numeric / NULLIF(n_live_tup, 0), 3) AS dead_ratio,
       last_autovacuum,
       autovacuum_count
FROM pg_stat_user_tables
WHERE n_dead_tup > 10000
ORDER BY n_dead_tup DESC
LIMIT 20;

A dead ratio that climbs across repeated samples is the actual symptom. A high ratio that holds steady on a write-heavy table is often normal.

Then check whether anything is pinning the cutoff:

-- Oldest transactions and the snapshot they hold
SELECT pid, state, backend_xmin,
       now() - xact_start AS xact_age,
       left(query, 60) AS query
FROM pg_stat_activity
WHERE backend_xmin IS NOT NULL
ORDER BY xact_age DESC NULLS LAST
LIMIT 10;

-- Replication slots pinning old rows (including forgotten ones)
SELECT slot_name, active, xmin, catalog_xmin FROM pg_replication_slots;

-- Prepared (two-phase) transactions nobody committed
SELECT gid, prepared, owner FROM pg_prepared_xacts;

An inactive replication slot is the sneakiest of the three: it holds the cutoff indefinitely, produces no query to blame in pg_stat_activity, and also grows WAL until the disk fills. A slot left behind by a decommissioned read replica or an abandoned CDC pipeline will quietly do both.

Takeaway: three queries — activity, slots, prepared transactions — explain the large majority of "vacuum runs but bloat grows" cases.

What actually blocks dead tuple removal?

Blocker	How you spot it	Fix
Long-running query	`xact_age` large, `state = 'active'`	Optimize or cap it; set `statement_timeout`
`idle in transaction`	`state = 'idle in transaction'`	`idle_in_transaction_session_timeout`; fix the client's commit path
Inactive replication slot	`pg_replication_slots.active = false`	Drop the slot if the consumer is gone
`hot_standby_feedback = on` on a replica	Replica running long reports	Trade-off: query cancels vs. primary bloat
Prepared transaction	Row in `pg_prepared_xacts`	`ROLLBACK PREPARED`; audit the XA client
Autovacuum genuinely too slow	Blockers clean, `n_dead_tup` still climbing	Tune thresholds and cost limits (below)

Only the last row is a tuning problem. The rest are application or topology problems that tuning will not fix.

Which autovacuum settings are worth changing?

The defaults are deliberately conservative and scale-based, and that's exactly where large tables lose. With autovacuum_vacuum_scale_factor at 0.2, a 50-million-row table waits for roughly 10 million dead rows before autovacuum even considers it. That threshold is fine for a 10,000-row table and absurd for a large one.

Set it per table rather than globally:

ALTER TABLE events SET (
  autovacuum_vacuum_scale_factor = 0.01,
  autovacuum_vacuum_threshold    = 5000,
  autovacuum_analyze_scale_factor = 0.02
);

If autovacuum starts often enough but never finishes before more garbage arrives, the constraint is throughput, not eligibility. Autovacuum sleeps according to autovacuum_vacuum_cost_delay (2ms by default since Postgres 12) once it has burned through its cost budget. On modern NVMe storage that pacing is usually far more cautious than the hardware needs:

ALTER SYSTEM SET autovacuum_vacuum_cost_delay = '1ms';
ALTER SYSTEM SET autovacuum_vacuum_cost_limit = 1000;
ALTER SYSTEM SET autovacuum_max_workers = 5;  -- requires restart
SELECT pg_reload_conf();

Raise the cost limit gradually and watch I/O. Autovacuum competing with peak traffic for the same disk is a real failure mode, just a less common one than autovacuum being throttled into irrelevance.

Two more worth knowing: maintenance_work_mem bounds how much dead-tuple state a vacuum can hold, and running out of it forces extra index scan passes over the same table. Postgres 17 reworked that storage so the old 1 GB effective ceiling no longer applies, which is a genuine reason to prioritize that upgrade on bloat-prone databases. And autovacuum_vacuum_insert_scale_factor (Postgres 13+) covers insert-only tables, which otherwise accumulate unfrozen pages and never get vacuumed on the dead-tuple path at all.

Takeaway: per-table thresholds for eligibility, cost delay and limit for throughput — they fix different failures and are not interchangeable.

What do I do about "database must be vacuumed within N transactions"?

This warning means transaction ID age is approaching the wraparound limit. If it keeps climbing, Postgres eventually refuses writes with database is not accepting commands to avoid wraparound data loss, and recovery requires single-user mode. Treat the warning as a page, not a log line.

Find the offenders by age:

SELECT c.relname,
       age(c.relfrozenxid) AS xid_age,
       pg_size_pretty(pg_total_relation_size(c.oid)) AS size
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind IN ('r', 'm')
  AND n.nspname NOT IN ('pg_catalog', 'information_schema')
ORDER BY xid_age DESC
LIMIT 10;

Compare against autovacuum_freeze_max_age (200 million by default). Then do the same blocker hunt — an anti-wraparound autovacuum is subject to the same cutoff as any other vacuum, so an open transaction stalls the thing keeping you online. Anti-wraparound vacuums also refuse to yield to lock requests the way ordinary autovacuum does, so a VACUUM (FREEZE) you kick off manually during a quiet window is often the calmer path than letting one start during peak traffic.

Takeaway: wraparound warnings are a deadline, and the fix is almost always removing the snapshot holder, not vacuuming harder.

The bloat is already there — now what?

Vacuum makes space reusable inside the table; it rarely returns it to the filesystem. To actually shrink files you need a rewrite.

VACUUM FULL rewrites the table compactly but takes an ACCESS EXCLUSIVE lock for the entire operation, so every reader and writer blocks until it finishes. It's the right call for a table you can take offline and the wrong call for anything on the request path.

If you need the rewrite without the outage, pg_repack does it online by maintaining a shadow copy and swapping at the end, requiring only a brief exclusive lock — at the cost of needing roughly double the table's disk space during the operation and a superuser-installed extension, which some managed providers don't offer. For measuring bloat honestly rather than estimating it from statistics, the pgstattuple extension scans the table and reports real free space, and that full scan is heavy enough that you should run it off-peak.

FAQ

How do I know if autovacuum is running right now?
Query pg_stat_progress_vacuum, which shows each active vacuum's phase, heap blocks scanned, and index vacuum count. If a vacuum has a high index_vacuum_count, it's making multiple index passes and needs more maintenance_work_mem.

Does VACUUM lock the table?
Ordinary VACUUM and autovacuum take a SHARE UPDATE EXCLUSIVE lock, so reads and writes continue normally; they only conflict with schema changes and other vacuums. VACUUM FULL is different — it takes ACCESS EXCLUSIVE and blocks everything for the duration.

Why is n_dead_tup high right after a vacuum finished?
Either the dead rows weren't removable yet because an older snapshot still exists, or the counter is simply stale — pg_stat_user_tables values are estimates updated by the stats collector. Check the vacuum's own log output for how many tuples it reported as not yet removable.

Bottom line

Diagnose before you tune: look at pg_stat_activity, pg_replication_slots, and pg_prepared_xacts first, because a held snapshot makes every autovacuum setting irrelevant. If those are clean and dead tuples still accumulate, lower the scale factor per table for eligibility and lower the cost delay for throughput. Treat wraparound warnings as an incident with a deadline. And if you're already carrying bloat you can't afford an outage to remove, pg_repack is the standard answer — assuming your provider lets you install it.

UUIDv7 vs ULID vs bigint: Which Primary Key Holds Up When the Table Gets Big?

Libme — Mon, 07 Sep 2026 23:45:49 +0000

If inserts into a Postgres table got slower as it grew and the primary key is a random UUID (v4), the key is a real suspect: random keys scatter B-tree writes across the whole index instead of concentrating them at one end. UUIDv7 and ULID fix that by putting a millisecond timestamp in the high bits, so new rows land next to each other. bigint identity is still the smallest and fastest option — it just leaks row counts and makes multi-writer ID generation someone else's problem.

This is the part of schema design that's cheap to get right on day one and genuinely expensive to change at row 400 million.

Why do random UUIDs slow down inserts as the table grows?

The symptom is unglamorous: insert latency that used to be flat starts creeping up, correlating with table size rather than traffic. Nothing shows up as a slow query, and EXPLAIN on the insert looks fine.

The mechanism is index locality. A B-tree index on a random value means each new row's key sorts into an essentially random leaf page. Once the index is bigger than the memory Postgres can keep it in, most inserts touch a leaf page that isn't in shared_buffers, so the write turns into a read first. That read is the actual cost.

There's a second-order effect that surprises people more. With full_page_writes on (the default), the first modification of a page after a checkpoint writes the entire page into the WAL, not just the row. Random inserts touch many distinct pages per checkpoint interval; sequential inserts keep hammering the same few. Same rows, meaningfully more WAL — which surfaces as replication lag and backup size, not query latency, so it gets diagnosed as a storage problem.

Random keys also age the index badly. Sequential inserts fill leaf pages to the fillfactor and move on. Random inserts land in already-full pages, splitting them, and the halves stay half-empty. The index ends up physically larger than its contents justify.

A random primary key doesn't make any single query slow; it makes the whole index stop fitting in cache sooner, which is much harder to spot.

How do you confirm it's the key and not something else?

Don't take my word for it — the diagnosis is three queries against your own database.

Start with how much of the index actually fits in memory, and how much dead space it's carrying:

-- index size vs table size
SELECT
  relname,
  pg_size_pretty(pg_relation_size(indexrelid))   AS index_size,
  idx_scan
FROM pg_stat_user_indexes
JOIN pg_class ON pg_class.oid = indexrelid
WHERE relname LIKE '%_pkey'
ORDER BY pg_relation_size(indexrelid) DESC
LIMIT 10;

-- density of the primary key index (needs the pgstattuple extension)
CREATE EXTENSION IF NOT EXISTS pgstattuple;
SELECT index_size, leaf_pages, avg_leaf_density, leaf_fragmentation
FROM pgstatindex('events_pkey');

avg_leaf_density well below ~85% on an append-only table is the fingerprint of random-key page splits; a freshly built index on sequential keys sits near the B-tree fillfactor of 90%.

Then check whether the buffer cache is losing:

SELECT
  sum(heap_blks_read)  AS heap_read,
  sum(heap_blks_hit)   AS heap_hit,
  round(100.0 * sum(idx_blks_hit) /
        nullif(sum(idx_blks_hit) + sum(idx_blks_read), 0), 2) AS idx_hit_pct
FROM pg_statio_user_tables;

If index hit percentage on a write-heavy table is drifting down over weeks while the working set hasn't changed shape, you're watching an index outgrow RAM.

Measure avg_leaf_density and index cache hit rate on your own table before rewriting a schema — key choice is worth changing when those two numbers say so, not on principle.

UUIDv7 vs ULID vs bigint: what actually differs?

	`bigint` identity	UUIDv4	UUIDv7	ULID
Storage in Postgres	8 bytes	16 bytes (`uuid`)	16 bytes (`uuid`)	16 bytes as `uuid`, ~27 as `text`
Insert locality	Sequential	Random	Sequential (ms granularity)	Sequential (ms granularity)
Generated by client?	No (needs the DB)	Yes	Yes	Yes
Sortable by creation time	Yes	No	Yes	Yes, also as a string
Standardized	SQL standard	RFC 9562	RFC 9562	Community spec, no RFC
Native Postgres type	Yes	Yes	Yes (`uuid`)	No
Leaks	Row count, insert order	Nothing	Creation time (ms)	Creation time (ms)

Two rows in that table decide most arguments.

The storage row is why bigint still wins on pure efficiency: 8 bytes versus 16 sounds trivial until you count every foreign key, every composite index that includes the key, and every index tuple's share of a page. Doubling key width on a schema with several FK columns per table is a real, permanent tax on how much of your database fits in cache.

The "generated by client" row is why people pay that tax anyway. If the ID has to exist before the row does — because the client creates it offline, because you're writing to several shards, because you want to build an object graph in memory and insert it in one round trip — then a database sequence is the wrong tool, and you're choosing among the UUID-shaped options.

Pick bigint unless something in your architecture genuinely needs an ID before the insert; if it does, the choice narrows to UUIDv7 in almost every case.

Is ULID worth giving up the native `uuid` type?

ULID's distinguishing feature is its canonical text form: 26 characters of Crockford base32 that sort lexicographically in the same order as the underlying bytes. If IDs travel through systems that only handle strings — log lines, sorted keys in an object store, a URL path you want to eyeball chronologically — that's a genuine convenience UUID hex doesn't give you.

The cost in Postgres is that there is no ULID type. Store it as text and you pay ~27 bytes plus varlena overhead per index entry, and collation-aware comparisons instead of a fixed 16-byte memcmp. Store the same 128 bits in a uuid column and convert at the application boundary — the approach I'd take — and you've kept only the string formatting.

ULID also has no RFC behind it. UUIDv7 was standardized in RFC 9562 (2024) and now has first-class support arriving across ecosystems: PostgreSQL 18 ships a built-in uuidv7() function, and Python's standard library gained uuid.uuid7() in 3.14. As of mid-2026, choosing ULID means choosing the option with less platform support for a formatting benefit.

ULID is a reasonable choice when the ID's string form is part of your product surface; if it isn't, UUIDv7 gets you the same insert locality with a native type.

How do you generate UUIDv7 on the Postgres version you actually have?

On PostgreSQL 18 or newer it's built in:

CREATE TABLE events (
  id         uuid PRIMARY KEY DEFAULT uuidv7(),
  payload    jsonb NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now()
);

On 13–17, generate it in the application (the uuid npm package and Python's uuid_utils both do v7), or add this function, which builds a valid v7 by overwriting the first 6 bytes of a v4 with a millisecond timestamp and flipping the version nibble to 0111:

CREATE OR REPLACE FUNCTION uuid_generate_v7() RETURNS uuid AS $$
BEGIN
  RETURN encode(
    set_bit(
      set_bit(
        overlay(
          uuid_send(gen_random_uuid())
          PLACING substring(
            int8send(floor(extract(epoch FROM clock_timestamp()) * 1000)::bigint)
            FROM 3
          )
          FROM 1 FOR 6
        ),
        52, 1
      ),
      53, 1
    ),
    'hex'
  )::uuid;
END
$$ LANGUAGE plpgsql VOLATILE;

gen_random_uuid() has been built in since PostgreSQL 13, so this needs no extensions. Verify it before trusting it — SELECT uuid_generate_v7(); twice a second apart should give you two values whose first hex characters are ascending.

What's the honest downside of sequential keys?

Everything monotonic concentrates writes on the rightmost leaf page of the index, and under high insert concurrency that page becomes a lock hotspot. This affects bigint sequences and UUIDv7 equally. In practice it's a problem at a scale where you're already thinking hard about write throughput, and it's the flip side of the cache benefit — you can't have locality without contention. If you hit it, that's when the random-key layout stops being a bug and starts being the design.

The other real cost is the timestamp: a UUIDv7 or ULID exposes its creation time to millisecond precision to anyone holding it. Fine for an internal events table. For a password reset token, or any ID where creation time is sensitive, use a random v4 — that's the case v4 is for, and it's why "always use v7" is bad advice.

FAQ

Should I migrate an existing UUIDv4 primary key to UUIDv7?
Usually not on its own. Changing a primary key type means rewriting the table and every foreign key referencing it, which is a lock-and-backfill project. Do it when the index-density and cache-hit numbers above show real degradation, or fold it into a migration you were already doing.

Is UUIDv7 slower to generate than UUIDv4?
No meaningfully. Both are dominated by the random-bytes call; v7 replaces 48 bits of randomness with a timestamp read. Generation cost is not the reason to choose between them.

Can I sort by primary key instead of by created_at with UUIDv7?
Within millisecond resolution, yes — UUIDv7 sorts by creation time. Rows created in the same millisecond have no defined order between them, so if you need a strict total order (cursor pagination, for instance), keep an explicit tiebreaker.

Bottom line

If your IDs can come from the database, use bigint identity — it's half the width and there's no clever alternative that beats it. If they can't, default to UUIDv7 stored in a native uuid column, using PostgreSQL 18's uuidv7() or the function above on older versions. Reach for ULID only when the 26-character sortable string is genuinely part of how your system works, and keep UUIDv4 for IDs where leaking a creation timestamp would be a problem. And before you migrate anything, run pgstatindex on the primary key you already have — the number either justifies the work or it doesn't.

curl Returns 200 but Your HTTP Client Gets a 404: Debugging Vary and CDN Cache Variants

Libme — Sun, 06 Sep 2026 03:18:23 +0000

If the same URL returns different responses depending on which client asks, you are almost certainly hitting two different cache entries, not two different origin states. A CDN keys its cache on the URL plus whichever request headers the origin listed in Vary — most often Accept-Encoding. curl sends no Accept-Encoding by default; nearly every HTTP library sends gzip, deflate. That one difference puts them on opposite sides of a cache split, and a stale error response can sit in one variant for hours while the other is perfectly healthy.

I lost most of an afternoon to this. A publishing script called an API endpoint for an article that had just gone live and got a clean 404 every single time. Pasting the identical URL into curl returned 200 with the full payload. Same machine, same network, same second.

Why do two clients get different responses from the same URL?

The cache key. RFC 9110 says a response's Vary header lists the request headers that participate in matching a stored response. So when the origin sends:

Vary: Accept-Encoding, Origin, X-Loggedin

the CDN stops storing "one response for /api/articles/123" and starts storing "one response per distinct combination of those three headers." Those are variants. They have independent TTLs, independent ages, and independent contents.

Now the client difference matters:

# curl sends no Accept-Encoding by default -> the "identity" variant
curl -sI https://example.com/api/articles/123

# --compressed sends Accept-Encoding: gzip -> a different variant
curl -sI --compressed https://example.com/api/articles/123

import requests
# requests/urllib3 send "Accept-Encoding: gzip, deflate" unless told otherwise
r = requests.get("https://example.com/api/articles/123")
print(r.status_code)

In my case, the resource had been requested during the brief window right after publication when the origin still answered 404. That 404 got stored in the gzip variant with a long edge TTL (x-accel-expires: 172800), and every subsequent library call matched it. The identity variant was populated later, after the origin was consistent, and served a fresh 200. Nothing retried its way out of that — the two variants had no idea the other existed.

The takeaway: when one client succeeds and another fails on the same URL, compare the request headers before you touch the origin.

How do you confirm it's a cache variant and not a flaky origin?

Diff the response headers of the two clients. Age, X-Cache, CF-Cache-Status, and Vary will tell you almost immediately.

for enc in "identity" "gzip"; do
  echo "--- Accept-Encoding: $enc"
  curl -sI -H "Accept-Encoding: $enc" https://example.com/api/articles/123 \
    | grep -iE '^(HTTP/|age|vary|x-cache|cf-cache-status|x-accel-expires)'
done

Two signals confirm the diagnosis:

The two runs return different status codes or different Age values. A large Age (mine was over 58,000 seconds) on the failing variant means you're reading something cached long before your current problem started.
Vary names a header that differs between your clients.

If both variants return identical status and near-zero Age, stop here — the problem is at the origin, and you should be reading origin logs instead.

The takeaway: a large Age on the failing request and a small one on the succeeding request is a cache-variant fingerprint, not an origin bug.

What can you actually do when the bad variant belongs to someone else's CDN?

This is the uncomfortable case: it's a third-party API, you can't purge their cache, and support tickets take longer than your deploy. Your only real lever is changing which variant you land on, by controlling the headers you send.

import requests

session = requests.Session()
# Bypass the poisoned gzip variant entirely by matching the identity cache key.
session.headers["Accept-Encoding"] = "identity"

r = session.get("https://example.com/api/articles/123", timeout=10)
r.raise_for_status()

That is what fixed my publishing script: one helper that every outbound call to that API goes through, pinning Accept-Encoding: identity. The cost is real — you give up response compression, so this is a reasonable trade for small JSON payloads and a bad one for large ones. Do it in a single wrapper function, not scattered at each call site, so you can reverse it in one edit later.

Options, honestly compared:

Approach	Works when	Real cost
Change your `Accept-Encoding`	Third-party CDN, `Vary: Accept-Encoding`	Loses compression; still cached, just a different variant
Add a cache-busting query param	You control nothing but the URL	Pollutes their cache, may violate rate limits or ToS
`Cache-Control: no-cache` request header	CDN honors it (many don't for anonymous traffic)	Frequently ignored; unreliable to depend on
Purge by URL	You own the CDN	Often purges only one variant — verify it clears all
Retry with backoff	The origin is genuinely flaky	Useless here; every retry matches the same poisoned key

That last row is the trap. Retries feel like the safe generic fix, and against a cache-key problem they are pure latency with a guaranteed failure at the end.

The takeaway: retries cannot fix a cache-key mismatch, because every retry computes the same key.

How do you stop your own service from creating this problem?

If you operate the origin, the fix is to keep the variant count small and predictable.

Normalize Accept-Encoding at the edge rather than passing raw client values through. Client header values are wildly diverse (gzip;q=1.0, deflate;q=0.8, br, and so on), and if each string becomes its own cache key you get a low hit ratio plus many more chances to store a bad response somewhere. Varnish's default VCL normalizes this for exactly that reason, and Fastly documents the same pattern:

sub vcl_recv {
    if (req.http.Accept-Encoding) {
        if (req.http.Accept-Encoding ~ "br") {
            set req.http.Accept-Encoding = "br";
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } else {
            unset req.http.Accept-Encoding;
        }
    }
}

Three more rules that have saved me repeatedly:

Never let an error response inherit a success TTL. A 404 or 5xx cached for two days is the actual damage here; the variant split only decided who saw it. Set short negative-caching TTLs explicitly.
Never Vary on high-cardinality headers. Vary: User-Agent multiplies your cache by thousands of variants and effectively disables it. Vary: Cookie on an authenticated endpoint is worse — one wrong key and you serve someone else's data.
Vary: Origin needs care with CORS. If you echo the request Origin into Access-Control-Allow-Origin without varying on it, a cached response hands the wrong origin's CORS header to the next caller. Varying on it is correct, but it is another cache split.

If you want a managed edge where this behavior is configurable rather than emergent, both Fastly (through VCL) and AWS CloudFront (through cache policies that let you specify exactly which headers enter the key) give you explicit control of the cache key instead of leaving it to whatever Vary your framework happens to emit. As of mid-2026, check your provider's current documentation before assuming a specific normalization default — this is exactly the kind of behavior that changes between platform versions.

The takeaway: your cache key should be something you designed, not a side effect of your framework's default headers.

FAQ

Why does curl work but Python requests return 404?
Because curl sends no Accept-Encoding header by default while requests sends gzip, deflate, and if the response carries Vary: Accept-Encoding those two requests hit different CDN cache entries. Reproduce it with curl --compressed — if that also fails, you've confirmed the cache variant is the cause rather than anything client-specific.

Does the Vary header affect the CDN cache key?
Yes. Every header listed in Vary becomes part of the cache key, so the CDN stores a separate copy of the response for each distinct combination of those header values. This is why purging "the URL" sometimes fails to clear the copy your client is actually receiving.

How do I force a request to bypass a bad cached variant?
Change a header that appears in the response's Vary list — most commonly by sending Accept-Encoding: identity — so your request maps to a different cache key. You cannot rely on Cache-Control: no-cache in the request, because many CDNs ignore it for anonymous traffic.

Bottom line

If two clients disagree about the same URL, diff their request headers and their Age values before you suspect the origin. When the poisoned variant lives on a CDN you don't control, your only reliable move is to change the headers that make up the cache key — pin Accept-Encoding in one shared HTTP helper and accept the loss of compression for small payloads. When you own the origin, normalize Accept-Encoding at the edge, keep Vary to low-cardinality headers, and give error responses their own short TTL. Retries are the wrong instinct here; they recompute an identical key and fail identically.

Your Logging Bill Is a Volume Problem: Model the Cost Before You Switch Vendors

Libme — Fri, 04 Sep 2026 23:01:09 +0000

If your log bill jumped and your traffic didn't, the vendor is usually not the problem — the shape of your log volume is. Every managed platform charges on some combination of bytes ingested, what gets indexed for search, and how long it is retained, and most teams push all three knobs to maximum for every line their app emits. Measure bytes per service per level first, cut the noise, and only then compare prices; switching vendors without doing that just moves the same volume to a different invoice.

Why does the log bill grow faster than traffic?

Log volume scales with code paths, not users. A new service, a debug line someone left in a hot loop, a retry wrapper that logs every attempt, a Kubernetes liveness probe hitting /healthz every five seconds across a dozen replicas — none of that shows up in your request graph, and all of it shows up on the invoice.

The other multiplier is structure. Structured JSON logging is the right call, but a log line that carries the full request headers, a serialized user object, and a 4 KB stack trace costs roughly ten times what the equivalent message-plus-fields line costs. You are billed on bytes, and nobody notices bytes until the finance channel does.

The takeaway: log spend correlates with the number of emit sites in your codebase, which grows monotonically and never gets reviewed.

What are you actually paying for?

Pricing pages hide behind different units, so compare models, not sticker numbers. As of September 2026, these are the structural differences that matter — verify current rates on each vendor's pricing page before you build a spreadsheet.

Platform	Primary billing unit	Where the surprise usually hides
Datadog Logs	Ingested GB, then separately indexed events by retention period	Ingest is cheap relative to indexing; teams index everything by default
AWS CloudWatch Logs	GB ingested + GB stored + GB scanned by Logs Insights queries	Query cost is per-scan, so debugging a bad week costs money every time
Grafana Cloud Logs (Loki)	Ingested GB + active series/labels	High-cardinality labels (user ID, request ID) blow up the index
Elastic Cloud	Provisioned cluster resources	You pay for capacity whether or not you fill it; retention is your storage problem
Axiom	Ingested GB with query included	Fewer knobs, less control when you want tiered retention
Self-hosted Loki	Object storage + compute you run	Your on-call rotation now owns the logging system

The distinction that saves the most money is ingest vs. index. Datadog's model exists precisely because most logs are written once and read never: you can send everything, index only the subset you actually query, and rehydrate the rest when you need it. If your team is on Datadog and hasn't configured exclusion filters, that is the single highest-leverage afternoon of work available to you.

The takeaway: you are not buying "logs," you are buying ingestion, searchability, and retention as three separately priced things.

How do I measure my log volume before I shop?

Do not start from the vendor dashboard — start from the bytes your app emits, grouped by the dimensions you can actually act on: service and level.

If you have a day of JSON logs on disk:

# Bytes and line counts grouped by service + level.
# Re-serialization makes this an approximation, but the ranking is what matters.
jq -rc '[(.service // "unknown"), (.level // "unknown"), (tostring | length)] | @tsv' app.jsonl \
  | awk -F'\t' '{b[$1"\t"$2] += $3; n[$1"\t"$2]++}
                END {for (k in b) printf "%.1f\t%d\t%s\n", b[k]/1048576, n[k], k}' \
  | sort -rn \
  | head -20

Output columns are MB, line count, service, level. In every audit I have run, the top three rows are the whole conversation — usually one chatty service, one debug level that was never turned off after an incident, and health checks.

On AWS, the equivalent first look is stored bytes per log group:

aws logs describe-log-groups \
  --query 'logGroups[].[logGroupName,storedBytes,retentionInDays]' \
  --output text \
  | sort -k2 -rn \
  | head -20

Watch for None in the retention column — that is "never expires," which is the CloudWatch default and quietly bills you forever for logs from a service you deleted last year.

The takeaway: if you cannot name your top three log sources by bytes, you are not ready to compare vendors.

What should you cut first?

In rough order of savings per hour of effort:

Drop health checks and readiness probes at the logger, not the vendor. They are pure volume with zero diagnostic value.
Turn off debug in production. Obvious, and still the most common finding.
Sample successful requests; keep every error. This is where the real reduction lives.
Stop logging objects you already have elsewhere. Full request headers and serialized ORM entities belong in a trace, not a log line.
Set retention per source. Access logs rarely need the same window as payment events.

Sampling is the one that scares people, because random sampling shreds traces — you get request 3 of a five-line story. Sample deterministically on the trace or request ID so a sampled request keeps all of its lines:

// sampling.js
import { createHash } from 'node:crypto'

// Deterministic: the same key always lands the same way.
export function sampled(key, rate) {
  if (rate >= 1) return true
  if (rate <= 0) return false
  const h = createHash('sha1').update(String(key)).digest()
  return h.readUInt32BE(0) / 0xffffffff < rate
}

Wired into an Express access log, with errors and slow requests exempt:

import express from 'express'
import pino from 'pino'
import { randomUUID } from 'node:crypto'
import { sampled } from './sampling.js'

const log = pino()
const app = express()

app.use((req, res, next) => {
  const start = process.hrtime.bigint()
  const requestId = req.headers['x-request-id'] ?? randomUUID()

  res.on('finish', () => {
    const ms = Number(process.hrtime.bigint() - start) / 1e6
    const interesting = res.statusCode >= 500 || ms > 1000
    const rate = interesting ? 1 : 0.05

    if (req.path === '/healthz') return
    if (!sampled(requestId, rate)) return

    log.info({ requestId, method: req.method, path: req.route?.path ?? req.path,
               status: res.statusCode, ms: Math.round(ms) }, 'request')
  })

  next()
})

Note req.route?.path rather than req.path: logging the templated route (/users/:id) instead of the raw URL keeps cardinality bounded, which matters a lot on label-indexed backends like Loki.

The takeaway: deterministic sampling on request ID cuts volume without ever handing you half a trace during an incident.

When does self-hosting actually cost less?

Self-hosted Loki backed by S3 or R2 is genuinely cheap on storage, because it indexes only labels and treats log bodies as compressed chunks in object storage. If you are running Kubernetes already, have someone who is comfortable owning a stateful service, and your pain is "we generate a lot of low-value logs we still want searchable," Grafana Loki is the option that makes that volume affordable rather than making you delete it.

The honest cost is not the infrastructure. It is that your logging system becomes a thing that can page you, and it tends to page you during exactly the incidents when you need it. Loki's query performance also degrades badly if your label scheme is wrong, and fixing a label scheme after the fact means reindexing your habits, not just your config.

For a team under roughly ten engineers with no dedicated platform person, a managed platform is almost always cheaper once you price your own time honestly. Better Stack sits reasonably in that gap if you want managed logs without Datadog's per-feature complexity, though its ecosystem of integrations is thinner than the incumbents'. If you are already deep in AWS and mostly need logs for post-incident forensics rather than daily querying, CloudWatch Logs with aggressive per-group retention is the boring answer that stops the bleeding without a migration.

Situation	Reasonable choice
Small team, AWS-native, logs read rarely	CloudWatch with per-group retention set
Want managed, dislike per-feature pricing complexity	Better Stack or Axiom
Already run Grafana + Kubernetes, high volume	Self-hosted or Grafana Cloud Loki
Need deep correlation with metrics/APM, have budget	Datadog with exclusion filters configured on day one

The takeaway: self-hosting trades a predictable invoice for an unpredictable on-call surface — take that trade only if you already run stateful services well.

FAQ

How do I reduce Datadog log ingestion costs?
Configure exclusion filters so high-volume, low-value logs are ingested but not indexed, and sample access logs at the application before they leave the host. Indexing, not ingestion, is usually the larger line item, so cutting what gets indexed changes the bill fastest.

Is CloudWatch Logs cheaper than Datadog?
For storage and ingestion of logs you rarely query, usually yes; for logs you query daily, CloudWatch's per-GB-scanned Logs Insights charges can erase the difference. Compare on your actual query frequency, not just ingested volume.

Does log sampling break debugging?
Not if you sample deterministically on the request or trace ID and exempt errors and slow requests, because every retained request keeps its complete set of lines. Random per-line sampling does break debugging, which is why it has a bad reputation.

Bottom line

Audit before you migrate: one jq pass over a day of logs tells you more than any vendor comparison. If you are on Datadog, exclusion filters and deterministic sampling will cut the bill more than switching would. If you are AWS-native and mostly write logs you never read, set CloudWatch retention per log group today. Self-host Loki only if you already operate stateful infrastructure and your volume is genuinely large — otherwise you are trading a line item for an on-call burden.

Every Deploy Throws a Few 502s: Where Graceful Shutdown Actually Breaks

Libme — Wed, 02 Sep 2026 21:40:41 +0000

If your error tracker shows a tight cluster of 502 Bad Gateway or ECONNRESET errors that starts the second a deploy begins and stops about ten seconds later, your app is almost certainly being killed while it still holds in-flight requests. The fix is not one setting. Four separate things have to be true — the process has to receive SIGTERM, handle it, stop accepting new connections while draining old ones, and stay alive long enough for the load balancer to stop routing to it — and most teams have exactly one or two of those in place.

I have debugged this on Kubernetes, on ECS, and on a plain Docker host, and the symptom is identical every time: a burst of errors that is too small to page anyone and too regular to be a coincidence. Here is how to find which of the four links is broken.

Why do deploys produce 502s at all?

Two clocks are running during a rollout, and nothing synchronizes them.

Clock one is the orchestrator killing your container. Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds (30 by default), then sends SIGKILL. docker stop sends SIGTERM and waits 10 seconds before SIGKILL.

Clock two is the routing layer forgetting about your pod. In Kubernetes, removing the pod from the Service endpoints propagates asynchronously to every kube-proxy and every ingress controller. On ECS behind an ALB, the task is deregistered from the target group and then the deregistration delay runs down.

These two happen concurrently, not in sequence. The orchestrator does not wait for the routing layer to catch up before it starts killing your process. So for some window — usually a fraction of a second to a couple of seconds — traffic is still being sent to a container that has already been told to die. If that container exits immediately, every request in that window becomes a 502.

The takeaway: a 502 during deploy is not a crash, it is a race between the kill signal and the routing update.

Is your process even receiving SIGTERM?

Before touching shutdown logic, confirm the signal arrives. The most common reason it does not is the container's PID 1.

# Shell form: your process runs as a child of /bin/sh, which does NOT forward SIGTERM.
CMD npm start

# Exec form: your process IS PID 1 and receives the signal.
CMD ["node", "server.js"]

The shell form runs /bin/sh -c "npm start". The shell becomes PID 1, receives SIGTERM, and does nothing with it — your Node process never hears about it and dies 10 or 30 seconds later by SIGKILL. Wrapper commands are the other frequent culprit: anything that spawns your server as a child process has to explicitly forward signals, and process-manager wrappers are a common place for them to get swallowed.

Verify it in about fifteen seconds, on any machine:

docker run -d --name shutdowntest myimage:latest
docker stop shutdowntest
docker inspect shutdowntest --format '{{.State.ExitCode}} {{.State.OOMKilled}}'

Exit code 0 means you handled the signal and exited cleanly. 143 is 128 + 15, meaning SIGTERM was delivered and the default handler killed you — the signal arrived but nothing handled it. 137 is 128 + 9: you were SIGKILLed after the grace period expired, which means either you never got the signal or your shutdown handler hung.

If you need a real init process — because your app legitimately spawns children, or you inherited a shell-form entrypoint you cannot change — Docker's built-in init flag runs tini as PID 1 and forwards signals to your process correctly. The drawback is that it is a runtime flag, so it only helps where you control the run command; on Kubernetes you add tini or dumb-init to the image instead.

The takeaway: exit code 143 or 137 after a docker stop means your shutdown handler is either missing or hanging, and no amount of load balancer tuning will fix that.

What a correct Node shutdown handler looks like

Calling server.close() is necessary but famously not sufficient. It stops accepting new connections and waits for active ones to finish — but an idle keep-alive connection counts as active, so on a busy service the callback may never fire. Node 18.2 added the two methods that close that gap:

const http = require('http');

const server = http.createServer(app);
server.listen(3000);

let shuttingDown = false;

// The readiness endpoint flips first, before anything stops working.
app.get('/readyz', (req, res) => {
  res.status(shuttingDown ? 503 : 200).send(shuttingDown ? 'draining' : 'ok');
});

function shutdown(signal) {
  if (shuttingDown) return;
  shuttingDown = true;
  console.log(`${signal} received, draining`);

  // Give the routing layer time to notice /readyz is failing.
  setTimeout(() => {
    server.close(async (err) => {
      if (err) console.error('close error', err);
      await pool.end();       // drain the DB pool
      process.exit(err ? 1 : 0);
    });
    // Idle keep-alive sockets would otherwise hold close() open forever.
    server.closeIdleConnections();
  }, 5000);

  // Hard ceiling, comfortably inside terminationGracePeriodSeconds.
  setTimeout(() => {
    console.error('drain timed out, forcing exit');
    server.closeAllConnections();
    process.exit(1);
  }, 20000).unref();
}

['SIGTERM', 'SIGINT'].forEach((sig) => process.on(sig, () => shutdown(sig)));

Three details matter more than the rest. The 5-second delay before server.close() is the whole point — you keep serving normally while the load balancer notices you are unready. The forced-exit timer must be shorter than the orchestrator's grace period, or you get SIGKILLed mid-drain and lose the requests you were trying to protect. And attaching a SIGTERM listener removes Node's default exit behavior, so if your handler has a bug the process now hangs until it is killed — you have made things worse, not better.

If you would rather not hand-roll this, the stoppable and http-terminator packages wrap the same connection-draining logic behind one call. The tradeoff is a production dependency in the shutdown path, which is the last place you want a surprise.

The takeaway: sleep first, then close — a shutdown handler that starts closing sockets the instant SIGTERM lands is just a faster way to drop requests.

Where the delay belongs on each platform

The "wait before you stop serving" step can live in the app or in the platform. Pick one place and be explicit about it.

Platform	What removes you from routing	Where to put the drain delay	Grace period knob
Kubernetes	Endpoint removal, propagated async to kube-proxy / ingress	`preStop` sleep hook, or in-app delay after `SIGTERM`	`terminationGracePeriodSeconds` (default 30)
ECS + ALB	Target deregistration, then deregistration delay	Rely on deregistration delay; keep serving until then	`stopTimeout` (task def), ALB deregistration delay
Plain Docker / Compose	Nothing — you are the routing layer	In-app delay	`docker stop -t`, `STOPSIGNAL`
Managed PaaS	Provider-controlled	In-app delay only	Usually fixed, check provider docs

On Kubernetes the preStop hook is the version most teams should start with, because it works for any language without touching application code:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]
terminationGracePeriodSeconds: 30

preStop runs before SIGTERM is sent, and the grace period clock only starts after it finishes — so your app keeps serving normally for those 5 seconds while endpoint removal propagates. Its real drawback is that sleep must exist in the image, which it does not in distroless or scratch builds; there you either use the built-in sleep action available in recent Kubernetes versions or move the delay into the app.

One rule that is easy to miss: readiness and liveness probes must behave differently during shutdown. Readiness should fail immediately so traffic stops. Liveness must keep passing, or the kubelet restarts a container that was in the middle of a clean exit.

The takeaway: put the drain delay in exactly one layer, and make sure your grace period is longer than your in-app forced-exit timer.

How do you prove it is fixed?

Do not trust a quiet dashboard — deploy under load and watch. Run a steady, low-rate request stream against the service and trigger a rollout:

# Any constant-rate client works; the point is a request every 100ms during the rollout.
while true; do
  curl -s -o /dev/null -w "%{http_code}\n" https://your-service/healthz
  sleep 0.1
done | sort | uniq -c

Roll the deployment in another terminal. A correct setup shows 200 for every line. If you see a handful of 502s, note how many: at 10 requests per second, six 502s means roughly a 600ms hole, which points at routing propagation rather than a missing handler. Dozens of them across the whole rollout points at the process dying instantly, and you should go back to the docker stop exit-code check.

The takeaway: the only convincing test for graceful shutdown is a real rollout under continuous traffic, because the bug only exists inside a window that idle traffic never hits.

FAQ

Why does my Kubernetes pod get 502 errors during rolling updates?
Because endpoint removal and SIGTERM happen concurrently, so traffic keeps arriving for a short window after your container starts shutting down. Add a preStop sleep of a few seconds so the pod keeps serving normally while the routing update propagates.

What is exit code 143 in Docker?
143 is 128 + 15, meaning the process was terminated by SIGTERM and used the default handler instead of shutting down cleanly. It confirms the signal was delivered — the problem is in your application, not in the container runtime.

Does server.close() in Node wait for in-flight requests?
Yes, but it also waits for idle keep-alive connections, so on a busy server the callback can be delayed indefinitely. Call server.closeIdleConnections() right after server.close() (available from Node 18.2) and keep a forced-exit timer as a backstop.

Bottom line

If you only do one thing, check the exit code after docker stop — that single number tells you whether you have a signal problem or a draining problem, and the two have completely different fixes. On Kubernetes, add a preStop sleep and a readiness endpoint that fails immediately on SIGTERM; that combination fixes the majority of deploy-time 502s without any application rewrite. Write the in-app handler when you have real cleanup to do — draining a database pool, finishing a queue job — and always give it a forced-exit timer shorter than the platform grace period. Then prove it with a rollout under constant traffic, because this is a bug that only exists during a window you will never hit by hand.

Postgres as a Job Queue vs Redis vs SQS: When Does "Just Use Your Database" Stop Working?

Libme — Wed, 02 Sep 2026 04:14:58 +0000

A Postgres table with SELECT ... FOR UPDATE SKIP LOCKED is a correct, durable job queue, and for most teams it stays correct well past the throughput they actually have. You move off it for two reasons only: sustained enqueue rates high enough that queue churn hurts your primary database, or a need for fan-out to multiple independent consumers. "It feels wrong to use the database" is not one of the reasons.

I've run all three — a Postgres queue for a scheduling app, Redis Streams for an ingestion pipeline, SQS for anything crossing an account boundary. What follows is the failure mode each one actually produced, because that's the part the "5 ways to build a queue" posts skip.

Why does a naive Postgres queue lose or double-process jobs?

The broken version everyone writes first looks like this:

-- DON'T: two workers can read the same row
SELECT id, payload FROM jobs WHERE status = 'queued' ORDER BY run_at LIMIT 1;
UPDATE jobs SET status = 'running' WHERE id = $1;

Between the SELECT and the UPDATE, another worker runs the same SELECT. Both get the job. If you "fix" it by wrapping the select in FOR UPDATE without SKIP LOCKED, you get the opposite symptom: every worker blocks on the same hot row and your throughput collapses to one job at a time, which shows up in pg_stat_activity as a pile of workers in Lock wait events.

The correct claim is a single statement:

UPDATE jobs
SET status = 'running',
    locked_at = now(),
    attempts = attempts + 1
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'queued' AND run_at <= now()
  ORDER BY run_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING id, payload, attempts;

SKIP LOCKED (Postgres 9.5 and later) tells the subselect to step over rows another transaction already holds instead of waiting for them. Each worker gets a different job, atomically, with no advisory-lock bookkeeping.

That handles concurrency. It does not handle a worker that gets OOM-killed halfway through a job — the row sits in running forever. You need a reaper:

UPDATE jobs
SET status = 'queued', locked_at = NULL
WHERE status = 'running'
  AND locked_at < now() - interval '10 minutes'
  AND attempts < 5;

That interval is a contract: your jobs must either finish inside it or heartbeat locked_at while they run. Pick it deliberately rather than copying ten minutes from a blog post.

The other thing that bites is dead tuples. A queue table is the highest-churn table you own — every job is an insert, one or more updates, and eventually a delete. Autovacuum's default scale factor is proportional to table size, which is wrong for a table that turns over completely every hour. Set it per-table:

ALTER TABLE jobs SET (
  autovacuum_vacuum_scale_factor = 0.0,
  autovacuum_vacuum_threshold = 1000,
  autovacuum_analyze_scale_factor = 0.0,
  autovacuum_analyze_threshold = 1000
);

Without this, index bloat makes your "instant" dequeue query slowly get slower over weeks, and the symptom — a queue that was fine in month one and mysteriously laggy in month three — looks nothing like its cause.

A Postgres queue's real operational cost is not the dequeue query; it's autovacuum tuning on a high-churn table.

When is Redis actually the better queue?

Redis is the right answer when you need low-latency dequeue at high rates and you can tolerate the durability model. The trap is which Redis primitive you pick.

BRPOP is at-most-once. The moment Redis hands the item to your worker, it's gone from the server. Worker crashes, job is gone, no trace. This is fine for cache warming and catastrophic for payment webhooks, and the API gives you no hint about which situation you're in.

The reliable patterns are BLMOVE (Redis 6.2+, replacing the deprecated BRPOPLPUSH) into a per-worker processing list, or Redis Streams with consumer groups:

import redis

r = redis.Redis(decode_responses=True)

# Create the group once; MKSTREAM makes the stream if it doesn't exist yet.
try:
    r.xgroup_create("jobs", "workers", id="0", mkstream=True)
except redis.ResponseError as e:
    if "BUSYGROUP" not in str(e):
        raise

while True:
    resp = r.xreadgroup("workers", "worker-1", {"jobs": ">"}, count=1, block=5000)
    if not resp:
        continue
    _, entries = resp[0]
    for entry_id, fields in entries:
        try:
            handle(fields)
            r.xack("jobs", "workers", entry_id)   # only now is it done
        except Exception:
            pass  # stays pending; XAUTOCLAIM will hand it to another worker

The unacknowledged entry stays in the group's pending list, and XAUTOCLAIM reassigns it after an idle threshold. That's the same reaper concept as the Postgres locked_at sweep, just built in.

The durability caveat is real and worth stating plainly: with the default appendfsync everysec, a hard crash can lose about a second of writes. Managed Redis with AOF enabled and replication narrows the window but doesn't close it. If you want Redis Streams with the operational surface managed for you, Upstash is the one that fits a serverless worker fleet, since it bills per request and doesn't hold a connection per worker.

Choose Redis for dequeue latency, and only with Streams or BLMOVE — BRPOP quietly makes your queue at-most-once.

What does SQS give you that neither gives?

SQS's value is that it is not your infrastructure. No vacuum tuning, no failover, no capacity planning, and a dead-letter queue you configure instead of build. It's the default when a queue crosses a service or account boundary.

The two things that surprise people:

Visibility timeout is not a lock, it's a timer. If your job takes longer than the timeout, SQS re-delivers it to another consumer while the first is still working. You must either set the timeout above your worst-case duration or heartbeat with ChangeMessageVisibility:

import boto3

sqs = boto3.client("sqs")
resp = sqs.receive_message(QueueUrl=QUEUE_URL, WaitTimeSeconds=20, MaxNumberOfMessages=1)

for msg in resp.get("Messages", []):
    handle_with_heartbeat(
        msg["Body"],
        extend=lambda: sqs.change_message_visibility(
            QueueUrl=QUEUE_URL,
            ReceiptHandle=msg["ReceiptHandle"],
            VisibilityTimeout=120,
        ),
    )
    sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=msg["ReceiptHandle"])

Note WaitTimeSeconds=20 — long polling. Leaving it at zero is the single most common way people turn an idle SQS queue into a surprising bill, because empty receives are still billable requests.

And standard queues are at-least-once with best-effort ordering. Duplicates are a documented property, not a bug. FIFO queues give ordering and deduplication within a message group, at lower throughput per group. Either way, your handler has to be idempotent — which is true of all three options here, so treat it as a fixed cost rather than a differentiator.

SQS trades local latency and cheap introspection for someone else being on call for the queue itself.

Decision table

	Postgres + SKIP LOCKED	Redis Streams	SQS
Durability	Same as your DB (WAL, PITR)	Config-dependent; sub-second loss window on crash	Managed, replicated
Delivery	At-least-once	At-least-once with XACK	At-least-once (standard)
Transactional with app writes	Yes — same commit	No	No
Ops burden	Autovacuum + reaper	Memory + persistence config	Effectively none
Debugging	`SELECT * FROM jobs`	`XPENDING`, `XINFO`	Console + CloudWatch, no ad-hoc query
Cost shape	Free-ish; costs you DB headroom	Instance or per-request	Per request
Breaks down when	Enqueue churn competes with app traffic	You need durable-by-default	You need transactional enqueue or sub-10ms latency

The row that decides it most often is "transactional with app writes." If enqueueing a job must be atomic with the row that caused it, Postgres wins outright — everything else needs an outbox table, at which point you've built a Postgres queue anyway and added a second system.

FAQ

Can Postgres handle a job queue in production?
Yes. With FOR UPDATE SKIP LOCKED, a partial index on pending rows, and per-table autovacuum tuning, a single Postgres instance handles job rates well beyond what most applications produce. The limit you hit first is usually contention with your application's own queries on the same instance, not the queue mechanics.

Is Redis a reliable message queue?
Only with Streams and consumer groups, or BLMOVE into a processing list. BRPOP and LPOP delete the item at delivery, so a worker crash loses the job. Even with Streams, Redis persistence is configurable and can lose a small window of writes on a hard crash.

What's the difference between SQS visibility timeout and a lock?
A lock is held until released; a visibility timeout expires on a schedule regardless of whether your worker is still running. If processing exceeds the timeout, SQS delivers the same message to another consumer, so long-running jobs must extend the timeout with ChangeMessageVisibility.

Bottom line

Start with Postgres if your jobs are enqueued by the same application that owns the database — the transactional guarantee is worth more than anything the alternatives offer, and one fewer system to operate is a real feature. Move to Redis Streams when dequeue latency or enqueue volume starts showing up in your database's wait events, and accept the persistence trade-off explicitly rather than by default. Reach for SQS when the queue spans services, teams, or AWS accounts, or when nobody on the team wants to own queue infrastructure. All three demand idempotent handlers, so build that first and the migration between them stays cheap.

Token Streaming Works Locally but Arrives All at Once in Production: Finding the Buffer

Libme — Tue, 01 Sep 2026 07:46:12 +0000

If your LLM response streams token-by-token on localhost but lands as a single blob in production, your application code is almost certainly fine. Something between your process and the browser is holding bytes: a compression layer, a reverse proxy with response buffering on, or a platform that buffers the entire response before returning it. The fix is not to change how you write chunks — it is to walk the request path one hop at a time and find which hop stopped forwarding.

I have now debugged this on three different stacks, and every time the instinct was wrong. The first suspect is always the streaming code. It is almost never the streaming code.

Why does streaming work on localhost but not in production?

On localhost there is exactly one hop: your process writes to a socket, the browser reads it. In production there are usually four or five — a load balancer, a reverse proxy, possibly a CDN, maybe a serverless runtime wrapper, and compression middleware inside your own app. Each of those is allowed to accumulate bytes before forwarding, and most of them do it by default because buffering makes normal request/response traffic faster.

Server-Sent Events (SSE) and chunked responses are the exception where buffering destroys the entire point. The bytes still arrive correctly, so nothing errors, no log line fires, and monitoring stays green. You get a silent latency bug: the user waits eight seconds and then sees a wall of text.

The takeaway: buffering is a correctness-preserving optimization, which is exactly why nothing in your stack will warn you about it.

How do I find which hop is buffering?

Bisect the path. Start at the origin process and add one hop per test. This takes about ten minutes and beats guessing.

Test the app directly, on the box, bypassing every proxy:

curl -N --no-buffer -H "Accept: text/event-stream" \
  http://127.0.0.1:3000/api/chat

-N disables curl's own output buffering — leave it off and you will misdiagnose your infrastructure because of your test client. If chunks appear one at a time here, your handler is correct and the problem is downstream.

To see arrival timing rather than just final output, stamp each line:

curl -sN https://example.com/api/chat | while IFS= read -r line; do
  printf '%s %s\n' "$(date +%T.%N)" "$line"
done

(%N is GNU coreutils; on macOS use gdate from coreutils, or pipe through ts from moreutils.) A healthy stream shows timestamps creeping forward. A buffered one shows every line stamped within the same millisecond at the end — that single observation tells you the response was assembled somewhere and released at once.

Then repeat against each hop: the internal service address, the proxy address, the public hostname. The first URL that produces same-millisecond timestamps is the hop that owns your bug.

Hop	Typical symptom	Quick check	Usual fix
Compression middleware (in-app)	Buffered until ~1KB accumulates, then bursts	Response has `content-encoding: gzip`	Exclude `text/event-stream` from the compression filter
nginx / reverse proxy	Fully buffered, released at end	Works on origin port, not through proxy	`proxy_buffering off` or send `X-Accel-Buffering: no`
CDN / edge layer	Buffered only on the public hostname	Origin hostname streams, public one does not	Bypass rule for the route; check edge compression
Serverless wrapper	Always fully buffered, no config helps	Same behavior everywhere including local emulation	Use a runtime with explicit response-streaming support
Browser client	Network tab shows chunks, UI updates once	`curl -N` streams fine	Read the body as a stream, do not `await res.text()`

Which layers buffer most often?

Compression middleware, inside your own app. This is the one people miss because it is not "infrastructure." Express's compression package compresses anything compressible considers text — and text/event-stream is text/*, so it qualifies. gzip needs a block of input before it emits output, so your tokens sit in the compressor. Exclude the content type:

const compression = require("compression");

app.use(
  compression({
    filter: (req, res) =>
      res.getHeader("Content-Type") !== "text/event-stream" &&
      compression.filter(req, res),
  })
);

The same applies to Starlette/FastAPI's GZipMiddleware — if it is installed, take it out of the path for the streaming route and re-test before touching anything else.

Reverse proxies. nginx buffers proxied responses by default (proxy_buffering on), which is the right default for HTML and the wrong one for SSE. Scope the exception to the route rather than turning it off globally:

location /api/chat {
    proxy_pass http://app_upstream;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;
    proxy_cache off;
    gzip off;
    proxy_read_timeout 3600s;
}

If you cannot edit the proxy config — a shared ingress, a managed load balancer — the application can ask for the same behavior with a response header. nginx honors X-Accel-Buffering: no per response, which makes it the one reverse proxy where a hosted app can fix its own streaming without touching infrastructure config.

Set the headers correctly at the origin regardless:

from fastapi.responses import StreamingResponse
import json

@app.get("/api/chat")
async def chat():
    async def gen():
        async for chunk in llm_stream():
            yield f"data: {json.dumps({'text': chunk})}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(
        gen(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache, no-transform",
            "X-Accel-Buffering": "no",
        },
    )

no-transform is the part people leave out. It tells intermediaries they may not re-encode the body, which is a standards-level way of asking CDNs not to compress your stream.

The takeaway: fix compression first, proxy buffering second — in that order, because a compression layer will keep buffering even after you turn proxy buffering off.

What if the platform itself refuses to stream?

Some runtimes buffer the whole response by design, and no header will change that. Classic API Gateway integrations return the response as a single payload — the Lambda finishes, then the gateway replies. If you are streaming from AWS, Lambda Function URLs with the response-streaming invoke mode and awslambda.streamifyResponse are the supported path, at the cost of moving off API Gateway and losing the request-level features you had there.

Platform-side, as of September 2026, Vercel, Cloudflare Workers, and Fly.io all support streamed responses on their standard runtimes, so a stream that dies there is usually your own middleware rather than the platform. Cloudflare Workers is the one I reach for when the workload is pure passthrough streaming, though the CPU-time model makes it a poor fit if you do heavy work in the same request. If you need a long-lived connection with ordinary Node or Python semantics and no execution-time ceiling, a container platform like Fly.io removes the constraint entirely — you trade the zero-ops story for managing a process that stays up.

The takeaway: before optimizing your streaming code, confirm the runtime is even allowed to send bytes before the handler returns.

Is the bug possibly in the browser?

Sometimes. If the network panel shows chunks arriving over time but the UI updates once, the server is fine and the client is collecting the body. await res.text() and await res.json() both wait for completion by definition. Read the stream:

const res = await fetch("/api/chat");
const reader = res.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  render(decoder.decode(value, { stream: true }));
}

Note { stream: true } — without it, a multi-byte UTF-8 character split across two chunks decodes to a replacement character. That one bites when non-English output enters the picture.

FAQ

Why does my SSE stream only work locally and not behind nginx?
nginx has proxy_buffering on by default, so it accumulates the proxied response before forwarding it. Set proxy_buffering off for that location, or have your application send the X-Accel-Buffering: no response header, which nginx honors per response.

Does gzip break server-sent events?
It can. gzip needs input to accumulate before it emits compressed output, so a compression layer holds your tokens even when every proxy in front is configured to stream. Exclude text/event-stream from compression and send Cache-Control: no-cache, no-transform.

How do I test if a response is actually streaming?
Run curl -N --no-buffer against the endpoint and timestamp each line as it arrives. If all lines carry effectively the same timestamp, something buffered the response; if the timestamps spread out, the stream is live. Test the origin port first, then each hop outward.

Bottom line

If tokens stream on localhost and not in production, spend your time bisecting hops rather than rewriting the handler. Check in-app compression first, then reverse-proxy buffering, then the CDN, then the runtime — in that order, since each earlier layer can mask a fix applied to a later one. Set Cache-Control: no-cache, no-transform and X-Accel-Buffering: no at the origin as a permanent default; they cost nothing and preempt the two most common causes. And keep the timestamped curl -N one-liner in your notes — it converts a vague "streaming feels broken" report into a specific hop in about ten minutes.

Exit Code 137: Why Your Container Gets OOMKilled While Node's Heap Looks Fine

Libme — Sun, 30 Aug 2026 12:08:58 +0000

Exit code 137 is 128 + 9 — your process was killed with SIGKILL, and in a container that almost always means the cgroup memory limit was hit and the kernel OOM killer picked your process. The reason your heap graph looks innocent is that the limit applies to the container's working set (RSS: V8 heap + native buffers + thread stacks + mapped code + allocator overhead), while heapUsed only measures one slice of that. Debugging this starts by measuring the right number, not by raising the limit.

I have burned entire afternoons on this, twice, because the dashboard I was staring at was accurate and irrelevant at the same time.

How do I confirm it was actually an OOM kill and not a crash?

A Node process that runs out of heap dies loudly. You get a stack trace and a nonzero-but-not-137 exit:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

An OOM kill is silent. There is no stack trace, no uncaughtException handler firing, no last log line — SIGKILL cannot be trapped. That silence is the diagnostic signal. Confirm it from the outside:

# Kubernetes
kubectl describe pod my-api-7d9f | grep -A3 'Last State'
#     Last State:     Terminated
#       Reason:       OOMKilled
#       Exit Code:    137

# Plain Docker
docker inspect my-api --format '{{.State.OOMKilled}} {{.State.ExitCode}}'
# true 137

# The kernel's side of the story, if you can reach the host
dmesg -T | grep -i 'memory cgroup out of memory'

If Reason: OOMKilled is absent and you still see 137, someone or something sent SIGKILL — a docker kill, a failed liveness probe escalating past its grace period, or a node draining. Those are different bugs with different fixes.

Takeaway: a clean stack trace means you ran out of heap; total silence plus 137 means you ran out of container.

Why does `heapUsed` stay flat while RSS climbs?

process.memoryUsage() returns five numbers, and most dashboards graph the wrong one:

setInterval(() => {
  const m = process.memoryUsage();
  const mb = (n) => Math.round(n / 1024 / 1024);
  console.log(JSON.stringify({
    rss: mb(m.rss),                   // what the kernel counts against your limit
    heapTotal: mb(m.heapTotal),       // V8 heap reserved
    heapUsed: mb(m.heapUsed),         // what most dashboards graph
    external: mb(m.external),         // C++ objects bound to JS objects
    arrayBuffers: mb(m.arrayBuffers), // Buffers, TypedArrays — off-heap
  }));
}, 15_000);

Every byte you read from a socket, a file stream, an image pipeline, or a database driver's binary protocol lands in arrayBuffers/external, not in old space. A service that buffers uploads or concatenates response bodies can hold hundreds of megabytes off-heap while heapUsed sits at a tidy 90 MB. V8's garbage collector is also under no pressure to run, because from its point of view there is plenty of room — the memory it cares about is not the memory that is running out.

Native modules make this worse in a way that is easy to misread. sharp, canvas, grpc, and similar addons allocate through their own libraries; leaked handles there never appear in a heap snapshot at all.

Takeaway: if RSS and heapUsed diverge, stop reading heap snapshots — the memory you are losing is not on the JavaScript heap.

What limit is Node actually enforcing inside a container?

Two limits exist, they are set in different places, and nothing makes them agree:

Limit	Set by	Covers	What happens when you hit it
V8 old-space max	`--max-old-space-size` / `NODE_OPTIONS`	JS heap only	Aggressive GC, then a fatal `heap out of memory` error with a stack trace
cgroup memory max	Docker `--memory`, k8s `resources.limits.memory`	Everything: RSS, page cache, allocator overhead	SIGKILL, exit 137, no output

Read the real limit from inside the container rather than trusting the manifest:

# cgroup v2 (the default on current distros as of mid-2026)
cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current

# cgroup v1
cat /sys/fs/cgroup/memory/memory.limit_in_bytes

Node tries to size its default heap from the memory it detects, but that detection depends on the runtime version and on which cgroup version the host uses, so I no longer rely on it in either direction. Set it explicitly, below the container limit, leaving room for everything off-heap:

# container limit 1024Mi → heap ceiling 768Mi, ~256Mi for buffers, stacks, allocator
ENV NODE_OPTIONS="--max-old-space-size=768"

The point of that gap is not politeness. A heap ceiling under the container ceiling converts a silent SIGKILL into a loud, debuggable JavaScript heap out of memory crash with a stack trace — you trade an unexplained restart for an actual error message.

Takeaway: set --max-old-space-size to roughly 70–80% of the container limit so leaks surface as crashes you can read instead of kills you can't.

Why does memory keep growing when there is no leak?

Three causes I hit far more often than an actual leak:

Allocator fragmentation. glibc's malloc creates per-thread arenas, and a process with many threads (Node's libuv pool, native addons) can hold on to a lot of freed-but-unreturned memory. Capping arenas with MALLOC_ARENA_MAX=2 or switching the image to jemalloc is worth testing — it flattened RSS growth in one image-processing service of mine and did nothing measurable in two others, so treat it as an experiment, not a fix.

Page cache counted as usage. memory.current includes file cache, which inflates the number without being a real problem. Kubernetes evaluates the working set, not raw usage, so graph container_memory_working_set_bytes — Prometheus with cAdvisor metrics exposes exactly this series, and it is the one that lines up with an OOM kill. Its drawback is granularity: at a 15–30 second scrape interval it will miss a spike that kills you in under a scrape.

Unbounded concurrency. Memory tracks in-flight requests. A queue consumer that pulls 500 messages at once, or an endpoint with no upload size cap, is not leaking — it is just allowed to use more memory than exists. If you want a managed alternative to burning a week on this, Datadog's container memory views correlate working-set spikes with the specific request traces that caused them, at the cost of another per-host bill.

For genuine leaks, capture a snapshot from a live process without a debugger attached:

node --heapsnapshot-signal=SIGUSR2 server.js
kill -USR2 <pid>   # writes a .heapsnapshot into cwd; diff two of them in Chrome DevTools

Take one snapshot after warmup and one an hour later, and compare — a single snapshot tells you almost nothing.

Takeaway: raising the memory limit is a legitimate fix only after you have ruled out fragmentation, page cache, and unbounded concurrency.

FAQ

What does exit code 137 mean in Docker?
It means the process received SIGKILL (128 + signal 9). In containers, this is nearly always the kernel OOM killer enforcing the cgroup memory limit. Check docker inspect --format '{{.State.OOMKilled}}' — if it prints true, it was memory, not your code.

Why is my Node.js container OOMKilled when heap usage is low?
Because the container limit counts RSS, which includes off-heap memory: Buffers and TypedArrays (arrayBuffers), native addon allocations, thread stacks, and allocator overhead. Log process.memoryUsage().rss alongside heapUsed; if they diverge, the growth is off-heap and heap snapshots will not show it.

Should I set --max-old-space-size in a container?
Yes. Set it explicitly at roughly 70–80% of the container's memory limit rather than relying on Node's automatic detection, which varies by version and cgroup setup. The gap leaves headroom for off-heap memory and turns silent OOM kills into readable heap-exhaustion errors.

Bottom line

If you see 137 with no logs, confirm OOMKilled first — that single check separates a memory problem from a lifecycle problem and saves you from debugging the wrong thing. Then graph RSS and working set, not heapUsed, and set --max-old-space-size under the container limit so future failures arrive with a stack trace attached. Reach for heap snapshots only once RSS and heapUsed are growing together; if they have diverged, look at buffers, native addons, and concurrency limits instead. Raising the limit is a valid last step, but it is a decision you should make with the working-set graph in front of you, not as a reflex.

Feature Flags: When a Managed Service Beats a Config Table You Own

Libme — Fri, 28 Aug 2026 21:47:54 +0000

If all you need is "turn this code path on for everyone or nobody," a feature_flags table plus a cached lookup will serve you for years and cost nothing. You start paying for a managed service when non-engineers need to flip flags, when you need percentage rollouts targeted at stable user buckets, and when someone has to answer "who turned that on, and when?" The trap in both directions is the same: teams underestimate the failure mode where the flag layer is unreachable, and they underestimate how much dead flag code accumulates.

What does a flag service actually sell you?

Not the boolean. Evaluating a boolean is five lines of code. What you're buying, roughly in order of how much they matter to a small team:

A UI safe enough for someone who can't deploy. The whole point of a kill switch is that it works at 2am when the person on call isn't the person who wrote the feature.
An audit trail. Who changed which flag, when, from what value. This is the thing you'll miss the first time a flag change causes an incident and nobody can reconstruct the timeline.
Consistent bucketing across services. A user in the 10% rollout in your API must land in the same bucket in your web frontend and your background worker. Get that wrong and you get the "the feature flickers on and off for one customer" bug.
Real-time propagation and targeting beyond percentages. Streaming updates so a flip takes effect in seconds, plus per-plan, per-org, per-region rules.

The first two are an afternoon of work. The last two are where homegrown implementations quietly rot.

Takeaway: you're not paying for flag evaluation, you're paying for safe non-engineer access and an audit trail you'll only value in hindsight.

What happens when the flag service is unreachable?

This is the failure mode that turns a cost decision into an availability decision, and it's the one I've actually been burned by. The symptom looks like this in your logs:

[warn] LaunchDarkly client initialization timed out after 5000ms
[warn] feature flag "new_checkout" evaluated to fallback value: false

Two independent things go wrong. First, SDK initialization is usually awaited during boot, so an unreachable provider adds its full timeout to startup — which, in a rolling deploy with a readiness probe, looks like a failed deploy rather than a flag problem. Second, when init fails every flag returns the fallback passed at the call site, and in most codebases half of those were typed as false without thinking. If false means "use the old code path," fine. If false means "disable the rate limiter," you've just had an outage caused by your flag vendor.

The fix has nothing to do with which vendor you choose:

// flags.js — never let the flag provider decide whether your app starts.
const DEFAULTS = require('./flag-defaults.json'); // committed to the repo

let snapshot = { ...DEFAULTS };

function withTimeout(promise, ms) {
  return Promise.race([
    promise,
    new Promise((_, reject) =>
      setTimeout(() => reject(new Error('flag init timeout')), ms).unref()
    ),
  ]);
}

async function initFlags(client) {
  try {
    // whatever your SDK calls its "ready" promise
    await withTimeout(client.ready(), 500);
    snapshot = await client.allFlags();
  } catch (err) {
    console.warn('flags: serving committed defaults', { reason: err.message });
  }
}

function flag(key) {
  return key in snapshot ? snapshot[key] : DEFAULTS[key] ?? false;
}

module.exports = { initFlags, flag };

Three properties matter: a short, explicit init timeout; a defaults file in version control, so the fallback is a reviewed decision; and a boot sequence that continues regardless. Most SDKs cache their last known ruleset in memory and ride out a network blip — but only if the process already initialized, which is exactly what doesn't hold during a deploy.

Takeaway: pick your fallback values deliberately and commit them to the repo, because a flag provider outage will read them all at once.

When is a managed flag service worth paying for?

Signal	Config table you own	Managed service
Only engineers flip flags	Fine	Overkill
Support or product needs to flip flags	Painful (you build the UI)	This is the product
Boolean on/off only	Fine	Overkill
Percentage rollouts across multiple services	Doable, easy to get subtly wrong	Solved, consistent bucketing
Need "who changed what, when"	You build audit logging	Built in
Flags read on hot paths (per-request)	Cache locally, trivial	Needs local eval or a proxy
Client-side / mobile flags	You build a public endpoint	Solved, with edge caching
Compliance or SSO requirements	Your problem	Usually an upper-tier feature
Fewer than ~10 live flags at a time	Fine	Hard to justify

Two rows deserve emphasis. Client-side flags are where homegrown solutions get expensive: you now need a public, cacheable, low-latency endpoint that doesn't leak targeting rules to the browser. And pricing — as of mid-2026 the common models are per-seat, per-monthly-active-context, or per-request for edge evaluation. Check which shape your usage is: a two-person team with millions of anonymous visitors gets a very different bill under a per-context model than under a per-seat one.

Takeaway: the moment a non-engineer needs to flip a flag, or a flag has to reach the browser, the build-your-own math stops working.

What does the homegrown version actually look like?

Smaller than people expect, if you keep the scope honest. A table:

create table feature_flags (
  key             text primary key,
  enabled         boolean not null default false,
  rollout_percent int not null default 0
                  check (rollout_percent between 0 and 100),
  updated_at      timestamptz not null default now(),
  updated_by      text
);

And deterministic bucketing, which is the part worth getting right:

const crypto = require('crypto');

function inRollout(flagKey, userId, percent) {
  if (percent >= 100) return true;
  if (percent <= 0) return false;
  // Hash the flag key with the user id so a user isn't in the same
  // bucket for every flag, and so the bucket is stable across services.
  const digest = crypto.createHash('sha256')
    .update(`${flagKey}:${userId}`)
    .digest();
  return digest.readUInt32BE(0) % 100 < percent;
}

Hashing flagKey:userId rather than userId alone matters: if you hash the user id only, the same unlucky 10% of users are the guinea pigs for every single rollout you ever do. Cache the table in memory with a short TTL, refresh in the background, and never block a request on the database read.

What the homegrown version does not give you is flag hygiene. Dead flags are the real cost: every stale flag is a branch no test covers and no one dares delete. Run something like this in CI and diff it against your flag store:

git grep -oh "flag('[a-z0-9_.-]*'" -- src | sort -u

Takeaway: a homegrown flag system is a weekend to build and a permanent tax to clean up, and the cleanup is the part teams skip.

How do the main options differ as of mid-2026?

If you want the mature, safe default with the deepest targeting and experimentation tooling, LaunchDarkly is the option that handles streaming updates, per-environment audit trails, and a self-hosted relay for high-volume evaluation — with the caveat that it is the priciest of the group and the relay is extra infrastructure you operate. If you want an open-source core you can self-host on your own Postgres and pay for only when you need SSO and richer access control, Unleash is the one designed to run in your infrastructure, though front-end flags require running its edge/proxy component so targeting rules never reach the browser. If you want open source with a hosted option and a simpler mental model that includes remote config alongside flags, Flagsmith covers that ground, with a smaller ecosystem and fewer third-party integrations than LaunchDarkly. If your team already runs PostHog for product analytics, its built-in flags are worth using before adding a second vendor, keeping in mind flags are a secondary product there rather than the main event.

Whichever you pick, wrap it behind OpenFeature, the vendor-neutral flag SDK standard under the CNCF, so a later migration is a provider swap rather than a codebase-wide refactor. The abstraction is thin, but provider maturity varies by language — check your runtime before committing.

Takeaway: choose on hosting model and access control first, because targeting features have largely converged across these tools.

FAQ

Should I use a database table for feature flags?
Yes, if only engineers flip flags, you have a handful of them, and they're boolean. Cache the table in memory with a short TTL so you're not adding a query to every request, and add an updated_by column from day one — reconstructing who flipped what is the first thing you'll want during an incident.

What happens if LaunchDarkly or my flag provider goes down?
Initialized SDK clients keep serving their last cached ruleset, so running processes usually survive a provider outage. The dangerous window is process startup: if the client can't initialize, every flag falls back to the default value at its call site, so commit an explicit defaults file and set a short init timeout instead of blocking boot.

How many feature flags is too many?
There's no universal number, but if you can't name what every live flag is currently gating, you have flag debt. Give each flag an owner and a removal date at creation time, and treat a permanently-on flag as a code change you owe the repo, not a setting.

Bottom line

If you're a small team where engineers own the flips and flags are on/off, build the table — it's an afternoon, and a managed service would mostly bill you for a UI nobody opens. Move to a managed service when support or product needs to flip flags without a deploy, when you need consistent percentage bucketing across more than one service, or when flags have to reach the browser. Self-host Unleash or Flagsmith if you'd rather spend ops time than budget; take LaunchDarkly if targeting depth and audit trails matter more than the invoice. Either way, set explicit fallback defaults and a removal date per flag on the day you create it — that discipline matters more than which vendor's logo is on the dashboard.

DEV Community: Libme

Webhook Signature Verification Fails: Your Framework Already Parsed the Body

What does the error actually look like?

Why does re-serializing the parsed body not work?

How do I get the raw body in each framework?

What else changes the bytes between sender and handler?

How do I test this without waiting for real events?

FAQ

Bottom line

Related reading

Playwright vs Cypress When Your E2E Tests Pass Locally but Fail in CI

What does "flaky in CI only" actually look like?

How do Playwright and Cypress wait differently?

What do retries and artifacts cost you in each tool?

When does parallelization change the decision?

FAQ

Bottom line

Related reading

Your Nightly Job Ran Twice on the DST Switch: Making Scheduled Jobs Timezone-Safe

What actually happens at the transition

Why "just set the scheduler's timezone" isn't enough

The two-clock rule

Which approach for which job?

How do you test this before November?

FAQ

Bottom line

Related reading

Postgres Autovacuum Isn't Keeping Up: Diagnosing Bloat, Long Transactions, and Wraparound Warnings

Why "autovacuum ran" and "dead rows were removed" are different things

How do I tell if autovacuum is falling behind?

What actually blocks dead tuple removal?

Which autovacuum settings are worth changing?

What do I do about "database must be vacuumed within N transactions"?

The bloat is already there — now what?

FAQ

Bottom line

Related reading

UUIDv7 vs ULID vs bigint: Which Primary Key Holds Up When the Table Gets Big?

Why do random UUIDs slow down inserts as the table grows?

How do you confirm it's the key and not something else?

UUIDv7 vs ULID vs bigint: what actually differs?

Is ULID worth giving up the native uuid type?

How do you generate UUIDv7 on the Postgres version you actually have?

What's the honest downside of sequential keys?

FAQ

Bottom line

Related reading

curl Returns 200 but Your HTTP Client Gets a 404: Debugging Vary and CDN Cache Variants

Why do two clients get different responses from the same URL?

How do you confirm it's a cache variant and not a flaky origin?

What can you actually do when the bad variant belongs to someone else's CDN?

How do you stop your own service from creating this problem?

FAQ

Bottom line

Related reading

Your Logging Bill Is a Volume Problem: Model the Cost Before You Switch Vendors

Why does the log bill grow faster than traffic?

What are you actually paying for?

How do I measure my log volume before I shop?

What should you cut first?

When does self-hosting actually cost less?

FAQ

Bottom line

Related reading

Every Deploy Throws a Few 502s: Where Graceful Shutdown Actually Breaks

Why do deploys produce 502s at all?

Is your process even receiving SIGTERM?

What a correct Node shutdown handler looks like

Where the delay belongs on each platform

How do you prove it is fixed?

FAQ

Bottom line

Related reading

Postgres as a Job Queue vs Redis vs SQS: When Does "Just Use Your Database" Stop Working?

Why does a naive Postgres queue lose or double-process jobs?

When is Redis actually the better queue?

What does SQS give you that neither gives?

Decision table

FAQ

Bottom line

Is ULID worth giving up the native `uuid` type?

Why does `heapUsed` stay flat while RSS climbs?