DEV Community: Libme

Your LLM Bill Jumped After You Added Context: Find the Cache Miss Before You Downgrade the Model

Libme — Sun, 04 Oct 2026 00:44:13 +0000

If your LLM API spend climbed after you added retrieval, a longer system prompt, or tool definitions, the cause is almost always input tokens being reprocessed at full price on every request — not the model you picked. Check the cache fields in the response usage object first: if cached reads are zero across repeated requests that share a prefix, you have a bug, not a pricing problem. Fixing prefix stability and breakpoint placement is free; downgrading the model costs you quality.

Why is the bill rising when output length didn't change?

The first thing to do is stop reasoning about the bill and start logging per-request token accounting. On the Anthropic API, every response carries a usage object that splits input tokens three ways:

import anthropic

client = anthropic.Anthropic()

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=2048,
    system=[
        {
            "type": "text",
            "text": STABLE_PREAMBLE,          # instructions, schema, few-shot examples
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": question}],
)

u = resp.usage
print(
    f"uncached={u.input_tokens} "
    f"written={u.cache_creation_input_tokens} "
    f"read={u.cache_read_input_tokens} "
    f"output={u.output_tokens}"
)

The field that trips people up is input_tokens: it is the uncached remainder, not the prompt size. Total prompt size is input_tokens + cache_creation_input_tokens + cache_read_input_tokens. I have watched a team conclude their prompt was small because input_tokens read 4K, while the other two fields accounted for ten times that. OpenAI's API exposes the analogous number as usage.prompt_tokens_details.cached_tokens; the arithmetic lesson is the same.

Run the same request twice in a row and compare. A healthy second request reads the shared prefix and writes only the delta. If the second request shows a cache write roughly equal to the full prompt and a read of zero, your prefix is not stable.

Takeaway: before you argue about model pricing, prove whether your repeated prefix is being read from cache or reprocessed from scratch.

What silently breaks prompt caching?

Prompt caching is a prefix match over the exact rendered bytes. The rendering order is tools → system → messages, and any byte that changes early invalidates everything after it. That makes the failure mode sneaky: the request still succeeds, the output is still correct, and only the bill notices.

The classic offender is a system prompt that looks like this:

# Every request produces a different prefix — nothing after this ever caches
system = f"""You are a support assistant.
Current time: {datetime.now().isoformat()}
User: {user.email} (plan: {user.plan})
...500 lines of instructions...
"""

Three invalidators in four lines. The fix is to freeze the prefix and push volatile values past the last breakpoint — into a later message, where a change at turn five invalidates nothing before turn five:

system = [{"type": "text", "text": STATIC_INSTRUCTIONS,
           "cache_control": {"type": "ephemeral"}}]

messages = [
    {"role": "user", "content": [
        {"type": "text", "text": f"Current time: {now}. Plan: {user.plan}."},
        {"type": "text", "text": question},
    ]},
]

Other invalidators worth grepping for in any code that builds a prompt: json.dumps() without sort_keys=True, iteration over a set, tool lists assembled per user (tools render at position zero, so a per-user tool set means no cross-user reuse), and conditional system sections where every feature-flag combination becomes its own distinct prefix. Switching models invalidates too — caches are model-scoped.

Takeaway: anything that varies per request belongs after the last cache breakpoint, and "per request" includes the clock.

Where should the cache breakpoint actually go?

Marking the end of the prompt feels right and is usually wrong. If the last block is the user's unique question or freshly retrieved rows, the breakpoint lands after bytes that will never repeat — so every request pays the write premium and no request ever reads it back. The signature in your logs is a cache write on every request while reads never cover the shared prefix.

Prompt shape	Where the breakpoint goes	Failure if you get it wrong
Big fixed preamble, varying question	End of the shared preamble	Write premium on every request, reads near zero
Growing multi-turn conversation	Last block of the newest turn	Whole history reprocessed each turn
Prefix differs from byte one per request	Nowhere — don't cache	Pure surcharge, no reads
Sections with different change rates	One breakpoint per stability boundary	Daily-changing context invalidates never-changing tools

Two more constraints that bite in practice, as of late 2026: there is a minimum cacheable prefix (model-dependent, and not monotonic across generations — newer models cache shorter prefixes than some older ones), and below it caching silently no-ops with no error. And parallel requests over an identical prefix all miss, because an entry only becomes readable once the first response starts streaming. For fan-out, send one request, wait for its first token, then fire the rest.

The economics are a ratio, not a mystery: a cache write costs more than a plain input token and a cache read costs a fraction of one, so a prefix read back even twice is already ahead. Longer-lived cache entries cost more to write, which means they only pay off across gaps that the default short-lived entry can't bridge. If you want the version of this that requires no breakpoint bookkeeping at all, the Anthropic API's automatic caching places the breakpoint for you and is the right default for ordinary multi-turn chat — it places exactly one breakpoint, so prompts with several stability boundaries still need explicit markers.

Takeaway: put the breakpoint at the end of what repeats, not at the end of the prompt.

When is the batch endpoint worth it?

For any workload where a human isn't waiting on the response, the batch endpoints from the major providers run asynchronously at roughly half the standard per-token rate (as of late 2026 — confirm the current discount in your provider's pricing docs). Nightly summarization, backfills, evaluation runs, bulk classification, and embedding generation all qualify.

Three things to plan for. Results arrive in arbitrary order, so key them by your own custom_id and never by position. Completion is a window, not a latency target — build the job so a delayed batch degrades a dashboard rather than blocking a user request. And batch does not compose with everything: pairing it with caching tricks or streaming-dependent code paths usually means reshaping the request.

Takeaway: if no human is blocked on the answer, half-price asynchronous processing is the highest-leverage change you can make without touching prompt quality.

Should you route to a smaller model?

Eventually, maybe — but it belongs last in the order, after the free wins, because it is the only lever that trades quality. Two things make cascades less attractive than the napkin math suggests. First, caches are model-scoped, so splitting traffic across two models forfeits cache reuse between them; a cascade that saves 40% on paper can lose most of it to cold prefixes. Second, the honest unit is cost per completed task, not cost per request — a cheaper model that needs a retry, a longer chain, or human cleanup is not cheaper.

Measure the simpler alternative first: the same capable model at a lower effort or reasoning setting, on the same traffic. On current-generation models, a reduced-effort run often lands where the previous generation's maximum effort did, and you keep one cache namespace. Tune that per route — classification and extraction routes usually hold quality at the low end, while coding and long-horizon agent loops do not.

For attributing spend to routes before you cut anything, an LLM observability layer such as Langfuse gives you per-trace token and cost breakdowns without writing your own aggregation, at the cost of running one more service (or paying for the hosted tier) and adding a logging dependency to your request path.

Takeaway: prove the cheap model wins on cost per completed task — including retries and lost cache reuse — before you ship the cascade.

FAQ

Why is cache_read_input_tokens always 0?
Something in the prefix changes between requests. Log two consecutive full request bodies, strip the cache_control markers, and diff them — the first difference inside the overlapping region is your invalidator. Timestamps, UUIDs, unsorted JSON, and per-user tool lists cause most of these.

Does prompt caching reduce latency or just cost?
Both, in practice: cached prefix tokens are not reprocessed, so time to first token drops on long prompts. Cost is the more reliable win; latency improvement depends on how much of the prompt was cacheable.

Is the LLM batch API worth it for a small app?
Yes, for any job a user isn't waiting on — roughly half price for the same model and prompt, as of late 2026. It is the rare optimization with no quality tradeoff, only a latency one.

Bottom line

Start with measurement: log input_tokens, cache writes, and cache reads per request, and treat a zero read rate on a repeated prefix as a bug with a ticket. Fix prefix stability and breakpoint placement before anything else — they cost nothing in quality and often account for most of the surprise. Move every job with no human in the loop to the batch endpoint. Only then consider lower effort settings, and only after that a smaller model, judged on cost per completed task rather than cost per call.

Postgres Multi-Tenancy: Row-Level Security, tenant_id Filters, or a Schema per Tenant?

Libme — Fri, 02 Oct 2026 15:28:02 +0000

For most B2B SaaS apps, a tenant_id column plus row-level security (RLS) on every table is the right default: you keep one schema and one migration path, and the database enforces isolation even when someone forgets a WHERE clause. Schema-per-tenant only pays off when you have a small number of large tenants with genuinely different needs (per-tenant restores, per-tenant data residency). Database-per-tenant is an operations decision, not a data-modeling one — pick it when tenants must be backed up, moved, or deleted independently.

The part nobody warns you about: RLS fails silently in both directions. Misconfigured one way, it returns zero rows and your app looks broken. Misconfigured the other way, it returns everyone's rows and nothing looks wrong at all.

What does "multi-tenancy" actually mean at the storage layer?

Three models, in increasing order of isolation and operational cost:

Model	Isolation enforced by	Migration cost	Per-tenant restore	Realistic tenant count
`tenant_id` column, app filters	Your application code	One migration	Hard (row-level surgery)	Unlimited
`tenant_id` column + RLS	Postgres, per query	One migration	Hard	Unlimited
Schema per tenant	Postgres, per `search_path`	One migration × N schemas	Easy (`pg_dump -n`)	Tens to low hundreds
Database per tenant	Postgres, per connection	One migration × N databases	Easy (`pg_dump`)	Tens

The first row is where most teams start and where most tenant-data leaks come from. A single endpoint that builds a query without the tenant predicate — a new report, an admin tool, a background job reusing a helper — is enough. Code review catches this most of the time, which is exactly the problem: "most of the time" is not an isolation guarantee.

Takeaway: if isolation depends on every future query being written correctly, you do not have isolation, you have a convention.

How do I set up RLS so it actually enforces anything?

Two statements per table, plus a policy:

ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
ALTER TABLE invoices FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON invoices
  USING (tenant_id = current_setting('app.current_tenant', true)::uuid)
  WITH CHECK (tenant_id = current_setting('app.current_tenant', true)::uuid);

ENABLE turns policies on. FORCE is the one people skip, and it matters: without it, the table's owner bypasses every policy. If your application connects with the same role that created the tables — the default in a lot of small setups — then ENABLE ROW LEVEL SECURITY alone does nothing for your app's queries. It looks configured. It enforces nothing.

USING filters reads (and which rows UPDATE/DELETE can see). WITH CHECK validates rows being written. Omit WITH CHECK and a tenant can insert rows labeled with someone else's tenant_id.

On the application side, set the tenant inside the transaction, never as a session-level SET:

// pg (node-postgres). set_config(..., true) == SET LOCAL, scoped to this transaction.
async function withTenant(pool, tenantId, fn) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    await client.query('SELECT set_config($1, $2, true)', ['app.current_tenant', tenantId]);
    return await fn(client);
  } finally {
    await client.query('COMMIT').catch(() => client.query('ROLLBACK'));
    client.release();
  }
}

SET LOCAL cannot take a bind parameter, which is why set_config($1, $2, true) is the version you want — string-concatenating a tenant id into a SET statement is an injection point in the one place you least want one.

Takeaway: ENABLE without FORCE, or USING without WITH CHECK, is a policy that reads as secure in a migration diff and is not.

Why does RLS return zero rows (or everyone's rows) in production?

These are the failure modes I keep running into, in the order they bite.

Zero rows, no error. current_setting('app.current_tenant', true) with the second argument true returns NULL when the setting is missing instead of raising unrecognized configuration parameter. tenant_id = NULL is never true, so every query returns an empty set. The symptom is a page that renders with no data and no stack trace, usually in a code path that got a connection outside your withTenant wrapper — a health check, a migration script, a queue worker. Drop the true during development so you get a loud error instead of an empty list.

Everyone's rows, no error. Three causes: the app role owns the tables and you did not FORCE; the app role has BYPASSRLS (superuser always does — this is why your app should never connect as postgres); or the table is new and nobody enabled RLS on it. The third is the common one. Put it in CI:

-- Fails the build if any table in the schema is missing RLS.
SELECT c.relname
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public'
  AND c.relkind = 'r'
  AND (NOT c.relrowsecurity OR NOT c.relforcerowsecurity);

Assert that query returns no rows, with an explicit allowlist for genuinely shared tables (plans, feature flags, country codes).

Cross-tenant bleed under a connection pooler. In transaction pooling mode, PgBouncer or Supavisor hand your next transaction whatever server connection is free. A session-level SET app.current_tenant survives on that server connection and is then inherited by a different tenant's transaction. SET LOCAL / set_config(..., true) resets at commit, which is what makes RLS and transaction pooling compatible at all.

Takeaway: test isolation with a real query as a real tenant role in CI — a passing migration proves the policy exists, not that it applies.

When is schema-per-tenant worth the operational cost?

When a tenant is a unit of operations, not just a unit of data. Concretely: you need pg_dump -n tenant_42 to restore one customer without touching the others, you have contractual data-residency requirements, or a few large tenants need different indexes or retention.

The cost is real. Every migration runs N times, so a schema change is now a job with partial-failure semantics rather than a single statement. Thousands of schemas bloat the system catalogs and make pg_dump, autovacuum scheduling, and query planning noticeably slower. Under a pooler, search_path has the same leak problem as session-level SET.

If you want per-tenant isolation without hand-rolling the migration fan-out, Neon's database branching gives you cheap copy-on-write databases you can create and drop per tenant or per environment, at the cost of tying that piece of your infrastructure to one vendor's control plane. If you are already on Postgres-as-your-backend and want RLS wired to your auth layer, Supabase enforces policies against JWT claims by default, which is genuinely good hygiene but means an incorrect policy is directly internet-facing rather than behind your API. For tenants that outgrow one machine, Citus distributes tables by tenant_id so a tenant's rows stay colocated on one node, with the tradeoff that cross-tenant analytical queries and some schema changes get more expensive.

Takeaway: choose schema- or database-per-tenant when you need per-tenant restore and delete, not because it feels more secure.

Does RLS cost performance?

It adds a predicate to every query on the table, so the honest answer is: only if your indexes ignore the tenant. Make tenant_id the leading column of the indexes that back your hot queries — (tenant_id, created_at DESC) rather than (created_at DESC) — and the RLS predicate rides along for free.

The subtler cost: Postgres evaluates RLS quals before user-supplied quals unless the operators involved are marked LEAKPROOF, since a non-leakproof function could otherwise expose values from rows you should never see. That restriction occasionally blocks a plan the planner would have chosen otherwise. Check EXPLAIN on your slowest tenant-scoped query with RLS on and off before assuming it is free.

Takeaway: RLS is cheap when tenant_id leads your indexes and worth measuring when it doesn't.

FAQ

Can I use row-level security with PgBouncer in transaction mode?
Yes, as long as you set the tenant with SET LOCAL or set_config('app.current_tenant', $1, true) inside the transaction. A plain session-level SET persists on the pooled server connection and will be inherited by another tenant's transaction.

Why does my RLS policy return no rows even though the data exists?
The configuration parameter the policy reads is unset on that connection, so current_setting('app.current_tenant', true) returns NULL and the comparison is never true. It almost always means a code path acquired a connection without going through your per-request transaction wrapper.

Do I need one database per tenant for compliance?
Usually no — RLS satisfies most logical-isolation requirements. Separate databases matter when you need per-tenant backup, restore, deletion, or data residency as an operational guarantee rather than a query-level one.

Bottom line

Start with a tenant_id column and RLS with both ENABLE and FORCE, an app role that is not the table owner and has no BYPASSRLS, and SET LOCAL inside every request transaction. Add a CI check that fails when a new table ships without a policy, because that is how the leak actually happens. Move to schema- or database-per-tenant only when per-tenant restore, deletion, or residency becomes a requirement — and accept that your migration pipeline becomes a fan-out job that day. Whatever you pick, write one test that logs in as tenant A and asserts it cannot read tenant B's row; it is the only evidence that any of this works.

Users Randomly Logged Out? Your Redis Eviction Policy Is Deleting Sessions

Libme — Thu, 01 Oct 2026 23:54:44 +0000

If a subset of your users get logged out at unpredictable times — no deploy, no session-secret change, worse during traffic peaks — check maxmemory-policy on your Redis instance before you touch cookie code. When Redis hits its memory ceiling with an allkeys-* policy, it deletes whatever keys are least recently used to make room, and it does not care that some of those keys are login sessions or queued jobs. The fix is not a bigger instance; it's separating keys you can afford to lose from keys you can't.

I lost most of a day to this on a side project where sessions, a page cache, and a BullMQ queue all shared one small managed Redis. The cache was doing its job — filling memory — and Redis was doing its job — throwing things out. Together they produced a third behavior nobody asked for: random logouts and jobs that silently never ran.

Why does Redis delete keys that still have time left on their TTL?

Redis with maxmemory set is a bounded store, and maxmemory-policy decides what happens when a write would cross that bound. The policies split into three families:

noeviction — refuse the write. The client gets OOM command not allowed when used memory > 'maxmemory'.
volatile-lru, volatile-lfu, volatile-random, volatile-ttl — evict only keys that have a TTL set. If there are no such keys, writes fail like noeviction.
allkeys-lru, allkeys-lfu, allkeys-random — evict anything, TTL or not.

The trap is that eviction is invisible from the application's side. A deleted session key is indistinguishable from an expired one: your middleware looks up the session, gets nil, and correctly concludes "not logged in." There's no error to catch, no log line, nothing that says this key was taken from you. Same for a queue: BullMQ or Sidekiq asks for a job hash that isn't there anymore and the job just doesn't exist.

Two details make this harder to reason about than it should be. First, volatile-* policies feel safe ("only expiring keys get evicted") right up until you realize sessions almost always have a TTL — they're prime eviction candidates, and the ones with the longest remaining life under volatile-ttl survive while your active-but-recently-renewed ones may not. Second, SELECT-ing a different logical database (db 0 vs db 1) does not isolate anything: all databases on an instance share one memory budget and one eviction policy, and Redis Cluster drops multiple databases entirely. Prefixing keys and switching DB numbers buys you tidiness, not safety.

Takeaway: eviction is a silent DEL issued by the server, so any key whose absence changes correctness must live somewhere eviction cannot reach.

How do I confirm eviction is what's happening?

Three commands, in this order. Don't trust the default documented in the docs — managed providers ship their own parameter defaults, and someone may have changed yours years ago.

# 1. What policy is actually in force, and how close to the ceiling are we?
redis-cli INFO memory | grep -E 'used_memory_human|maxmemory_human|maxmemory_policy'

# 2. The smoking gun: a non-zero, *increasing* counter.
redis-cli INFO stats | grep -E 'evicted_keys|expired_keys'

evicted_keys is a monotonic counter since the last restart. Any non-zero value on an instance holding sessions or jobs is a bug report. Sample it twice a minute apart — a rising number during the window your users complain about is as close to proof as you'll get after the fact.

If you need to catch it live, keyspace notifications will tell you exactly which keys are going:

# 'E' = keyevent channels, 'e' = evicted events
redis-cli CONFIG SET notify-keyspace-events Ee
redis-cli PSUBSCRIBE '__keyevent@*__:evicted'

Each message carries the evicted key name, so you'll see sess:... or bull:mail:... scroll past and the argument ends there. Two honest caveats: notifications are fire-and-forget (a disconnected subscriber misses events entirely, so this is a debugging tool, not an audit log), and CONFIG SET doesn't survive a restart unless you CONFIG REWRITE or change it in your provider's parameter group.

The dead ends I burned time on first, so you can skip them: rotating the session secret, SameSite/Secure cookie flags, sticky-session config on the load balancer, and clock skew on token expiry. All plausible causes of logouts, none of which produce a rising evicted_keys.

Takeaway: evicted_keys > 0 on an instance storing sessions or jobs is not a tuning opportunity, it's a correctness defect.

What policy should each workload get?

Pick per role, not per cluster. The point of the table is that "which policy is best" is the wrong question — the right one is "what does this instance hold."

Instance role	Policy	Failure mode you're accepting	Alert on
Page/query cache	`allkeys-lru` (or `allkeys-lfu` for skewed access)	Cache misses, extra DB load	Hit ratio drop
Sessions / auth	`noeviction` + TTL on every key	Writes fail loudly at the ceiling	`evicted_keys > 0`, OOM errors
Job queue / broker	`noeviction`	New enqueues rejected; existing jobs safe	Any OOM error, memory > 70%
Rate limit counters	`volatile-lru`, short TTLs	Some limits reset early (fails open)	Memory trend only
Locks / idempotency keys	`noeviction`	Lock acquisition fails instead of double-granting	`evicted_keys > 0`

noeviction feels scary because it turns a memory problem into visible request failures. That's the point: an OOM error is a page you can respond to, while an evicted lock key is a duplicate charge a customer tells you about. Sidekiq's documentation has recommended noeviction for its Redis for years for exactly this reason.

Enforce it at boot rather than trusting a runbook. This has caught a mis-restored parameter group for me more than once:

import Redis from 'ioredis'

const cache = new Redis(process.env.REDIS_CACHE_URL)      // evictable
const durable = new Redis(process.env.REDIS_DURABLE_URL)  // sessions, queue, locks

export async function assertRedisPolicies() {
  const [, policy] = await durable.config('GET', 'maxmemory-policy')
  if (policy !== 'noeviction') {
    throw new Error(
      `durable Redis has maxmemory-policy=${policy}; sessions and jobs can be silently evicted`
    )
  }
}

Fail the deploy on that, and the class of bug disappears instead of getting rediscovered next year.

Is a second instance worth the cost?

For most small teams, yes, and it's often cheaper than the upsize you were about to buy. Cache is the part that wants headroom; sessions, locks, and queue metadata for a modest app are megabytes, so the durable node can stay tiny.

If you want a Redis-compatible store where eviction isn't a correctness risk at all, Amazon MemoryDB is the one that treats memory as a cache over a durable multi-AZ transaction log rather than as the only copy of your data — it costs meaningfully more than ElastiCache per node, and it inherits the same single-threaded hot-key ceiling, so it solves durability, not throughput. If you'd rather not pay for a second always-on node, Upstash bills per request instead of per instance, which makes creating one database per role effectively free — the tradeoff is that a chatty cache client becomes a line item, so you watch command volume there instead of memory. And if you moved to Valkey after the Redis licensing change, note that it kept the same eviction semantics and INFO fields, so both the bug and every diagnostic above migrate with you unchanged.

Takeaway: the second instance isn't about capacity, it's about giving eviction a blast radius you chose on purpose.

FAQ

Why are my Redis keys disappearing before the TTL expires?
Almost always eviction under maxmemory pressure. Run redis-cli INFO stats | grep evicted_keys — if that number is climbing, Redis is deleting keys to stay under its memory limit, and an allkeys-* policy lets it take keys that have plenty of TTL left.

Is it safe to use the same Redis for caching and sessions?
Only if the policy is noeviction, and then your cache stops absorbing growth gracefully and starts failing writes instead. Separate instances are the safe answer; separate logical databases (SELECT 1) are not, because all databases on one instance share the same memory budget and eviction policy.

What is the best maxmemory-policy for Redis?
There is no single best one — it depends on whether losing a key is acceptable. Use allkeys-lru for pure caches, and noeviction for sessions, job queues, locks, and idempotency keys where a missing key changes program behavior rather than just costing you a recompute.

Bottom line

If users report random logouts or jobs that vanish without a trace, check evicted_keys and maxmemory-policy before you audit a single line of auth code. Caches should be evictable and everything whose absence changes correctness should not be, which in practice means two Redis instances with two policies and a boot-time assertion that nobody has quietly changed them. Keep noeviction on the durable side even though it converts memory pressure into loud request failures — loud is the feature. Bigger instances only move the date this happens.

Plausible vs Umami vs PostHog: When Self-Hosting Analytics Stops Being Cheaper

Libme — Wed, 30 Sep 2026 15:27:08 +0000

If all you need is pageviews, referrers, and top pages, self-hosting Umami on a small VPS is genuinely cheap and stays cheap. If you need session replay, funnels, and feature flags, self-hosting PostHog is the most expensive "free" decision on this list. Plausible sits in between: pleasant to run, but it drags a ClickHouse instance along with it, and that's the part people underestimate.

I've run all three — Umami and Plausible self-hosted, PostHog on their cloud — on side projects and a small internal product. What follows is the decision I'd hand a colleague, including the failure mode that costs people a week before they notice.

What are you actually buying when you pay for analytics?

Not the dashboard. The dashboard is the easy part, which is why there are a dozen good open-source ones.

You're paying for storage and retention of an append-only event stream, an ingestion endpoint that survives your traffic spikes, and — the underrated one — somebody else owning the upgrade path of a columnar database you don't want to learn. Analytics data is write-heavy, immutable, and queried with wide aggregations, which is exactly the workload that makes people reach for ClickHouse and exactly the workload that makes a single Postgres box feel fine right up until it doesn't.

The honest framing: hosted analytics is a bet that your time is worth more than the subscription, and self-hosting is a bet that it isn't.

Plausible, Umami, PostHog: what is each one actually for?

	Umami	Plausible	PostHog
Core job	Lightweight web analytics	Web analytics + goals/funnels	Product analytics suite
Data store (self-host)	Postgres or MySQL	Postgres and ClickHouse	ClickHouse, Kafka, Redis, Postgres
License (self-host)	MIT	AGPL (Community Edition)	MIT core, some features source-available
Session replay	No	No	Yes
Feature flags / experiments	No	No	Yes
Realistic self-host floor	One small VPS	One mid-size VPS	Not a small-VPS workload
Cloud pricing model (as of Sep 2026)	Event/site tiers, free tier	Pageview tiers	Usage-based per product, generous free tier

The shape of that table is the whole decision. Umami answers "what pages do people read." Plausible answers that plus "did they do the thing I care about." PostHog answers "why did this cohort drop off at step three, and what happens if I flip this flag for 10% of them."

If you want the lightest possible self-hosted option that runs next to your app on the database you already operate, Umami is the one that doesn't add a second data store to your stack.

Drawbacks, because every tool gets one:

Umami: the reporting is deliberately shallow. No funnel debugging worth the name, and once your events table gets large on Postgres you will be the one adding indexes and thinking about partitioning.
Plausible: the Community Edition is intentionally behind the hosted product on some features, and self-hosting means you now operate ClickHouse — including its memory appetite and its own backup story.
PostHog: enormous surface area, and as of mid-2026 their self-host guidance steers small teams toward the hobby Docker deployment rather than a supported production cluster. Session replay also makes your event volume — and your bill — grow in a way pageview counting never does.

Takeaway: pick the tool by which question you're going to ask at 11pm, not by which dashboard looks nicest in the screenshots.

Why does my analytics show fewer visits than my server logs?

This is the first real bug everyone hits, and it isn't a bug.

Client-side analytics scripts are blocked by content blockers, and the block rate skews hard by audience — a developer-tools site loses far more than a recipe blog. Your access logs count requests; your analytics counts successfully executed JavaScript. Those numbers will never match, and if they do match exactly, something is double-counting.

The partial fix is to serve the script and the event endpoint from your own domain so they aren't third-party requests. For Plausible on Next.js:

// next.config.js
module.exports = {
  async rewrites() {
    return [
      {
        source: '/stats/js/script.js',
        destination: 'https://plausible.io/js/script.js',
      },
      {
        source: '/stats/api/event',
        destination: 'https://plausible.io/api/event',
      },
    ];
  },
};

<script
  defer
  data-domain="example.com"
  data-api="/stats/api/event"
  src="/stats/js/script.js"
></script>

The data-api attribute is the part people forget. Proxy the script but leave the event endpoint pointing at the vendor and you've solved nothing — the beacon is still the third-party request that gets blocked.

If you terminate TLS yourself, the Caddy equivalent:

example.com {
    handle /stats/js/script.js {
        rewrite * /js/script.js
        reverse_proxy https://plausible.io {
            header_up Host plausible.io
        }
    }

    handle /stats/api/event {
        rewrite * /api/event
        reverse_proxy https://plausible.io {
            header_up Host plausible.io
            header_up X-Forwarded-For {remote_host}
        }
    }

    reverse_proxy localhost:3000
}

Here's the failure mode that wastes the week. If you proxy the event endpoint without forwarding the real client IP, every visitor arrives from your proxy's address. Analytics tools that derive country and the daily visitor hash from the client IP will then report your entire audience as one country and collapse distinct visitors together. The dashboard doesn't error — it just quietly goes wrong, and it looks like a traffic change rather than a config change. If your geography chart flattens to a single row on the day you shipped a proxy, that's your bug, not your users.

The same rule applies when self-hosting behind nginx or a CDN: the analytics container must see a trustworthy X-Forwarded-For, and it must be configured to trust that header from your proxy only.

Takeaway: proxying analytics recovers blocked traffic, but it moves client IP handling into your infrastructure — and every downstream metric that depends on IP silently degrades if you get it wrong.

What does self-hosting actually cost?

Not the VPS. The VPS is the cheap part.

Memory, not CPU. ClickHouse is happy to use whatever RAM you give it, and a Plausible or PostHog stack on an undersized box gets OOM-killed during ingestion bursts, not during queries. Budget for headroom you aren't using yet.
Backups you've actually restored. An analytics database nobody backs up is a decision, just an unstated one. Restoring ClickHouse is not the same procedure as restoring Postgres, and the time to learn it is not the morning after.
Upgrades. Minor version bumps are fine. The painful ones are the migrations that change the event schema; those arrive on the maintainer's schedule, not yours.
Retention. Hosted plans often trim raw event retention on lower tiers. Self-hosting gives you unlimited retention and the corresponding unlimited disk growth — set a TTL on day one instead of discovering it at 90% full.

Rough rule from running these: a self-hosted analytics stack costs a few hours to stand up, then a few hours a year, until the day it breaks — and then it costs whatever your afternoon is worth, at the least convenient moment.

Takeaway: self-hosting analytics isn't free, it's prepaid in hours, and you pay in a currency you can't budget in advance.

When is the paid cloud tier worth it?

Situation	What I'd do
Side project, pageviews only	Self-host Umami, one VPS, done
Client sites you bill for	Hosted tier — you're selling reliability, not operating it
Privacy/compliance is the reason you're here	Self-host Plausible CE, budget for ClickHouse
You need funnels, replay, and flags	PostHog Cloud, and cap your event volume early
Traffic is spiky and unpredictable	Hosted — burst ingestion is the hardest part to self-run

The tipping point isn't traffic volume, it's whether losing a week of data would matter. Hobby projects can absorb a gap; anything you report to someone else can't.

If you want product analytics without operating a multi-service data pipeline, PostHog Cloud is the option that gives you funnels, replay, and feature flags behind one SDK — with the tradeoff that event volume, not user count, drives what you pay.

Takeaway: pay when the data becomes evidence someone else relies on.

FAQ

Is self-hosted Plausible free?
The Community Edition is free software under the AGPL, but running it requires both Postgres and ClickHouse, plus backups and upgrades you perform yourself. It's free of licensing cost, not free of operating cost.

Why does Google Analytics show different numbers than Plausible or Umami?
They count different things. GA4 models sessions and applies its own filtering and modeling; privacy-focused tools typically count a visitor via a daily rotating hash with no cross-day identity. Neither number is wrong — they answer different questions, so never compare them directly.

Can I self-host PostHog on a single small server?
You can run the hobby Docker deployment for evaluation, but as of mid-2026 PostHog directs production users to its cloud rather than supporting large self-hosted clusters. If you're choosing PostHog for its full feature set, plan on the hosted version.

Bottom line

Choose Umami if you want pageviews on infrastructure you already run, and accept shallow reporting in exchange. Choose Plausible when goals and conversions matter and you want a clean, privacy-respecting dashboard — self-host it if compliance demands it, otherwise let them operate the ClickHouse. Choose PostHog when your real question is about product behavior rather than traffic, and pay for the cloud tier instead of assembling the pipeline yourself. Whichever you pick, proxy the script through your own domain on day one and verify your geography chart afterward — that single check catches the most common silent misconfiguration in this entire category.

Scrubbing Test Data Without Nulling It Out: The Reserved Ranges That Are Safe to Seed

Libme — Wed, 30 Sep 2026 06:42:58 +0000

If your scrub script sets phone = NULL and email = NULL, every feature that depends on those fields silently stops existing in your test environment — SMS opt-in, required-field validation, notification fan-out. The fix is not random generation, which can dial a stranger. Standards bodies and telecom regulators have set aside ranges that are syntactically valid but guaranteed unroutable, and writing those into your scrubbed data keeps the code paths alive while making delivery impossible.

A commenter on an earlier post made this point about phone numbers, and it generalizes to almost every PII column you touch.

What actually breaks when you NULL a column?

Three distinct failures, and only the first one is obvious.

Validation paths never execute. Your checkout form requires a phone number. In the scrubbed environment nobody has one, so the edit form renders a blank required field, the E.164 normalizer is never called, and the branch that handles "user has a number but it failed verification" is unreachable. You ship the bug that only fires for users who do have data.

Downstream code gets a type it did not plan for. A NULL phone reaches your SMS client and you get TypeError: Cannot read properties of null (reading 'replace') in your formatter — or, if the null makes it to the provider, a rejection like Twilio's 21211 Invalid 'To' Phone Number. Neither error tells you anything about the feature you were testing.

Randomly generated data is worse than NULL. The moment someone points a test environment at live credentials — a misconfigured secret, a provider client that defaults to production when an env var is missing — randomly generated digits are routable digits. Random data fails open: it reaches a real person. Reserved data fails closed.

The takeaway: NULL removes the code path, random data removes the safety, and reserved ranges remove neither.

Strategy	Shape is valid	Exercises the code path	Safe if creds leak	Deterministic
`NULL` everything	No	No	Yes	Yes
Random digits/strings	Usually	Yes	No	Only if seeded
Scrubbed production values	Yes	Yes	No	Yes
Reserved ranges	Yes	Yes	Yes	Yes, if hashed from the PK

Which ranges are actually reserved?

These are set aside by the relevant authority specifically so that fiction and documentation cannot collide with a real subscriber or host.

Data	Reserved range	Set aside by
Phone, North America	`555-0100` to `555-0199` in any area code	NANPA, for fictional use
Phone, UK mobile	`07700 900000`–`900999`	Ofcom drama range
Phone, UK landline	`020 7946 0000`–`0999`, `01632 960000`–`960999`	Ofcom drama range
Email domain	`example.com`, `example.net`, `example.org`	RFC 2606
Hostname / TLD	`.test`, `.example`, `.invalid`, `.localhost`	RFC 2606
IPv4	`192.0.2.0/24`, `198.51.100.0/24`, `203.0.113.0/24`	RFC 5737 (documentation)
IPv6	`2001:db8::/32`	RFC 3849 (documentation)
Card numbers	Your gateway's published test numbers, sandbox only	Payment provider docs

Two details that bite people. The whole 555 prefix is not reserved — only 555-0100 through 555-0199; 555-1212 is real directory assistance in much of the NANP. And other regulators publish their own drama ranges, but the specific blocks differ by country, so look them up at the regulator rather than pattern-matching from the US ones.

For email, example.com is safe to write but mail to it goes nowhere, which means you cannot open a signup confirmation. If your tests need to read the message, keep the reserved domain in the database and point your provider at a mail-capture inbox in the preview environment's config — separate concerns, separate places.

The takeaway: reserved ranges are valid enough to pass your validators and dead enough that no carrier or DNS resolver will deliver anything.

How do I generate them deterministically?

Hash the primary key. Same input, same fake value, every run — so a re-scrub does not churn your test fixtures, and the same user looks identical in users, invoices, and your search index.

-- Run only on a machine allowed to see production data.
CREATE OR REPLACE FUNCTION fake_phone_e164(seed text) RETURNS text AS $$
  SELECT '+1' || npa || '555' || lpad((100 + (h % 100))::text, 4, '0')
  FROM (
    SELECT h,
           (ARRAY['202','212','312','415','503','617','713','808'])
             [1 + ((h / 100) % 8)] AS npa
    FROM (SELECT ('x' || substr(md5(seed), 1, 6))::bit(24)::int AS h) x
  ) y;
$$ LANGUAGE sql IMMUTABLE;

UPDATE users SET
  phone      = fake_phone_e164(id::text),
  email      = 'user-' || substr(md5(id::text), 1, 12) || '@example.com',
  full_name  = 'Test User ' || substr(md5(id::text), 1, 6),
  last_ip    = ('203.0.113.' || (1 + (('x' || substr(md5(id::text), 1, 4))::bit(16)::int % 250)))::inet;

bit(24)::int is always positive, which saves you from the abs() overflow footgun on the more common bit(32)::int version of this trick.

Keep the shape of the data, not just the type. If 20% of your users have no phone in production, preserve that: CASE WHEN phone IS NULL THEN NULL ELSE fake_phone_e164(id::text) END. A column where every row is populated tests a world you do not ship to.

The takeaway: derive fakes from the primary key so the scrub is idempotent and referentially consistent across tables.

How do I prove the scrub actually worked?

Assert it in the same transaction, and let the assertion fail the job. A scrub that half-ran is more dangerous than no scrub, because everyone downstream assumes it completed.

DO $$
DECLARE leaked int;
BEGIN
  SELECT count(*) INTO leaked FROM users
   WHERE phone IS NOT NULL AND phone !~ '^\+1\d{3}55501\d{2}$';
  IF leaked > 0 THEN
    RAISE EXCEPTION 'scrub incomplete: % rows outside reserved phone range', leaked;
  END IF;

  SELECT count(*) INTO leaked FROM users
   WHERE email IS NOT NULL AND email !~* '@example\.(com|net|org)$';
  IF leaked > 0 THEN
    RAISE EXCEPTION 'scrub incomplete: % rows with non-reserved email domain', leaked;
  END IF;
END $$;

Run this as the last step of the export job, and again as the first step of the import job on the untrusted side. The second run is the one that catches a snapshot that was copied from the wrong bucket.

If you want the managed version of this, PostgreSQL Anonymizer is the open-source extension that lets you declare masking rules as security labels on the columns themselves, so the rule lives next to the schema instead of in a script that drifts. Its cost is real: it is a C extension, so it has to be available on your managed Postgres provider, and the rules need database-level privileges to manage. Neosync covers similar ground as a service with referential-integrity-aware transformers if you would rather not run an extension at all, with the usual tradeoff of sending your schema to a third party.

The takeaway: a scrub without a machine-checked assertion is a hope, and hopes do not fail CI.

What about unique constraints and volume?

This is where the approach has a hard edge. The North American reserved block gives you exactly 100 numbers per area code. Eight area codes is 800 distinct values; every valid NPA is on the order of tens of thousands. If users.phone carries a unique index and you have more rows than that, the scrub will fail on a duplicate key.

Do not widen the range to fix that. The right response is to shrink the dataset: a preview or test database needs hundreds of representative rows, not your full user table. Subset first, scrub second, and the ceiling stops mattering. If you genuinely need a full-size dataset for load testing, drop the unique constraint in that environment explicitly and write down why — as of late 2026 there is no reserved range large enough to satisfy a million-row unique phone column, and pretending otherwise means routable numbers.

The takeaway: running out of reserved values is a signal that your test dataset is too big, not that the range is too small.

FAQ

What phone numbers are safe to use for test data?
In North America, 555-0100 through 555-0199 in any area code are reserved by NANPA for fictional use and will not connect to a subscriber. In the UK, Ofcom reserves 07700 900000–900999 for mobile and 020 7946 0000–0999 for London landlines. Other prefixes, including the rest of 555, can be real.

Can I send test emails to example.com?
You can safely store @example.com addresses — RFC 2606 reserves the domain so nobody can register it or receive mail there. But nothing will be delivered, so if your test needs to open the email, route the preview environment's SMTP to a mail-capture inbox instead of relying on the address.

How do I keep fake data consistent across tables?
Generate every fake value from a hash of the row's primary key rather than from a random source. The same user ID produces the same phone number and email on every run and in every table that references it, which makes the scrub idempotent.

Bottom line

Null-based scrubbing is the default because it is one line of SQL, and it quietly deletes the test coverage you thought you had. Write reserved-range values instead: they pass validation, exercise the same code paths as real data, and cannot reach a human even if someone wires the environment to live credentials by mistake. Derive them deterministically from the primary key, assert the result with a query that can fail the job, and subset your data before you scrub it so unique constraints never force you outside the reserved block. If you would rather declare the rules than maintain a script, PostgreSQL Anonymizer is the sane starting point on self-managed Postgres.

CORS Errors Only in Production: Preflight, Credentials, and the Cache That Hides Your Fix

Libme — Tue, 29 Sep 2026 15:26:02 +0000

If your frontend talks to your API fine on localhost and dies with a CORS error in production, the cause is almost never "CORS is broken." It is that locally the browser saw one origin (a dev-server proxy) and in production it sees two, and the first cross-origin request your app makes is now a preflight OPTIONS that your backend answers wrong — often with a 401, a redirect, or a 500 that carries no CORS headers at all. Fix it in this order: read the actual preflight response, decide where CORS belongs (proxy, app, or gateway), then make sure your CDN is not caching the answer for the wrong origin.

Why does it work locally and fail in production?

In development, Vite, Next.js, and Create React App all proxy /api to your backend. The browser sees http://localhost:5173/api/items — same origin, no CORS at all. In production you deploy the frontend to https://app.example.com and the API to https://api.example.com, and every request is suddenly cross-origin.

Cross-origin does not automatically mean preflight. The browser sends a preflight only when the request is not "simple": a method other than GET, HEAD, or POST, a Content-Type outside application/x-www-form-urlencoded, multipart/form-data, and text/plain, or any header you set yourself. That is why the failure often looks arbitrary: your GET /health works, and POST /v1/items with Content-Type: application/json and an Authorization header fails. The JSON content type alone is enough to trigger the preflight.

The first thing to establish is not "is CORS configured" but "did this request preflight at all" — those are two different bugs with two different fixes.

How do I read the real error instead of guessing?

Chrome's console messages are specific, and each one points at a different header:

Console message (abridged)	What it actually means
`No 'Access-Control-Allow-Origin' header is present`	The response never went through your CORS layer — wrong route, error path, or middleware order
`Response to preflight request doesn't pass access control check`	The `OPTIONS` response is the problem, not your `POST`
`Request header field authorization is not allowed by Access-Control-Allow-Headers`	Preflight answered, but the allow-list is missing a header you send
`Method PATCH is not allowed by Access-Control-Allow-Methods`	Same, for the method
`...must not be the wildcard '*' when the request's credentials mode is 'include'`	You are sending cookies; `*` is illegal in that mode
`Redirect is not allowed for a preflight request`	Your `OPTIONS` got a `301`/`302` (usually HTTP→HTTPS or a trailing-slash rule)

Then reproduce the preflight by hand. This is the single most useful command in the whole investigation, because it shows you exactly what the browser sees without the browser in the way:

curl -i -X OPTIONS https://api.example.com/v1/items \
  -H 'Origin: https://app.example.com' \
  -H 'Access-Control-Request-Method: POST' \
  -H 'Access-Control-Request-Headers: content-type,authorization'

You want 204 (or 200) plus Access-Control-Allow-Origin, Access-Control-Allow-Methods, and Access-Control-Allow-Headers covering what you asked for. What I usually find instead is a 401: the preflight carries no cookies and no Authorization header by design, so any auth middleware mounted before the CORS layer rejects it. The browser then reports a generic CORS failure and never sends the real request.

A curl -i -X OPTIONS that returns 401 or 302 is your answer — stop reading CORS docs and fix middleware order or redirect rules.

Where should CORS live: proxy, app, or gateway?

This is the decision that actually matters, and most teams make it by accident.

Approach	Good when	Real drawback
Same-origin path routing (`/api/*` → backend at the edge)	You control DNS and want CORS to disappear entirely	Another hop to operate; cookie `Domain`/`Path` and cache rules need review
CORS in the app (framework middleware)	Small number of services, origins known at deploy time	Every service re-implements it; error responses easily bypass it
CORS at the gateway	Many services behind one entry point	Two places can emit the headers, and duplicated `Access-Control-Allow-Origin` is itself a hard failure

Same-origin routing is underrated. If your frontend already sits behind a CDN, forwarding app.example.com/api/* to the API removes preflights, third-party cookie problems, and the entire Allow-Headers allow-list in one move. If you want that without running a reverse proxy yourself, Cloudflare Workers can sit in front of the existing hostname and forward /api/* to the backend, which keeps the browser on one origin. The cost is real: you now own a routing layer, and you have to be deliberate about what it caches.

If you keep CORS in the app, mount it first and make sure it also wraps error handlers. In Express:

const express = require('express')
const cors = require('cors')

const app = express()
const allowed = new Set(['https://app.example.com', 'https://staging.example.com'])

app.use(cors({
  origin: (origin, cb) => {
    // no Origin header: curl, server-to-server, same-origin
    if (!origin || allowed.has(origin)) return cb(null, true)
    return cb(new Error(`origin not allowed: ${origin}`))
  },
  credentials: true,
  methods: ['GET', 'POST', 'PATCH', 'DELETE'],
  allowedHeaders: ['Content-Type', 'Authorization'],
  maxAge: 600,
}))

app.use(express.json())
app.post('/v1/items', (req, res) => res.status(201).json({ ok: true }))

app.listen(3000)

Two notes from getting this wrong. The cors middleware already answers OPTIONS when mounted with app.use, so a separate catch-all OPTIONS route is unnecessary — and on Express 5 a bare app.options('*', ...) no longer behaves as it did on 4, because the path-matching rules changed. Second, if auth middleware comes before this block, you are back to the 401 preflight.

In FastAPI the equivalent is CORSMiddleware, and the trap is different: passing allow_origins=["*"] together with allow_credentials=True is invalid per spec, and frameworks resolve that contradiction inconsistently rather than erroring.

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware

app = FastAPI()
app.add_middleware(
    CORSMiddleware,
    allow_origins=["https://app.example.com"],
    allow_credentials=True,
    allow_methods=["GET", "POST", "PATCH", "DELETE"],
    allow_headers=["Content-Type", "Authorization"],
    max_age=600,
)

Never trust the config object — trust the curl -i output, because the header your framework actually emits is the only thing the browser evaluates.

What changes the moment cookies are involved?

Once the frontend sends credentials: 'include', three rules bind at once: Access-Control-Allow-Origin must be an exact origin (no *), Access-Control-Allow-Credentials: true must be present, and wildcards in Allow-Headers or Allow-Methods stop counting. A cookie crossing sites also needs SameSite=None; Secure — so the "CORS error" may be a cookie that was never sent.

The fastest way to tell them apart: if the preflight passes and the real request returns 401, CORS is fine and the cookie is the problem. Check the Network tab's request headers for Cookie before touching any CORS config.

Why does the fix work for one user and not another?

Because something cached it. If you echo the request's Origin into Access-Control-Allow-Origin — which you must do for credentialed requests — the response varies by a request header, and any shared cache in front of you needs to know that:

Access-Control-Allow-Origin: https://app.example.com
Access-Control-Allow-Credentials: true
Vary: Origin

Without Vary: Origin, a CDN can serve app.example.com's allow-header to staging.example.com and back, and the symptom is the worst kind: intermittent, user-specific, unreproducible on your machine. I have also watched a correct fix look like a non-fix because a stale preflight was still cached — browsers cache preflight results per Access-Control-Max-Age, and Chromium caps that at two hours (as of mid-2026). Test fixes in a fresh incognito window, or keep maxAge low until things are stable.

Any response whose CORS headers depend on the request's Origin must send Vary: Origin, or your cache will eventually hand the wrong answer to the wrong site.

FAQ

Why does my API work in Postman but not in the browser?
Postman and curl are not browsers and do not enforce CORS — they never send a preflight and never check response headers. A request succeeding in Postman tells you the endpoint works; it tells you nothing about CORS.

Can I fix a CORS error from the frontend?
No. CORS headers come from the server that owns the resource, so the only frontend-side "fixes" are avoiding the cross-origin request entirely (proxy the call through your own origin) or, for third-party APIs you do not control, calling them from your backend.

Why does my preflight return 401?
Preflight OPTIONS requests deliberately carry no cookies and no Authorization header, so authentication middleware rejects them unless CORS handling runs first. Mount your CORS layer before auth, or exempt OPTIONS from authentication.

Bottom line

Debug in this order: confirm whether a preflight is involved, reproduce it with curl -i -X OPTIONS, and only then edit configuration. If you control both hostnames, same-origin path routing at the edge is the most durable fix, because it deletes the problem class instead of configuring it. If you keep CORS in the app, mount it before auth, list your origins explicitly, and send Vary: Origin. And if the preflight passes but the real request 401s, you have a cookie problem wearing a CORS costume.

Docker Image Works on Your Mac but Fails on the Server: Fixing exec format error for Good

Libme — Mon, 28 Sep 2026 15:30:22 +0000

If your container runs fine locally on an Apple Silicon Mac and dies on the server with exec /usr/local/bin/app: exec format error, you built an arm64 image and deployed it to an amd64 host. The fix is not a flag on docker run — you need a multi-architecture image, built either by cross-compiling inside the Dockerfile or by building each architecture on its own native machine and merging the results into one manifest. Emulation via QEMU will also work, and it is the slowest option by a wide margin on compile-heavy images.

Why does `exec format error` only show up on the server?

docker build produces an image for the architecture of whatever machine ran the build, unless you tell it otherwise. On an M-series Mac that is linux/arm64. Your EC2 box, your Hetzner VPS, most CI runners, and most managed container platforms are still linux/amd64. The kernel on that host loads your ELF binary, reads a machine type it cannot execute, and returns ENOEXEC. Docker surfaces that as exec format error, which reads like a corrupt binary and is actually a passport problem.

You can confirm the mismatch in about ten seconds:

# What did I actually build?
docker image inspect myapp:latest --format '{{.Os}}/{{.Architecture}}'

# What architectures does the pushed tag serve?
docker buildx imagetools inspect ghcr.io/me/myapp:latest

If the second command lists only one platform, every host that isn't that platform is one deploy away from the same error. The related failure you'll hit on the way is no matching manifest for linux/amd64 in the manifest list entries — that is the same problem caught earlier, at pull time instead of exec time, which is strictly better.

There's also a quieter variant. Run an amd64 image on an Apple Silicon Mac and Docker prints WARNING: The requested image's platform (linux/amd64) does not match the detected host platform and then runs it anyway under emulation. That warning is the one people learn to scroll past, and it is exactly the signal that your local and production architectures have drifted apart.

Takeaway: exec format error is never a corrupted binary — it means the image's architecture and the host's architecture disagree, and one imagetools inspect proves which side is wrong.

What are the three ways to build for both architectures?

Approach	arm64 + amd64 build speed	Setup cost	Works with CGO / native deps	Best when
QEMU emulation on one runner	Slowest; the emulated half dominates the build	One extra action, no Dockerfile changes	Yes, but painfully slow	Interpreted apps, images you rebuild rarely
Cross-compile in the Dockerfile	Fast — no emulation at all	Dockerfile rework	No (needs a cross toolchain)	Go and Rust services
Native runner per architecture, then merge manifests	Fast, and both halves run in parallel	Two jobs plus a merge job	Yes	Anything, including Python wheels and node-gyp

A fourth option is to rent someone else's native builders. If you want a managed version of the two-runner setup without maintaining the matrix yourself, Depot runs native amd64 and arm64 builders behind a single docker build call and keeps the layer cache on persistent volumes between runs; the trade-off is that your build context now leaves your CI provider for a third party, which some compliance reviews will care about. Docker Build Cloud is the equivalent from Docker itself and has the advantage of reusing your existing Docker Hub identity, with the caveat that its pricing is metered in build minutes, so a repo with a noisy push trigger can spend more than you expect.

Takeaway: emulation is the cheapest to configure and the most expensive to run, so treat it as a stopgap rather than a resting place.

How do I set up buildx and QEMU correctly?

The single-runner path needs the QEMU binfmt handlers registered and a buildx builder that uses the docker-container driver — the default docker driver cannot build more than one platform at a time:

docker run --privileged --rm tonistiigi/binfmt --install all
docker buildx create --name multi --driver docker-container --use
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t ghcr.io/me/myapp:1.4.0 \
  --push .

The thing that trips people up here: you cannot --load a multi-platform build into your local image store. Docker's classic image store holds one manifest per tag, so --load with two platforms fails outright. Either push to a registry (as above), or build a single platform locally for testing with --platform linux/amd64 --load. As of mid-2026 the containerd image store in Docker Desktop removes this limitation, but it's an opt-in setting, so don't write a team runbook that assumes it.

In GitHub Actions the equivalent is two setup steps before the build:

- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
- uses: docker/build-push-action@v6
  with:
    platforms: linux/amd64,linux/arm64
    push: true
    tags: ghcr.io/${{ github.repository }}:latest

Takeaway: --platform with two values and --load are mutually exclusive on the classic image store — push to a registry, or build one platform at a time locally.

How do I stop the emulated half from dominating build time?

For compiled languages, skip emulation entirely. BuildKit hands you BUILDPLATFORM and TARGETARCH; pin the build stage to the native platform and let the compiler do the cross-targeting:

# syntax=docker/dockerfile:1
FROM --platform=$BUILDPLATFORM golang:1.24-alpine AS build
ARG TARGETOS TARGETARCH
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
    go build -o /out/app ./cmd/app

FROM alpine:3.20
COPY --from=build /out/app /usr/local/bin/app
ENTRYPOINT ["/usr/local/bin/app"]

Every stage now runs natively; only the tiny final COPY differs per architecture. The honest limitation is CGO_ENABLED=0: the moment you need cgo, sqlite bindings, or any C library, you're back to installing a cross toolchain, and that is usually more work than the third option below.

For everything else — Python with compiled wheels, Node with native addons, anything with a make install that assumes a real compiler — build each architecture on a machine that actually is that architecture, then stitch the digests together. As of mid-2026 GitHub offers hosted Linux arm64 runners under labels like ubuntu-24.04-arm, which makes this pattern available without self-hosting:

jobs:
  build:
    strategy:
      matrix:
        include:
          - platform: linux/amd64
            runner: ubuntu-24.04
          - platform: linux/arm64
            runner: ubuntu-24.04-arm
    runs-on: ${{ matrix.runner }}
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: build
        uses: docker/build-push-action@v6
        with:
          platforms: ${{ matrix.platform }}
          outputs: type=image,name=ghcr.io/${{ github.repository }},push-by-digest=true,name-canonical=true,push=true
      - run: |
          mkdir -p /tmp/digests
          touch "/tmp/digests/${DIGEST#sha256:}"
        env:
          DIGEST: ${{ steps.build.outputs.digest }}
      - uses: actions/upload-artifact@v4
        with:
          name: digests-${{ strategy.job-index }}
          path: /tmp/digests/*

  merge:
    needs: build
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/download-artifact@v4
        with:
          path: /tmp/digests
          pattern: digests-*
          merge-multiple: true
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - working-directory: /tmp/digests
        run: |
          docker buildx imagetools create \
            -t ghcr.io/${{ github.repository }}:latest \
            $(printf 'ghcr.io/${{ github.repository }}@sha256:%s ' *)

The two build jobs run in parallel, so wall-clock time is roughly the slower of the two native builds rather than the sum of a native build and an emulated one. What tripped me up the first time: the merge job needs its own registry login, because imagetools create reads the source digests from the registry rather than from any local state.

Takeaway: parallel native builds plus a manifest merge is the only approach that is both fast and language-agnostic, and it costs you one extra CI job.

Which one should you actually pick?

Start from what breaks if you choose wrong. If your image is a Go or Rust binary, cross-compiling in the Dockerfile is the smallest change with the biggest win, and you can do it in one commit. If your build installs native dependencies, go straight to the matrix-plus-merge workflow — you will fight the cross toolchain for longer than it takes to write the merge job. Reach for emulation when the image is a thin layer over an interpreter and you rebuild it a few times a week; the build being slow simply won't matter at that frequency.

And regardless of approach, make the mismatch impossible to ship. Adding a pull-time architecture assertion to your deploy script catches the problem before a container ever restarts in a loop:

set -euo pipefail
want="linux/amd64"
docker buildx imagetools inspect "$IMAGE" | grep -q "$want" \
  || { echo "image $IMAGE has no $want variant"; exit 1; }

Takeaway: pick the approach by whether your build needs a C compiler, then verify the published manifest in CI so architecture drift fails the pipeline instead of the deploy.

FAQ

What does exec /usr/local/bin/app: exec format error mean in Docker?
It means the binary inside the image was compiled for a different CPU architecture than the host running the container — almost always an arm64 image built on an Apple Silicon Mac being run on an amd64 server. Rebuild the image with docker buildx build --platform linux/amd64 or publish a multi-architecture manifest covering both.

Can I build a multi-platform Docker image and load it into my local Docker?
Not with the classic image store: docker buildx build --platform linux/amd64,linux/arm64 --load fails because a local tag can only point at one manifest. Push to a registry instead, or build a single platform with --load for local testing. Docker Desktop's containerd image store lifts this restriction, but it is opt-in as of mid-2026.

Is --platform linux/amd64 on an M1 Mac enough for production?
It produces a correct amd64 image, but it runs your whole build under QEMU emulation, which is dramatically slower for anything that compiles code and occasionally exposes emulator bugs in native toolchains. It's fine as a one-off; for CI, build each architecture on a native runner and merge the manifests.

Preview Environments Per Pull Request: How to Seed Them Without Cloning Production

Libme — Sun, 27 Sep 2026 15:29:30 +0000

Spinning up a preview deploy per pull request is the easy half — every modern host does it with a config flag. The hard half is the database: a shared staging database turns previews into a queue of people breaking each other's data, and a restored production snapshot puts real customer records in an environment that anyone with the PR link can reach. The pattern that survives is an ephemeral database per PR, created from schema plus a small deterministic seed, with production data used only after it has been scrubbed on a machine that is allowed to see it.

What actually breaks in a preview environment?

The deploy almost never fails. What fails looks like this:

Everything points at one staging database. Two PRs land migrations in different orders and the third developer gets ERROR: relation "order_line_items" does not exist from a branch that never ran the migration adding it — or someone's test loop truncates a table another person was demoing from.

Connection exhaustion. Each preview app opens its own pool. Ten open PRs at ten connections each, against a small instance with a 100-connection ceiling, gives you FATAL: sorry, too many clients already in whichever environment connected last — usually the one the reviewer just opened. Previews multiply idle connections rather than traffic, which is why they surface pooling problems first.

Migrations that only run forward. Preview infrastructure will create a database for you; it will not un-apply your migration when you force-push a rewritten one. If your runner tracks applied versions in the database, recreate the environment rather than reusing it.

Seed scripts that assume an empty database. Run them twice on a reused environment and you get duplicate key value violates unique constraint "users_email_key". Write every seed as an upsert (INSERT ... ON CONFLICT DO NOTHING) and you stop caring which state you started from.

Secrets. Preview builds inherit project-level environment variables by default on most hosts — that is how a preview ends up holding a live payment key. Give the preview scope its own secret set, test-mode credentials only.

The takeaway: a preview environment is only as isolated as its database and its secrets, and both default to shared.

Where should the preview database come from?

Four options, and they differ mostly in how fast they are and how much real data they expose.

Approach	Setup cost	Time to fresh env	Data realism	Main risk
Shared staging DB	None	Instant	Medium	Cross-PR interference, migration drift
Ephemeral Postgres container + seed	Low	Seconds	Low (what you seed)	Seed rot: diverges from real schema use
Branch on managed Postgres (copy-on-write)	Low	Seconds	High (mirrors parent)	Real data in a low-trust environment
Restore of a scrubbed prod snapshot	High	Minutes	High	Scrub pipeline must be airtight

For most small teams the ephemeral container plus a seed file is the correct default, and it is free. You reach for branching when the bugs you keep shipping are data-shaped — queries that are fast on 200 seeded rows and hopeless on the real distribution.

If you want copy-on-write branches without building them, Neon is the managed Postgres whose branching is the product itself: a branch is a cheap pointer at the parent's storage, so a per-PR database costs you the diff rather than a full copy. Supabase offers git-linked preview branches if your app already lives on its stack, with the tradeoff that the branch is a whole project and carries that startup latency. As of mid-2026, treat either as a pre-production surface no matter how convenient it is — a branch of production is production data.

The takeaway: pick branching for data realism, containers for isolation and cost, and never a shared database for either.

How do you seed data without copying production?

Build the seed from three layers, in this order.

1. Schema from your migration runner, never from a dump. The whole point is to exercise the migrations the PR contains.

2. A committed deterministic seed. Fixed UUIDs and fixed timestamps, so a failing test is reproducible and reviewers can bookmark /orders/0000...0001.

-- seed/01_core.sql — idempotent, safe to re-run
INSERT INTO users (id, email, display_name, created_at) VALUES
  ('00000000-0000-0000-0000-000000000001', 'owner@example.test',  'Test Owner',  '2026-01-02T00:00:00Z'),
  ('00000000-0000-0000-0000-000000000002', 'viewer@example.test', 'Test Viewer', '2026-01-02T00:00:00Z')
ON CONFLICT (id) DO NOTHING;

INSERT INTO orders (id, user_id, status, total_cents, created_at)
SELECT
  ('00000000-0000-0000-0000-0000000100' || lpad(n::text, 2, '0'))::uuid,
  '00000000-0000-0000-0000-000000000001',
  (ARRAY['pending','paid','refunded'])[1 + (n % 3)],
  1000 + n * 37,
  timestamptz '2026-01-03T00:00:00Z' + (n || ' hours')::interval
FROM generate_series(1, 50) AS n
ON CONFLICT (id) DO NOTHING;

Use a reserved test domain like example.test for every address. Seeds leak into outbound email eventually, and a domain that cannot resolve is the cheapest guardrail there is.

3. Volume, generated rather than copied. If the PR touches a query, 50 rows will lie to you. generate_series into the hot table buys realistic row counts and index behavior without a single real record:

INSERT INTO events (user_id, kind, payload, created_at)
SELECT
  '00000000-0000-0000-0000-000000000001',
  'page_view',
  jsonb_build_object('path', '/p/' || (n % 500)),
  now() - (n || ' minutes')::interval
FROM generate_series(1, 200000) AS n;
ANALYZE events;

That ANALYZE matters. Freshly bulk-loaded tables have no statistics, and the planner will pick a plan that has nothing to do with what production does.

When you genuinely need production shapes, scrub inside the trusted environment and ship only the output — never pg_dump production to a laptop first:

#!/usr/bin/env bash
set -euo pipefail

pg_dump "$PROD_URL" --no-owner --no-privileges \
  --exclude-table-data='audit_log' \
  --exclude-table-data='sessions' \
  -Fc -f /tmp/raw.dump

pg_restore -d "$SCRUB_URL" --no-owner --clean --if-exists /tmp/raw.dump
psql "$SCRUB_URL" -v ON_ERROR_STOP=1 -f scrub.sql
pg_dump "$SCRUB_URL" --no-owner -Fc -f /tmp/preview-seed.dump

-- scrub.sql: irreversible, and verified by a test
UPDATE users SET
  email        = 'user' || id || '@example.test',
  display_name = 'User ' || left(md5(id::text), 6),
  phone        = NULL;
DELETE FROM payment_methods;

The scrub needs a test in the same job that fails loudly — a query asserting zero rows still match your real email domain. A scrub nobody verifies is a scrub that stopped matching the schema three migrations ago. If you would rather not hand-roll the rules, PostgreSQL Anonymizer is the extension that lets you declare masking rules on columns, so they live with the schema instead of in a script that drifts.

The takeaway: generate volume, scrub only where you must, and assert the scrub worked in CI rather than trusting it.

Wiring it into CI without leaving databases behind

The failure mode here is cost, not correctness: ephemeral environments are only ephemeral if something deletes them. Tie creation and teardown to the same PR lifecycle events.

name: preview-db
on:
  pull_request:
    types: [opened, synchronize, reopened, closed]

concurrency:
  group: preview-db-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  up:
    if: github.event.action != 'closed'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Create or reset the PR database
        env:
          ADMIN_URL: ${{ secrets.PREVIEW_ADMIN_URL }}
          DB: pr_${{ github.event.pull_request.number }}
        run: |
          psql "$ADMIN_URL" -v ON_ERROR_STOP=1 \
            -c "DROP DATABASE IF EXISTS \"$DB\" WITH (FORCE)" \
            -c "CREATE DATABASE \"$DB\""
      - run: ./scripts/migrate.sh
        env:
          DATABASE_URL: ${{ secrets.PREVIEW_BASE_URL }}/pr_${{ github.event.pull_request.number }}
      - run: psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f seed/01_core.sql
        env:
          DATABASE_URL: ${{ secrets.PREVIEW_BASE_URL }}/pr_${{ github.event.pull_request.number }}

  down:
    if: github.event.action == 'closed'
    runs-on: ubuntu-latest
    steps:
      - name: Drop the PR database
        env:
          ADMIN_URL: ${{ secrets.PREVIEW_ADMIN_URL }}
        run: |
          psql "$ADMIN_URL" -v ON_ERROR_STOP=1 \
            -c "DROP DATABASE IF EXISTS \"pr_${{ github.event.pull_request.number }}\" WITH (FORCE)"

WITH (FORCE) (Postgres 13+) is what makes the drop reliable — without it, one leftover connection leaves you with database "pr_412" is being accessed by other users and a database that lives forever. Add a scheduled job that drops any pr_* database whose PR is closed, because the closed event does get missed.

On the app side, Vercel's preview deployments are the least-effort way to get a URL per PR for a frontend, while Render's preview environments declared in render.yaml cover the case where the PR needs a long-running backend process. Both still expect you to supply the database strategy above; neither isolates your data for you.

The takeaway: every create path needs a matching delete path plus a sweeper, because webhook-driven teardown is not reliable enough to be the only one.

What about webhooks and OAuth callbacks that can't reach a preview URL?

Third-party services cannot deliver to an unpredictable per-PR hostname, and OAuth providers reject unregistered redirect URIs. Best first: register one stable proxy URL that routes to the right preview by header or path segment; use the provider's CLI forwarding where it exists; or stub the integration and exercise the real thing only in staging. Do not register thirty redirect URIs — that list becomes permanent.

When is paid branching worth it?

The honest baseline is a small always-on Postgres instance plus the CI job above: a few dollars a month and an afternoon of setup. Branching starts paying when your data has enough real-world skew that seeds stop predicting production, when the seed job lands on the critical path of every review, or when reviewers are non-engineers who need plausible data to judge a UI.

The takeaway: pay for branching to buy data realism, not to skip the seed file — you need the seed file either way.

FAQ

Can I just use a copy of production for preview environments?
Only after scrubbing, and only if the scrub runs inside the environment that is already allowed to hold production data. A preview URL is typically reachable by anyone with the link and protected by far weaker controls than production, so unscrubbed copies turn every open PR into an additional place a breach can start.

How do I give each pull request its own database on Postgres?
Create a database named after the PR number in a CI job triggered by pull_request, run your migrations against it, apply an idempotent seed, and drop it on the closed event with DROP DATABASE ... WITH (FORCE). Add a scheduled sweeper for the PRs whose close event got lost, otherwise abandoned databases accumulate.

Why does my preview environment run out of database connections?
Each preview app holds its own connection pool, so idle previews consume connections even with no traffic. Either put a pooler in front of the instance and let previews connect through it in transaction mode, or cap preview pools to one or two connections each — previews serve one reviewer, not production traffic.

Bottom line

With a handful of open PRs, run an ephemeral database per PR with a committed idempotent seed plus a generate_series block for volume: free, isolated, and it exercises your migrations on every push. If seeded data keeps failing to predict production, move to copy-on-write branching on a managed Postgres and apply production-level access controls to those branches. If you need production shapes without production risk, build the scrub pipeline once, run it inside the trusted environment, and test the scrub in CI. Whichever you pick, write the teardown job before the create job — the environments that cost real money are the ones nobody remembers to delete.

Why Page 500 Is Slow and Shows Duplicate Rows: Offset vs Keyset Pagination

Libme — Sat, 26 Sep 2026 15:10:23 +0000

LIMIT 20 OFFSET 10000 asks the database to produce 10,020 rows in sort order and throw 10,000 of them away, so the deeper a user scrolls the more work each request does. Worse, offsets describe a position in a result set that keeps changing — if rows are inserted or deleted between two requests, the reader sees the same item twice or never sees it at all. Keyset pagination (also called cursor or seek pagination) filters on the last row you sent instead of counting rows, which makes every page cost the same and makes the sequence stable.

This is the kind of bug that never shows up in development, because your seed data has 200 rows and nobody inserts anything while you click.

What actually happens when you say OFFSET 10000?

There is no "skip ahead" primitive in a B-tree index scan. Postgres (and MySQL, and SQLite) implements OFFSET by fetching rows in order and discarding them until the count is satisfied. You can see it in the plan — the Limit node reports the rows it emitted, but the node underneath it has already produced offset + limit rows:

EXPLAIN (ANALYZE, BUFFERS)
SELECT id, title, created_at
FROM posts
WHERE feed_id = 42
ORDER BY created_at DESC, id DESC
LIMIT 20 OFFSET 10000;

Look at actual rows on the child node and at the Buffers: shared hit/read counts. Both scale with the offset, not with the page size. That is the whole problem: page 1 and page 500 are not the same query in disguise, they are a cheap query and an expensive one that happen to share syntax.

If the sort column isn't indexed it's worse — the planner sorts the entire filtered set before it can discard anything, and deep pages can tip into a disk-based sort (Sort Method: external merge). But even with a perfect index, offset cost grows linearly with depth.

The takeaway: an offset is not a bookmark, it's an instruction to re-walk the list from the beginning every time.

The bug that isn't slowness: duplicated and skipped rows

Slowness gets noticed. This one gets filed as "the API is flaky."

A client reads page 1 (OFFSET 0 LIMIT 20) of a feed sorted newest-first. Before it requests page 2, three new rows are inserted. Now the row that was at index 19 has shifted to index 22 — so OFFSET 20 returns rows the client already displayed. Deletions produce the mirror image: rows shift up, and items slide past the window unseen.

For an infinite-scroll feed this shows up as duplicate cards in the list. For a background job that pages through a table to export or reprocess it, the same mechanism silently skips records, which is much more expensive to discover. I have chased "missing rows" in a nightly export that turned out to be nothing but OFFSET racing against concurrent inserts.

The takeaway: if anything writes to the table while a client is paging through it, offset pagination is not just slow — it is incorrect.

How do I write a keyset pagination query in Postgres?

Keyset pagination replaces "skip N rows" with "give me the rows after this one." You need a sort key that is totally ordered — that means adding a unique tiebreaker to whatever the user actually sorts by, because created_at alone will have ties and rows will be dropped or repeated at page boundaries.

-- page 1
SELECT id, title, created_at
FROM posts
WHERE feed_id = 42
ORDER BY created_at DESC, id DESC
LIMIT 20;

-- page N+1: pass the last row's (created_at, id) back in
SELECT id, title, created_at
FROM posts
WHERE feed_id = 42
  AND (created_at, id) < ($1, $2)
ORDER BY created_at DESC, id DESC
LIMIT 20;

The row-constructor comparison (created_at, id) < ($1, $2) is the important part. It is not the same as created_at < $1 AND id < $2 (that drops rows), and unlike the hand-expanded created_at < $1 OR (created_at = $1 AND id < $2), Postgres can turn the row comparison into a single index seek on a composite index:

CREATE INDEX posts_feed_created_id_idx
  ON posts (feed_id, created_at DESC, id DESC);

Now every page is the same cost: seek to a position in the index, read 20 entries, stop. Page 500 costs what page 1 costs.

Two constraints to check before you ship this:

All sort columns must point the same direction. Row comparison assumes a single ordering. ORDER BY created_at DESC, id ASC can't be expressed as (created_at, id) < (...); either make the directions agree or write the expanded OR form and accept a worse plan.
NULLs break it. A nullable sort column with NULLS LAST won't compare the way you expect. Sort on a NOT NULL column, or on a COALESCE(...) expression that you also index.

The takeaway: keyset pagination is a WHERE clause that mirrors your ORDER BY exactly — the moment they diverge, rows go missing at page boundaries.

How should the cursor be encoded in the API?

Don't expose ?created_at=...&last_id=... as separate query parameters. Clients will start constructing them by hand, and then you can never change the sort key. Encode the position into one opaque token:

// cursor.js
const encode = (row) =>
  Buffer.from(JSON.stringify({ t: row.created_at.toISOString(), i: row.id }))
    .toString('base64url');

const decode = (cursor) => {
  const { t, i } = JSON.parse(Buffer.from(cursor, 'base64url').toString());
  if (typeof t !== 'string' || !Number.isInteger(i)) throw new Error('bad cursor');
  return { t, i };
};

async function listPosts(db, feedId, cursor, limit = 20) {
  const after = cursor ? decode(cursor) : null;
  const rows = after
    ? (await db.query(
        `SELECT id, title, created_at FROM posts
         WHERE feed_id = $1 AND (created_at, id) < ($2, $3)
         ORDER BY created_at DESC, id DESC LIMIT $4`,
        [feedId, after.t, after.i, limit + 1])).rows
    : (await db.query(
        `SELECT id, title, created_at FROM posts
         WHERE feed_id = $1
         ORDER BY created_at DESC, id DESC LIMIT $2`,
        [feedId, limit + 1])).rows;

  const hasMore = rows.length > limit;
  const page = rows.slice(0, limit);
  return { items: page, nextCursor: hasMore && page.length ? encode(page.at(-1)) : null };
}

Two details worth copying: fetching limit + 1 rows is how you answer "is there a next page" without a second COUNT(*), and validating the decoded shape matters because a cursor is user-controlled input that goes into a WHERE clause. Base64 is encoding, not protection — if the sort key is something you'd rather not leak, sign the cursor or store it server-side.

The takeaway: an opaque cursor is a version boundary — it lets you change the sort key later without breaking every client that saved a URL.

When is OFFSET still the right call?

Keyset pagination cannot jump to an arbitrary page, because page 47 has no meaning without reading pages 1 through 46. If your UI has numbered page buttons over an admin table of a few thousand rows that nobody is writing to, OFFSET is fine and simpler.

	Offset/limit	Keyset/cursor
Cost of deep pages	Grows with depth	Flat
Stable under concurrent writes	No — duplicates and skips	Yes
Jump to page N	Yes	No
Total page count	Easy (`COUNT(*)`)	Needs a separate estimate
Arbitrary user-chosen sort	Any column	Needs an index per sort order
Implementation cost	Trivial	Cursor encode/decode + composite index

Total counts are the real trade. COUNT(*) on a large filtered set is its own performance problem, so most feeds that switch to cursors also drop the exact total and show "load more" instead. If you need a number, an approximate count from planner statistics is usually enough for a UI hint — just never present an estimate as an exact figure.

Framework support, as of mid-2026: Django REST Framework ships CursorPagination out of the box, and it is the one that gets you a correct opaque cursor without writing SQL — at the price of requiring a stable ordering field and giving you no total count. In the Node/TypeScript world, Prisma's cursor + take arguments implement the same idea, and the awkward part is that the cursor must be a unique field, so a compound "sort by date, break ties by id" ordering still needs care. If your list is served from a search engine rather than a relational database, Elasticsearch's search_after is the equivalent primitive, and it needs a tiebreaker field plus a point-in-time to stay consistent across pages.

The takeaway: choose offset for bounded, browsable, mostly-static tables; choose keyset for anything that grows or that a machine pages through.

FAQ

Why does OFFSET get slower on higher page numbers?
Because the database has to generate and discard every skipped row in sort order before it can return your page. OFFSET 10000 LIMIT 20 reads 10,020 rows internally. The cost grows linearly with the offset, so the last pages of a long list are the most expensive queries in your system.

Does keyset pagination require a unique column?
It requires a sort key that is unique as a whole. Sorting by a non-unique column like created_at is fine as long as you append a unique tiebreaker (id) to both the ORDER BY and the cursor comparison. Without the tiebreaker, rows sharing a timestamp at a page boundary get duplicated or skipped.

Can I still show a total page count with cursor pagination?
Not cheaply. You can run a separate COUNT(*) with the same filters, but that query scans the matching rows and often costs more than the page itself. Most cursor-paginated APIs replace the page count with a "has more" boolean derived from fetching one extra row.

Bottom line

If your endpoint backs an infinite scroll, a public feed, or any job that walks a table while other processes write to it, switch to keyset pagination — the correctness argument is stronger than the performance one, because offset-based paging genuinely loses rows under concurrent inserts. Keep offset for admin tables with page-number UI, bounded row counts, and low write traffic. The migration is usually one composite index, one row-constructor WHERE clause, and an opaque cursor in the response; the thing you give up is jumping to an arbitrary page, so confirm the UI can live with "load more" before you start.

PgBouncer vs Supavisor vs RDS Proxy: Which Postgres Pooler Survives Transaction Mode?

Libme — Fri, 25 Sep 2026 23:38:35 +0000

Picking a Postgres connection pooler is really picking which session features you are willing to lose. All three of these poolers give you the same win — hundreds of app connections multiplexed onto a few dozen backend connections — and all three do it by taking your connection away between transactions, which is exactly what breaks prepared statements, session SETs, LISTEN/NOTIFY and cross-transaction advisory locks. PgBouncer gives you the most control and the most operational work, Supavisor is the reasonable pick if you're already on Supabase or want a clustered pooler you don't hand-tune, and RDS Proxy is the one to choose when IAM auth and failover handling matter more than pooling efficiency.

Why does adding a pooler break my app?

The failure that sends you looking for a pooler is familiar:

FATAL: sorry, too many clients already

Postgres allocates a backend process per connection, so max_connections is a real ceiling, and serverless functions or a scaled-out app blow through it. A pooler fixes that by keeping a small set of server connections and lending them out.

The catch is when it takes the connection back. In session pooling, a client holds one server connection until it disconnects — safe, but it barely helps, because your idle app connections still pin backends. In transaction pooling, the server connection returns to the pool at COMMIT, so 500 clients can share 20 backends. That's the mode everyone actually wants, and it's the mode that changes the semantics of your database connection.

Your driver notices first:

prepared statement "S_1" already exists

or, with asyncpg:

prepared statement "__asyncpg_stmt_3__" does not exist

Two different clients landed on the same server connection, and the second one's prepared statement cache is now describing a different backend than it thinks. The dangerous version of this bug is the one that throws no error at all: pg_advisory_lock() taken outside a transaction, released to the pool mid-flight, and silently held by a connection your next request never gets back. If your code assumes anything persists between transactions on the same connection, transaction pooling will break it — sometimes loudly, sometimes not.

What exactly breaks in transaction mode?

Feature	Transaction mode behavior	Fix
Protocol-level named prepared statements	Collide across clients unless the pooler tracks them	PgBouncer `max_prepared_statements` (1.21+), or disable in the driver
`SET` outside a transaction	Leaks to the next client on that server connection	Use `SET LOCAL`, or set it in the connection string / server defaults
`pg_advisory_lock()` (session-scoped)	Held by an unrelated connection; may never release	Use `pg_advisory_xact_lock()`
`LISTEN` / `NOTIFY`	Listener silently loses its connection	Dedicated direct connection, not through the pooler
Temp tables, `WITH HOLD` cursors	Disappear after `COMMIT`	Keep it inside one transaction, or use session mode
Long transactions	Hold a backend for their whole duration	Fix the transaction; the pooler can't help

The driver-side escape hatch is usually one flag. psycopg3:

import psycopg

# Through a transaction-mode pooler: don't let psycopg promote
# frequently-used queries to named prepared statements.
conn = psycopg.connect(
    "postgresql://app@pooler.internal:6432/app",
    prepare_threshold=None,
)

asyncpg needs both the statement cache off and unique statement names disabled:

import asyncpg

conn = await asyncpg.connect(
    "postgresql://app@pooler.internal:6432/app",
    statement_cache_size=0,
)

You pay for this: unprepared queries re-plan on every execution. For short OLTP queries the cost is small; for a complex analytical query in a hot path it isn't. That trade is the reason PgBouncer's prepared-statement support matters.

Before you compare poolers, audit your own code for session state — most "the pooler is broken" tickets are an app holding state the pooler never promised to keep.

PgBouncer: most control, most of it yours to operate

PgBouncer is the default answer because it's small, battle-tested and runs anywhere. Since 1.21 it tracks protocol-level named prepared statements in transaction mode, which removes the single most common driver breakage:

[databases]
app = host=db.internal port=5432 dbname=app

[pgbouncer]
pool_mode = transaction
max_client_conn = 2000
default_pool_size = 20
max_prepared_statements = 200
server_reset_query = DISCARD ALL

Watch the pool from the admin console rather than guessing:

-- psql "postgresql://pgbouncer@pooler.internal:6432/pgbouncer"
SHOW POOLS;
-- cl_waiting > 0 consistently means default_pool_size is too small
-- or transactions are running too long
SHOW STATS;

cl_waiting is the number that tells you the truth. If it's persistently above zero, clients are queuing on the pooler and your p99 now includes pool wait time that your database metrics will never show you.

The honest drawback: PgBouncer is essentially single-threaded, so one process saturates a core under heavy traffic and you scale it by running several processes with so_reuseport or putting instances behind a load balancer — plus you now own a network hop that can page you at 3am. If you want that control and are willing to run it, PgBouncer is the pooler with the fewest surprises about what it's doing to your connections.

Supavisor: a clustered pooler you don't hand-tune

Supavisor is Supabase's open-source pooler, written in Elixir, designed to be multi-tenant and to run as a cluster rather than as a single process per host. It exposes transaction mode and session mode on separate ports, so the usual pattern is app traffic on the transaction port and migrations or LISTEN-style work on the session port. It has added named prepared statement support in transaction mode, but treat that as version-dependent and verify it with your actual driver before relying on it.

If you're on Supabase, this is not really a choice — the platform's pooler endpoints are Supavisor, and the decision collapses to "which port". Outside Supabase, it's a credible self-hosted option when you want horizontal scaling without assembling it yourself from PgBouncer processes. Supavisor is the option worth considering when you want a pooler that scales across nodes instead of one you shard by hand.

The drawback is ecosystem maturity: PgBouncer has a decade more field reports, and when something strange happens at 2am, the number of existing answers matters.

RDS Proxy: what are you actually paying for?

RDS Proxy is the managed option for RDS and Aurora, and pooling is arguably not its best feature. What you're buying is IAM authentication, credentials pulled from Secrets Manager instead of your app config, and connection handling across failovers — the proxy holds client connections while the underlying instance fails over, which shortens the error window your app sees.

Its pooling has a specific gotcha: pinning. When RDS Proxy sees session state it can't safely share, it stops multiplexing and dedicates a backend connection to that client for the rest of the session. Your pooler quietly becomes a passthrough. Check it directly:

CloudWatch → RDS → Proxy metrics
  DatabaseConnectionsCurrentlySessionPinned
  DatabaseConnectionsBorrowLatency

If the pinned count tracks your client count, you're paying for a proxy that isn't pooling. As of mid-2026, AWS bills RDS Proxy per vCPU-hour of the underlying database instance rather than per connection or per request, with a floor for small instances — check the current pricing page, but note the shape: cost scales with your database size, not your traffic, so a small instance with bursty Lambda traffic is where it looks best and a large instance with modest connection counts is where it looks worst. RDS Proxy is the right call when IAM auth and failover behavior are requirements, not when raw pooling efficiency is the goal.

Which one should you run?

	PgBouncer	Supavisor	RDS Proxy
Operate it yourself	Yes	Yes (or managed on Supabase)	No
Named prepared statements in transaction mode	Yes, 1.21+	Yes, version-dependent	Pinning risk
Scaling model	Multiple processes / instances	Clustered	Managed by AWS
Auth integration	Postgres auth, auth_query	Postgres auth	IAM + Secrets Manager
Failover handling	You handle it	You handle it	Built in
Cost shape	Instance you run	Instance you run, or platform	Per vCPU-hour of the DB instance
Best when	You want control and predictability	You're on Supabase, or want a clustered pooler	You're on RDS/Aurora and need IAM + failover

FAQ

What's the difference between session pooling and transaction pooling in PgBouncer?
Session pooling assigns a server connection to a client until it disconnects, preserving all session state but providing little multiplexing. Transaction pooling returns the server connection to the pool after each COMMIT, which is what lets hundreds of clients share a few dozen backends, but it breaks anything that depends on session state — prepared statements, session SETs, LISTEN/NOTIFY and session-level advisory locks.

Why do I get "prepared statement already exists" through PgBouncer?
Your driver created a named prepared statement on one server connection, then landed on a different one for the next query. Fix it by enabling max_prepared_statements in PgBouncer 1.21 or later so the pooler tracks statements per server connection, or by disabling named prepared statements in the driver (prepare_threshold=None in psycopg3, statement_cache_size=0 in asyncpg).

Does RDS Proxy replace PgBouncer?
Only partly. RDS Proxy pools connections and adds IAM authentication and failover handling, but it pins connections whenever it detects session state it can't share, which can eliminate the pooling benefit entirely. Watch the DatabaseConnectionsCurrentlySessionPinned CloudWatch metric before assuming it's pooling.

Bottom line

If you run your own Postgres and want the most predictable behavior, run PgBouncer in transaction mode with max_prepared_statements set, and watch SHOW POOLS for cl_waiting. If you're on Supabase, use Supavisor's transaction port for app traffic and the session port for migrations and listeners — that decision is already made for you. If you're on RDS or Aurora and your real requirements are IAM auth and clean failovers, RDS Proxy earns its cost, provided you check the pinned-connection metric and fix whatever session state is causing it. Whichever you pick, do the app-side audit first: SET LOCAL instead of SET, pg_advisory_xact_lock() instead of pg_advisory_lock(), and a direct connection for anything that listens.

Uptime Monitor Says 100% While Users See Errors: What to Check, and When Paid Monitoring Is Worth It

Libme — Thu, 24 Sep 2026 15:58:55 +0000

An uptime monitor that reports 100% during an outage is almost always measuring the wrong thing: a CDN-cached page, a health endpoint that returns 200 whether or not the database answers, or a check that only looks at the status code. The fix is two health endpoints (shallow for the load balancer, deep for the external monitor), Cache-Control: no-store on both, and an assertion on the response body. Paid monitoring is worth it once your customers notice outages before you do — what you are buying is a tighter check interval, multi-region confirmation, and a way to page a human.

Why did the monitor say "up" when the site was down?

The incident that made me rewrite my monitoring: a customer emailed a screenshot of a 500 page while the uptime dashboard showed a flat green line. The app had been throwing ECONNREFUSED on every database call for about forty minutes.

The monitor was hitting https://example.com/, which sat behind a CDN with a long cache TTL. The edge kept serving cached HTML while the origin was on fire: status 200, fast response, green line. The monitor was accurately reporting that the CDN was up.

Three patterns cause almost every "100% uptime during an outage" report:

The checked URL is cached. The CDN answers instead of your origin. curl -sI https://example.com/ | grep -iE 'age|cache|x-cache' will show a non-zero Age or a HIT header if this is happening to you.
The health endpoint is shallow. /health returns {"status":"ok"} as long as the process is alive. Process alive, database gone, still 200.
The check only asserts on the status code. Some frameworks render an error page with a 200. Some maintenance pages do too. A status-code-only check cannot tell the difference between "the app works" and "the app produced HTML."

An uptime check is only as honest as the endpoint it hits, and a cached 200 is the most common lie.

How should a health endpoint be built so a monitor can trust it?

You need two endpoints with different jobs, because the load balancer and the external monitor are asking different questions.

The load balancer asks "should I keep routing to this instance?" That check must be shallow. If it includes the database, one database blip makes every instance fail readiness at the same moment, the load balancer pulls all of them, and a thirty-second hiccup becomes a full outage.

The external monitor asks "can a user actually get served?" That check should be deep: touch the database, touch the cache, and fail loudly with a 503 if any of them do not answer within a tight timeout. Here is the shape I use, in Express with pg and redis (ESM, so top-level await works):

import express from "express";
import pg from "pg";
import { createClient } from "redis";

const app = express();
const db = new pg.Pool({ connectionString: process.env.DATABASE_URL });
const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

function withTimeout(promise, ms, name) {
  let timer;
  const timeout = new Promise((_, reject) => {
    timer = setTimeout(
      () => reject(new Error(`${name} timed out after ${ms}ms`)),
      ms
    );
  });
  return Promise.race([promise, timeout]).finally(() => clearTimeout(timer));
}

// Shallow: "is this process alive?" — for the load balancer only.
app.get("/health", (req, res) => {
  res.set("Cache-Control", "no-store");
  res.json({ status: "ok" });
});

// Deep: "can this instance serve a real request?" — for the external monitor.
app.get("/health/deep", async (req, res) => {
  res.set("Cache-Control", "no-store");
  const checks = {
    postgres: withTimeout(db.query("SELECT 1"), 2000, "postgres"),
    redis: withTimeout(redis.ping(), 1000, "redis"),
  };

  const results = {};
  let healthy = true;
  for (const [name, check] of Object.entries(checks)) {
    try {
      await check;
      results[name] = "ok";
    } catch (err) {
      results[name] = err.message;
      healthy = false;
    }
  }

  res
    .status(healthy ? 200 : 503)
    .json({ status: healthy ? "ok" : "degraded", checks: results });
});

app.listen(3000);

Three details matter more than the rest. The timeouts are short and explicit, because a health check that hangs for thirty seconds on a dead connection pool is worse than one that fails fast. Cache-Control: no-store goes on both endpoints, and you then verify the CDN honors it — some edge rules cache regardless of origin headers, in which case you route /health/* around the cache entirely. And the deep check covers only dependencies you own. Put a third-party API in it and their outage pages you at 3 a.m. for something you cannot fix.

Then configure the monitor to hit /health/deep and assert on the body, not just the code. Every serious uptime tool supports a keyword or JSON match; use it to require "status":"ok" in the response. That single assertion closes the gap where an error page with a 200 slips through.

The load balancer gets the shallow check and the monitor gets the deep one; giving both the same endpoint is how a database blip turns into a total outage.

What are you actually paying for with an uptime monitoring subscription?

Free tiers are generous enough that the paid decision comes down to a handful of capabilities. As of September 2026, pricing models cluster around monitor counts and check intervals, but the numbers move often enough that you should read the current plan page rather than trust any post.

What you pay for	Free tier reality	Why it matters
Check interval	Every few minutes	Interval is your worst-case detection delay; a 3-minute outage can fit inside a 5-minute gap
Multi-region confirmation	Usually one location	One vantage point produces false alarms and misses regional DNS or routing failures
Body / JSON assertions	Often limited	Without them you are back to status-code-only checks
Browser (synthetic) checks	Rare	The only way to verify "login and checkout still work," not just "the URL answers"
On-call routing	Email only	Phone or SMS escalation is what wakes someone up at night
Status page	Basic	Cuts support email during an incident

The hidden costs are the ones that scale. Per-monitor pricing feels cheap until every microservice and cron job has its own check. SMS and voice alerts are frequently metered separately. Browser checks are typically priced per run, so a Playwright flow every minute from three regions costs meaningfully more than a ping.

The four tools I keep coming back to each own a distinct slice of this problem.

If you just need a free ping every few minutes with email alerts, UptimeRobot is the one most people start with, and for a single side project it is often enough. Its limit is that the free interval is coarse, and assertions and status pages get thin once you need more than the basics.

If you want the managed version of the whole loop, Better Stack is the one that bundles uptime checks with on-call scheduling and a status page so a failed check pages a human without a second tool. The drawback is that you are paying for incident management you may not need yet, and seats plus phone alerting add up for a small team.

If your real question is whether a user can still log in and check out, Checkly is the one that runs Playwright scripts on a schedule from multiple regions and keeps the check definitions as code in your repo. The trade-off is maintenance: browser checks break when your UI changes, and per-run pricing punishes aggressive schedules.

For cron jobs and queue workers that have no URL to hit, Healthchecks.io is the one built around the dead-man's switch: the job pings it on success, and you get alerted when the ping stops arriving. It does not do HTTP monitoring at all, so it complements rather than replaces the others.

Pay for monitoring when the cost of a missed outage exceeds the subscription, and for most small products that line is crossed the first time a customer reports the outage before your monitor does.

When is the free tier genuinely enough?

A free ping monitor is fine when you have one or two public endpoints, a five-minute detection delay is acceptable, email is a reasonable alert channel because nobody is on call anyway, and a deep health endpoint like the one above is what it hits. Solo side projects and internal tools usually qualify.

Move to a paid tier at the first of these signals: paying customers who would churn over an unnoticed outage, a second person who needs to be paged, a purchase flow that can break while the homepage stays up, or an SLA you signed. Fix the endpoint first either way; a paid monitor pointed at a cached homepage costs more and still lies.

The free tier stops being free the day a customer becomes your monitoring system.

FAQ

Why does my uptime monitor show 100% uptime when my site is down?
Because it is checking something other than your application: a CDN-cached page, a shallow health endpoint that returns 200 while the database is unreachable, or a check that only looks at the status code. Point it at a deep health endpoint with Cache-Control: no-store and assert on the response body.

Should a health check endpoint check the database?
Yes for the endpoint your external monitor hits, no for the one your load balancer uses. A deep check with short timeouts tells you users are being served; a shallow liveness check keeps a brief database blip from making the load balancer drop every instance at once.

How often should an uptime monitor check my site?
The check interval is your maximum detection delay. Every five minutes is acceptable for hobby projects; every minute with confirmation from a second region is the baseline once customers depend on the service.

Bottom line

Fix the endpoint before you buy anything: a deep /health/deep with dependency timeouts, no-store headers, and a body assertion in the monitor closes the "100% uptime during an outage" hole on any tool's free tier. Solo projects and internal tools can stay on UptimeRobot's free ping and Healthchecks.io for cron jobs. Teams with paying customers should pay for one-minute multi-region checks and phone escalation, which is where Better Stack fits. If a broken login flow would go unnoticed while the homepage stays green, add Checkly's browser checks for the one or two flows that actually make money.

WebSocket Closes Every 60 Seconds With Code 1006: Finding the Proxy Idle Timeout and Fixing It With Heartbeats

Libme — Wed, 23 Sep 2026 02:51:43 +0000

If your WebSocket or Server-Sent Events connection dies at a suspiciously round interval (60 seconds is the classic, 55 or 30 also show up) and the browser reports close code 1006 with no reason, something between your client and your server has an idle timeout and is cutting the TCP connection when no bytes flow. The fix is not a bigger timeout; it's a server-initiated heartbeat that keeps bytes moving at a shorter interval than the strictest hop, plus a client that reconnects without panicking. This post walks through how to prove which hop is killing you and what the fix looks like on each.

What does close code 1006 actually mean?

1006 is defined by the WebSocket spec as "abnormal closure": the connection ended without either side sending a close frame. Your server did not say goodbye, and neither did the browser. That is precisely the signature of a middlebox dropping the TCP socket: an L7 proxy tracking idle time decides the connection is dead, closes both sides, and neither endpoint gets a WebSocket-level close.

In the browser console it looks like this:

WebSocket connection to 'wss://app.example.com/ws' failed:
WebSocket is closed before the connection is established.

or, more commonly on an already-open socket, an onclose event with event.code === 1006 and event.wasClean === false. On the Node side you often see nothing at all, or a plain ECONNRESET on the underlying socket. Local development works perfectly because there is no proxy between localhost:3000 and your browser.

The takeaway: a 1006 on a regular interval is a timer in the network path, not a bug in your message handling.

Which hop has the timer?

Every layer between the browser and your process can have its own idle clock, and only application bytes reset it. TCP keepalive probes do not help here: they are exchanged between adjacent TCP peers only, and an L7 proxy terminates TCP on both sides, so a keepalive from your server never reaches the load balancer's "was there data?" counter.

The usual suspects, as of mid-2026 (defaults change, so verify against your provider's current docs):

Hop	Default idle behavior	Where to change it
nginx (`proxy_pass`)	Closes if the upstream sends nothing for `proxy_read_timeout` (60s)	`proxy_read_timeout` / `proxy_send_timeout` in the `location` block
ingress-nginx (Kubernetes)	Same 60s defaults, inherited from nginx	`nginx.ingress.kubernetes.io/proxy-read-timeout` annotation
AWS Application Load Balancer	Idle timeout, 60s default, applies to WebSockets	Load balancer attribute `idle_timeout.timeout_seconds`
Google Cloud external HTTP(S) LB	Backend service timeout (30s default) is treated as the maximum lifetime of a WebSocket, idle or not	Backend service `timeoutSec`
Heroku router	Terminates after 55s with no data in either direction	Not configurable; heartbeat is the only option
Cloudflare (proxied)	WebSockets supported; timeouts depend on plan and whether the origin responds	Check the current docs for your plan; heartbeat regardless

The Google Cloud row is the one that surprises people. Raising it to 3600 does not make the problem go away; it just makes your sockets die once an hour instead of every 30 seconds, which is a harder bug to notice and reproduce. Your client must handle reconnection no matter what, and the heartbeat only prevents the idle kills.

To identify the hop quickly: open a socket, send nothing, and time the close. 60s points at nginx or ALB. 55s is Heroku. 30s that ignores your heartbeat entirely is the GCP max-lifetime behavior. If it dies at a different interval after you add a heartbeat, you have found a second timer stacked behind the first.

The takeaway: measure the time-to-death with a silent connection before touching any config, because the interval tells you which layer to look at.

How do you keep the connection alive without relying on the client?

Send the heartbeat from the server. Client-side timers are unreliable: Chrome throttles setInterval in background tabs (intensive throttling aligns timers to once per minute after a tab has been hidden for a while, as of current stable), which is exactly long enough to trip a 60-second proxy timeout. The server has no such constraint and it's the only party that also needs to detect dead clients for cleanup.

With the ws library in Node, the pattern from its own README is a ping every N seconds and a terminate() for any client that did not pong since the last round:

import { WebSocketServer } from 'ws';

const wss = new WebSocketServer({ port: 3000 });

wss.on('connection', (ws) => {
  ws.isAlive = true;
  ws.on('pong', () => { ws.isAlive = true; });
});

const HEARTBEAT_MS = 25_000; // must be < the strictest proxy idle timeout

const interval = setInterval(() => {
  for (const ws of wss.clients) {
    if (ws.isAlive === false) {
      ws.terminate(); // no pong since last tick: assume the peer is gone
      continue;
    }
    ws.isAlive = false;
    ws.ping();
  }
}, HEARTBEAT_MS);

wss.on('close', () => clearInterval(interval));

Two details matter. First, the ping frame counts as application data to every L7 proxy in the path, so it resets each idle clock. Second, terminate() rather than close() on the unresponsive branch: a peer that cannot pong will not complete a close handshake either, and close() would leave a zombie for another full interval.

Pick the interval as roughly half of the shortest timeout you found in the table. 25 seconds is a safe default under a 55–60 second limit; it's also what Socket.IO ships as its pingInterval default, which is why Socket.IO users rarely hit this class of bug until they move to a stricter proxy. If you'd rather not own any of this, Socket.IO is the library that handles heartbeats, reconnection with backoff, and transport fallback out of the box; the cost is a custom protocol on top of WebSockets, so non-JS clients need a Socket.IO client library rather than a plain WebSocket.

For Server-Sent Events the same principle applies, and it's even simpler because SSE has a comment syntax that clients ignore:

// Express handler for an SSE stream
app.get('/events', (req, res) => {
  res.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache',
    'Connection': 'keep-alive',
  });
  res.write('retry: 3000\n\n'); // tell EventSource how long to wait before reconnecting

  const keepalive = setInterval(() => res.write(': keepalive\n\n'), 25_000);
  req.on('close', () => clearInterval(keepalive));
});

A line starting with : is a comment per the SSE spec; EventSource discards it but the proxy sees bytes. Note the retry: field: EventSource reconnects on its own, and this lets you set the delay instead of relying on the browser default.

The takeaway: a server-sent ping every 25 seconds fixes the idle-timeout class of disconnect on every proxy in the table except the one that enforces a maximum lifetime.

Should you raise the proxy timeout instead?

Raise it as well, not instead. Heartbeats solve the idle problem, but a 60-second proxy_read_timeout is also what kills legitimately long silent responses like a slow LLM completion or a large export, so it's worth loosening for the specific WebSocket location:

location /ws {
    proxy_pass http://app_upstream;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
}

Keep the large value scoped to the upgrade path. Applying it globally means a stuck upstream on a normal HTTP request can hold a worker connection for an hour.

The one-line rule: heartbeats are mandatory and portable; timeout increases are a per-environment optimization that you'll forget to carry over to the next hosting provider.

How should the client reconnect?

The browser's WebSocket does not reconnect on its own (EventSource does). A minimal client needs exponential backoff with jitter so that a proxy restart doesn't turn ten thousand clients into a synchronized reconnect storm:

function connect(url, { onMessage }) {
  let attempt = 0;
  let ws;

  function open() {
    ws = new WebSocket(url);
    ws.onopen = () => { attempt = 0; };
    ws.onmessage = (e) => onMessage(JSON.parse(e.data));
    ws.onclose = (e) => {
      const base = Math.min(30_000, 1000 * 2 ** attempt);
      const delay = base / 2 + Math.random() * (base / 2); // jitter
      attempt += 1;
      setTimeout(open, delay);
    };
    ws.onerror = () => ws.close(); // ensure onclose fires and schedules a retry
  }

  open();
  return () => ws && ws.close(1000, 'client shutdown');
}

What this does not solve is missed messages during the gap. If a client must not lose events, you need a resumable stream: either a Last-Event-ID (native to SSE) or a sequence number you send on reconnect so the server can replay from its buffer. That is real work, and it's the point at which a managed service earns its fee. If you want the managed version of this, Ably is the one that handles connection state recovery and message replay across reconnects without you building the buffer; the trade-off is per-message and per-connection pricing that makes chatty, high-fanout workloads expensive compared to a socket you own, so run the math on your peak message rate before committing.

The takeaway: reconnection with jitter is table stakes, and replay after reconnect is where you decide between building a buffer and buying one.

FAQ

Why does my WebSocket close after 60 seconds behind nginx?
Because nginx's proxy_read_timeout defaults to 60 seconds and closes the upstream connection when no data has been transmitted in that window. Send a ping from the server at least every 30 seconds, and raise proxy_read_timeout on the WebSocket location if you also have long silent responses.

Does TCP keepalive prevent WebSocket idle timeouts on a load balancer?
No. TCP keepalive probes only travel between adjacent TCP peers, and an L7 load balancer terminates TCP on each side, so its idle timer only resets on application data. Use WebSocket ping frames or SSE comment lines instead.

What is the difference between WebSocket close code 1006 and 1001?
1001 means one side sent a close frame because it is "going away" (page navigation, server shutdown). 1006 means the connection dropped with no close frame at all, which almost always indicates a proxy, load balancer, or network cut rather than either application.

Bottom line

If you own a WebSocket or SSE endpoint and it runs behind anything other than your own process, ship a server-initiated heartbeat at about 25 seconds and a client that reconnects with jittered backoff; do that before adjusting a single timeout. Then raise the proxy's read timeout on the upgrade path so slow-but-legitimate silences survive, and check whether your load balancer enforces a maximum connection lifetime (Google Cloud does by default) that no heartbeat can defeat. Teams that also need guaranteed delivery across reconnects should either budget the time for a replay buffer or pay a managed realtime provider for one.

DEV Community: Libme

Your LLM Bill Jumped After You Added Context: Find the Cache Miss Before You Downgrade the Model

Why is the bill rising when output length didn't change?

What silently breaks prompt caching?

Where should the cache breakpoint actually go?

When is the batch endpoint worth it?

Should you route to a smaller model?

FAQ

Bottom line

Related reading

Postgres Multi-Tenancy: Row-Level Security, tenant_id Filters, or a Schema per Tenant?

What does "multi-tenancy" actually mean at the storage layer?

How do I set up RLS so it actually enforces anything?

Why does RLS return zero rows (or everyone's rows) in production?

When is schema-per-tenant worth the operational cost?

Does RLS cost performance?

FAQ

Bottom line

Related reading

Users Randomly Logged Out? Your Redis Eviction Policy Is Deleting Sessions

Why does Redis delete keys that still have time left on their TTL?

How do I confirm eviction is what's happening?

What policy should each workload get?

Is a second instance worth the cost?

FAQ

Bottom line

Related reading

Plausible vs Umami vs PostHog: When Self-Hosting Analytics Stops Being Cheaper

What are you actually buying when you pay for analytics?

Plausible, Umami, PostHog: what is each one actually for?

Why does my analytics show fewer visits than my server logs?

What does self-hosting actually cost?

When is the paid cloud tier worth it?

FAQ

Bottom line

Related reading

Scrubbing Test Data Without Nulling It Out: The Reserved Ranges That Are Safe to Seed

What actually breaks when you NULL a column?

Which ranges are actually reserved?

How do I generate them deterministically?

How do I prove the scrub actually worked?

What about unique constraints and volume?

FAQ

Bottom line

Related reading

CORS Errors Only in Production: Preflight, Credentials, and the Cache That Hides Your Fix

Why does it work locally and fail in production?

How do I read the real error instead of guessing?

Where should CORS live: proxy, app, or gateway?

What changes the moment cookies are involved?

Why does the fix work for one user and not another?

FAQ

Bottom line

Related reading

Docker Image Works on Your Mac but Fails on the Server: Fixing exec format error for Good

Why does exec format error only show up on the server?

What are the three ways to build for both architectures?

How do I set up buildx and QEMU correctly?

How do I stop the emulated half from dominating build time?

Which one should you actually pick?

FAQ

Related reading

Preview Environments Per Pull Request: How to Seed Them Without Cloning Production

What actually breaks in a preview environment?

Where should the preview database come from?

How do you seed data without copying production?

Wiring it into CI without leaving databases behind

What about webhooks and OAuth callbacks that can't reach a preview URL?

When is paid branching worth it?

FAQ

Bottom line

Related reading

Why Page 500 Is Slow and Shows Duplicate Rows: Offset vs Keyset Pagination

What actually happens when you say OFFSET 10000?

The bug that isn't slowness: duplicated and skipped rows

How do I write a keyset pagination query in Postgres?

How should the cursor be encoded in the API?

When is OFFSET still the right call?

FAQ

Bottom line

Why does `exec format error` only show up on the server?