DEV Community: Libme

FastAPI Latency Spikes Under Load: How One Blocking Call in an async Endpoint Stalls Every Request

Libme — Sat, 24 Oct 2026 22:08:17 +0000

TL;DR: An async def endpoint runs directly on the event loop. If it calls anything that blocks (time.sleep, requests.get, a sync database driver, heavy CPU work), the loop cannot serve any other request until that call returns. The fix is to either declare the endpoint with plain def so the framework runs it in a thread pool, push the blocking call into a thread with asyncio.to_thread, or switch to a genuinely async client. The symptom that sends people searching is odd: health checks time out and p99 explodes while CPU looks fine.

What does the symptom look like?

The pattern I see over and over: an endpoint is fast in isolation, and a load test or a traffic burst makes unrelated endpoints slow, including /health. Uptime probes fail, the load balancer marks the instance unhealthy, and the instance restarts under load, which makes things worse.

Here is a minimal reproduction. Python 3.11+, with fastapi, uvicorn and httpx installed:

# app.py
import time
from fastapi import FastAPI

app = FastAPI()

@app.get("/health")
async def health():
    return {"ok": True}

@app.get("/slow-async")
async def slow_async():
    time.sleep(0.5)  # blocks the event loop
    return {"done": True}

@app.get("/slow-sync")
def slow_sync():
    time.sleep(0.5)  # runs in a worker thread
    return {"done": True}

Run it with uvicorn app:app and fire ten concurrent requests at each slow route. Against /slow-async, the ten requests are served one after another, so the last one waits roughly ten times 0.5 seconds. That is arithmetic, not a benchmark: a single-threaded loop can only do one blocking call at a time. During that window /health also waits, because it is queued behind them on the same loop.

# hammer.py
import asyncio, time, sys
import httpx

async def main(path: str, n: int = 10):
    async with httpx.AsyncClient(base_url="http://127.0.0.1:8000", timeout=30) as c:
        start = time.perf_counter()
        # fire n slow requests, then probe /health while they are in flight
        slow = [asyncio.create_task(c.get(path)) for _ in range(n)]
        await asyncio.sleep(0.1)
        h0 = time.perf_counter()
        await c.get("/health")
        print(f"/health took {time.perf_counter() - h0:.2f}s while {path} was busy")
        await asyncio.gather(*slow)
        print(f"all {n} done in {time.perf_counter() - start:.2f}s")

asyncio.run(main(sys.argv[1]))

Run python hammer.py /slow-async and then python hammer.py /slow-sync. The first shows /health stuck for seconds; the second answers it almost immediately.

Takeaway: async def is a promise that you never block, and the framework cannot enforce it for you.

Why does `def` work when `async def` doesn't?

FastAPI (via Starlette and AnyIO) treats the two kinds differently. A plain def endpoint is dispatched to a worker thread pool, so a blocking call occupies a thread, not the loop. An async def endpoint is awaited on the loop itself. As of late 2026 the AnyIO thread limiter defaults to 40 concurrent threads, so a def endpoint is not free of limits either: the 41st concurrent blocking request queues.

The common mistake is the well-meaning "everything should be async" refactor. Converting def to async def without converting the calls inside it makes the endpoint worse, because you moved it off the thread pool onto the loop.

Takeaway: converting def to async def is a performance regression unless every call inside it was converted too.

How do I find the blocking call?

Do not guess. Two tools find it quickly.

1. asyncio debug mode. Start the server with PYTHONASYNCIODEBUG=1 uvicorn app:app. The loop logs any callback that holds it longer than slow_callback_duration (0.1 s by default), with the task and the coroutine involved:

Executing <Task pending name='Task-7' coro=<slow_async() running at app.py:13> ...> took 0.501 seconds

That line names the endpoint. Debug mode adds overhead, so use it in staging or a local repro, not on a hot production box.

2. A sampling profiler. If you cannot reproduce locally, py-spy dump --pid <pid> shows where each thread currently is. When the loop thread sits in time.sleep, socket.recv, or a driver's execute, that is your culprit. The honest limitation of py-spy: it needs permission to attach to the process (ptrace on Linux, root on macOS), which is often locked down in containers.

The usual suspects, in the order I find them: requests inside an async handler, a sync ORM session or psycopg2 call, open().read() on large files, json.loads on multi-megabyte payloads, password hashing (bcrypt/argon2 are deliberately slow), and image or PDF processing.

Takeaway: the slow-callback log turns "something is blocking" into a file and line number in one run.

Which fix should I use?

Situation	Fix	Trade-off
Handler is mostly sync code (sync ORM, `requests`)	Change `async def` to `def`	Limited by the thread pool size; no `await` inside
One blocking call inside an otherwise async handler	`await asyncio.to_thread(fn, ...)`	Same thread pool limits; code must be thread-safe
HTTP calls to other services	Use an async client such as `httpx.AsyncClient`	Needs one shared client, not one per request
CPU-bound work (hashing, image resize, big parsing)	Process pool or a job queue	Serialization cost; more moving parts

For the one-call case:

import asyncio
import bcrypt
from fastapi import FastAPI

app = FastAPI()

def _hash(password: bytes) -> bytes:
    return bcrypt.hashpw(password, bcrypt.gensalt())

@app.post("/signup")
async def signup(password: str):
    hashed = await asyncio.to_thread(_hash, password.encode())
    return {"hash_len": len(hashed)}

For outbound HTTP, create the client once at startup rather than per request:

from contextlib import asynccontextmanager
import httpx
from fastapi import FastAPI, Request

@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.http = httpx.AsyncClient(timeout=httpx.Timeout(5.0))
    yield
    await app.state.http.aclose()

app = FastAPI(lifespan=lifespan)

@app.get("/proxy")
async def proxy(request: Request):
    r = await request.app.state.http.get("https://example.com")
    return {"status": r.status_code}

If you want an HTTP client that works in both sync and async code with the same API, httpx is the one I reach for; its drawback is that it is a bit slower than lower-level clients such as aiohttp on very high request rates, and the default timeouts differ from requests, so set them explicitly.

CPU-bound work is the case threads do not rescue: threads share the GIL, so heavy pure-Python computation in to_thread still starves other Python threads (C extensions that release the GIL, such as bcrypt, are fine). Use ProcessPoolExecutor via loop.run_in_executor or hand it to a worker queue.

Takeaway: match the fix to the kind of blocking: threads for I/O and GIL-releasing libraries, processes or queues for pure-Python CPU work.

Can running more workers hide the problem?

Yes, and that is why it survives so long. uvicorn --workers 4 (or Gunicorn with Uvicorn workers) gives you four independent loops, so one stuck loop no longer stalls everything. But four workers each blocked by one slow request is just a smaller version of the same outage. More workers are good capacity planning and bad bug-hiding; fix the block first, then size the workers.

If you want the guard rail rather than the fix, Uvicorn's --timeout-keep-alive and your proxy's upstream timeout will at least turn a silent stall into a visible 504, which is easier to alert on.

Takeaway: extra workers raise the number of requests a blocking call can hurt at once; they do not remove the block.

FAQ

Why is my FastAPI app slow when I use async def?
Almost always because something inside the handler blocks the event loop, such as requests, a sync database driver, or time.sleep. Run with PYTHONASYNCIODEBUG=1 to see which coroutine holds the loop.

Should I use def or async def in FastAPI?
Use async def only when every I/O call inside it is awaited. Use plain def when you depend on sync libraries; FastAPI runs it in a thread pool.

Does asyncio.to_thread make blocking code non-blocking?
It moves the call off the event loop onto a worker thread, so other requests keep being served. It does not speed up the call, and pure-Python CPU work still contends for the GIL.

Bottom line

If you use sync libraries, write plain def endpoints and stop worrying. If you are committed to async def, treat every library call as guilty until proven awaitable, and keep asyncio debug mode in your staging setup so a new blocking dependency gets caught in review rather than at 2 a.m. For pure CPU work, move it out of the web process entirely.

GitHub Push Rejected: "File Exceeds 100 MB" (GH001) — Fix It With filter-repo or LFS

Libme — Fri, 23 Oct 2026 22:26:52 +0000

Deleting the big file in a new commit does not fix this error, because Git still carries the file in an earlier commit and GitHub rejects the whole push. You have to either rewrite history so the blob never existed (git filter-repo), or move the file to Git LFS and rewrite history so old commits point to LFS instead (git lfs migrate). Pick filter-repo if the file should not be in the repo at all, LFS if it should be versioned but is legitimately large.

What does the error actually look like?

remote: error: Trace: 3f1c...
remote: error: See https://gh.io/lfs for more information.
remote: error: File data/embeddings.bin is 143.21 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
 ! [remote rejected] main -> main (pre-receive hook declined)
error: failed to push some refs to 'github.com:you/project.git'

As of late 2026, GitHub prints a warning for files over 50 MB and blocks pushes containing files over 100 MB. The check runs on every blob in the commits you are pushing, not just the tip of your branch. That is the part that surprises people.

Takeaway: GH001 is about every commit in the push, so the fix has to change history, not just the working tree.

Why doesn't `git rm` and a new commit fix it?

I have watched this dead-end happen more than once: the developer runs git rm data/embeddings.bin, commits, pushes, and gets the identical error. The file is deleted from the latest snapshot, but the commit that added it is still an ancestor of your branch, and it is part of what you are pushing.

Other dead ends that look plausible:

git commit --amend only works if the big file is in the very last commit.
Adding it to .gitignore after the fact does nothing for a file already tracked or already committed.
git reset --soft and recommit works for unpushed local history and is actually fine for a small number of commits, but gets tedious fast and loses your commit messages.

First, find out which commit and which path is the problem:

# list the 10 largest blobs anywhere in history
git rev-list --objects --all \
  | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
  | awk '$1 == "blob" {print $3, $4}' \
  | sort -rn \
  | head -10

The first column is bytes, the second is the path. If the same path shows up in multiple versions, every version is in history and every one over the limit will be rejected.

Takeaway: before touching anything, list the largest blobs in history so you know whether you are removing one file or twenty.

Option A: Remove the file entirely with git filter-repo

Use this when the file was committed by accident (a dataset, a build artifact, a database dump, a model checkpoint) and does not belong in the repository.

git filter-repo is the tool the Git project itself recommends over the older git filter-branch. Install it with pip install git-filter-repo or your package manager. Work on a fresh clone, because filter-repo deliberately refuses to run on a repo that looks like it has remotes and local work you could lose.

git clone git@github.com:you/project.git project-clean
cd project-clean

# remove one path from all history
git filter-repo --path data/embeddings.bin --invert-paths

# or: remove every blob over a size threshold, whatever its path
git filter-repo --strip-blobs-bigger-than 50M

filter-repo removes the origin remote as a safety measure, so re-add it and force-push:

git remote add origin git@github.com:you/project.git
git push --force-with-lease origin main

Two honest drawbacks. First, this rewrites every commit hash from the first affected commit onward, so every collaborator must re-clone or hard-reset, and open pull requests will break. Second, if the file contained a secret, rewriting history on GitHub does not make the secret safe: assume it leaked and rotate it. Cached views of old commits can persist on the platform even after a force push.

If you want a single-purpose tool with a simpler interface, BFG Repo-Cleaner does the same job, but it is less flexible with path filtering than filter-repo.

Takeaway: use git filter-repo --invert-paths when the file was a mistake, and treat every collaborator clone as needing a reset afterward.

Option B: Keep the file but move it to Git LFS

Use this when the file is a legitimate part of the project, like design assets, test fixtures, or a small model, and you want it versioned.

git lfs install
# rewrite ALL history so matching files become LFS pointers
git lfs migrate import --include="*.bin,data/*.parquet" --everything
git push --force-with-lease origin main

--everything rewrites every branch and tag; without it only the current branch is migrated and the other refs still contain the big blobs. Check the result with git lfs ls-files, and confirm .gitattributes now contains the tracking rules, since that file has to be committed for teammates to get the same behavior.

The drawback is that LFS is a separate storage and bandwidth quota on GitHub. The free allowance is limited, large files in CI re-download on every checkout unless you cache them, and forks have their own quirks with LFS objects. Check GitHub's current LFS billing page before committing a multi-gigabyte dataset to it.

Takeaway: LFS keeps big files versioned, but you are trading a hard push error for a storage and bandwidth quota you now have to watch.

Which fix should you use?

Situation	Best fix	History rewritten?	Ongoing cost
Accidental commit of dump, build output, `.env`, `node_modules`	`git filter-repo --invert-paths`	Yes	None
Several oversized blobs, unsure which	`filter-repo --strip-blobs-bigger-than`	Yes	None
Large assets that must be versioned	`git lfs migrate import`	Yes	LFS quota
Large file only in unpushed local commits	`git reset --soft` and recommit	Local only	None
Dataset or model that changes often	Object storage (S3, R2) plus a download script	No	Storage fees

The last row is the one I would choose most often for ML-ish repos: store the artifact in a bucket and keep a small script plus a checksum in Git. It keeps clones fast and leaves no history problem to fix later.

How do you stop this from happening again?

Add the pattern to .gitignore before the file exists, and add a pre-commit check so the failure happens on your laptop instead of at push time:

#!/usr/bin/env bash
# .git/hooks/pre-commit — block staged files over 50 MB
limit=$((50 * 1024 * 1024))
for f in $(git diff --cached --name-only --diff-filter=AM); do
  size=$(wc -c < "$f")
  if [ "$size" -gt "$limit" ]; then
    echo "Refusing to commit $f ($size bytes, limit $limit)" >&2
    exit 1
  fi
done

Make it executable with chmod +x .git/hooks/pre-commit. Hooks in .git/ are not shared, so for a team use a framework such as pre-commit, which has a built-in large-file check.

FAQ

Why does GitHub reject my push after I deleted the large file?
Because the commit that added the file is still in your branch history, and GitHub scans every commit being pushed. You must rewrite history with git filter-repo or git lfs migrate to remove the blob from those commits.

What is GitHub's maximum file size for a regular Git push?
GitHub blocks files larger than 100 MB and warns above 50 MB, as of late 2026. Larger files need Git LFS or external storage.

Is it safe to force-push after running git filter-repo?
It is safe for your repo if you use --force-with-lease and coordinate with collaborators, who will need to re-clone. It is not a fix for leaked secrets, which must be rotated regardless.

Bottom line

If the file should never have been committed, run git filter-repo on a fresh clone and force-push once. If you genuinely need large files versioned alongside code, Git LFS is the standard route on GitHub, with a quota to monitor. For datasets and model weights that change often, keep them in object storage and let Git hold only the script that fetches them.

Sentry vs GlitchTip vs Self-Hosted Sentry: Error Tracking for a Small Team When the Event Quota Keeps Running Out

Libme — Thu, 22 Oct 2026 21:31:45 +0000

For most small teams, hosted Sentry is the right default, and the thing that decides whether it stays affordable is not the vendor but how many duplicate events your app emits. GlitchTip is a credible cheaper alternative if you only need error tracking and are happy with the Sentry SDKs. Self-hosting full Sentry is the option I would pick last, because the cost moves from a bill to your on-call time.

Why does the error quota run out in the first week?

The usual story: you add error tracking, everything looks calm, then a downstream API goes down for twenty minutes. A retry loop or a queue consumer throws the same exception on every attempt, and a plan priced by event volume absorbs thousands of identical events. The dashboard shows one issue with a huge count. The quota is gone and the useful errors that arrive afterwards are dropped or throttled.

Hosted tools price by events (or similar units) per month, and exact tiers change, so check the current pricing page rather than trusting a blog post. What does not change is the shape of the problem: volume is driven by your worst incident, not your normal day. A single noisy failure mode can cost more quota than a month of ordinary errors.

Takeaway: error-tracking cost is set by your noisiest incident, so cap duplicates in the client before comparing vendors.

Fix the volume first: client-side dedupe and sampling

Before you switch tools, drop repeats at the source. The Sentry JavaScript/Node SDK exposes beforeSend, ignoreErrors and sampleRate, and GlitchTip accepts events from the same SDKs, so this work carries over if you move.

import * as Sentry from "@sentry/node";

const WINDOW_MS = 60_000;
const MAX_PER_WINDOW = 5;
const seen = new Map<string, { count: number; start: number }>();

function fingerprintOf(event: Sentry.ErrorEvent): string {
  const ex = event.exception?.values?.[0];
  return `${ex?.type ?? "unknown"}:${ex?.value ?? ""}`;
}

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  // Network noise you cannot act on
  ignoreErrors: ["ECONNRESET", "AbortError"],
  beforeSend(event) {
    const key = fingerprintOf(event);
    const now = Date.now();
    const entry = seen.get(key);
    if (!entry || now - entry.start > WINDOW_MS) {
      seen.set(key, { count: 1, start: now });
      return event;
    }
    entry.count += 1;
    // Returning null drops the event before it counts against quota
    return entry.count > MAX_PER_WINDOW ? null : event;
  },
});

Two caveats from using this pattern. The Map grows with distinct messages, so if your error text embeds IDs you should key on the type and a normalized message, or add periodic cleanup. And a per-process limiter is per process: ten replicas each send their own five events per minute. That is still a large reduction from unbounded.

Hosted Sentry also offers server-side controls such as spike protection and inbound filters (as of late 2026 these are plan-dependent in places, so verify on your tier). Server-side filtering is a safety net, but dropping events in the client means they never reach the quota at all.

Takeaway: a ten-line beforeSend limiter removes the failure mode that makes every error-tracking plan look too small.

Sentry vs GlitchTip vs self-hosted Sentry: what actually differs?

	Hosted Sentry	GlitchTip (hosted or self-hosted)	Self-hosted Sentry
SDK	Native Sentry SDKs	Sentry-compatible SDKs, change the DSN	Native Sentry SDKs
Scope	Errors, performance, session replay, profiling, more	Mostly error tracking, plus basic uptime and performance features	Broad, close to hosted feature set
Ops burden	None	Low: a small app plus Postgres and a queue	High: many services to run and upgrade
Cost driver	Event volume and add-ons	Hosted: plan tier; self-hosted: your server	Server size and your time
Main drawback	Quota anxiety, add-ons priced separately	Smaller feature set, smaller ecosystem	Upgrades and resource needs

GlitchTip is open source, and because it speaks the Sentry ingestion protocol you can point an existing SDK at it by swapping the DSN. In my testing of that swap on a small Node service, errors, stack traces and breadcrumbs arrived without code changes. What you give up is breadth: if you rely on session replay, deep performance tracing or the newer AI-assisted triage features, expect gaps, and verify against GlitchTip's current docs before committing.

Self-hosted Sentry is the full product, but it runs as a large set of containers (databases, queues, ingestion workers and more) and its own docs describe a substantial minimum resource footprint. Upgrades are guided by an install script and are not a casual weekend task. For a team of two to five people, I have seen the hidden cost land as "nobody wants to own the upgrade", which is how an error tracker quietly stops working.

Takeaway: the SDK is portable across all three, so the real choice is how much of the platform you want to operate.

When is hosted Sentry worth paying for?

If you want one place for errors, performance traces and release health with no infrastructure, hosted Sentry is the product that covers all of that out of the box, and a paid plan is justified once errors are tied to revenue and someone is on call. The drawbacks are real: pricing is usage-based, so a bad incident can push you into overage or throttling, and optional features are billed separately.

If you only need "tell me when production throws and show me the stack trace", GlitchTip is the one that does that with the least overhead, and it is cheap or free to run on a server you already have. Its limitation is that anything beyond core error tracking is thinner than the original.

Self-hosted Sentry makes sense mainly when data residency or policy forbids a third-party SaaS and you have someone who will own upgrades.

Takeaway: pay for hosted Sentry when breadth matters, pick GlitchTip when errors are all you need, and self-host full Sentry only under a compliance constraint.

FAQ

Is GlitchTip compatible with the Sentry SDK?
Yes, GlitchTip accepts events from Sentry client SDKs, so you usually only change the DSN. Features beyond error ingestion, like replay, may not be available.

How do I stop one error from using my whole Sentry quota?
Drop repeats in the SDK with beforeSend and ignoreErrors, and use sampleRate for high-volume noise. Server-side spike protection helps, but client-side filtering keeps events from being counted at all.

Is self-hosting Sentry cheaper than the hosted plan?
Only if your time is free. The server cost is modest compared with the upgrade and maintenance effort of running a multi-service stack.

Bottom line

Start with hosted Sentry and a beforeSend limiter; most teams never need to leave. Move to GlitchTip if your bill is driven by volume and you only use error tracking. Self-host full Sentry only when policy forces it and an owner is named. Whatever you choose, test an incident: throw the same error 10,000 times in staging and see what the dashboard and your quota do.

Docker "toomanyrequests: You have reached your pull rate limit" in CI: Log In, Mirror, or Run a Pull-Through Cache?

Libme — Tue, 20 Oct 2026 21:10:08 +0000

If your CI pipeline suddenly fails with toomanyrequests on docker pull or FROM, Docker Hub is throttling the IP your runners share, not your code. The fastest fix is to authenticate the pull with docker login; the durable fix is to stop pulling the same base images from Docker Hub on every run, using a registry mirror or a pull-through cache. Which one you need depends on how many pulls you make and who owns the IP.

What does the error actually look like?

The text varies slightly by Docker version, but the shape is stable:

ERROR: failed to solve: ubuntu:24.04: failed to resolve source metadata for
docker.io/library/ubuntu:24.04: unexpected status from HEAD request to
https://registry-1.docker.io/v2/library/ubuntu/manifests/24.04: 429 Too Many Requests

toomanyrequests: You have reached your pull rate limit. You may increase the limit
by authenticating and upgrading: https://www.docker.com/increase-rate-limit

What tripped me up the first time: the job had passed for weeks and nothing in the repo changed. The limit is counted per source, not per project. Unauthenticated pulls are counted against the client IP, and hosted CI runners share IP ranges with many other tenants. Your pipeline can fail because a stranger's pipeline pulled too much. Once you authenticate, the count moves to your account.

Takeaway: a rate-limit failure that appears with no code change usually means the shared IP's quota ran out, not that your pipeline got heavier.

How do I check how many pulls I have left?

Docker documents a special repository, ratelimitpreview/test, that returns your remaining quota in response headers without spending a pull. Run this from the machine (or CI step) that is failing:

TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)

curl -sI -H "Authorization: Bearer $TOKEN" \
  https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest \
  | grep -i ratelimit

You will see ratelimit-limit and ratelimit-remaining, each with a window (for example ;w=21600 for six hours). To see the authenticated numbers, pass your credentials to the token request with curl -u "$USER:$PAT". Comparing the two outputs is the quickest way to confirm that logging in actually changes your bucket.

The exact limits and windows have changed over the years and Docker has revised the free-tier numbers more than once, so read the current figures on Docker's pull-usage documentation rather than trusting any blog post, including this one (as of October 2026, I would not hardcode a number anywhere).

Takeaway: read ratelimit-remaining before changing anything, so you know whether you are fixing the right problem.

Fix 1: authenticate every pull (five minutes)

In GitHub Actions, log in before anything touches the registry. Use a personal access token with read-only scope, not your account password:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: false

Two gotchas. First, docker/login-action makes the login visible to Buildx builds, but a plain docker pull inside a nested container or a services: image may use a different credential path. Second, secrets are empty in pull requests from forks (I wrote about that separately), so fork PRs fall back to anonymous pulls and can still hit the limit.

Takeaway: logging in is the cheapest fix, but it only helps jobs that can see the secret.

Fix 2: stop pulling from Docker Hub at all

Most of the pulls in a typical pipeline are a handful of official base images: node, python, postgres, redis. Two options remove them from Docker Hub's accounting.

Use a mirror of the official images. Amazon publishes copies of Docker's official images on its public registry, so FROM public.ecr.aws/docker/library/node:22-slim pulls the same library image from a different provider. The limitation is that only the official library images are mirrored there, not arbitrary third-party images, and you depend on that copy staying in sync (check the tag you pin actually exists before you rewrite every Dockerfile).

Run a pull-through cache. A cache sits between your runners and Docker Hub, stores each layer once, and serves it to every later job. The open source registry image can do this in a few lines:

# config.yml for the registry:2 image
version: 0.1
storage:
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
proxy:
  remoteurl: https://registry-1.docker.io
  username: your-dockerhub-user
  password: your-dockerhub-read-token

Then point the Docker daemon at it with "registry-mirrors": ["http://your-cache:5000"] in /etc/docker/daemon.json. Note that this mirror setting only applies to Docker Hub images, which is exactly the case you are fixing. Cloud registries offer managed versions of the same idea (Amazon ECR's pull-through cache can front Docker Hub when you give it credentials), which removes the server you would otherwise babysit.

The drawbacks are real. A self-run cache is another service that can go down and block every build, it needs disk cleanup, and on ephemeral hosted runners there is no private network to put it on. If your builds run on self-hosted runners in one network, a cache pays off quickly; on hosted runners, prefer the mirror or login.

Takeaway: a cache turns N pulls per day into one, but only if your runners can actually reach it.

Which option fits which setup?

Situation	Best fix	Main drawback
Hosted CI, few images, secrets available	`docker login` with a read-only token	Fork PRs still anonymous
Hosted CI, mostly official base images	Mirror via `public.ecr.aws/docker/library`	Official images only; tag sync
Self-hosted runners, many pipelines	Pull-through cache	You run and patch another service
Already on AWS with ECR	ECR pull-through cache	Setup of credentials and repo prefixes
Publishing your own images too	Push to GHCR or ECR, keep Hub as a pull source	Migration of image names

If you are choosing a place for your own images, GitHub's container registry is the lowest-friction choice when your code already lives on GitHub, since GITHUB_TOKEN can authenticate the push without another secret.

Takeaway: pick by where your runners live, not by which fix sounds most robust.

FAQ

Why does docker pull say toomanyrequests when I only pulled once?
Unauthenticated pulls are counted per IP address, so shared CI or office NAT addresses burn the quota across many users. Authenticating moves the count to your account.

Do HEAD requests count toward the Docker Hub rate limit?
Docker's documentation says manifest HEAD requests do not count as pulls, which is why the ratelimitpreview check above is free. The 429 in a build log can still appear on a HEAD when you are already over the limit.

How do I avoid Docker Hub rate limits without paying?
Authenticate with a free-account token, pin base images to a mirror such as the Amazon ECR public registry copy of the official images, and cache layers in CI so unchanged layers are not re-pulled.

Bottom line

If you are on hosted CI with a handful of images, add docker/login-action with a read-only token today and check ratelimit-remaining to confirm it worked. If most of your pulls are official images, move the FROM lines to a mirror and you are mostly done. Only build a pull-through cache when you run self-hosted runners at a scale where a mirror's gaps start to hurt, and plan for it being one more thing that can break your builds.

ERR_REQUIRE_ESM in Node: Fix CommonJS/ESM Conflicts and Pick tsx, ts-node or tsup

Libme — Mon, 19 Oct 2026 22:04:47 +0000

If you see Error [ERR_REQUIRE_ESM]: require() of ES Module ... not supported, a CommonJS file is trying to require() a package that now ships only ES modules. As of late 2026 the quickest fix is often upgrading Node (20.19+ or 22.12+ can require() ES modules), but the durable fix is making your project's module format consistent: either go all-ESM, or pin the dependency and bundle.

This post covers the error variants you will actually hit, why each happens, and which TypeScript runner (tsx, ts-node, tsup) to use so you stop fighting it.

What does ERR_REQUIRE_ESM actually mean?

Node has two module systems. CommonJS uses require() and module.exports. ES modules use import/export. For years, CommonJS could not load ESM synchronously, because ESM loading is asynchronous by design. The classic failure looks like this:

Error [ERR_REQUIRE_ESM]: require() of ES Module /app/node_modules/chalk/source/index.js
from /app/dist/index.js not supported.
Instead change the require of index.js in /app/dist/index.js to a dynamic import() which is available in all CommonJS modules.

This usually appears after a routine npm update. A maintainer of a dependency (chalk, node-fetch and nanoid are well-known past examples) released a major version that dropped CommonJS, and your lockfile picked it up. Nothing in your code changed, which is why it feels random.

The trap: TypeScript compiled with "module": "commonjs" rewrites your import chalk from 'chalk' into a require('chalk'). Your source looks like ESM, your output is CommonJS, and the crash happens at runtime.

Takeaway: the error is about the format of the file doing the requiring, not the syntax in your source.

Does newer Node make this go away?

Partly. As of late 2026, require() of a synchronous ES module works without flags on Node 22.12+ and 20.19+ (earlier versions needed --experimental-require-module). Check with node -v before debugging anything else; in my experience a stale Node in a Docker base image is a common reason the error persists on a server while it works on a laptop.

It has real limits:

A module that uses top-level await still fails, with ERR_REQUIRE_ASYNC_MODULE.
require() returns the module namespace object, so a default export arrives as .default:

// CommonJS file requiring an ESM-only package on a supporting Node version
const mod = require('some-esm-only-package');
const fn = mod.default ?? mod; // default export lives on .default

Bundlers and some older tooling (Jest in its default CommonJS mode, for example) have their own resolution and may still choke.

Takeaway: upgrading Node fixes the common case, but top-level await and default-export shape are the two places it still bites.

How do you fix it permanently?

You have three honest options, in order of how much I recommend them for new code.

1. Go all-ESM. Set "type": "module" in package.json, and in tsconfig.json use a Node-aware setting:

{
  "compilerOptions": {
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "target": "ES2022",
    "outDir": "dist",
    "strict": true
  }
}

Relative imports then need explicit extensions, and this is the part that trips people up. Write import { db } from './db.js' even though the file on disk is db.ts. TypeScript maps it, and Node needs the .js at runtime.

2. Stay CommonJS and use dynamic import for the one offender. This works in any CommonJS file on any supported Node:

async function main() {
  const { default: chalk } = await import('chalk');
  console.log(chalk.green('ok'));
}
main();

One catch: if TypeScript is compiling to CommonJS, it will rewrite import() into require() and bring the error back. With module: "NodeNext" in a CommonJS package it preserves dynamic import(); with the older module: "commonjs" it does not.

3. Pin the last CommonJS major. Fast, and a legitimate stopgap, but you stop receiving security fixes for that dependency. Treat it as a ticket, not a solution.

Takeaway: pick one module format per package and make TypeScript's module setting match it; most of these errors are a mismatch between the two.

"Unknown file extension .ts": which TypeScript runner should you use?

The second error people search for right after the first is:

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /app/src/index.ts

It means Node's ESM loader was handed a .ts file with no loader that understands TypeScript. This is the classic symptom of running ts-node against an ESM project without the right loader flags. Your options:

Tool	What it does	Type checks?	Best for	Real drawback
`tsx`	Runs TS/ESM/CJS files directly, using esbuild under the hood	No	Scripts, dev servers, mixed CJS/ESM repos	Won't catch type errors; run `tsc --noEmit` separately
`ts-node`	Compiles with the TypeScript compiler at runtime	Yes (unless `--transpile-only`)	Existing projects already configured for it	ESM support needs careful config and tends to produce exactly the error above
`tsup`	Bundles TS into `dist/` (CJS, ESM or both) with `.d.ts` output	Declarations only, via a flag	Publishing libraries, shipping a single built artifact	A build step; not a runtime launcher
Node native type stripping	Newer Node versions can run `.ts` files directly	No	Simple scripts with no enums or path aliases	Supports only erasable TypeScript syntax; check your Node version

A working dev setup I would start from today:

{
  "type": "module",
  "scripts": {
    "dev": "tsx watch src/index.ts",
    "typecheck": "tsc --noEmit",
    "build": "tsup src/index.ts --format esm --target node20 --clean",
    "start": "node dist/index.js"
  }
}

If you want a runner that stops you thinking about CommonJS versus ESM during development, tsx is the one that handles both without configuration. It will not tell you about type errors, so the typecheck script in CI is not optional.

If you are publishing a library that consumers on both module systems will import, tsup is the tool that emits both formats plus type declarations from one command. You still need to get the exports map in package.json right, and tsup does not do that for you.

Takeaway: use tsx to run, tsc --noEmit to check, and tsup to ship; ts-node is only worth keeping if it already works in your repo.

FAQ

Why does ERR_REQUIRE_ESM only happen after npm update?

A dependency released a major version that is ESM-only, and your version range allowed it. Check the package's changelog and "type"/"exports" fields in its package.json.

Can I use require() on an ES module?

Yes on Node 22.12+ and 20.19+, as long as the module has no top-level await. Older versions throw ERR_REQUIRE_ESM, and you need dynamic import() instead.

Why do I need .js extensions in TypeScript imports?

Node's ESM resolver does not guess extensions, and TypeScript does not rewrite your import paths. You write the extension the file will have after compilation, which is .js.

Bottom line

For a new project, go all-ESM with "type": "module", NodeNext module settings, tsx during development, and tsup if you publish. For an existing CommonJS codebase hit by one ESM-only dependency, upgrade Node first, then use dynamic import() for that package, and pin the old major only as a short-lived stopgap. Whichever way you go, keep tsc --noEmit in CI, because the fast runners all skip type checking.

Node.js "fetch failed" with ETIMEDOUT or ENETUNREACH While curl Works: The Broken IPv6 Route

Libme — Sun, 18 Oct 2026 22:11:28 +0000

Short answer: if a Node.js request dies with TypeError: fetch failed, AggregateError [ETIMEDOUT], or connect ENETUNREACH 2606:... but curl to the same host succeeds, the hostname probably resolves to both IPv6 and IPv4 addresses, and your machine or container has an IPv6 address family it cannot actually route. Confirm it with curl -4 versus curl -6, then either fix the network or tell Node to prefer IPv4 and fall back faster.

What does the error actually look like?

The surface message from the built-in fetch is unhelpful:

TypeError: fetch failed
    at node:internal/deps/undici/undici:...
  [cause]: AggregateError [ETIMEDOUT]:
      at internalConnectMultiple (node:net:...)
    [errors]: [
      Error: connect ETIMEDOUT 2606:4700:...:443
      Error: connect ENETUNREACH 2606:4700:...:443
      Error: connect ETIMEDOUT 104.18.x.x:443
    ]

The detail is in error.cause. Log it, or you will chase the wrong problem. The giveaway is the mix of IPv6 and IPv4 addresses in one AggregateError: Node tried several addresses for the same hostname and none connected in time.

try {
  await fetch("https://api.example.com/health");
} catch (err) {
  console.error(err.cause?.errors ?? err.cause ?? err);
}

Takeaway: always print err.cause; fetch failed is a wrapper, and the per-address errors are the diagnosis.

Why does curl work when Node does not?

Two behaviors changed across recent Node versions, and they interact badly with half-configured networks. As of Node 17, dns.lookup returns addresses in the order the resolver gives them (verbatim), so IPv6 addresses typically come first instead of being sorted behind IPv4. As of Node 20, net.connect tries multiple addresses using "happy eyeballs" (autoSelectFamily), but each attempt has a short default timeout (documented as 250 ms), which is tight on slow or cross-region links.

curl, by contrast, has had happy-eyeballs logic for years and tends to fall back to IPv4 quickly. So the same host can feel fine from curl and flaky from Node, especially on:

Docker containers or CI runners where the host has no IPv6 route but DNS still returns AAAA records.
VPNs and corporate networks that advertise IPv6 but drop the packets.
Home or office routers with a half-working IPv6 prefix.

What tripped me up: the failure is often intermittent, because it depends on which address wins the race and how loaded the link is. It looks like a flaky upstream when it is really your network.

Takeaway: AAAA records plus no working IPv6 route equals a slow or failed first attempt, and Node's default fallback window is short.

How do I confirm it is IPv6?

Force each address family from the same machine (or inside the same container) that fails:

curl -4 -sS -o /dev/null -w "ipv4: %{http_code} %{time_total}s\n" https://api.example.com/health
curl -6 -sS -o /dev/null -w "ipv6: %{http_code} %{time_total}s\n" https://api.example.com/health
getent ahosts api.example.com   # which addresses, in which order

If -4 succeeds quickly and -6 hangs or prints "Network is unreachable", you have your answer. Run it inside the container, not on your laptop; the host and the container often have different routes.

You can also reproduce it in Node without any HTTP library:

import dns from "node:dns/promises";
import net from "node:net";

const host = "api.example.com";
for (const family of [4, 6]) {
  try {
    const { address } = await dns.lookup(host, { family });
    await new Promise((resolve, reject) => {
      const s = net.connect({ host: address, port: 443, timeout: 3000 }, () => { s.destroy(); resolve(); });
      s.on("timeout", () => { s.destroy(); reject(new Error("timeout")); });
      s.on("error", reject);
    });
    console.log(`IPv${family} OK (${address})`);
  } catch (e) {
    console.log(`IPv${family} FAILED: ${e.code ?? e.message}`);
  }
}

Takeaway: test each address family separately, from inside the environment that fails.

Which fix should I use?

Fix	Scope	Trade-off
Fix the network (enable IPv6 on the Docker network, or stop advertising AAAA)	Whole machine	Correct but needs infra access
`NODE_OPTIONS=--dns-result-order=ipv4first`	Whole process, no code change	Applies to all lookups, including hosts where IPv6 is the good path
`dns.setDefaultResultOrder("ipv4first")` at startup	Same, in code	Must run before the first lookup
`net.setDefaultAutoSelectFamilyAttemptTimeout(500)`	Gives slow links more time per address	Does not help if IPv6 is truly dead; only reduces false fallbacks
Disable IPv6 via sysctl in the container	Container	Blunt; breaks hosts that are IPv6-only

The least invasive one I reach for first:

// top of your entrypoint, before any network call
import dns from "node:dns";
import net from "node:net";

dns.setDefaultResultOrder("ipv4first");
net.setDefaultAutoSelectFamilyAttemptTimeout(500); // ms per address; Node 20+

or, with no code change, in your Dockerfile or deployment config:

ENV NODE_OPTIONS="--dns-result-order=ipv4first"

Caveats worth knowing. These settings affect dns.lookup-based connections, which is what net, http, and fetch use by default; libraries that do their own resolution may behave differently. Pinning IPv4 first is a workaround, not a diagnosis: if your platform later becomes IPv6-only on the edge, you will want the real fix. And on Node versions before 20, autoSelectFamily is opt-in, so the attempt-timeout call does nothing useful there.

If you run on Docker and want the proper fix, either enable IPv6 on the daemon's network with a valid prefix, or accept IPv4-only and make sure your resolver stops handing back AAAA records the container can't use. As of late 2026 I still see Docker's default bridge network without IPv6 on most hosts, so this class of bug keeps showing up in containers first.

Takeaway: prefer ipv4first as a fast, reversible mitigation, and treat the network fix as the real one.

Is this a reason to pay for anything?

Not really, and that is the point. This is a configuration problem, not a tooling gap. The one case where money enters the picture is when you do not control the network: if a locked-down CI runner or serverless platform has flaky IPv6 and you cannot change it, the cheap fix is the environment variable above rather than migrating providers. I would only move platforms if the vendor cannot fix routing on their side and it blocks you repeatedly.

FAQ

Why does Node say "fetch failed" instead of the real error?
Node's built-in fetch (undici) wraps the underlying socket error. The real reason is in error.cause, and for multi-address failures in error.cause.errors.

Why does curl work but Node fetch times out?
curl falls back across IPv4 and IPv6 quickly, while Node's per-address connect attempt has a short default timeout. If IPv6 is unroutable, Node can exhaust its attempts before reaching a working IPv4 address.

How do I make Node prefer IPv4?
Set NODE_OPTIONS=--dns-result-order=ipv4first, or call dns.setDefaultResultOrder("ipv4first") before any network request.

Bottom line

If you see fetch failed with a mix of IPv6 and IPv4 addresses in the cause, test curl -4 against curl -6 from inside the failing environment before touching application code. For a quick, reversible mitigation, set --dns-result-order=ipv4first and raise the attempt timeout; for a lasting one, fix IPv6 routing or stop publishing AAAA records your network cannot use. Do not add retries around it: retries on a dead address family just make the same slow failure repeat.

ngrok vs Cloudflare Tunnel vs Tailscale Funnel: Which One for Testing Webhooks on Localhost?

Libme — Sat, 17 Oct 2026 22:59:38 +0000

Short answer: For a throwaway webhook test, Cloudflare's quick tunnel is the lowest-friction option because it needs no account. For a URL you register once with Stripe, GitHub, or Slack and keep for months, you want a stable hostname: ngrok with a reserved domain, a named Cloudflare Tunnel, or Tailscale Funnel if your machine is already on a tailnet. The deciding factors are URL stability, what the tunnel does to your requests, and whether you already own a domain.

All three behaviors below are as of October 2026. Tunnel vendors change free-tier terms often, so check the current docs before you commit.

What problem are you actually solving?

A webhook provider needs a public HTTPS URL. Your handler runs on localhost:3000. A tunnel gives you a public hostname that forwards to that port.

Before comparing tools, start with a handler that shows you what really arrives. Most "the tunnel is broken" reports I have debugged turned out to be the handler, not the tunnel.

// server.js: run with: node server.js
import http from "node:http";

http.createServer((req, res) => {
  const chunks = [];
  req.on("data", (c) => chunks.push(c));
  req.on("end", () => {
    const body = Buffer.concat(chunks);
    console.log(req.method, req.url, `${body.length} bytes`);
    console.log("headers:", req.headers);
    res.writeHead(200, { "content-type": "text/plain" });
    res.end("ok");
  });
}).listen(3000, () => console.log("listening on :3000"));

Once this prints a request through your tunnel, you know the path works end to end. Then swap in your real handler.

Takeaway: prove the tunnel with a dumb echo server first, so you never debug your handler and your tunnel at the same time.

How do the three compare?

	ngrok	Cloudflare Tunnel	Tailscale Funnel
Account needed	Yes	Quick tunnel: no. Named tunnel: yes, plus a domain on Cloudflare	Yes, plus a tailnet
Throwaway URL	Random, changes per session unless you reserve a domain	`trycloudflare.com` random URL, changes every run	Stable per machine name
Stable URL, free	Typically one static domain on the free plan (verify current terms)	Yes, with a named tunnel on your own domain	Yes, `<machine>.<tailnet>.ts.net`
Request inspector / replay	Built in, the main reason people pick it	None built in	None built in
Ports	Any local port	Any local port	Public side limited to a few ports (443, 8443, 10000)
Main gotcha	Free-tier browser interstitial page	Quick tunnels are for testing, with documented limits	Needs HTTPS and MagicDNS enabled on the tailnet

When is ngrok the right pick?

ngrok's real advantage is the request inspector. You open its local web UI, see every incoming webhook with headers and body, and replay one with a click. When a provider sends a payload once and your handler crashed on it, replay saves you from re-triggering the event. For debugging signature verification, that matters; if you are fighting a failing signature check, being able to replay the exact bytes is worth a lot.

ngrok http 3000
# with a reserved domain from your dashboard:
ngrok http --url=your-name.ngrok-free.app 3000

The flag name for the domain has changed between ngrok CLI versions, so run ngrok http --help rather than trusting a blog post, including this one.

Limitations I have hit: the free plan can show an interstitial warning page to browser traffic. Webhook providers are not browsers, so they are usually unaffected, but if you test with a browser or a tool that sends a browser-like user agent, you may see the interstitial instead of your app. Also, your traffic and your inspector history go through a third party, which matters if payloads contain real customer data.

If you want the managed version of this with request replay built in, ngrok is the one that gives you inspection and a stable domain without owning DNS.

Takeaway: pick ngrok when you need to replay and inspect payloads, and accept that your webhook traffic transits its service.

When does Cloudflare Tunnel win?

Two modes, and people conflate them.

# Quick tunnel: no account, random URL, dies when the process stops
cloudflared tunnel --url http://localhost:3000

# Named tunnel: stable hostname on a domain you manage in Cloudflare
cloudflared tunnel login
cloudflared tunnel create dev-webhooks
cloudflared tunnel route dns dev-webhooks hooks.example.com
cloudflared tunnel run --url http://localhost:3000 dev-webhooks

The quick tunnel is the fastest path from zero to a working public URL. Cloudflare documents it as a testing feature with limits, and as of my last check it does not support Server-Sent Events, so if your handler streams, test with a named tunnel instead. The URL changes every run, which means re-registering the webhook each time. Fine for a ten-minute experiment, painful for a week of development.

The named tunnel gives you a permanent hostname, but only if your domain's DNS is on Cloudflare. If it is not, you are signing up for a DNS migration just to test a webhook. If you already have a domain there, Cloudflare Tunnel is the one that gives you a permanent hostname at no extra cost and with no third-party hostname in your provider's settings.

Drawback: there is no built-in request inspector. You add logging to your own handler, which is why the echo server above is useful.

Takeaway: Cloudflare Tunnel is the best stable free option only if your domain already lives in Cloudflare; otherwise use the quick tunnel for short tests.

Is Tailscale Funnel a good fit for webhooks?

If your laptop or home server is already on a tailnet, it is nearly free to set up.

tailscale funnel 3000
tailscale funnel status

You get https://<machine>.<tailnet>.ts.net, stable across sessions, with a valid certificate. You must have HTTPS certificates and MagicDNS enabled for the tailnet, and Funnel has to be allowed in the access policy; the first run prints what is missing.

The catch is exposure model. Funnel makes a service on your machine reachable from the public internet, and bandwidth limits exist that you cannot configure. The public ports are restricted to a short list, so you cannot pick an arbitrary one. For a webhook receiver this is acceptable; for anything heavier, it is not the tool.

Because the hostname contains your tailnet name, you are also publishing that name in whatever dashboard you register it in. Some people do not want that in a shared Stripe account.

Takeaway: Funnel is the right answer when you are already a Tailscale user and want a stable URL without touching DNS.

FAQ

Why does my webhook work with ngrok but fail signature verification?
Usually something re-serialized the body before you hashed it. Verify against the raw bytes, not a parsed-and-restringified JSON object; tunnels forward bytes unchanged.

Can I use a Cloudflare quick tunnel for a permanent webhook endpoint?
No. The URL is random and changes on every run, so the provider's registered URL breaks as soon as you restart.

Do I need a tunnel if I use the provider's CLI?
Often not. Some providers ship a CLI that forwards events to localhost for you, which skips the public URL entirely, but it only covers that one provider.

Bottom line

Use Cloudflare's quick tunnel for a ten-minute smoke test. Use ngrok when you are debugging payloads and want replay. Use a named Cloudflare Tunnel if your domain is already on Cloudflare and you want a permanent URL you control. Use Tailscale Funnel if you already live on a tailnet. Whichever you choose, run the echo server first.

Postgres "deadlock detected": Fix the Lock Order First, Then Add a Retry

Libme — Fri, 16 Oct 2026 22:11:13 +0000

Postgres raises ERROR: deadlock detected (SQLSTATE 40P01) when two transactions each hold a lock the other needs. The durable fix is to make every code path lock rows in the same order; a retry on 40P01 is the safety net, not the cure. If you only add the retry, you hide the bug and pay for it in latency.

What does the error actually say?

This is the shape of the message you get in the application and the server log:

ERROR:  deadlock detected
DETAIL:  Process 8121 waits for ShareLock on transaction 90417; blocked by process 8130.
Process 8130 waits for ShareLock on transaction 90416; blocked by process 8121.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (0,12) in relation "accounts"

Read it as a cycle. Process 8121 is waiting for a transaction that process 8130 owns, and 8130 is waiting for one that 8121 owns. Postgres notices the cycle after deadlock_timeout (1 second by default), picks one transaction as the victim, and aborts it. The other one proceeds as if nothing happened.

Two things trip people up. The victim is not necessarily the "wrong" transaction, so you cannot fix this by blaming one code path. And the CONTEXT line names only the row the victim was waiting on when it was killed, not the whole story. The server log, not the client error, has the full query text for both sides.

Takeaway: a deadlock is a cycle between two or more transactions, and the error message is only the last edge of that cycle.

Why does it only happen under load?

A deadlock needs two transactions to interleave at exactly the wrong moment, so it passes every test you run alone. The classic version is a transfer between two accounts:

-- Transaction A (transfer 1 -> 2)
BEGIN;
UPDATE accounts SET balance = balance - 10 WHERE id = 1;
UPDATE accounts SET balance = balance + 10 WHERE id = 2;
COMMIT;

-- Transaction B (transfer 2 -> 1), running at the same moment
BEGIN;
UPDATE accounts SET balance = balance - 5 WHERE id = 2;
UPDATE accounts SET balance = balance + 5 WHERE id = 1;
COMMIT;

A locks row 1, B locks row 2, then each reaches for the row the other holds. Neither can finish. Nothing about this is a Postgres quirk; any database with row locks behaves this way.

The less obvious sources, in the order I tend to find them:

Batch updates with no stable order. UPDATE ... WHERE id IN (...) locks rows in whatever order the plan visits them. Two workers updating overlapping sets can lock in opposite orders. Postgres does not guarantee the order from the IN list.
Foreign keys. Inserting a child row takes a FOR KEY SHARE lock on the parent row. Concurrent inserts of children plus an update of the same parent can form a cycle you never wrote explicitly.
Upserts and unique indexes. Two transactions inserting overlapping sets of keys with INSERT ... ON CONFLICT in different orders can deadlock on the index entries.
Mixed application paths. One endpoint locks orders then inventory; a background job locks inventory then orders.

Takeaway: most production deadlocks come from two code paths touching the same rows in different orders, not from a single bad query.

How do you find the two queries involved?

Turn on lock-wait logging so the server log tells you who waited on whom. These are standard settings; set them in postgresql.conf or through your managed provider's parameter UI:

ALTER SYSTEM SET log_lock_waits = on;
SELECT pg_reload_conf();

With log_lock_waits on, any wait longer than deadlock_timeout is logged, and a deadlock entry includes the statements for each process involved. Pair that with log_line_prefix containing the process id (%p) so you can match Process 8121 to its query. If you ship logs to a hosted service, search for deadlock detected and read the two statements side by side. The ordering mismatch is usually visible in thirty seconds.

Takeaway: turn on log_lock_waits before you need it, because a deadlock you cannot attribute to two specific statements is a deadlock you cannot fix.

How do you fix it: consistent lock ordering

Lock the rows you are about to modify up front, in a deterministic order, then do the writes. For the transfer example:

BEGIN;

SELECT id
FROM accounts
WHERE id = ANY(ARRAY[1, 2])
ORDER BY id
FOR UPDATE;

UPDATE accounts SET balance = balance - 10 WHERE id = 1;
UPDATE accounts SET balance = balance + 10 WHERE id = 2;

COMMIT;

Both directions of transfer now lock id 1 before id 2. The second transaction waits for the first instead of forming a cycle, and waiting is fine; only circular waiting is fatal.

In application code, sort the keys before you touch them. A Python sketch with psycopg 3:

import psycopg

def transfer(conn: psycopg.Connection, src: int, dst: int, amount: int) -> None:
    with conn.transaction():
        # Lock both rows in a fixed order, regardless of transfer direction.
        conn.execute(
            "SELECT id FROM accounts WHERE id = ANY(%s) ORDER BY id FOR UPDATE",
            ([src, dst],),
        )
        conn.execute(
            "UPDATE accounts SET balance = balance - %s WHERE id = %s",
            (amount, src),
        )
        conn.execute(
            "UPDATE accounts SET balance = balance + %s WHERE id = %s",
            (amount, dst),
        )

For batch jobs, the same idea applies: select the target ids with ORDER BY id FOR UPDATE, or use FOR UPDATE SKIP LOCKED if workers should divide work rather than queue behind each other. Also keep transactions short. A transaction that holds locks while it calls an external API widens the window for every cycle.

Takeaway: if every transaction acquires locks in one global order (for example ascending primary key), a lock cycle cannot form.

What should the retry look like?

Ordering fixes the cycles you know about. Foreign-key and index locks can still produce rare ones, so wrap write transactions in a retry that catches only the deadlock error:

import random
import time
import psycopg
from psycopg import errors

def run_with_deadlock_retry(conn, fn, attempts=4):
    for attempt in range(attempts):
        try:
            return fn(conn)
        except errors.DeadlockDetected:  # SQLSTATE 40P01
            if attempt == attempts - 1:
                raise
            time.sleep(random.uniform(0, 0.05 * 2 ** attempt))

Retry the whole transaction, not the single statement that failed, because Postgres has already rolled it back. Make sure the function is safe to run twice, which means no external side effects inside it. Jitter matters: without it, the two transactions can collide again in lockstep.

Approach	Removes the cycle?	Cost	Use when
Consistent lock order (`ORDER BY id FOR UPDATE`)	Yes	One extra statement per transaction	Always, as the primary fix
Retry on `40P01`	No, it recovers	Latency on the rare collision	As a safety net behind ordering
Shorter transactions	Narrows the window	Refactoring	Locks held across slow calls
`SKIP LOCKED`	Avoids waiting	Rows may be skipped this pass	Worker pools draining a queue
Raise `deadlock_timeout`	No	Slower detection	Rarely; it just delays the failure

Takeaway: ordering prevents deadlocks, retries absorb the leftovers, and raising timeouts does neither.

FAQ

Is a Postgres deadlock a bug in my database?

No. Postgres is behaving correctly by detecting the cycle and aborting one transaction; the bug is in the application's lock ordering.

Should I just retry on deadlock detected?

Retry as a safety net, but fix the ordering first. A retry on its own turns a correctness bug into intermittent latency spikes.

What is SQLSTATE 40P01?

It is the code Postgres returns for deadlock_detected. Catch that code specifically instead of retrying on every error.

Bottom line

If you see deadlock detected, enable log_lock_waits, find the two statements, and make both paths lock rows in the same order, usually ascending primary key via SELECT ... ORDER BY id FOR UPDATE. Then wrap write transactions in a small jittered retry on 40P01 for the cycles you cannot predict. Do not treat a longer deadlock_timeout as a fix. As of late 2026 this behavior is stable across supported Postgres versions, though the exact log wording can differ slightly.

Postgres "could not resize shared memory segment: No space left on device" in Docker: Why It Happens and How to Fix It

Libme — Thu, 15 Oct 2026 22:11:16 +0000

Short answer: this error means a parallel query tried to allocate more shared memory than the container's /dev/shm allows. Docker gives every container a 64MB /dev/shm by default, regardless of how much RAM the host has. Raise it with shm_size (Docker/Compose) or a memory-backed emptyDir mounted at /dev/shm (Kubernetes); disabling parallelism is a stopgap, not a fix.

What does the error look like?

ERROR:  could not resize shared memory segment "/PostgreSQL.1672401134" to 67146816 bytes: No space left on device

The confusing part is that the disk is not full. df -h on your volumes looks fine, the container has gigabytes of memory headroom, and the same query works on your laptop's native Postgres install. It also tends to be intermittent: small queries pass, a bigger report query or a larger dataset after a data load triggers it.

The "No space left on device" refers to /dev/shm, a tmpfs. Check it from inside the container:

docker exec -it my-postgres df -h /dev/shm
# Filesystem      Size  Used Avail Use% Mounted on
# shm              64M     0   64M   0% /dev/shm

If you see 64M, that is your ceiling.

Why does a parallel query need /dev/shm?

When the planner chooses a parallel plan, the leader process and its workers need to share data: tuple queues, and for a Parallel Hash join, the hash table itself. Postgres allocates that as dynamic shared memory. On Linux the default dynamic_shared_memory_type is posix, which creates segments named /PostgreSQL.<number> in /dev/shm.

You can confirm a query is a candidate by looking for parallel nodes:

EXPLAIN (ANALYZE, BUFFERS)
SELECT o.customer_id, count(*)
FROM orders o
JOIN order_items i ON i.order_id = o.id
GROUP BY o.customer_id;
-- Look for: Gather, Workers Planned: 2, Parallel Hash Join

A Parallel Hash builds one shared hash table sized by work_mem multiplied by the number of participants (as of Postgres 13+ the shared hash table can use up to work_mem × hash_mem_multiplier per participant). With work_mem = 64MB and two workers plus the leader, one query can ask for hundreds of megabytes of shared memory. Against a 64MB tmpfs, that fails.

Takeaway: the error is a tmpfs size limit that only parallel plans with large hash tables hit, which is why it shows up after data growth rather than on day one.

What are the fixes, and which one should I use?

Fix	What it does	Cost / drawback
Raise `/dev/shm` size	Gives parallel queries the memory they were planned for	Shared memory counts toward container memory in some setups; size it deliberately
`max_parallel_workers_per_gather = 0`	Stops parallel plans entirely	Slower large scans and aggregations; hides the problem
Lower `work_mem`	Smaller hash tables, fewer shm bytes	Can push sorts and hashes to disk, slowing other queries
`dynamic_shared_memory_type = mmap`	Backs segments with files under `pg_dynshmem` instead of `/dev/shm`	Disk-backed, so slower, and not recommended for production by the Postgres docs

For almost everyone, the first row is the right answer.

Docker and Compose

services:
  db:
    image: postgres:16
    shm_size: 256mb
    environment:
      POSTGRES_PASSWORD: example

With plain docker run, the flag is --shm-size=256m. Recreate the container (a restart is not enough if you changed the Compose definition; run docker compose up -d) and re-check df -h /dev/shm.

How big? Start from your worst parallel query: roughly (workers + 1) × work_mem × hash_mem_multiplier is a reasonable upper bound for one hash join, and concurrent queries add up. I would rather size it to a few hundred MB and watch than set it to something huge, because tmpfs pages are real memory once written.

Kubernetes

Kubernetes has no shm_size field. The standard approach is an in-memory emptyDir mounted over /dev/shm:

spec:
  containers:
    - name: postgres
      image: postgres:16
      resources:
        limits:
          memory: 2Gi
      volumeMounts:
        - name: dshm
          mountPath: /dev/shm
  volumes:
    - name: dshm
      emptyDir:
        medium: Memory
        sizeLimit: 512Mi

The thing that tripped me up here: memory-backed emptyDir usage is charged against the container's memory limit. If shared memory fills while shared_buffers and backend memory are already near the limit, you swap a "No space left on device" error for an OOMKill. Leave headroom in limits.memory beyond what you used to give the pod.

If you run Postgres through an operator such as CloudNativePG or Zalando's postgres-operator, check whether it already mounts /dev/shm for you before adding your own volume; operators commonly handle this and a duplicate mount will conflict.

Takeaway: raise /dev/shm to a few hundred MB and treat that memory as part of the container's budget, not free space.

How do I verify the fix?

Re-run the failing query and confirm the plan is still parallel, so you did not "fix" it by silently turning parallelism off:

docker exec -it my-postgres psql -U postgres -c "SHOW dynamic_shared_memory_type;"
docker exec -it my-postgres df -h /dev/shm

Then run EXPLAIN (ANALYZE) and check that Workers Launched matches Workers Planned. If launched is lower than planned, the cluster is hitting max_parallel_workers, which is a different limit.

Do managed databases have this problem?

Not in a way you control. On managed offerings such as Amazon RDS for PostgreSQL, shared memory is part of the instance configuration, so you will not edit /dev/shm yourself. If you want to avoid owning this kind of container-level detail entirely, Amazon RDS for PostgreSQL is one option that removes it; the tradeoff is less control over kernel-level settings and a bill that scales with instance class. For a local dev database or a small self-managed deployment, the fix above is a one-line change and not a reason to migrate.

FAQ

Why does Postgres say "No space left on device" when my disk is empty?
Because the error refers to /dev/shm, an in-memory tmpfs, not your data volume. Docker defaults it to 64MB.

What should I set shm_size to for Postgres in Docker?
256MB is a sensible starting point for development and modest workloads. Size up if your parallel queries use large work_mem values, and verify with df -h /dev/shm during a heavy query.

Is it safe to set max_parallel_workers_per_gather = 0 to make the error go away?
It stops the error, but it also disables parallel query for that setting's scope, so large scans and aggregates will run slower. Use it to unblock yourself, then raise /dev/shm and turn parallelism back on.

Bottom line

If you run Postgres in a container and see "could not resize shared memory segment," set shm_size in Docker or mount a memory-backed emptyDir at /dev/shm in Kubernetes, and budget that memory inside your limits. Use max_parallel_workers_per_gather = 0 only as a temporary workaround, and avoid mmap shared memory for production. If you would rather not manage container internals at all, a managed Postgres service removes this class of problem at the cost of control.

npm ci Says package.json and package-lock.json Are Not in Sync: Causes, Fixes, and How pnpm and Yarn Compare

Libme — Wed, 14 Oct 2026 22:15:26 +0000

npm ci fails with "package.json and package-lock.json are not in sync" when the lockfile cannot satisfy what package.json declares, or when it was written by a different npm version than the one running in CI. The fix is almost always to regenerate the lockfile with the same npm version CI uses, commit it, and pin that version so it cannot drift again. This post covers the four real causes, a diagnostic order, and how npm, pnpm and Yarn differ in enforcing lockfiles (as of October 2026).

What does the error actually mean?

The message looks like this:

npm error `npm ci` can only install packages when your package.json and
package-lock.json or npm-shrinkwrap.json are in sync. Please update your
lock file with `npm install` before continuing.
npm error Missing: zod@3.23.8 from lock file
npm error Invalid: lock file's esbuild@0.21.5 does not satisfy esbuild@0.24.0

Unlike npm install, npm ci never edits the lockfile. It deletes node_modules, compares the lockfile against package.json, and refuses to proceed on any mismatch. That strictness is the feature: CI should fail loudly instead of quietly resolving new versions.

The takeaway: this error is npm ci doing its job, so the real question is why the two files disagree.

Why do the two files disagree?

In rough order of how often I see them:

Someone edited package.json by hand and never ran npm install, so the lockfile has no entry for the new range.
Merge conflicts resolved badly. Two branches both changed dependencies, and the lockfile conflict was fixed by picking one side.
Different npm versions. Older npm majors write older lockfileVersion formats and resolve peer dependencies differently. A laptop on one major and CI on another can each consider the other's lockfile incomplete.
Platform-specific optional dependencies. Packages such as esbuild, Rollup, SWC and Lightning CSS ship native binaries as optional dependencies per OS and CPU. A lockfile generated on macOS can lack the Linux entries, producing errors like Cannot find module @rollup/rollup-linux-x64-gnu in CI.

The takeaway: three of the four causes are human workflow problems, and only the optional-dependency one is a genuine tooling quirk.

How do I diagnose it in the right order?

Start by making the failure reproducible locally with the exact versions CI uses:

# Match CI's Node/npm first
node -v && npm -v

# Reproduce in a clean clone, not your working directory
git clone . /tmp/repro && cd /tmp/repro
npm ci

If it fails in a clean clone, the committed lockfile is the problem. If it passes locally but fails in CI, the npm versions differ. Compare the lockfileVersion field at the top of package-lock.json against what CI's npm would write.

To fix cases 1 to 3:

# Use the same npm major as CI
npm install            # reconciles the lockfile with package.json
git add package-lock.json
npm ci                 # must pass before you push

For case 4, a cleanly regenerated lockfile on a recent npm often includes the other platforms' optional entries. If it does not, delete node_modules and package-lock.json once, reinstall, and verify with npm ci inside a Linux container that matches CI. Dead ends I wasted time on: npm ci --force and --legacy-peer-deps do not repair a stale lockfile. They change resolution rules and can hide the drift rather than remove it.

The takeaway: reproduce in a clean clone with CI's exact npm version before touching the lockfile.

How do I stop it from coming back?

Pin the tool version in the repository so laptops and CI cannot disagree:

{
  "packageManager": "npm@10.9.0",
  "engines": { "node": ">=20", "npm": ">=10" }
}

The packageManager field is read by Corepack, and engines only warns by default unless engine-strict is set in .npmrc. Then add a cheap guard job that fails pull requests for drift before the slow pipeline starts:

- uses: actions/setup-node@v4
  with:
    node-version-file: .nvmrc
    cache: npm
- run: npm ci --ignore-scripts
- run: git diff --exit-code package-lock.json

The last line catches the case where install silently rewrote the lockfile locally but nobody committed it.

The takeaway: a pinned package manager version removes the most common cause, and git diff --exit-code on the lockfile catches the rest.

How do npm, pnpm and Yarn enforce lockfiles?

	Strict CI command	Behavior on mismatch	Notable limitation
npm	`npm ci`	Always fails, never writes lockfile	Output can be verbose; sensitive to npm version differences
pnpm	`pnpm install --frozen-lockfile`	Fails; this is the default when pnpm detects a CI environment	Different `node_modules` layout can expose code relying on undeclared dependencies
Yarn (Berry)	`yarn install --immutable`	Fails if the lockfile would change	Migration from classic Yarn changes config and plugins
Yarn (classic)	`yarn install --frozen-lockfile`	Fails	Classic is in maintenance mode

If you want strict lockfile enforcement without remembering a flag, pnpm is the one that turns it on by default in CI. Its real cost is that its non-flat node_modules will surface imports of packages you never declared, which means a migration day of fixing those. If you stay on npm, npm ci is already strict enough; your gap is version pinning, not the tool. Yarn Berry's --immutable is equally strict, but expect to learn its configuration model.

The takeaway: all three package managers can fail CI on drift, so switching tools matters less than pinning the tool version and running the strict command.

FAQ

Why does npm ci fail but npm install works?

npm install updates the lockfile to match package.json, hiding the mismatch, while npm ci refuses to modify it and fails instead. Run npm install, commit the updated lockfile, and npm ci will pass.

Should I delete package-lock.json to fix npm ci?

Only as a last resort, because regenerating from scratch re-resolves every dependency to the newest version its range allows. Try npm install with the CI-matching npm version first, which keeps existing resolutions where possible.

Why does npm ci fail on Linux with a missing @rollup or esbuild binary?

The lockfile was generated on another OS and lacks the Linux optional dependency entries. Regenerate the lockfile with a recent npm and verify with npm ci in a Linux container.

Bottom line

If you use npm, pin the npm version with packageManager and make npm ci plus a lockfile diff check your first CI step. If drift keeps happening on a team with mixed machines, pnpm's default frozen-lockfile behavior in CI removes one class of mistakes, at the price of a migration. Yarn Berry fits teams already invested in it. Whichever you pick, never paper over this error with --force.

Intermittent 502s and "read ECONNRESET" Behind a Load Balancer: The Keep-Alive Timeout Mismatch

Libme — Tue, 13 Oct 2026 22:14:30 +0000

Short answer: if you see a trickle of 502s (or read ECONNRESET / socket hang up in clients) that has no pattern in your app logs, check whether your app server closes idle connections sooner than the proxy in front of it. The rule: the component behind must keep idle connections open longer than the component in front. Node's default keepAliveTimeout is 5 seconds; many load balancers default to 60. That gap is a race condition, and it only fires under moderate traffic.

What does the failure look like?

The symptoms I've seen, and that are commonly reported for this problem:

A low, steady rate of 502 Bad Gateway (or 504) responses from the proxy, often well under 1% of requests.
The app's own access log shows nothing for those requests. They never arrived, or arrived on a socket that was already dead.
Errors cluster in "quiet-ish" periods, not at peak. At peak, connections are reused so fast they never idle out. At zero traffic, nothing is sent. The middle is where it hurts.
Client-side, you may see Error: read ECONNRESET or Error: socket hang up from Node, ConnectionResetError: [Errno 104] Connection reset by peer from Python, or upstream prematurely closed connection while reading response header from upstream in the nginx error log.

Dead ends I (and most people) try first: raising app memory, adding retries in the app, blaming the database, increasing proxy read timeouts. None of those touch the cause.

Why does it happen?

HTTP keep-alive lets the proxy reuse one TCP connection to your app for many requests. Both sides run an idle timer. If your app's timer is 5 seconds and the proxy's is 60, the proxy happily believes a connection is still good at second 5.0 while your app is closing it. If the proxy sends a request in that exact window, the request lands on a socket whose FIN is already in flight. The app never processes it; the proxy sees a reset and returns 502.

The window is tiny, which is why the error rate is low and why it looks random. It also scales with traffic volume in the "moderate" band, because you need enough requests for some to land in the window, but enough gaps for connections to idle at all.

Takeaway: an idle-timeout mismatch is a race, so it produces rare, patternless errors that no amount of app-level debugging will explain.

Which defaults conflict? (as of October 2026)

Defaults change between versions, so verify against your own versions. These are the ones I'd check first:

Component	Setting	Typical default	Notes
Node.js `http` server	`server.keepAliveTimeout`	5 seconds	Too short for most proxies
Node.js `http` server	`server.headersTimeout`	60 seconds (in recent versions)	Must exceed `keepAliveTimeout`
AWS Application Load Balancer	idle timeout	60 seconds	Configurable per load balancer
nginx (as a proxy to upstream)	`keepalive_timeout` inside `upstream {}`	60 seconds	Only applies if you enable upstream `keepalive` connections
Gunicorn	`--keep-alive`	2 seconds	Very short; matters when a proxy reuses connections
Go `net/http` server	`IdleTimeout`	falls back to `ReadTimeout`, else no limit	Set it explicitly

The correct ordering is always: client idle < proxy idle < app idle, measured at each hop. So for an ALB at 60 seconds, the app should be comfortably above that.

How do you fix it in Node.js?

Set the app's keep-alive timeout above the proxy's idle timeout, and keep headersTimeout above that:

const http = require("http");

const server = http.createServer((req, res) => {
  res.end("ok");
});

// Proxy (e.g. an ALB) idles out at 60s, so stay above it.
server.keepAliveTimeout = 65_000;
// headersTimeout must be greater than keepAliveTimeout, or Node can
// still drop connections early.
server.headersTimeout = 66_000;

server.listen(3000);

Frameworks like Express and Fastify sit on top of this, so you set it on the underlying http.Server (the value returned by app.listen() in Express). Fastify exposes keepAliveTimeout as a server option.

For Go, set it explicitly instead of inheriting:

srv := &http.Server{
    Addr:        ":3000",
    Handler:     mux,
    IdleTimeout: 65 * time.Second,
}

For Gunicorn, --keep-alive 65 (and note that the default sync worker doesn't really support keep-alive; a threaded or async worker class does).

Takeaway: pick one number for the proxy's idle timeout, then set every server behind it to at least 5 seconds more.

How do you prove it's the cause before changing anything?

Don't just apply the fix and hope. A quick reproduction: set the app timeout very low and hammer it at intervals near that value.

// repro.js: expects the server above with keepAliveTimeout = 1000
const http = require("http");
const agent = new http.Agent({ keepAlive: true, maxSockets: 1 });

function hit() {
  return new Promise((resolve) => {
    http
      .get({ host: "localhost", port: 3000, agent }, (res) => {
        res.resume();
        res.on("end", () => resolve("ok"));
      })
      .on("error", (e) => resolve(e.code));
  });
}

(async () => {
  const results = {};
  for (let i = 0; i < 200; i++) {
    const r = await hit();
    results[r] = (results[r] || 0) + 1;
    // Wait close to the server's idle timeout to land in the race window
    await new Promise((r) => setTimeout(r, 1000));
  }
  console.log(results); // ECONNRESET shows up in the tally
})();

With the server timeout at 1 second and the client waiting about 1 second, you should see some ECONNRESET results. Raise the server timeout to 5 seconds and rerun, and they should disappear. You are reproducing the race at small scale. In production, correlate by checking the proxy's logs: ALB access logs record the backend status as - for these cases, and nginx logs the prematurely closed line above.

One more detail: since Node 19, the global HTTP agent uses keep-alive by default, so outbound calls from your Node services to other services can hit the same race in the opposite direction. If a downstream closes idle sockets quickly, you get ECONNRESET on your side. Retrying idempotent requests once on ECONNRESET is a reasonable second line of defense, but fix the timeout first.

Is a managed load balancer worth it for this?

The mismatch exists with any proxy, managed or self-hosted. The difference is visibility. If you want the idle timeout as a single documented setting you can read in one place, AWS Application Load Balancer is the one that exposes it per load balancer with a clear default. Its limitation is that it only solves your side of the bargain; it can't know what your app does, and the fix still lives in your code. Self-hosted nginx gives more control, including upstream keepalive tuning, but you own every timeout and every default across versions.

FAQ

Why do I get intermittent 502 errors from my load balancer when my app logs show nothing?
Most often the app closed an idle keep-alive connection just as the load balancer reused it, so the request never reached your code. Raise the app's keep-alive timeout above the load balancer's idle timeout.

What should Node.js keepAliveTimeout be set to behind an AWS ALB?
Higher than the ALB idle timeout, which defaults to 60 seconds, so something like 65 seconds, with headersTimeout set slightly above that.

Is read ECONNRESET always a keep-alive problem?
No. It also appears when a process crashes, a container is OOM-killed, or a firewall drops idle connections. The keep-alive race is the likely cause when it is rare, patternless, and your app logs and metrics look healthy.

Bottom line

If you run Node, Gunicorn, or any app server behind a load balancer or reverse proxy, set the app's keep-alive timeout explicitly rather than inheriting defaults, and make it longer than the proxy's idle timeout. Reproduce with a low timeout if you need to convince yourself or your team. If errors persist after the fix, move on to crashes, memory limits, and network devices that drop idle flows.

Scale-to-Zero vs Provisioned Managed Postgres: Which Pricing Model Fits Your Traffic?

Libme — Mon, 12 Oct 2026 17:10:58 +0000

TL;DR: Scale-to-zero Postgres (billed mostly on compute time actually used) wins when your database is idle most of the day, such as side projects, staging, and per-PR previews. A provisioned instance (billed per hour whether or not anyone queries it) wins once your database is busy for a large fraction of the month, because you stop paying a premium for elasticity you no longer use. The break-even is a ratio of active hours to total hours, and you can compute it in ten lines of code with your own provider's current rates.

I'm deliberately not publishing a price table. Rates change, and a stale number in a reference post does more harm than none. What lasts is the shape of each billing model, and that shape is what decides your bill. Everything provider-specific below is as of October 2026; check the pricing page before you commit.

What are the three billing models for managed Postgres?

Almost every managed Postgres offering falls into one of these:

Provisioned instance. You pick an instance size and pay per hour while it exists, plus storage, backup storage, and sometimes provisioned IOPS. Amazon RDS for PostgreSQL is the canonical example. Running idle costs the same as running flat out.
Plan-based project. You pay a flat monthly tier per project that bundles a compute size, storage and bandwidth, with overage charges beyond the bundle. Supabase works this way as of this writing. Free-tier projects have historically been paused after a period of inactivity, which is a limitation, not a feature.
Usage-based, scale-to-zero. Compute is billed by the time (and size) it is actually running, and it suspends when idle. Neon is the best-known Postgres example, and Aurora Serverless v2 added the ability to scale down to zero capacity with auto-pause in late 2024.

The key takeaway: the pricing model decides your bill more than the headline "per GB" or "per vCPU" rate does.

Why does scale-to-zero stop being cheaper?

Two mechanics erode the advantage, and both bit me when I modeled a small production workload.

Rate premium. Per-hour compute on elastic offerings is usually priced higher than the equivalent always-on capacity, because the provider is carrying idle capacity for you. That premium is invisible while your database sleeps 20 hours a day. It dominates once it sleeps 2.

Cold starts keep it awake. A scale-to-zero database suspends after an idle timeout. If anything pings it more often than that timeout (a health check, a cron job, a monitoring probe, an ORM that opens a connection at boot), it never sleeps, and you are paying the elastic premium for 24 hours a day. If you adopt this model, the first thing to audit is who is touching the database on a schedule. I covered the health-check side of that in a separate post on deep health checks; the same probe that makes monitoring honest can quietly defeat auto-suspend.

The key takeaway: scale-to-zero only saves money if the database actually reaches zero, so find every periodic connection before you assume it will.

How do I compute the break-even for my workload?

Plug in your provider's current numbers. The script below compares a provisioned instance against an elastic one at a given active-time fraction, and finds the fraction where they cross.

def monthly_cost(active_fraction, elastic_rate, provisioned_rate,
                 storage_gb=20, storage_rate=0.0, hours=730):
    """Rates are dollars per compute-hour. Replace with current provider numbers."""
    elastic = elastic_rate * hours * active_fraction + storage_gb * storage_rate
    provisioned = provisioned_rate * hours + storage_gb * storage_rate
    return elastic, provisioned


def break_even(elastic_rate, provisioned_rate):
    # Elastic is cheaper while active_fraction < provisioned_rate / elastic_rate
    return provisioned_rate / elastic_rate


if __name__ == "__main__":
    # Placeholder rates for illustration only: NOT real provider prices.
    ELASTIC, PROVISIONED = 0.16, 0.10
    print(f"break-even active fraction: {break_even(ELASTIC, PROVISIONED):.0%}")
    for frac in (0.05, 0.25, 0.50, 0.75, 1.00):
        e, p = monthly_cost(frac, ELASTIC, PROVISIONED)
        print(f"active {frac:>4.0%}  elastic ${e:7.2f}  provisioned ${p:7.2f}")

With the made-up rates above, the elastic option is cheaper until the database is active about 62% of the month. The numbers are fake on purpose; the structure is the point. If your elastic rate is 1.6 times your provisioned rate, you need to be idle more than roughly 38% of the time to come out ahead. Real offerings also add minimum compute floors, storage that is billed even when compute is asleep, and per-project fees, so extend the function before trusting it.

The key takeaway: the break-even active fraction equals the provisioned hourly rate divided by the elastic hourly rate, and nothing else about the sticker price matters until you know it.

Which model fits which workload?

Workload	Best-fit model	Why	Main drawback
Side project, a few visits a day	Scale-to-zero	Sleeps most of the month	Cold-start latency on the first request
Staging and per-PR preview databases	Scale-to-zero	Dozens of mostly idle copies	Branching and cleanup need discipline
Early-stage app with steady daily use	Plan-based project	Predictable monthly line item	Overage charges and plan jumps are lumpy
Production with constant traffic, 24/7	Provisioned instance	No elasticity premium, tunable parameters	You pay for idle headroom and manage sizing
Spiky B2B app, busy in work hours only	Scale-to-zero or plan-based	Nights and weekends are free	Autoscaling can lag behind a sudden burst

Preview environments are where scale-to-zero is hardest to beat. If you are creating a database per pull request, a provisioned instance per PR is absurd, and a branchable, suspend-on-idle database is the natural fit. I wrote about seeding those environments without cloning production data in an earlier post.

The key takeaway: match the billing model to the duty cycle of the workload, not to the size of the data.

What are the hidden costs that aren't compute?

Compute is rarely the surprise. These are:

Storage billed while compute sleeps. Suspended does not mean free.
Backups and point-in-time recovery. Retention windows are priced separately on most providers; longer windows cost more.
Multi-AZ or high availability. On RDS, a standby replica roughly doubles the compute line. Whether you need it is a real decision, not a default.
Connection limits and pooling. Small compute sizes have low connection caps, which pushes you toward a pooler. Serverless functions make this worse, as covered in the post on transaction-mode pooling.
Egress. Data leaving the provider's network is billed on most clouds, and cross-region traffic more so.

None of these appear in the "from $X per month" headline.

The key takeaway: price the full bundle (compute, storage, backups, HA, egress) at your expected duty cycle, because the headline number covers only the first item.

FAQ

Is scale-to-zero Postgres production-ready?
Yes for workloads that tolerate a cold start of up to a second or so on the first request after idle time, and no for latency-sensitive paths that can't. Measure your own cold-start time on your provider and region; don't trust a blog post, including this one.

Why is my scale-to-zero database never suspending?
Something is connecting more often than the idle timeout, usually a health check, cron job, monitoring probe, or connection pool with keepalive. Check the connection logs for recurring clients and lengthen or remove the periodic connection.

Is RDS cheaper than Neon or Supabase?
It depends on the duty cycle. RDS is typically cheaper per active hour for a database that is busy around the clock, while scale-to-zero and plan-based options win when the database is idle for much of the month.

Bottom line

If your database is idle most of the day, start with a scale-to-zero option such as Neon, and audit every scheduled connection so it can actually suspend. If you want a flat monthly bill bundled with auth and storage for an early product, Supabase is the plan-based choice, with the caveat that plan jumps and overages are lumpy. If your database is busy most of the month and you need tuned parameters or high availability, a provisioned Amazon RDS for PostgreSQL instance is usually the cheaper and more controllable home. Whichever you pick, run the break-even script with this month's real rates and re-run it when your traffic shape changes.

DEV Community: Libme

FastAPI Latency Spikes Under Load: How One Blocking Call in an async Endpoint Stalls Every Request

What does the symptom look like?

Why does def work when async def doesn't?

How do I find the blocking call?

Which fix should I use?

Can running more workers hide the problem?

FAQ

Bottom line

Related reading

GitHub Push Rejected: "File Exceeds 100 MB" (GH001) — Fix It With filter-repo or LFS

What does the error actually look like?

Why doesn't git rm and a new commit fix it?

Option A: Remove the file entirely with git filter-repo

Option B: Keep the file but move it to Git LFS

Which fix should you use?

How do you stop this from happening again?

FAQ

Bottom line

Related reading

Sentry vs GlitchTip vs Self-Hosted Sentry: Error Tracking for a Small Team When the Event Quota Keeps Running Out

Why does the error quota run out in the first week?

Fix the volume first: client-side dedupe and sampling

Sentry vs GlitchTip vs self-hosted Sentry: what actually differs?

When is hosted Sentry worth paying for?

FAQ

Bottom line

Related reading

Docker "toomanyrequests: You have reached your pull rate limit" in CI: Log In, Mirror, or Run a Pull-Through Cache?

What does the error actually look like?

How do I check how many pulls I have left?

Fix 1: authenticate every pull (five minutes)

Fix 2: stop pulling from Docker Hub at all

Which option fits which setup?

FAQ

Bottom line

Related reading

ERR_REQUIRE_ESM in Node: Fix CommonJS/ESM Conflicts and Pick tsx, ts-node or tsup

What does ERR_REQUIRE_ESM actually mean?

Does newer Node make this go away?

How do you fix it permanently?

"Unknown file extension .ts": which TypeScript runner should you use?

FAQ

Why does ERR_REQUIRE_ESM only happen after npm update?

Can I use require() on an ES module?

Why do I need .js extensions in TypeScript imports?

Bottom line

Related reading

Node.js "fetch failed" with ETIMEDOUT or ENETUNREACH While curl Works: The Broken IPv6 Route

What does the error actually look like?

Why does curl work when Node does not?

How do I confirm it is IPv6?

Which fix should I use?

Is this a reason to pay for anything?

FAQ

Bottom line

Related reading

ngrok vs Cloudflare Tunnel vs Tailscale Funnel: Which One for Testing Webhooks on Localhost?

What problem are you actually solving?

How do the three compare?

When is ngrok the right pick?

When does Cloudflare Tunnel win?

Is Tailscale Funnel a good fit for webhooks?

FAQ

Bottom line

Related reading

Postgres "deadlock detected": Fix the Lock Order First, Then Add a Retry

What does the error actually say?

Why does it only happen under load?

How do you find the two queries involved?

How do you fix it: consistent lock ordering

What should the retry look like?

FAQ

Is a Postgres deadlock a bug in my database?

Should I just retry on deadlock detected?

What is SQLSTATE 40P01?

Bottom line

Related reading

Postgres "could not resize shared memory segment: No space left on device" in Docker: Why It Happens and How to Fix It

What does the error look like?

Why does `def` work when `async def` doesn't?

Why doesn't `git rm` and a new commit fix it?