The error was FATAL: sorry, too many clients already, thrown by a Postgres instance that, by every dashboard we had, was using 40% of its CPU and half its RAM. Plenty of headroom. It rejected the connection anyway, which is the specific kind of outage that makes people distrust their own monitoring.
The number nobody had looked at
max_connections was set to 100, the Postgres default nobody changes unless something forces the question. We weren't near it on any given request — we were near it in aggregate, because every one of our six app instances kept its own connection pool of 20, and 6 × 20 is 120 before Postgres even finishes its own reserved slots for replication and superuser access.
SELECT count(*) FROM pg_stat_activity;
-- 97, climbing toward 100 during traffic spikes
Each connection, busy or idle, holds a backend process in Postgres with its own memory overhead — a few megabytes each, which is why "just raise max_connections to 1000" is the advice that fixes the symptom and quietly creates a memory problem three weeks later.
What actually fixed it
PgBouncer in transaction mode, sitting between the app and Postgres, multiplexing hundreds of app-side connections onto a much smaller pool of real backend connections:
[databases]
mydb = host=127.0.0.1 port=5432 dbname=mydb
[pgbouncer]
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 20
App-side, nothing changed except the port it connected to. Real Postgres connections dropped from 97 peak to a steady 20, with headroom that didn't depend on how many app instances we happened to be running that week.
Why this stayed invisible so long
The connection ceiling scales with instance count × pool size, and both of those numbers grow independently, usually for unrelated reasons — you scale instances for traffic, you scale pool size because someone hit a slow-query timeout once and bumped it. Nobody multiplies the two together until Postgres does it for you, out loud, during a traffic spike.
This is also where the machine underneath quietly matters. We'd been running Postgres on a shared-tenancy VPS where "half your RAM" was a number we trusted less than we should have — on an oversold host, headroom on a dashboard isn't the same guarantee it looks like. Since moving that database onto a Krova Cube, RAM is reserved 1:1 — no overselling, no thin provisioning — so when pg_stat_activity and free -h say there's room, there actually is room, and pooling fixes are fixes rather than guesses against an unknown neighbor's usage. I run Krova, so take the specific plug as informed rather than neutral, but the pooling fix works regardless of host.
What I'd check first
SELECT count(*) FROM pg_stat_activity; against SHOW max_connections;, and separately, (number of app instances) × (pool size per instance). If the second number is anywhere near the first, you don't have a Postgres problem, you have an arithmetic problem that Postgres is enforcing on your behalf.
Top comments (0)