The problem
Every few weeks, the same alert fired: API p99 latency climbing past 4 seconds under peak traffic, timeouts cascading through the stack, and a Slack thread full of people staring at a Postgres primary that was, by every dashboard we had, nowhere near maxed out. CPU sat at 40-50%. Disk I/O was fine. Memory was fine.
The fix that "worked" every time was the same one: bump the connection pool. Someone would raise max_connections on Postgres, raise the app-side pool size, redeploy, and latency would recover for a day or two. Then it would come back worse. Over about three months we crept from a sane pool configuration to 20 app pods each holding up to 100 connections — 2,000 potential connections aimed at a 16-core primary. The outages got worse, not better, and we kept reaching for the same lever because the error message ("pool exhausted," "too many clients already") looked identical every time.
Why it happens
Postgres uses a process-per-connection model — every connection is a full OS process, not a lightweight thread. Coordinating shared state across those processes (the buffer pool, the lock manager, WAL insertion) happens through internal lightweight locks. Up to a point, adding connections adds throughput, same as adding lanes to a highway. Past that point — the concurrency "knee" — every new connection adds more coordination overhead than it adds useful work. You get more processes competing for the same finite cores, more context switching, more time spent acquiring and waiting on internal locks relative to time spent executing queries.
This is coherency delay, the term Neil Gunther's Universal Scalability Law uses for it, and it's why the curve doesn't plateau — it retrogrades. More concurrent connections can mean fewer completed queries per second, because most of those connections are idle-in-transaction or queued behind a lock, and the scheduler is burning cycles switching between hundreds of mostly-idle processes instead of finishing the handful doing real work. That's also why CPU looked fine: the bottleneck wasn't compute, it was coordination, and coordination overhead doesn't show up as "CPU busy" on a basic dashboard.
This matches what you'll see in any pgbench scaling test on a fixed-core box: throughput climbs, peaks around roughly 2-4x the core count, and then falls as you keep adding concurrent connections beyond that.
What to do about it
1. Compute a target pool size instead of guessing. A widely-cited starting heuristic (popularized by HikariCP's Brett Wooldridge, borrowed from Baron Schwartz): connections ≈ (core_count × 2) + effective_spindle_count. For a 16-core primary on NVMe storage (effective spindle count ~1), that's around 33 — call it 40 with headroom. Not 2,000.
2. Size the pool for the database, not for the fleet. The mistake was letting "how many connections does Postgres allow" be a function of "how many app pods happen to be running." Put a pooler in front — PgBouncer in transaction pooling mode is the standard choice — so the app fleet can open however many logical connections it wants against PgBouncer, while PgBouncer multiplexes them down to a small, fixed number of real backend connections. App-side scaling and database-side connection count become two separate knobs instead of one.
3. Validate with Little's Law. L = λ × W: the concurrency you need (L) is your throughput (λ) times your average latency (W). At 2,000 queries/sec and 5ms average query time, you need roughly 10 connections doing real work at any instant — not 2,000. Most teams are wildly overprovisioned on paper for concurrency they don't actually need, while starving on the throughput they do.
4. Look for idle-in-transaction connections specifically. They hold locks and snapshots without doing work, and they're usually the actual leak — "not enough connections" and "too many connections" throw the identical error message, but the fixes are opposite. Check pg_stat_activity for idle in transaction before you touch the pool size.
After we put PgBouncer in transaction mode between the fleet and the primary and capped real backend connections around 40, p99 latency went from 4.1s to roughly 80ms — same hardware, same queries, no code changes. DB CPU dropped too, because it was spending cycles executing instead of switching between mostly-idle processes.
Key takeaways
- More connections isn't more capacity. Past the concurrency knee, added connections increase contention and can reduce throughput.
- Low CPU utilization doesn't mean you have room for more connections — check lock waits and idle-in-transaction time, not just CPU%.
- Size the pool for the database's hardware (cores + effective spindles), not for how many app instances happen to be running.
- Put a transaction-mode pooler between your fleet and the database so app-side scaling and DB-side connection count scale independently.
- Little's Law tells you the concurrency you actually need. Compute it before you touch the number.
Top comments (0)