A few months ago we spent most of a morning chasing a latency problem that had every symptom of database overload and none of the causes.
The alert was on our main API's p95, which had climbed from about 180ms to over 9 seconds during an ordinary weekday afternoon. Not a traffic spike — request volume was within 10% of the previous day. Requests weren't failing outright at first, they were just queuing somewhere and eventually hitting our 10-second gateway timeout.
The first place we looked was the database, because that's where "slow" always seems to live. Postgres was at 15% CPU. The slow query log was empty. Replication lag was under a second. Every individual query we ran by hand came back in single-digit milliseconds. The database was, by every metric we had, bored.
Where the time was actually going
We had request timing instrumentation, but it measured the wrong boundary. It started the clock when a handler issued a query and stopped when the rows came back. That interval was fine — 4ms, 7ms, 11ms. What it didn't measure was the gap between "the handler wants a connection" and "the handler has one."
Once we put a span around connection acquisition specifically, the picture inverted immediately. Query execution: 6ms. Waiting for a connection from the pool: 8.4 seconds.
Our pool had a max size of 20. That number had been set roughly two years earlier by copying a default, and had never been revisited. It was fine for a long time, because our handlers held connections for a few milliseconds each. What changed wasn't the traffic — it was what the handlers did while holding a connection.
The actual bug: holding a connection across a network call
A feature we'd shipped a few weeks earlier did this, in effect:
conn = pool.acquire()
record = conn.query("select ... from jobs where id = $1", id)
enriched = http.get(vendor_api, record.external_id) # 300-800ms, external
conn.execute("update jobs set ... where id = $1", id)
conn.release()
The database work here is trivial. But the connection stays checked out across an external HTTP call that averaged around 400ms and had a p99 near two seconds. With 20 connections and a handler occupying one for ~400ms, we could serve roughly 50 of those requests per second before the pool became the ceiling — and every other endpoint in the service, including ones that touched no jobs at all, queued behind them.
That's the part that makes this class of bug nasty. The failure isn't localized to the slow feature. A pool is a shared resource, so one handler with a bad hold time degrades every endpoint sharing it. Our health check endpoint was timing out, which is exactly what made this look like an infrastructure problem rather than a code problem.
What we changed
Three things, in order of how much they mattered.
We stopped holding connections across anything that isn't a query. The fix in the code above is to release the connection before the HTTP call and acquire a second one for the update — two short holds instead of one long one. That alone took pool wait from 8.4 seconds back to sub-millisecond.
We added a hold-time metric and alerted on it. Not pool utilization — hold time per checkout, at p99. Utilization tells you the pool is full; hold time tells you why. We alert when p99 hold time exceeds 100ms, which is generous for a pool that should only be serving queries, and it catches this class of regression the day it ships instead of weeks later.
We set an explicit acquisition timeout. Previously a handler would wait indefinitely for a connection, which is how a saturated pool turns into an unbounded queue and then into memory pressure. Now acquisition fails after 2 seconds and the request returns a 503. Failing fast isn't a fix, but it keeps one degraded dependency from consuming the entire service's capacity.
We did also raise the pool size, from 20 to 40 — and that was the least important change of the three. We were deliberately cautious about it, because a bigger pool would have masked the real problem for a few more weeks and then pushed the eventual failure down onto the database itself, where it would have been considerably harder to recover from.
The general lesson
Connection pools fail in a way that points at the wrong component. Every symptom — timeouts, queuing, latency that climbs with load — looks like a database problem, while the database sits at 15% CPU insisting it isn't. The only way to see it is to measure the wait for a resource separately from the use of that resource.
We now do that for every pooled resource we have: database connections, HTTP client pools, worker slots. It's about ten lines of instrumentation, and it's the difference between a twenty-minute diagnosis and a four-hour one.
If a pool has a max size, something will eventually sit at that ceiling. Better to hear about it from a metric than from a pager.
We write these up as we run into them. We're the engineering team at Edilec, where we build and maintain backend systems for growing products.
Top comments (1)
Measuring acquisition separately from query execution is the diagnostic that matters here.
One correctness caveat with the two-short-checkouts fix: releasing the connection before the vendor call also removes the transaction snapshot/lock that connected the initial read to the later update. The job can change while the network call is in flight.
I would make that boundary explicit:
updated_at).enrichment_in_progressstate/lease in a short transaction.UPDATE ... WHERE id = ? AND version = ?(or compare-and-swap the lease owner), then check the affected-row count.That keeps the pool healthy without silently overwriting a concurrent change. For higher-value side effects, a durable outbox/state machine makes crash recovery visible instead of leaving “vendor succeeded, database update did not” indeterminate.
Operationally, I would graph acquisition wait, checkout duration, active/idle/waiting counts, request deadline remaining at checkout, and holds by endpoint/stack trace. The pool cap also needs to be budgeted across every process and replica, with database headroom—not tuned per instance in isolation.
Finally, the acquisition timeout should be shorter than the request deadline and paired with load shedding. Otherwise the service can spend two seconds waiting for a connection after there is no longer enough budget to complete the operation.