Little's Law Explains Why Your Connection Pool Fix Actually Worked

The problem

Earlier this week I wrote about a team that cut their Postgres connection pool by 90% and watched P99 latency drop from 4.1s to 80ms. Every time that story comes up, someone asks the same question: how do you know the right pool size instead of just guessing and hoping?

There's a formula for this. Most engineers met it once, filed it under "queueing theory, not my problem," and never touched it again. That's a mistake — it's one of the few genuinely load-bearing formulas in systems work.

Why it happens

Little's Law: L = λW

• L = average number of things in the system (requests in flight, jobs queued, connections checked out)
• λ = average arrival rate (requests/sec)
• W = average time each thing spends in the system (latency)

What makes it unusual is that it holds for any stable system, regardless of arrival pattern, service time distribution, or scheduling policy. It's not a statistical model with assumptions you can violate — it's closer to a conservation law, like counting cars on a highway. It only breaks if the system isn't in steady state.

Here's the part that explains the connection pool story: L isn't a free variable you get to pick independently. If λ is fixed (your traffic isn't going away) and you increase L (more concurrent connections, bigger pool), the law says W has to move too — and not always in the direction you want.

Past a certain point, more concurrency doesn't buy you more throughput; it buys you contention. Locks, cache lines, a connection limit on the database side — something starts queueing invisibly. That contention inflates W. And because L = λW isn't optional, an inflated W with steady λ means L keeps climbing: more requests in flight, not because you're serving more, but because everything is stuck waiting longer. The pool that was supposed to give the database room to work was actually the thing manufacturing the queue.

What to do about it

Use the law forward, not just as an explanation after the fact:

1. Measure λ — your actual arrival rate, from real traffic, not capacity-planning guesses.
2. Decide your target W — the latency you're actually willing to tolerate.
3. Solve for L — that's your target concurrency: pool size, worker count, whatever "in flight" means for your system.

If your current L is far above λ × W_target, you're not under-provisioned — you're backed up, and adding more capacity to the same contended resource will make W worse, not better. That was the trap in this week's story: the fix wasn't more connections, it was fewer, because fewer meant less contention, which meant lower W, which the law forces back into a smaller L for the same throughput.

The other use is live, during an incident. Queue depth (L) and request rate (λ) are usually easy to read off a dashboard even when your latency percentiles are lying to you — sampled, delayed, or only counting requests that finished. W = L / λ gives you a real average latency number you can trust in the middle of a page, before your APM tooling catches up.

Key takeaways

• Little's Law (L = λW) holds for any stable system — no distribution assumptions required.
• It's a relationship between averages, not a target to hit — you don't get to move one variable without the others responding.
• More concurrency past a contention point doesn't raise throughput; it raises W, which the law forces back into a bigger L — a growing backlog dressed up as "more capacity."
• It only holds in steady state — apply it to a bursty, non-equilibrium window and it'll look "wrong," when really the assumption broke, not the law.
• During an incident, L and λ are often more trustworthy signals than your latency dashboard — use them to back into W directly.