On October 10, 2026, Inngest's database connection pool collapsed twice within two hours, stalling run scheduling and queue processing for most accounts. According to Inngest's own October 15 incident report, the first collapse ran from 14:00 to 14:48 UTC and the second from 15:30 to 16:25, and the fix the team applied between the two outages caused a third, separate round of intermittent delays that lasted until the following morning.
Jump straight to what we checked on our own pooling setup if you run PgBouncer in front of Postgres today.
What Caused the Connection Pool Collapse
Inngest runs two PgBouncer poolers in front of its primary Postgres database. Short client-side deadlines on frequent database lookups interacted badly with how PgBouncer handles a query that is still waiting for a connection when its deadline expires. The driver cancelled the query and closed its own client connection, but PgBouncer had nothing left to cancel on its side, so it later ran the now-abandoned query to completion on a real database connection, then closed that connection entirely instead of returning it to the pool.
Each pooler opens new database connections one at a time, and connections were being destroyed by this pattern faster than new ones could replace them. According to the report, both collapses began at the top and bottom of the hour, when cron-scheduled work roughly triples the query rate, and a newly rolled-out set of lookups added to the cancellation volume at the same moment.
Why Restarting the Poolers Didn't Fix Anything
Restarting the collapsed poolers did not reliably bring them back. Raising the pool size limit did not help either, because the bottleneck was the rate of opening new connections, not the ceiling on how many could exist.
The team's mitigation moved traffic to poolers run alongside the database by its provider, and that stopped the second collapse between 16:10 and 16:25. But the new route reached the database through network address translation gateways that silently dropped some connections, and queries on a dropped connection hung for up to 16 minutes before failing, the source of a third, slower-burning incident that ran into October 11.
What We Checked on Our Own Infrastructure
We checked every service at Krova Cloud that sits behind a connection pooler for the exact pattern Inngest described: a client-side query deadline that is shorter than how long PgBouncer might hold a queued request. We found one internal billing service with a 400ms deadline against a pool that can legitimately queue for longer than that under load, the same mismatch that triggered Inngest's collapse. We raised that deadline and added a metric that counts connections PgBouncer destroys instead of reusing, since that number, not pool utilization, was the actual leading indicator in Inngest's own timeline.
What to Check in Your Own Pooler Setup
- Compare your database driver's query-cancellation timeout against PgBouncer's own connection-wait behavior. A cancelled client request does not guarantee PgBouncer cancels the same request on its side.
- Watch for a cron-driven top-of-hour query spike that lines up with any pooler instability you have dismissed as noise.
- Track connections destroyed per minute, not just pool size or utilization, since that was the real signal here and a healthy-looking pool size masked it.
- Test what happens when you move database traffic through a provider-managed pooler path during an incident, including whether that path runs through NAT gateways that can silently drop long-lived connections.
The catch is that none of Inngest's individual mitigations was wrong on its own. Restarting a stuck pooler is a reasonable first move, and routing around a collapsed pooler to a provider-managed alternative is a reasonable second move. The problem was that the second move introduced a new failure mode, the silently dropped NAT connection, that nobody had reason to test for until it was the only path left during a live incident.
Inngest's architectural response goes further than a patch: the company is consolidating from three data centers to one, which removes the cross-data-center dependency that caused a separate October 9 outage in the same incident window, the kind of structural fix I've argued for before when a postmortem's real lesson is architectural rather than a single line of config.
The Rule
A connection pool that looks healthy on a dashboard showing size and utilization can still be destroying connections faster than it replaces them, and that number does not show up unless someone adds it as its own metric. If your incident response plan for a collapsed pooler is "restart it, then fail over to a different pooler," test the failover path itself before you need it during a live outage, because that is exactly where Inngest's second mitigation introduced a new sixteen-minute hang nobody had seen before.
Frequently Asked Questions
What actually caused Inngest's connection pool collapse?
A mismatch between short client-side query deadlines and PgBouncer's handling of cancelled requests. When a deadline expired while a query waited for a connection, the driver cancelled it, but PgBouncer ran the abandoned query anyway and then destroyed the connection instead of returning it, per Inngest's October 15 incident report.
Did restarting the database poolers fix the outage?
No. Restarting the collapsed PgBouncer instances did not reliably recover them, and increasing the pool size limit did not help, because new connections were being destroyed faster than they could be opened. The team had to reroute traffic to a different pooler path entirely.
Why did the fix cause a second, separate incident?
The replacement route sent database traffic through network address translation gateways that silently dropped some connections. Queries on a dropped connection hung for up to 16 minutes before failing, which stalled processing again until the team reverted to the original poolers and improved the gateway routing.
Is this kind of connection pool collapse specific to Inngest's setup?
No. Any service running a client-side query deadline shorter than PgBouncer's own wait behavior for a queued connection can hit the same pattern. We found one internal example in our own infrastructure once we checked specifically for that mismatch rather than for pool size or utilization.
I'm Rohit, founder of Krova Cloud: disposable VMs that get torn down the moment a CI job finishes, so a connection pool that slowly destroys itself over hours of sustained load never gets the chance to. More debugging stories at debugly.dev.
Top comments (1)
The abandoned-query detail is the nastiest part — the pooler doing work nobody asked for, on a connection nobody owns anymore. It rhymes with a lesson I learned the hard way: intermediate layers report success about their own step, not the outcome. I spent a week trusting an SES 200 OK before I learned 200 means "accepted," not "delivered" — the email died silently downstream. Same shape here: PgBouncer "handled" the cancelled query by running it anyway. The top-of-the-hour cron tripling the query rate is the tell I'd put in a runbook — pool collapse always correlates with a traffic shape, never a random moment. Did your own audit find the pattern in your infrastructure, or was the deadline mismatch the whole story?