Two short deadlines took down Inngest's database path twice in one afternoon, and neither was misconfigured. They expired exactly on time. That is what made the failure so effective.
On 10 October 2026, both of Inngest's connection poolers lost every connection to their primary database at 14:00 UTC. They lost them again at 15:30. Run scheduling and queue processing stalled for most accounts in both windows, 14:00–14:48 and 15:30–16:25, and the mitigation that followed caused intermittent delays until 04:17 the next morning.
The report is worth reading because the mechanism generalises to any stack that puts a connection pooler with a wait queue between the application and Postgres.
The sequence, in four steps
- A frequently used lookup carried a short deadline.
- The deadline expired while its query was still queued inside PgBouncer, waiting for a database connection. The pgx driver cancelled the query and closed its client connection.
- PgBouncer had nothing running to cancel, because the query had never been handed a server connection. It later ran the abandoned query on a database connection.
- It then closed that connection entirely instead of returning it to the pool.
Step 4 is the whole failure. The client's timeout cost the system a database connection, not a request, and the cost landed after the client had stopped watching. No single request looked expensive. Individually, every one of these is a wasted query. Collectively they are a pool that empties itself.
Two amplifiers arrived at the same moment. Both collapses began at the top and bottom of the hour, when cron-scheduled work roughly triples the query rate, and a newly rolled out set of lookups was failing in large numbers, adding cancellations to a pool that was already busy. Inngest reproduced the behaviour in a test environment.
Why restarting and resizing both failed
Restarting the poolers did not reliably recover them. Raising the pool size did not help either. The report is explicit on why: opening connections was the bottleneck, not the pool limit.
Each pooler opens new database connections one at a time. So the refill path is serial while the destruction path is parallel. As a steady-state sketch rather than a model of this incident, if usable connections are destroyed at d per second and a pooler needs t seconds on average to establish a replacement, the refill ceiling is 1/t and the pool drains whenever:
d > 1/t
The sketch assumes constant destruction and handshake times, and it ignores the point at which the pooler stops trying. Inngest published neither number, so treat this as the shape of the constraint rather than a measurement of it. To see the shape: at 100 ms per connection a single pooler refills at ten per second; at 50 ms, twenty.
The second collapse ended by changing paths, not by growing the pool. The team moved all services onto poolers run alongside the database by the database provider, and processing recovered between 16:10 and 16:25.
The fix that caused a second incident
The new route reached the database through NAT gateways, and some connections through those gateways were silently dropped. Queries on a dropped connection hung for up to 16 minutes before failing, which stalled processing intermittently and most visibly hit cron functions. Customers reported delays at 03:01 on 11 October; the affected service moved back to the original poolers at 04:17, and gateway routing was improved on 12 October.
Sixteen minutes of silent hang is worse than a fast failure for a scheduler. A connection that errors invites a retry. One that simply stops answering consumes a worker slot and a deadline, and you learn about it from a customer. A failover drill that measures successful requests will not catch this. One that measures how long a hung connection stays hung will.
The contract that broke underneath it
The part of the report with the longest reach is not about pools. Internal failures counted against functions' configured retries, and a step that failed permanently on an internal error left its run in Running instead of marking it failed. onFailure handlers did not fire. Functions waiting on that run through step.invoke were not notified.
That is a broken guarantee, not an availability number. A durable execution engine sells the promise that every run reaches a terminal state and that failure handlers run. Under this bug the engine could hold a run in Running indefinitely, after spending the retry budget it was given — the customer's allowance consumed by the platform's own faults, and the transition that would have reported it never sent.
The recovery instructions confirm the state of those runs: they will not continue on their own, and customers can cancel and replay them, with the caveat that a replay starts from the beginning, so completed steps run again. Anyone with non-idempotent side effects in a completed step had a decision the platform could not make for them.
Four things to check in your own stack
Alert on net connection drain, not pool utilisation. Graph opens and closes per pooler per second, and alert when the close rate exceeds the open rate for a sustained interval. A pool at 30% with a negative slope is in more danger than one at 70% that is stable, and utilisation alone shows neither.
Bound the wait, not just the query. PgBouncer's query_wait_timeout disconnects a client whose query has not been assigned a server within the limit.
; pgbouncer.ini — the ordering matters more than the value
query_wait_timeout = 5 ; must sit below your client's own deadline
The point of setting it below the client deadline is to remove the queued entry before the client gives up. That is the reasoning, not proof that it preserves backend connections. Inngest describes the pooler running the abandoned query and then closing that connection; an independent load test from July 2025 puts the waste a step earlier, with the pooler finishing a login for a client that has already gone and discarding the fresh backend connection. The same engineer reported that lowering the timeout helped without being enough on its own. Test it against your driver's real cancellation path before adopting it.
Know what your driver does on cancel. In pgx v5, the thread reports replacing the default cancellation behaviour with a context watcher handler that sets a connection deadline instead of closing immediately, and describes the effect in their load test as large. If your code path cancels queued queries, this is the boundary to inspect.
Fence the lease before you force a terminal state. If you mark a run failed because its step lease expired without a heartbeat, late writes from a worker that is still running can land afterwards, and a replay produces a second copy of the side effects. Mark it failed only when the engine can reject writes from the old lease, or when the step is confirmed not to be in flight.
What the report does not establish
Inngest disclosed no connection counts, handshake timings, pool sizes or customer totals, so the arithmetic here is the report's timestamps and nothing more. The 44-minute, 48-minute and 55-minute windows are my subtraction on its UTC boundaries, not figures it published.
The report also links pgx issue 1780 as the known interaction behind the collapse. I read that issue: it was opened on 26 October 2023 and was still open at its latest comment in February 2026. The upstream record offers a compatible description of the boundary and no confirmed root cause, so Inngest's account is a well-documented instance of an unresolved interaction rather than evidence that it is fixed.
The structural change is the one to watch. Inngest is consolidating onto a single data centre acting as one cell, which removes the cross-data-centre failure class behind the 9 October incident. That trades one failure shape for another: fewer paths that can break, and more consequence when one does.
Sources
Thanks for reading :)
Top comments (0)