CircleCI's Postgres disk space exhaustion on October 9, 2026 stopped all pipeline and workflow processing for roughly 12 hours, and the trigger was a scheduled cleanup job, not an unexpected load spike. According to CircleCI's own October 12 incident report, the fix the team shipped after a nearly identical October 6 outage removed one source of database pressure but never accounted for a second one already scheduled to fire three days later.
Jump to what we checked in our own queueing setup if you run Temporal, or any workflow engine, behind a size-limited log volume.
What Caused the Postgres Disk Space Exhaustion
CircleCI runs Temporal, backed by a Multi-AZ Postgres instance, as the reliable queue behind every pipeline and job. That Postgres instance stored its write-ahead log on a Dedicated Log Volume capped at 1TB and 3,000 IOPS, limits AWS does not allow raising. When the database writes its log faster than it can archive that log to backup storage, free space on the volume drops, and once it fills, Postgres stops accepting writes entirely.
The October 6 incident happened because routine table maintenance generated log writes faster than archiving could keep up. The fix moved the transaction log back onto its dedicated volume and changed Temporal's cleanup settings to reduce a major source of those writes. What the fix did not account for: Temporal deletes completed workflow history a few days after it finishes, so the heavy October 6 workload was scheduled to be deleted on October 9.
How a Scheduled Cleanup Became an Outage
At 08:00 on October 9, Temporal began processing those scheduled deletions. Free space on the log volume dropped from 0.8TB to 0.1TB by 10:10. Monitoring alerted at 10:28, the team declared an incident at 10:31, and by 11:15 the volume was full and Postgres went fully read-only, stopping all Temporal processing. The same fix that worked on October 6, moving the log onto the data volume, brought the database back by 12:59, and the backlog that had queued since 11:15 cleared by 14:40.
A second, slower failure started at 15:15. Routing-cleanup tables that normally hold tens of thousands of rows awaiting deletion held tens of millions of rows by late afternoon, and every Temporal operation had to scan past them, slowing the whole system down.
Manual vacuum jobs made this worse by competing with live queries for CPU, and the team cancelled them at 17:45. At 20:48, engineers realized Postgres had shifted from a log-writing bottleneck to CPU thrashing, compounded by a retry storm: upstream systems were resubmitting work that had already failed, adding load on top of the real backlog. Pausing all new and retried work was what finally let the database catch up.
What We Checked in Our Own Queue
We checked every queue-backed service at Krova Cloud for a scheduled cleanup or retention job that fires independently of current load, the exact shape of CircleCI's second incident. We found one internal metrics pipeline that purges records on a fixed nightly schedule regardless of how much backlog already exists that day, and added a check that skips or staggers the purge when the queue is already under pressure.
What to Check in Your Own Queueing Stack
- Confirm whether your workflow engine's history deletion runs on a fixed schedule or on current load, since a scheduled job that ignores system state can fire during exactly the wrong hour.
- Alert on write-ahead-log archive lag and write rate directly, not just on disk free space, since CircleCI's postmortem says this would have caught the problem hours earlier.
- Treat a fix for one incident as unverified until you have checked it against every other job scheduled to run in the following days, not just the triggering condition you already found.
- Review whether your own retry logic backs off when a downstream dependency is already struggling, since CircleCI's retry storm added real load on top of a real outage.
The verdict here is not that CircleCI's October 6 fix was wrong. It genuinely removed the write pressure it targeted. The catch is that a fix scoped to the condition you just diagnosed can still leave a second, delayed trigger sitting on the calendar, and nothing about the first incident's symptoms would have pointed at it.
This has the same shape as the connection pool collapse I wrote about at Inngest: a real mitigation for incident one created, or in this case simply failed to anticipate, the exact conditions for a second, separate incident within days of the first.
The Rule
A postmortem fix that targets the specific write pattern you just diagnosed is not the same as a fix that accounts for everything else already scheduled to touch the same resource. CircleCI's own deletion job, set in motion days earlier by the very incident it was supposed to clean up after, is the clearest version of that gap: the fix and the next failure were already on the same calendar, and nobody checked the date.
Frequently Asked Questions
What caused CircleCI's October 9, 2026 outage?
A scheduled Temporal history-deletion job began processing a backlog from the October 6 incident, generating write-ahead-log writes faster than Postgres could archive them. The dedicated log volume filled, and Postgres went read-only, per CircleCI's October 12 incident report.
Was this the same problem as CircleCI's October 6 outage?
A related but distinct trigger. Both incidents involved the same write-ahead-log volume filling because writes outpaced archiving. The October 6 fix addressed routine table maintenance load but did not account for a scheduled deletion job from that same incident's backlog, which fired three days later.
Why did CircleCI's recovery take so long after the database came back online?
A second, separate problem appeared once the database recovered: cleanup tables that normally hold thousands of rows held tens of millions, slowing every database operation. A retry storm from upstream systems resubmitting failed work added further load on top of that backlog.
What is CircleCI doing to prevent this from recurring?
CircleCI split Temporal into multiple sharded instances with independent databases, removed the fixed-size log volume, added alerting on log archive lag and write rate, and changed how its systems retry against a struggling Temporal instance.
I'm Rohit, founder of Krova Cloud: disposable VMs that never carry a multi-day backlog into a second incident, since nothing on the box outlives the job it was spun up for. More debugging stories at debugly.dev.
Top comments (0)