Around 3,500 job starts failed because PostgreSQL’s WAL generation outpaced archival during routine anti-wraparound autovacuum.
The dedicated log volume filled, and CircleCI’s orchestration database went offline for 50 minutes. A maintenance operation had turned into a developer-feedback outage.
How Routine Vacuum Work Filled the WAL Disk
PostgreSQL autovacuum is routine maintenance, not an optional background nicety. CircleCI's report says an anti-wraparound autovacuum ran on several of its largest tables. On a very large table the process reads every page and produces a large volume of transaction logs. Around 12:40 UTC, this maintenance output plus normal application writes exceeded the rate at which the database could archive and reclaim the dedicated log volume.
The failure was a rate mismatch, not simply a database that had “too little disk.” If WAL is produced at rate g and archived at rate a, free space falls whenever g > a. Disk headroom buys time, but does not correct the mismatch. CircleCI did not publish those rates or the volume size, so the inequality explains the mechanism without pretending to reconstruct its capacity numbers.
Alerts began showing brief orchestration errors at 10:02. They cleared within about 15 minutes and looked similar to alerts responders had seen before. As the WAL volume filled, slowdowns became intermittent around 13:30, then continuous by 15:30. The incident was declared at 14:54. CircleCI's retrospective says the noisy, short-lived alerts and the absence of direct alerts on WAL volume and archive lag delayed recognition of the combined failure.
The more interesting part? Bringing the database back wasn’t enough. The storage change cut processing speed to roughly half its normal rate, and the backlog kept hurting customers for hours. It’s a good reminder that database recovery and service recovery are two very different things.
We broke down the failure mechanism, recovery timeline, architectural bottleneck, and the safeguards worth adding.
Full postmortem: https://buildopsy.com/system-design/circleci-wal-autovacuum-postmortem.html
Top comments (1)
Looks like they had complete sla breach 😅