CircleCI was not hacked. Its database filled a disk with its own diary.

Tuesday, October 6. You push a commit the way you have a thousand times, and the pipeline does not start. You check your config, because the config is always the first suspect. Nothing changed. You check the status page. It is not you. It is everyone.

Intermittent delays started around 13:30 UTC. From 15:30 to 20:30, pipeline and workflow and job starts were delayed across the platform. From 15:48 to 16:38, nothing started at all. Pipelines triggered from 16:02 onward failed. The UI went stale. Notifications stopped. Commit statuses never reached GitHub. For five hours, CircleCI could not reliably move work forward.

Nobody broke in. No deploy went wrong. No data was lost. The database behind CircleCI's workflow orchestration filled up a disk with its own diary, and then it could not write another word.

The diary every database keeps

Behind every pipeline you trigger sits an orchestration service. It tracks the state of every pipeline, every workflow, every job, and hands ready jobs to the machines that run them. All of that state lives in a database.

Databases keep a diary. Before a database applies any change to its actual data, it writes that change into a transaction log first. This is called write-ahead logging, and the idea is simple: if the power dies halfway through a write, the database replays the diary on restart and recovers every change. The log is the record of what happened, in order. It is what makes your database durable.

CircleCI's database keeps its diary on a dedicated storage volume, a disk reserved just for these logs. The logs are archived continuously: copied somewhere durable, then the space is reused. That only works if archiving keeps up with writing.

On Tuesday, routine maintenance ran against the database's largest tables. Big tables mean big changes, and big changes mean the diary fills up fast. The maintenance wrote transaction logs faster than the archiver could drain them. The dedicated volume filled to the top.

A database with a full transaction-log volume cannot accept writes. There is nowhere to record the next change, so every change waits. And a workflow orchestrator whose database cannot accept writes cannot move any work forward. No new pipeline state, no new job state, nothing handed to the execution fleet.

That is the whole incident in one sentence: the diary filled the disk, and the database went quiet.

The fix that made it worse for 50 minutes

The engineers could see where this was going. A database that cannot write to its log volume will eventually lock itself into read-only mode, and then recovery gets much harder. So they moved the transaction logs to the database's main storage volume, where there was room.

Moving a live database's transaction logs is not a copy and paste. It took the database offline for 50 minutes.

Sit with that for a second, because this is the part that stings. The fix for "the database is about to stop" was "take the database down ourselves, on purpose, for almost an hour." They traded an uncontrolled failure for a controlled one, because the controlled one had a known end. That is the kind of decision that only looks obvious from the outside. From the inside, at 16:00 UTC with every pipeline on the platform stuck, it is a brave call.

The second act

When the database came back, it processed work at about half its normal rate. Pipelines started moving again, but slowly, like traffic after a road reopens with one lane closed. At 18:19 UTC the engineers changed a database setting, and throughput recovered. Job start times were back to normal by 20:30 UTC. Customers whose jobs failed during the window were told to rerun them.

CircleCI published the postmortem on October 7. It is refreshingly plain about the chain: routine maintenance, archive lag, a full volume, a forced move, a slow recovery. No villains. Just a diary that outran its archivist.

The operator's checklist

If you run a database with write-ahead logging, and you almost certainly do, this incident hands you three things to check on Monday:

1. Monitor archive lag, not just disk used. "Disk 70% full" tells you nothing about whether archiving is keeping up. The number that matters is the gap between the newest log written and the newest log archived, and whether that gap is growing. Alert on the lag trend, not the disk gauge. Something shaped like this, with your own metric names:

- alert: TransactionLogArchiveFallingBehind
  # Fire when the archiver has not kept up for 15 minutes straight.
  expr: db_wal_archive_lag_seconds > 300
  for: 15m
  annotations:
    summary: "Log archiver falling behind on {{ $labels.instance }}"

2. Throttle heavy maintenance on your biggest tables. The maintenance that triggered this was routine. That is the point. Routine operations against your largest tables are the ones most likely to outrun your archiving. Run them in smaller batches, or run them while someone is watching the lag.

3. Size and watch the log volume like it is its own blast radius. A dedicated log volume is good design. It is also a single point of failure with a countdown on it. Know how fast it fills at your peak write rate, and make sure the person on call knows where that number lives before Tuesday afternoon.

The diary metaphor is doing real work here: the mechanism that saves your database during a crash is the same mechanism that takes it down when nobody is watching the archivist.

What is the equivalent in your stack? The safety mechanism that quietly becomes the failure mode when it falls behind?