At 15:48 UTC on October 6, 2026, CircleCI stopped starting work. Not some work. All of it.
You could still trigger a pipeline. The button worked. Nothing happened after it. For 50 minutes, no workflows or jobs started at all. Pipelines triggered from 16:02 failed outright. About 3,500 jobs failed to start during the incident, according to CircleCI's own post incident report. Job start times did not return to normal until 20:30 UTC, five hours after the first delays.
Nothing had been deleted. No code had been deployed badly. The cause was quieter than that, and it had been building for hours before anyone treated it as an emergency. A routine cleanup job filled a disk, and that disk was the one thing every pipeline depended on.
This is a story about a log file. If you run a database, or you depend on one, it is worth five minutes of your attention.
The machine in the middle
When you trigger a pipeline on CircleCI, your request goes to a workflow orchestration service. Think of it as a dispatcher. It tracks the state of every workflow and job, decides when each job is ready to run, and hands ready jobs to the machines that actually run them. It also drives the job data you see in the user interface, the notifications, and the commit statuses sent back to your version control provider.
That dispatcher stores its state in a database. Every change to that state is first written to a transaction log. A transaction log is a running receipt book: before the database applies a change, it writes down what is about to change. If the machine crashes mid-change, the receipt book lets it rebuild what happened. Nothing is trusted until it is in the log.
The database keeps those logs on a dedicated storage volume, a separate disk set aside just for them. It archives the logs continuously, clearing old ones so the space can be reused. That archive step is the quiet hero of this story, and also its failure point. If archiving falls behind and the volume fills, the database stops accepting writes. No writes means no new work can move forward. The dispatcher cannot dispatch.
The cleanup job nobody watches
Databases also run periodic maintenance to keep their internal records valid over time. CircleCI's report calls this process autovacuum. On a very large table, it reads every page of the table and writes a large volume of transaction logs as it goes.
On October 6, that maintenance started on several of the database's largest tables at around 12:40 UTC. It ran alongside the regular workload, which was generating its own transaction logs. Together, they wrote logs faster than the database could archive them. The log volume started to fill.
Here is the timeline, from CircleCI's report, all times UTC:
- 10:02: Monitoring raised short-lived alerts about errors in the orchestration service. Each cleared on its own within about 15 minutes. Engineers had seen similar brief alerts before, so they followed the runbook each time.
- 12:40: Maintenance on the largest tables began writing logs faster than archiving could clear them.
- 13:30: The orchestration service began to slow down, in short bursts at first.
- 14:54: Engineers declared an incident, after alerts from several services piled up together.
- 15:30: Delays became continuous. Some jobs began to fail.
- 15:43: The transaction log volume was close to full. If it filled, the database would stop accepting writes entirely.
- 15:47: Engineers began moving the logs onto the database's main storage volume, which had plenty of free space. The move required taking the database offline, for a length of time they could not predict in advance.
- 15:48 to 16:38: No workflows or jobs started. This was the 50 minute outage.
- 16:38 to 18:19: The database was back, but with logs and regular data now sharing the same storage, it processed work at about half its normal rate. Jobs waited tens of minutes, up to about 50 minutes at the longest.
- 18:19: Engineers changed a database setting to cut the time each write spent waiting on storage. Processing speed recovered.
- 20:30: Job start times returned to normal.
Read that timeline again and notice what did not happen. Nobody caught the disk filling in time, because CircleCI had no alert on how full the transaction log volume was, and no alert on how far archiving had fallen behind. Their own report says so plainly. Monitoring caught the slowdown. It did not catch the cause.
Why the fix took five hours
The obvious question is why moving logs to a bigger disk did not fix everything at 16:38. The answer is that the fix created a second problem. With logs and regular data sharing one volume, every write now competed for the same storage. The database ran at about half speed for nearly two hours while it worked through a backlog of unarchived logs. Only after engineers changed a setting that cut how long each write waited on storage did the backlog begin to clear. Extra capacity added later helped drain it faster.
There is a lesson in that shape. The first failure was a disk filling up. The second failure was the recovery running slowly because the emergency arrangement was worse than the original one. Backlogs do not just wait politely. While jobs queue up, the queue itself becomes load: more retries, more status checks, more work arriving on top of work that has not finished. CircleCI's report notes that during the incident, some jobs failed because their own systems gave up after retrying. A slow recovery can keep hurting after the root cause is gone.
CircleCI also prepared a full replacement stack of the service during the incident and chose not to switch to it, because switching carried a risk of data loss. That is a decision worth respecting. The fastest recovery is not always the right one when it can lose the very state you are trying to protect.
The guard you can add this week
CircleCI's own list of fixes starts with the missing piece: alerts on how full the transaction log volume is, and on how far archiving has fallen behind. You can build the first half of that in a small script, and it works for any database that keeps its logs on a known path. The idea is simple. Do not just check how full the disk is right now. Track how fast it is filling, and estimate how many minutes you have left. A disk at 60 percent is fine if it grows slowly. A disk at 60 percent that gains a percent every ten minutes is an incident in about seven hours, and you want to know that at noon, not at midnight.
Here is a small Python guard you can adapt. It reads the disk holding your log directory, compares it with its last reading, and exits with a warning or critical code your monitoring can alert on:
import json
import os
import shutil
import sys
import time
LOG_PATH = "/var/lib/postgresql/wal" # change to your log directory
STATE_FILE = "/var/tmp/wal_guard_state.json"
WARN_PERCENT = 70
CRITICAL_PERCENT = 85
CRITICAL_MINUTES_LEFT = 120
def read_disk(path):
usage = shutil.disk_usage(path)
used_percent = 100 * (usage.used / usage.total)
return used_percent, usage.free
def main():
used_percent, free_bytes = read_disk(LOG_PATH)
now = time.time()
minutes_left = None
if os.path.exists(STATE_FILE):
with open(STATE_FILE) as f:
previous = json.load(f)
elapsed_minutes = (now - previous["time"]) / 60
growth = used_percent - previous["used_percent"]
if elapsed_minutes > 0 and growth > 0:
minutes_left = (100 - used_percent) / (growth / elapsed_minutes)
with open(STATE_FILE, "w") as f:
json.dump({"time": now, "used_percent": used_percent}, f)
message = f"{LOG_PATH} is {used_percent:.1f}% full, {free_bytes // 1024 ** 3} GB free"
if minutes_left is not None:
message += f", filling in about {minutes_left:.0f} minutes at the current rate"
if used_percent >= CRITICAL_PERCENT or (
minutes_left is not None and minutes_left <= CRITICAL_MINUTES_LEFT
):
print(f"CRITICAL: {message}")
return 2
if used_percent >= WARN_PERCENT:
print(f"WARNING: {message}")
return 1
print(f"OK: {message}")
return 0
if __name__ == "__main__":
sys.exit(main())
Run it every few minutes from your scheduler, and wire exit codes 1 and 2 to alerts that page a human. Two details matter. First, alert on the rate, not just the level: the minutes-left estimate is what turns a disk check into an early warning. Second, keep the archive lag check alongside it. A disk can look healthy while archiving quietly falls behind, and by the time the disk fills, your warning window is gone. If your database exposes an archive lag metric, alert on that too, with the same urgency.
The deeper fix, the one CircleCI is still evaluating, is to remove the single bottleneck: split the service across multiple database instances, each with its own transaction log, or move to a distributed database that spreads writes across many nodes. Either way, one full log would slow down part of the workload instead of all of it. That is a bigger project. The disk guard is not. It is an afternoon of work, and it covers the exact gap that turned a routine cleanup into a five hour incident.
The quiet part
The most unsettling detail in CircleCI's report is not the disk. It is the morning. Short-lived alerts fired from 10:02, cleared on their own, and were treated as noise because similar alerts had cleared before. By the time the alerts from several services arrived together at 13:35, the log volume had already been filling for nearly an hour. CircleCI's own verdict is blunt: they should have recognised the combination of alerts sooner.
That is a pattern worth checking in your own systems. A single alert that clears itself is noise. The same alert plus related alerts from neighbouring services, at the same time, is a story. If your runbook only says what to do about one alert at a time, it will keep telling you everything is fine right up until it is not.
So here is the question to take back to your team: do you have an alert on your database log volume, and on how fast it is filling? If the answer is no, you are one routine cleanup job away from finding out the hard way. What would your morning look like if that cleanup started today?
Source: CircleCI, "Post Incident Report: Oct 6, 2026 - Delays in pipelines, workflows, UI data, and notifications", discuss.circleci.com.
Top comments (0)