DEV Community: Kashif Manzer

The timeout that ate the database pool: how Inngest lost every connection twice in one afternoon

Kashif Manzer — Wed, 04 Nov 2026 14:10:46 +0000

At 14:00 UTC on October 10, 2026, both of Inngest's database connection poolers lost every connection to the primary database. Not most of them. All of them. And then, at 15:30, it happened again.

Inngest runs background jobs for other companies. When you trigger a function there, a scheduler has to look up your account and your function before any work can start. Those lookups go through the database. With no usable connections, run scheduling stalled and queue processing stopped for most accounts.

The team restarted the poolers. That did not reliably fix it. They raised the pool size. That did not help either. The pool was not too small. It was being emptied faster than it could be refilled, by the very timeouts meant to protect it.

This is a story about a timeout doing exactly what it was told to do, in exactly the wrong place.

The pooler in the middle

First, the pieces, plainly.

A database connection is expensive to open. So most services do not open a fresh one for every request. They keep a pool of open connections and hand them out as needed.

A pooler sits between your service and the database. PgBouncer is one such pooler. Your service connects to PgBouncer, PgBouncer connects to the database, and PgBouncer decides which real database connection serves each query. It lets many application connections share a smaller number of database connections.

A database driver is the library your code uses to talk to the database. Inngest's report points to pgx, the Go driver for Postgres, and to how it interacts with PgBouncer. You do not need to know pgx to follow this story. You only need to know that the driver is the part that sends your query, waits for an answer, and gives up when a deadline passes.

A deadline, in Go, is usually a context with a timeout. You say: this lookup has 200 milliseconds, and if it is not done by then, stop waiting. That sounds safe. It usually is.

The loop that ate the pool

Here is the sequence Inngest describes, slowed down.

Several frequently used lookups had short deadlines. Under normal load, a lookup waits briefly in PgBouncer for a free database connection, gets one, runs, and returns. The deadline never fires.

But suppose the lookup is still waiting in PgBouncer when its deadline expires. The driver cancels the query and closes its connection to PgBouncer. That is the normal, polite thing to do when you have given up.

The problem is that PgBouncer had nothing running to cancel. The query had not started yet. It was still queued, waiting for a connection.

So PgBouncer later ran the abandoned query on a database connection anyway. Then, instead of returning that connection to the pool for reuse, it closed the connection entirely. One cancelled wait, one dead connection.

Now repeat that at scale. Each pooler opens new database connections one at a time. That is a slow refill. If cancellations are destroying connections faster than the pooler can open replacements, the pool drains toward zero. Thousands of clients queue behind the empty pool. Their deadlines expire too. More cancellations. More dead connections. The timeout meant to shed load becomes the thing that keeps the load going.

Inngest reproduced this behaviour in a test environment, so this is not a guess about what might have happened. It is the mechanism the report names.

Two details made October 10 worse. Both collapses started at the top and bottom of the hour, when cron scheduled work roughly triples the query rate. Cron work is work on a schedule, like a job that runs every hour on the hour, and a lot of it lands at once. At the same moments, a new set of lookups that had been rolled out to all accounts before the weekend began failing in large numbers, adding to the cancellations. A spike plus a fresh source of failures, feeding the same loop.

That is also why the obvious fixes failed. Restarting the poolers did not reliably recover them, because the loop restarted with them. Raising the pool size did not help, because opening connections, not the pool limit, was the bottleneck. If you can only refill one connection at a time, a bigger bucket does not help when the bucket has a hole in it.

What actually stopped it

The second collapse ended when Inngest moved all services to connection poolers run alongside the database by its database provider. Processing recovered between 16:10 and 16:25.

That move introduced a new problem, and it is worth naming because it shows how incident fixes trade one risk for another. The new route reached the database through network address translation gateways, and some connections through those gateways were silently dropped. Queries on a dropped connection hung for up to 16 minutes before failing, which intermittently stalled processing, most visibly for cron functions. Inngest moved the affected service back to the original poolers at 04:17 on October 11, and improved the gateways' routing on October 12.

The lasting fix for the pool collapse itself was different, and it is the part you can take home. Inngest changed the lookups involved so that a deadline now ends the caller's wait without cancelling the underlying query. The caller stops waiting. The query is allowed to finish, and its connection goes back to the pool instead of being destroyed. That removes the trigger for this collapse on those lookups. Inngest says it is reviewing the remaining database calls on the execution path and their timeouts, which is the honest version of this fix: it is not done everywhere yet.

The guard you can steal this week

The pattern below is a simplified illustration, not Inngest's code. It shows the split that matters: one deadline for how long the caller waits, and a separate life for the query itself.

The dangerous shape is one deadline doing two jobs:

ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer cancel()

// If this is still waiting in the pooler when the deadline fires,
// the cancellation can destroy the very connection you were waiting for.
row := pool.QueryRow(ctx, "SELECT name FROM accounts WHERE id = $1", accountID)

The safer shape separates the two:

result := make(chan lookupResult, 1)

go func() {
    // queryCtx is deliberately not tied to the caller's deadline.
    // The query runs to completion and its connection returns to the pool.
    row := pool.QueryRow(queryCtx, "SELECT name FROM accounts WHERE id = $1", accountID)
    var name string
    err := row.Scan(&name)
    result <- lookupResult{name: name, err: err}
}()

select {
case r := <-result:
    return r.name, r.err
case <-time.After(200 * time.Millisecond):
    // The caller stops waiting. It does not cancel the query.
    return "", errors.New("lookup timed out")
}

Use this only where it is safe, which is the part people skip. This shape makes sense for a read-only lookup that can safely run to completion even after the caller has given up. It does not make sense as a blanket rule for writes, where a query finishing late could do something the caller already assumed had failed. Check that before you copy it.

Then add three checks around it, because code alone will not save you from the next spike:

Separate your timeouts in your head before you separate them in code. Ask, for each database call: is this deadline for waiting in the queue, for running the query, or for both? If you cannot answer, you have found a candidate for this bug.
Alert on pool health, not just on slow queries. A pool draining toward zero usable connections, and a rising count of cancelled waits, is the early signal. Slow queries alone will mislead you, because in this incident the queries were not slow. They never started.
Load test at the spike, not at the average. Test with the top-of-hour burst, and with a newly deployed lookup failing at the same time. A pool that survives average load can still collapse in the first minute of a burst, and that minute is when everyone is watching.

The wider lesson from the same weekend

October 10 was the second incident in two days, and the first one explains why the second hurt so much. On October 9, from 21:05 to 21:49, one data center provider lost connectivity between its data center and Inngest's other data centers, plus outbound internet access. Inngest runs across three data centers, and function execution depends on all three. Accounts processed in the cut-off data center could not reach the primary database. Accounts elsewhere with run state stored there could not make progress. And accounts with no dependency on that data center were still delayed, because they shared an event-processing pipeline with the affected accounts, and failing work built a backlog there.

Inngest's structural answer is to consolidate onto a single data center that acts as a single cell, which removes this whole class of cross-data-center failure. That is a bigger bet than a timeout fix, and it is still under way. But the two incidents share one lesson: a dependency you treat as background, a shared pipeline or a pooler queue, can become the whole incident when it fails in a way your timeouts and retries were not designed for.

In both cases, the internal failures also counted against functions' configured retries, and a step that failed permanently on an internal error left its run stuck in a Running state instead of marking it failed, so failure handlers were not notified. Inngest has since changed that, so such runs are now marked failed. A stuck state that looks alive is its own kind of outage, quieter than a pool collapse and harder to notice.

So check your own code this week. Find one database lookup with a short deadline that goes through a pooler. Ask what happens if that deadline fires while the query is still waiting, not running. If the answer is "I am not sure", you have found your next test.

What is the shortest deadline in your codebase, and do you know whether it cancels a waiting query or just stops the wait?

The database log that filled up and stopped every CircleCI pipeline

Kashif Manzer — Mon, 02 Nov 2026 14:09:12 +0000

At 15:48 UTC on October 6, 2026, CircleCI stopped starting work. Not some work. All of it.

You could still trigger a pipeline. The button worked. Nothing happened after it. For 50 minutes, no workflows or jobs started at all. Pipelines triggered from 16:02 failed outright. About 3,500 jobs failed to start during the incident, according to CircleCI's own post incident report. Job start times did not return to normal until 20:30 UTC, five hours after the first delays.

Nothing had been deleted. No code had been deployed badly. The cause was quieter than that, and it had been building for hours before anyone treated it as an emergency. A routine cleanup job filled a disk, and that disk was the one thing every pipeline depended on.

This is a story about a log file. If you run a database, or you depend on one, it is worth five minutes of your attention.

The machine in the middle

When you trigger a pipeline on CircleCI, your request goes to a workflow orchestration service. Think of it as a dispatcher. It tracks the state of every workflow and job, decides when each job is ready to run, and hands ready jobs to the machines that actually run them. It also drives the job data you see in the user interface, the notifications, and the commit statuses sent back to your version control provider.

That dispatcher stores its state in a database. Every change to that state is first written to a transaction log. A transaction log is a running receipt book: before the database applies a change, it writes down what is about to change. If the machine crashes mid-change, the receipt book lets it rebuild what happened. Nothing is trusted until it is in the log.

The database keeps those logs on a dedicated storage volume, a separate disk set aside just for them. It archives the logs continuously, clearing old ones so the space can be reused. That archive step is the quiet hero of this story, and also its failure point. If archiving falls behind and the volume fills, the database stops accepting writes. No writes means no new work can move forward. The dispatcher cannot dispatch.

The cleanup job nobody watches

Databases also run periodic maintenance to keep their internal records valid over time. CircleCI's report calls this process autovacuum. On a very large table, it reads every page of the table and writes a large volume of transaction logs as it goes.

On October 6, that maintenance started on several of the database's largest tables at around 12:40 UTC. It ran alongside the regular workload, which was generating its own transaction logs. Together, they wrote logs faster than the database could archive them. The log volume started to fill.

Here is the timeline, from CircleCI's report, all times UTC:

10:02: Monitoring raised short-lived alerts about errors in the orchestration service. Each cleared on its own within about 15 minutes. Engineers had seen similar brief alerts before, so they followed the runbook each time.
12:40: Maintenance on the largest tables began writing logs faster than archiving could clear them.
13:30: The orchestration service began to slow down, in short bursts at first.
14:54: Engineers declared an incident, after alerts from several services piled up together.
15:30: Delays became continuous. Some jobs began to fail.
15:43: The transaction log volume was close to full. If it filled, the database would stop accepting writes entirely.
15:47: Engineers began moving the logs onto the database's main storage volume, which had plenty of free space. The move required taking the database offline, for a length of time they could not predict in advance.
15:48 to 16:38: No workflows or jobs started. This was the 50 minute outage.
16:38 to 18:19: The database was back, but with logs and regular data now sharing the same storage, it processed work at about half its normal rate. Jobs waited tens of minutes, up to about 50 minutes at the longest.
18:19: Engineers changed a database setting to cut the time each write spent waiting on storage. Processing speed recovered.
20:30: Job start times returned to normal.

Read that timeline again and notice what did not happen. Nobody caught the disk filling in time, because CircleCI had no alert on how full the transaction log volume was, and no alert on how far archiving had fallen behind. Their own report says so plainly. Monitoring caught the slowdown. It did not catch the cause.

Why the fix took five hours

The obvious question is why moving logs to a bigger disk did not fix everything at 16:38. The answer is that the fix created a second problem. With logs and regular data sharing one volume, every write now competed for the same storage. The database ran at about half speed for nearly two hours while it worked through a backlog of unarchived logs. Only after engineers changed a setting that cut how long each write waited on storage did the backlog begin to clear. Extra capacity added later helped drain it faster.

There is a lesson in that shape. The first failure was a disk filling up. The second failure was the recovery running slowly because the emergency arrangement was worse than the original one. Backlogs do not just wait politely. While jobs queue up, the queue itself becomes load: more retries, more status checks, more work arriving on top of work that has not finished. CircleCI's report notes that during the incident, some jobs failed because their own systems gave up after retrying. A slow recovery can keep hurting after the root cause is gone.

CircleCI also prepared a full replacement stack of the service during the incident and chose not to switch to it, because switching carried a risk of data loss. That is a decision worth respecting. The fastest recovery is not always the right one when it can lose the very state you are trying to protect.

The guard you can add this week

CircleCI's own list of fixes starts with the missing piece: alerts on how full the transaction log volume is, and on how far archiving has fallen behind. You can build the first half of that in a small script, and it works for any database that keeps its logs on a known path. The idea is simple. Do not just check how full the disk is right now. Track how fast it is filling, and estimate how many minutes you have left. A disk at 60 percent is fine if it grows slowly. A disk at 60 percent that gains a percent every ten minutes is an incident in about seven hours, and you want to know that at noon, not at midnight.

Here is a small Python guard you can adapt. It reads the disk holding your log directory, compares it with its last reading, and exits with a warning or critical code your monitoring can alert on:

import json
import os
import shutil
import sys
import time

LOG_PATH = "/var/lib/postgresql/wal"  # change to your log directory
STATE_FILE = "/var/tmp/wal_guard_state.json"
WARN_PERCENT = 70
CRITICAL_PERCENT = 85
CRITICAL_MINUTES_LEFT = 120


def read_disk(path):
    usage = shutil.disk_usage(path)
    used_percent = 100 * (usage.used / usage.total)
    return used_percent, usage.free


def main():
    used_percent, free_bytes = read_disk(LOG_PATH)
    now = time.time()

    minutes_left = None
    if os.path.exists(STATE_FILE):
        with open(STATE_FILE) as f:
            previous = json.load(f)
        elapsed_minutes = (now - previous["time"]) / 60
        growth = used_percent - previous["used_percent"]
        if elapsed_minutes > 0 and growth > 0:
            minutes_left = (100 - used_percent) / (growth / elapsed_minutes)

    with open(STATE_FILE, "w") as f:
        json.dump({"time": now, "used_percent": used_percent}, f)

    message = f"{LOG_PATH} is {used_percent:.1f}% full, {free_bytes // 1024 ** 3} GB free"
    if minutes_left is not None:
        message += f", filling in about {minutes_left:.0f} minutes at the current rate"

    if used_percent >= CRITICAL_PERCENT or (
        minutes_left is not None and minutes_left <= CRITICAL_MINUTES_LEFT
    ):
        print(f"CRITICAL: {message}")
        return 2
    if used_percent >= WARN_PERCENT:
        print(f"WARNING: {message}")
        return 1
    print(f"OK: {message}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Run it every few minutes from your scheduler, and wire exit codes 1 and 2 to alerts that page a human. Two details matter. First, alert on the rate, not just the level: the minutes-left estimate is what turns a disk check into an early warning. Second, keep the archive lag check alongside it. A disk can look healthy while archiving quietly falls behind, and by the time the disk fills, your warning window is gone. If your database exposes an archive lag metric, alert on that too, with the same urgency.

The deeper fix, the one CircleCI is still evaluating, is to remove the single bottleneck: split the service across multiple database instances, each with its own transaction log, or move to a distributed database that spreads writes across many nodes. Either way, one full log would slow down part of the workload instead of all of it. That is a bigger project. The disk guard is not. It is an afternoon of work, and it covers the exact gap that turned a routine cleanup into a five hour incident.

The quiet part

The most unsettling detail in CircleCI's report is not the disk. It is the morning. Short-lived alerts fired from 10:02, cleared on their own, and were treated as noise because similar alerts had cleared before. By the time the alerts from several services arrived together at 13:35, the log volume had already been filling for nearly an hour. CircleCI's own verdict is blunt: they should have recognised the combination of alerts sooner.

That is a pattern worth checking in your own systems. A single alert that clears itself is noise. The same alert plus related alerts from neighbouring services, at the same time, is a story. If your runbook only says what to do about one alert at a time, it will keep telling you everything is fine right up until it is not.

So here is the question to take back to your team: do you have an alert on your database log volume, and on how fast it is filling? If the answer is no, you are one routine cleanup job away from finding out the hard way. What would your morning look like if that cleanup started today?

Source: CircleCI, "Post Incident Report: Oct 6, 2026 - Delays in pipelines, workflows, UI data, and notifications", discuss.circleci.com.

The new machine with the old map: how GitHub routed writes to servers that no longer existed

Kashif Manzer — Sun, 01 Nov 2026 14:09:22 +0000

On October 7, 2026, GitHub broke in a strange, specific way. You could read almost anything. You could not save almost anything.

Push a commit, open a pull request, file an issue, start an Actions run, sign in: those are all writes, actions that save or change data. Reads, looking at code and pages that already existed, were largely unaffected. That split is the first clue, and it is worth sitting with before you look for a cause. When reads work and writes do not, the data itself is probably fine. The road that new data travels is not.

That is the mystery this article works through. It ends with a small guard, in Python, that you can steal for your own routing config. The guard does one job: it refuses to use a map that points to servers that no longer exist.

Clue one: only the writing road was broken

GitHub had two outages that day with the same cause, totalling about 19 minutes. The first ran from 15:06 to 15:16 UTC, about 10 minutes. The second ran from 16:52 to 17:01 UTC, about 9 minutes.

In the first window, 5.6 percent of all web and API requests failed, peaking at 8.9 percent, and more than 12,000 organizations saw failed requests. Up to 26 percent of Git authentication requests failed, and 17 percent of Actions workflow runs failed with infrastructure errors. In the second window, 6.2 percent of web and API requests failed, peaking at 9.3 percent, and more than 11,000 organizations were affected. Up to 25 percent of Git authentication requests failed, and 19 percent of Actions workflow runs failed with infrastructure errors.

No databases lost data, according to GitHub, and operations retried after recovery succeeded. So this was not a story about lost work. It was a story about work that could not find its destination.

GitHub's own status report, posted October 7, 2026, names the affected areas in those windows: Git operations, pull requests, issues, GitHub Actions, the API, webhooks, sign-in and authentication, GitHub Apps and OAuth, code scanning, Codespaces, and Copilot. That list looks random until you ask what those things share. They all need to save something new.

Clue two: a new machine with an old map

To understand the cause, you need three plain ideas.

A database server is a machine that stores data and answers questions about it. A routing config is a list that says which database servers should receive new and updated data. Think of it as a map: it tells each write where to go.

A quorum is a group of machines that must agree before a change counts. A leader is the machine in that group whose word the others accept for now. Leader election is just the process of choosing which machine leads. If a machine with a wrong map becomes leader, its wrong map becomes the map everyone follows.

GitHub relies on an internal service that keeps track of which database servers should receive new and updated data. While replacing machines that run this service as part of regular maintenance, one of the new machines started with an outdated copy of its records. That copy still listed database servers GitHub had already retired. Retired here means taken out of service, gone, not answering anymore.

When this node came online, it incorrectly modified routing configuration with the outdated records, routing write requests to non-existent database servers. Writes were sent to machines that were not there. Reads did not need that road in the same way, so they mostly kept working.

Each outage ended when one of GitHub's established machines, with the correct configuration, became the leader of the quorum and sent the correct configuration. The good map replaced the bad map. That is why the first outage was only about 10 minutes long. The system healed itself, once the right leader was in charge.

Clue three: the pause that was not honoured

Here is the part that should worry you more than the first outage. After the first outage, GitHub's engineers identified the new machine as the source and paused automated maintenance to stop it from being leader again. Then, at 16:52 UTC, an automated maintenance task that did not honour the pause put it back in charge, and the outage happened again.

Read that slowly. The team did the right thing after the first failure. They found the bad machine and they paused the automation around it. But the pause was not honoured by every piece of automation. One task did not know about the pause, or did not check it, and it undid the fix.

GitHub shut that machine down at 17:02 UTC and has since removed it. GitHub also says the automation now honours the pause, and that this failure to honour it was a new regression, a bug introduced by an update to its Puppet infrastructure. Puppet is a tool for managing machine configuration automatically, which is exactly the kind of tool that can quietly reintroduce a problem while nobody is looking.

This is the pattern to remember. The first failure was a stale map. The second failure was a safety control that only some of the system respected. A pause that lives in one place, and is ignored by another, is not a pause. It is a hope.

The guard: never trust a map you have not checked

GitHub says it is now doing several things to prevent a repeat: clearing retired database servers from the service's records before any new machine is added, keeping new machines from being elected leader until their records have been checked, rejecting configuration that points to servers that do not exist and keeping the last known-good settings instead, giving responders a faster way to take a misbehaving machine out of service, and adding alerts so changes like this are spotted within seconds.

You can borrow the middle idea, the one about rejecting a bad map, in a few lines. The rule is simple. Before you apply a new routing config, check every destination against the live inventory, the list of servers that actually exist right now. If even one destination is missing, reject the whole config and keep the last known-good one. Do not apply the good half and skip the bad half. A half applied map is still a wrong map.

def apply_routing_config(new_config, live_servers, last_good_config):
    """Apply new_config only if every destination really exists.

    new_config: a list of server names the new map points writes to.
    live_servers: a set of server names that exist right now.
    last_good_config: the last config we know worked.
    """
    unknown = [server for server in new_config if server not in live_servers]

    if unknown:
        print(f"Refusing bad routing config. Unknown servers: {unknown}")
        return last_good_config

    return new_config

That is the whole guard. The rest is discipline around it. Check the map at the point where it is applied, not only at the point where it is written. A config file can be correct when it is saved and stale by the time it is used, which is exactly what happened at GitHub: the records were outdated when the new machine started.

A boundary test makes the rule concrete. Test three cases: a config with no unknown servers, which should pass; a config with one unknown server among many good ones, which should fail; and a config where every server is retired, which should also fail. The one-unknown case is the important one. It is tempting to let it through, because most of the map looks fine. That temptation is the bug.

live = {"db-a", "db-b", "db-c"}
last_good = ["db-a", "db-b"]

assert apply_routing_config(["db-a", "db-b"], live, last_good) == ["db-a", "db-b"]
assert apply_routing_config(["db-a", "db-gone"], live, last_good) == last_good
assert apply_routing_config(["db-gone", "db-also-gone"], live, last_good) == last_good

Two more checks belong next to this guard, and they come straight from GitHub's own list. First, do not let a new machine become leader until its records have been checked. A new member of a quorum should prove its map is fresh before it is allowed to lead. Second, make a pause a single, checked thing, not a note in one tool. If a maintenance task can run without asking whether it is paused, it will, on the worst day, run at the worst time.

The question

You probably have a map like this somewhere: a config, a service discovery list, a set of connection strings, a deploy script that names servers. When was it last checked against what actually exists?

Find the newest machine in your system, the one added most recently, and ask a blunt question about it. Does it hold a copy of anything that was written before something was retired? If you cannot answer that in a minute, that is the audit, and it is worth doing this week.

When did you last verify that every destination in your routing config still exists?

The migration that kept running after the rollback

Kashif Manzer — Fri, 30 Oct 2026 13:09:49 +0000

You push a deploy. The dashboard accepts it. Then nothing happens.

No error. No crash. No failed build log to read. Your live service keeps serving traffic happily, as if nothing is wrong. The deploy just sits in the queue, waiting for a worker that never picks it up.

That is what happened at Render on Wednesday, October 14, 2026. Between 12:12 and 15:08 PDT, builds and deploys across all regions failed to start, stalled, or were badly delayed, in two periods of impact totalling about 2 hours and 20 minutes. Live user services were not affected. Only the machinery that starts new work was stuck.

That split is the clue worth sitting with. When the thing that is already running works fine and the thing that starts new work does not, the problem is usually not the whole platform. It is in the path that new work takes, and in whatever that path shares with something else.

What happened, in order

Render runs an internal job scheduler. It coordinates builds, deploys, and other critical platform operations. On October 14, the team was rolling out improvements to how that scheduler distributes work, after a successful practice run in a test environment. The planned version upgrade included a database schema migration.

A schema migration is a change to the shape of a database: adding a column, changing a type, adding an index, that kind of thing. Some migrations are quick. Some need a lock on a table, which means other work on that table has to wait until the migration finishes.

This migration locked a large table and drove database CPU use to 100 percent. That database stores two kinds of thing: the scheduler's core job state, and secondary data used for search and reporting. With the database that busy, the scheduler could not read or write job state reliably. Builds and deploys stopped or slowed.

Here is the timeline from Render's postmortem, posted October 22, 2026. All times are PDT on October 14.

12:13: The upgrade starts, including the migration. Database load rises and builds and deploys stop starting.

12:24: Impact identified, incident escalated.

12:34: Engineers revert the upgrade and stop the client performing the migration.

12:37: The scheduler recovers. Builds and deploys resume, but slowly.

13:17: Engineers restart one unhealthy instance of the system to rebalance load. Database connections and errors increase. The second period of impact starts.

13:37: Free tier deploys are paused to reduce load.

13:49: Engineers find the migration still running on the database and terminate it.

14:16: The database is restarted. An attempt to add database capacity fails because of an error from the underlying cloud provider.

14:30: Builds and deploys start to recover, slowly, as the backlog is processed.

15:08: Deploy start times return to normal and database load returns to pre-incident levels.

15:18: Incident resolved.

Read that timeline again and notice the two moments that hurt most. The first is 12:34. The team reverted the upgrade and stopped the client doing the migration, and the system briefly looked better. The second is 13:49, more than an hour later, when they found the migration still running on the database.

Stopping the client did not stop the migration.

The lesson most people miss: a migration lives on the server

This is the part of the story that translates directly to your own work, even if you never run a platform like Render.

When you run a migration, you usually run a small program on your machine, or in CI, that sends a statement to the database and waits. It is easy to think of that program as the migration. It is not. The migration is the work happening inside the database. Your program is just the messenger waiting for an answer.

If you kill the messenger, the database may keep working. If you roll back the deploy that started it, the database may keep working. If you restart your app, the database may keep working. The statement you sent is still there, still holding its lock, still driving CPU up, until it finishes, or until someone kills it on the database side.

That is why the partial recovery at 12:37 is so instructive. Deploys resumed, but deploy start times were about 20 seconds instead of a few hundred milliseconds. That is not a healthy system. That is a system still carrying a heavy load it has not named yet. The migration was still running, quietly, underneath a recovery that looked good enough.

Then a well meaning restart at 13:17, meant to rebalance load, made things worse. Connections and errors rose, and the second, longer period of impact began. When a system is already struggling with a hidden load, adding a restart on top can tip it from slow to stuck.

There were two more unlucky breaks in this incident, and both are worth noting because they are the kind of thing you only discover when you need them. A pre-upgrade step to create a database clone for rollback could not be completed, because of an error from the underlying cloud provider. So a fast rollback was not available. Later, an attempt to add database capacity failed for the same kind of reason. Two escape routes, both blocked by the same dependency. If your rollback plan depends on a step that can fail, test that step, not just the plan that assumes it worked.

Three guards you can add this week

None of this requires a platform team. The guards below are for the database you already run. Render's postmortem does not name its database engine or the exact migration statement, so treat the code here as a general guard for your own migrations, not as a description of Render's setup.

1. Put a leash on every migration

A migration should never be allowed to run forever, holding a lock, while you wonder what happened. Set a lock timeout before you run DDL, which is the database term for statements that change schema. On Postgres, that looks like this:

SET lock_timeout = '30s';
SET statement_timeout = '10min';
ALTER TABLE jobs ADD COLUMN IF NOT EXISTS retry_count integer NOT NULL DEFAULT 0;

A lock timeout says: if you cannot get the lock in 30 seconds, give up loudly instead of queueing behind everything else. A statement timeout says: if the whole statement takes longer than 10 minutes, stop it. The numbers are yours to choose. The point is that a migration fails fast and visibly, instead of succeeding slowly while it strangles the system. A failed migration you can see is better than a running migration you cannot.

If you use a migration tool, check whether it sets these for you. Many do not, by default. That one setting is the difference between a bad deploy you roll back in minutes and a bad afternoon you reconstruct from a timeline.

2. Kill it on the database side, and prove it is dead

When you stop a migration, do not trust the client. Check the database itself for long running statements, and terminate the one you mean to terminate. On Postgres, you can look like this:

SELECT pid, now() - query_start AS running_for, state, query
FROM pg_stat_activity
WHERE state = 'active'
  AND query_start < now() - interval '5 minutes'
ORDER BY query_start;

pg_stat_activity is Postgres's view of what every connection is doing right now. pid is the process id of that connection on the database server. If you find your migration there after you stopped the client, terminate that process id on the database side, then run the query again and confirm it is gone. The second check matters. The Render timeline shows a gap of about an hour and a quarter between stopping the client and finding the migration still running. A check that takes thirty seconds would have closed that gap.

In Python, the same habit looks like this: never assume a cancelled task cancelled the work. If you cancel a migration runner, follow it with a check against the database before you declare the rollback done.

def migration_is_still_running(conn, needle):
    rows = conn.execute(
        "SELECT pid, query FROM pg_stat_activity "
        "WHERE state = 'active' AND query ILIKE %s",
        ("%" + needle + "%",),
    ).fetchall()
    return rows

Pass in a distinctive fragment of the migration statement, and treat a non-empty result as "not rolled back yet". That is a small function, and it is the one Render's team would have wanted at 12:35.

3. Keep hot job state away from reporting data

The scheduler database held both core job state and secondary data for search and reporting. That is a quiet design choice with a loud failure mode. When one store does two jobs, a problem in the less important job can block the important one. A reporting query, a search index rebuild, or a migration on a shared table can all hold up the thing that actually starts your deploys.

You do not need to split the database on day one. But you can ask one question before your next migration: does this table also carry the state that decides whether work starts? If it does, that migration deserves a production sized test, a lock timeout, and a named person watching it. A practice run in a test environment passed for Render, and the migration still locked a large table in production. Test data is usually small. Production tables are not. A migration that is instant on a thousand rows can be a lock storm on millions. If you cannot test at production size, test at a size that at least makes the lock visible, and treat "it was fast in test" as weak evidence, not a guarantee.

Render's planned fixes point the same way: blue-green deploys for this component, so the system is not upgraded under load, cross-zone high availability for the database, and storage mechanisms better suited to the scheduler workload. Blue-green means running the new version alongside the old one and switching traffic over only when the new one is healthy, instead of upgrading the live system in place. That is a bigger change than a lock timeout, but it is the same idea at platform scale: never upgrade the only copy of a critical path while it is carrying traffic.

What to check before your next migration

Look at the last migration you ran, or the next one in your queue, and ask four questions.

First, if you stop the client mid-migration, what kills the statement on the database? If you cannot answer that, find the answer before the migration, not during the incident.

Second, what timeout bounds it? If there is none, add one.

Third, does the table it touches also hold the state that starts work? If it does, that migration is critical path work, whatever its size looks like in test.

Fourth, does your rollback depend on a step that can fail on its own, like a clone or a capacity increase? If it does, test that step separately, and have a second way back.

The Render outage did not come from a exotic bug. It came from an ordinary migration, on an ordinary upgrade, with a rollback plan that had a hole in it and a running statement nobody could see from the client side. That is what makes it worth learning from. Your next migration will also look ordinary, right up until it does not.

Sources: Render postmortem, "Builds, deploys, and queued deploys are not starting in all regions", October 14, 2026 incident, postmortem posted October 22, 2026, https://postmortem.io/incidents/render--2026-10-14--job-scheduler-schema-migration-builds-deploys/

When was the last time a migration surprised you, and did you check the database side before you called it rolled back?

The cluster that would not start, because 1000000 stopped being a number

Kashif Manzer — Wed, 28 Oct 2026 23:56:14 +0000

You create a new Kubernetes cluster. You wait. Nothing crashes. Nothing errors in a useful way. The cluster just sits there, with core components pending, as if it is waiting for something that will never arrive.

That is how this one started, on October 17, 2026, for people creating new Linode Kubernetes Engine-Enterprise clusters, known as LKE-E. Existing clusters kept working. Ongoing workloads kept working. Scaling existing clusters kept working. Only new clusters were stuck.

That split is the first clue. When old things work and new things do not, the problem is usually not the whole system. It is in the path that only new things take.

What happened

Starting October 17, 2026, customers were unable to create new LKE-E clusters globally. The impact was limited to newly provisioned clusters, while existing clusters, ongoing customer workloads, and cluster scaling remained fully functional throughout the event.

Investigation found the trigger after Linode Virtual Private Cloud subnet IDs crossed 1,000,000, following growth on October 17, 2026.

A Virtual Private Cloud, or VPC, is a private network inside a cloud provider. A subnet is a slice of that network. Each subnet gets an ID, a label that other parts of the system use to find it.

Due to a software logic flaw, the Cloud Controller Manager parsed these 7-digit IDs into scientific notation instead of raw integers, rendering the IDs invalid and causing required core components to remain pending on newly created clusters.

The Cloud Controller Manager is the part of Kubernetes that talks to the cloud provider. It asks for load balancers, routes, and network resources, and it keeps those resources in step with what the cluster wants. If it cannot match a subnet ID, it cannot finish that work, so the components that depend on it stay pending.

Akamai deployed a fix to enforce strict integer formatting and patched existing custom resources to restore the affected clusters. Internal testing confirmed that newly provisioned test clusters were fully operational, and the impact was considered mitigated at around 19:49 UTC on October 20, 2026. A postmortem update followed at 15:49 UTC on October 21, 2026.

That is the whole incident, in one paragraph. The interesting part is not Linode. It is the pattern, because it shows up in ordinary code far more often than it should.

An ID is not a number you do math on

An ID looks like a number, but you should treat it as a label. You do not add two user IDs together. You do not average subnet IDs. You only need the label to survive every hop unchanged, from the database, through the API, through JSON, through your code, and back out again.

A float breaks that promise.

A float, short for floating point number, is how computers store numbers with fractions. It is good for measurements. It is bad for exact labels, because it does not store every integer exactly, and because formatting a float can change how it looks.

Take 1000000. As an integer, it is seven digits, nothing more. As a float, a formatter can turn it into scientific notation, something like 1e+06. That means 1 times 10 to the power of 6. It is the same value to a human doing math, but it is not the same string. A downstream system that expects digits will reject it, or look up the wrong thing, or fail to match anything at all.

That is why the boundary matters. 999999 has six digits. 1000000 has seven. A bug that only appears when an ID grows by one digit can sit quietly for years, then break everything on the day a counter crosses that line. The system did not get worse that day. The data got bigger, and a hidden assumption finally broke.

You have probably seen a version of this before. A phone number stored as a number loses its leading zero. A credit card number stored as a number loses precision. A timestamp stored as a float loses the last digits. The lesson is the same every time. If you do not do math on it, do not store it as a number you can do math on.

The guard, in Go

In Go, do not parse an ID through float64. Parse it as an integer, and fail loudly if it is not digits.

raw := "1000000"
id, err := strconv.ParseInt(raw, 10, 64)
if err != nil {
    return err
}
_ = id

That code is small on purpose. ParseInt with base 10 and 64 bits will reject text that is not a plain integer. If the input is "1e+06", it fails, and you find out at the boundary, not three systems later when a cluster is pending.

JSON needs the same care. By default, Go's JSON decoder turns numbers into float64. If an ID arrives in JSON, tell the decoder to keep numbers as tokens instead.

dec := json.NewDecoder(r)
dec.UseNumber()
var v struct {
    SubnetID json.Number `json:"subnet_id"`
}
if err := dec.Decode(&v); err != nil {
    return err
}
if _, err := strconv.ParseInt(v.SubnetID.String(), 10, 64); err != nil {
    return err
}

UseNumber does not fix the ID for you. It stops the decoder from turning it into a float before you get a chance to check it. The ParseInt check after it is the guard.

The guard, in Python

Python has the same trap, in a different shape. JSON numbers become int or float depending on the value, and a generic formatter can still change the form. The guard is to reject a float ID, and to validate the digit form before you use it.

def subnet_id(value):
    if isinstance(value, bool):
        raise ValueError("subnet id must be digits, not a boolean")
    if isinstance(value, float):
        raise ValueError("subnet id must not come through as a float")
    text = str(value)
    if not text.isdigit():
        raise ValueError("subnet id must be digits only")
    return int(text)

The boolean check looks odd until you remember that in Python, bool is a subclass of int, so True would otherwise pass as 1. That is exactly the kind of small detail that makes a boundary guard worth writing once and reusing.

If you want to keep the ID as a string all the way through, that is often the best choice. A string does not get reformatted by a number formatter, and it does not lose leading zeros. Store it as a string, pass it as a string, and only convert it when you have a real reason to, and a check to go with it.

The test that would have caught it

The fix here was strict integer formatting. The test that catches this class of bug is even simpler. Test at the boundary where the form changes.

Test with 999999. Test with 1000000. Test with 1000001.

If your code passes the first and fails the second, you have found the bug before your users do. If you only test with small IDs like 1, 2, and 42, you will never see it, because small IDs survive almost every bad conversion.

Add that test at the place where an ID enters your system, not just where it leaves. By the time a bad ID reaches the cloud controller, the database, or the downstream service, you have lost the context that would tell you where it changed form. Check it at the boundary, log the original form when it fails, and fail loudly. A pending cluster with no useful error is worse than a clear rejection, because it looks like slowness, not a bug.

There is a second lesson in this incident, and it is about how you read a failure. Existing clusters worked. New clusters did not. That tells you the core system was healthy, and the creation path was not. When you see that split, do not restart everything. Trace the path that only the failing case takes, and check every place where a value changes form along that path. In this case, that path was subnet creation, and the form change was integer to scientific notation.

What to check this week

Look for IDs in your own system that pass through a float, a generic number formatter, or a JSON parser that does not preserve the number form. That includes subnet IDs, user IDs, order IDs, and any other identifier that looks like a number but is really a label.

Ask three questions for each one. First, does anything do math on this ID? If not, why is it a number at all? Second, what happens when it crosses a power of ten, from 999 to 1000, or from 999999 to 1000000? Third, if the form changes, where do you find out, at the boundary, or days later when something is pending?

If you cannot answer the third question, that is the bug to fix first. A clear error at the boundary is a feature. A silent pending state is not.

Sources: Linode status summary for the October 20, 2026 LKE-E incident, as reproduced on Statusfield, https://statusfield.com/services/linode/incidents

Where in your own system does an ID pass through a float, a generic formatter, or a JSON parser that could change its form?

Your app was patched. Your PHP was not: PHP2Shell explained

Kashif Manzer — Sun, 25 Oct 2026 20:57:34 +0000

On Monday, October 20, 2026, the secret got out early. Details of two critical PHP flaws leaked before the planned disclosure. The PHP team moved its security release up a day and shipped the fixes on Wednesday, October 21.

That rushed week has a name now: PHP2Shell. It is not a WordPress plugin bug. It is not a theme bug. It is a flaw in PHP itself, the runtime underneath all of those things.

If you run anything on PHP, this one is worth understanding properly, because the usual advice, update your app, does not fix it.

What PHP2Shell actually is

PHP2Shell is the name researchers gave to two flaws in PHP core:

CVE-2026-109618, rated 9.2 out of 10 under CVSS v4.0
CVE-2026-109619, also rated 9.2

Both can let an attacker who is not logged in run code on a vulnerable web server by sending a crafted HTTP request. No username. No password. No click from a victim.

A quick glossary, because the rest only makes sense with it:

Runtime: the program that runs your code. Your app is the recipe. PHP is the kitchen. If the kitchen is broken, a perfect recipe does not help.
Remote code execution (RCE): an attacker gets to run their own commands on your server, as the web server user.
Worker: a long lived PHP process that handles one request after another. PHP-FPM and Apache mod_php both use workers.

Both flaws live in how PHP sets up and cleans up after a web request. That is why they are so wide reaching. The vulnerable code runs before your application code gets a chance to defend itself.

Flaw one: the half written form

CVE-2026-109618 happens when PHP runs out of memory in the middle of setting up request data.

Think of it like a clerk filling in a form. Halfway through, the clerk runs out of paper and stops. Later, someone else picks up that half finished form and treats it as complete. They follow instructions that were never meant to be real.

In PHP terms, out of memory handling interrupts the setup while internal request data is only partly built. During cleanup, PHP can then treat that invalid data as valid. That confusion in memory is what an attacker can steer toward running code.

This is the flaw Cisco Talos describes as out of memory handling interrupting PHP while internal request data is only partially initialized, with cleanup then treating invalid data as valid.

Flaw two: the stale sticky note

CVE-2026-109619 is a close cousin, and it is easier to picture.

Imagine a worker reuses a desk. The last person left a sticky note on it. The new worker is supposed to throw that note away and start fresh, but for a brief moment the old note is still there, and the new worker reads it as if it were their own instructions.

In PHP terms, request specific state survives after the memory behind it has been freed. A later request on the same worker can reach that stale state before normal setup resets it. Again, memory that should be dead gets treated as live, and an attacker can use that window.

Neither flaw needs a bug in your code. Your code can be clean, reviewed, and fully patched, and still sit on top of a broken runtime.

Why updating WordPress did not fix it

This is the part most people missed in the first wave of headlines.

PHP powers WordPress, Drupal, Joomla, Laravel apps, and a long list of vendor tools and control panels. When you update WordPress, you update the app. You do not update PHP. Those are two different layers, updated by different people, on different schedules.

CISA, the US cybersecurity agency, lists the affected setups plainly: Apache with mod_php, PHP-FPM, php-cgi, FrankenPHP, and LiteSpeed. It calls Apache mod_php the highest risk, because the demonstrated exploit there triggers before any site code runs.

So the checklist question is not "is my CMS up to date?" It is "is my PHP up to date?" If you cannot answer the second question, you have not finished the job.

Why finding every PHP install is the hard part

Patching sounds simple: install the fixed version and move on. In practice, PHP hides.

It hides inside Docker images built months ago. It hides inside vendor software that never mentions PHP in its marketing. It hides on a shared hosting account you forgot about, and in an internal tool someone set up years ago and never documented.

CISA makes the same point in its alert: PHP is often a component of other software, and its presence may not be directly visible. That is why the first step is an inventory, not a patch. You cannot patch what you have not found.

A practical way to start:

List your servers, containers, and hosting accounts.
On each one, check the runtime itself, not the app. On a server, php -v tells you the version of the command line PHP, which is a start, but check what your web server actually uses too. A system can have more than one PHP installed.
Check your container images and rebuild any that bundle an old PHP, even if the app inside them is current.
Ask your hosting provider or platform what PHP version they run for you, if you do not control the server. Managed WordPress hosts patched many customers automatically. Unmanaged servers did not patch themselves.

What to do today

The fixed versions, for supported branches, are:

PHP 8.5.12
PHP 8.4.27
PHP 8.3.36
PHP 8.2.35

Pick the fixed version for the branch you already run. Do not jump branches in a panic unless you have tested it. A branch jump can break an app in ways a patch release should not.

Then do the step people skip: restart every PHP worker. Restart PHP-FPM, restart Apache if you use mod_php, restart queue workers. A running process keeps the old, vulnerable code in memory until it restarts. Installing the patch on disk is not enough while the old worker is still alive.

If you run PHP 8.1 or earlier, there is no official patch for your branch. Those versions are out of support. Your real options are to upgrade to a supported branch, or, as CISA puts it, take the PHP based system offline until it can be patched or updated. That sounds drastic. It is the honest version of "we have no fix for this version."

A web application firewall may block some attack attempts, and it is worth turning on if you have one. But it is not a complete fix. The PHP team and CISA are clear on this: patching is the only complete fix. A firewall rule can miss a variant. A patched runtime cannot be exploited through these two flaws at all.

Finally, watch for the signs of a worker gone wrong: a PHP worker process launching a shell, a scripting interpreter, or a download tool. That is not normal web server behaviour. If you see it, treat it as an incident, not a curiosity.

What about the AI angle?

You will have seen the other headline: these flaws were found with help from frontier AI models. Researchers at Calif and Wiz have said AI tools helped them trace and verify a complex chain that human researchers might have dismissed as impractical.

That part matters, but not for the reason the hype suggests. It does not mean AI is now finding every bug overnight. It means the cost of finding a hard, decade old bug just dropped. A flaw that sat in PHP since the 5.4 era, according to the researchers, stayed hidden for years partly because exploiting it looked too complex to bother with. Once that complexity barrier falls, the patching window matters more than ever.

The lesson is boring and useful: your patch process is now part of your security posture, not an admin chore you get to later. The gap between "patch available" and "patch installed everywhere, workers restarted" is the gap attackers live in.

Reports of early attack attempts circulated before and just after the patch, including from Cloudflare. But nuance matters here: CISA said that as of its alert, it had not observed successful exploitation in the wild. Early attempts are not the same as confirmed break ins. Do not let either extreme mislead you. The flaws are real and critical, the patch exists, and unpatched servers are the ones at risk.

The takeaway

Most security advice tells you to patch your app. PHP2Shell is a reminder to patch the layer underneath your app too.

Check your PHP version. Check it in every place PHP could be hiding, not just the one server you remember. Upgrade to a fixed release. Restart the workers. And if you are on an unsupported version, make the upgrade plan this week, not someday.

Your app was never the only thing an attacker could reach. This week proved it.

What did you find when you checked? Did PHP turn up somewhere you did not expect?

Sources

CISA alert: Critical Vulnerabilities in PHP Require Immediate Patching: https://www.cisa.gov/news-events/alerts/2026/10/21/critical-vulnerabilities-php-require-immediate-patching
Cisco Talos: Two critical RCEs in PHP core: https://blog.talosintelligence.com/two-critical-rces-in-php-core/
Tenable FAQ on CVE-2026-109618 and CVE-2026-109619: https://www.tenable.com/blog/php2shell-cve-2026-109618-cve-2026-109619-frequently-asked-questions-about-critical-php-remote
DigitalOcean: Critical PHP RCE Vulnerabilities, patch now: https://www.digitalocean.com/blog/critical-php-rce-vulnerabilities-patch-now

Why the Linux OOM killer killed your container when memory looked fine

Kashif Manzer — Fri, 23 Oct 2026 16:12:01 +0000

Why the Linux OOM killer killed your container when memory looked fine

Your phone buzzes at 3 AM. A pod restarted. kubectl describe pod says the last state was OOMKilled.

So you check the node. It has gigabytes free. Nothing is close to full. You stare at the two facts side by side: killed for lack of memory, and plenty of memory. One of them has to be lying.

Neither is. There are two different out-of-memory killers on that machine, and only one of them cares about the node's free memory.

The two killers

Every container on your node lives inside a cgroup, short for control group. A cgroup is the kernel's way of putting a fence around a group of processes and saying: this group gets this much CPU, this much memory, no more.

When you set resources.limits.memory: 512Mi on a container, you are setting the fence height for its cgroup. When the processes inside cross that line, the kernel steps in and kills something inside the cgroup. That is a cgroup OOM kill. The node can have 30 GB free. It does not matter. The fence is the fence.

The other killer is the system OOM killer. That one fires when the whole node is genuinely out of memory. It picks a victim from anywhere on the machine.

So the first question to answer after any OOM kill is: which one fired? The kernel tells you, in the machine's log:

dmesg | grep -i "out of memory"

You will see a line like this:

Out of memory: Killed process 18421 (java) total-vm:2147484kB ...

Right above it, the log names the cgroup the victim belonged to. If it names your container's cgroup, the container hit its own limit, not the node's. That is the 3 AM answer nine times out of ten.

How the killer picks its victim

Once the kernel decides a kill has to happen, it does not kill at random, and it does not simply kill the biggest process. It scores every candidate and kills the highest scorer.

The score lives in a file you can read right now:

cat /proc/self/oom_score

Mine says 666. Yours will say something else. The number is roughly 0 to 1000 and it is mostly this: how much memory the process uses, as a share of the memory it is allowed to use.

That "allowed to use" part is the whole trick. Inside a container, your allowance is the cgroup limit, not the node's memory. A process using 400 MB inside a 512 MB container scores far higher than a process using 2 GB on a 64 GB node, because the first one is at 78 percent of its allowance.

The kernel adds small adjustments on top. Root-owned processes get a small discount. And then there is the knob.

The one knob that changes the score

Every process carries a second file:

cat /proc/self/oom_score_adj

This one says 0 by default, and you (or your orchestrator) can set it anywhere from -1000 to +1000. The kernel simply adds it to the score. Set it to -1000 and the process is unkillable by the OOM killer. Set it to +1000 and it is first in line.

This is the mechanism behind one of the most confusing behaviors in Kubernetes. Kubernetes sets oom_score_adj for you based on the pod's quality of service class, a label Kubernetes derives from your resource settings:

Guaranteed (every container sets equal memory requests and limits): the kernel is told -998. Almost never killed.
BestEffort (no requests or limits at all): the kernel is told 1000. First in line, always.
Burstable (anything in between): somewhere from 2 to 999, scaled from how big your memory request is compared to the node.

So two identical processes can have wildly different survival odds, purely because of the QoS class on the pod spec. Your BestEffort batch job is not unlucky. It volunteered.

What to actually check

When a container dies with OOMKilled and the node looks healthy:

Read the kernel log. dmesg | grep -i "out of memory" names the cgroup. If it is your container's cgroup, the limit was the ceiling, not the node.
Compare limit to reality. kubectl top pod shows actual usage. If usage was brushing the limit, the limit was too small, or the app leaks.
Check the QoS class. kubectl describe pod shows it. A BestEffort pod on a busy node is living on borrowed time.
If it is Java, check the heap. The classic trap: the JVM sizes its heap from the node's memory unless told otherwise, then the container's cgroup fence kills it. Size the heap to fit inside the container limit, with room for the JVM's own overhead.

None of this requires new tools. The score, the knob, and the log are all plain files and one shell command. The killer is not mysterious once you know it scores against the fence, not the field.

What is the strangest OOM kill you have ever chased? I want to hear the one where the numbers made no sense until they did.

One zone lost power in London, and the banks went dark with it

Kashif Manzer — Wed, 21 Oct 2026 16:13:49 +0000

Tuesday, just after 7am

Tuesday morning in London. People tapped their bank cards at the till and the payment was declined. Revolut transfers froze mid-send. Lloyds and HSBC customers could not get into their apps. In courts across England and Wales, video hearings stalled because the courts' video platform was down. Royal Mail tracking went quiet, and delivery networks wobbled.

None of these companies had a power cut. A Google Cloud data center zone did.

Just after 7am London time on Tuesday, October 20, a power event knocked out europe-west2-b, one of three availability zones inside Google Cloud's London region. Google's own status text put it plainly: "A power event caused an outage within europe-west2-b zone. The power event is mitigated."

What followed is the part worth studying. One zone, one of three, lost power. And banks, courts, and couriers across the country felt it.

Google has said it will publish a full analysis once its internal investigation is finished. Until then, nobody outside Google knows the complete chain of events. What follows is what the status page and the press record show, plus the mechanics of what a zone failure actually breaks. If you run anything in the cloud, this is your blast radius lesson for the week.

What Google said

The timeline, from Google's status page and press reports:

Just after 7am London time (06:08 UTC), Tuesday Oct 20. The power event hits zone europe-west2-b. Google: "Multiple services are experiencing issues in the europe-west2 region."
Through the morning. "Google Cloud services are unavailable in europe-west2-b zone; some services in europe-west2 region may experience elevated errors and intermittent network issues." The incident page listed dozens of affected products: Compute Engine, Google Kubernetes Engine, Cloud Run, Cloud SQL, Cloud Storage, BigQuery, Pub/Sub, Memorystore, Persistent Disk, Cloud Load Balancing, Virtual Private Cloud, and more.
Around midday. Most services were back to normal by about 12:10pm London time, roughly six hours in. Full recovery of existing workloads was tentatively expected by early afternoon.
Late afternoon. The Ministry of Justice confirmed the courts' Cloud Video Platform had still not fully recovered. Some wounds take longer to close.

The customer list reads like a tour of British daily life: Revolut, Lloyds, HSBC, Halifax, Barclays, Bank of Scotland, NatWest, Nationwide, credit-reference firm Equifax, Royal Mail, DPD, property site Rightmove. The Telegraph reported cards declined at tills. The Guardian reported courts disrupted and millions unable to move money.

What a zone is, and what dies with it

A quick glossary, because the rest of this piece depends on it.

A region is a geographic area where a cloud provider operates, like London. Inside each region are availability zones: separate buildings, or clusters of buildings, each with its own power feed, its own cooling, and its own network connections. The London region, europe-west2, has three: a, b, and c. The idea is simple. If one building has a bad day, the other two keep going.

Your virtual machine lives in exactly one zone. Your disk lives in one zone too, with copies kept elsewhere for safety, but those copies are managed by the platform, not by you.

So when the power goes in zone b, everything physically in zone b stops at once. Virtual machines halt instantly, with no graceful shutdown. Disks stop being reachable. Anything that had its database primary in zone b stops answering. There is no orderly wind-down. The lights go out and the machines stop.

This is the part people nod along to and then get wrong in practice, because "we run in multiple zones" feels like a shield. It is a shield with holes. Here are the holes.

Why the surviving zones did not just absorb it

Failover takes time. Nothing in a distributed system switches over instantly. A database has to notice its primary is gone, hold an election, and promote a replica. A load balancer has to fail several health checks before it stops sending traffic to dead backends. DNS answers are cached, and each cached answer has a TTL, a countdown timer after which it expires. During every one of those windows, requests fail. Multi-zone does not mean zero-downtime. It means downtime measured in minutes instead of hours, if everything works as designed.

Your dependencies have zones too. Your app can run happily in zone a and still die, because it calls something that lived in zone b. A database primary. A NAT gateway. A load balancer component. A queue. Google's own status text admitted the mechanism: services across the region saw "elevated errors and intermittent network issues" because of shared infrastructure dependencies. A zone is not a sealed box. The region shares plumbing, and the plumbing runs through every zone.

You could not build replacements during the fire. Read this line from the status page again: "Customers may not be able to yet create new workloads." The system that creates virtual machines and clusters, the control plane, was itself degraded. Your running apps are the data plane. The API that manages them is the control plane. They are different systems, and this event hurt both. The usual escape plan, spin up fresh capacity in the surviving zones, was partially locked during the exact hours people needed it. The fire escape was bolted during the fire.

Two zones doing the work of three is not the same as three zones. Failover plans quietly assume spare capacity. When a third of a region's machines vanish, the survivors absorb their load on top of their own. Everything gets slower, queues grow, and some things fail that would have survived a smaller shock. One engineer's account in the press said the peer zones did not fully absorb the load. Capacity is the quiet part of every failover plan, and it is the part nobody load-tests.

The single-region trap

Here is the structural fact that made this outage national news instead of a footnote: europe-west2 is Google's only UK region. Customers who had built everything in London, sensibly, to keep data close to users, had no second UK location to fail over to.

Multi-zone protects you against one zone failing. Nothing inside a single region protects you against the region's shared systems failing, or against a zone failure cascading the way this one did through shared dependencies. Real protection is multi-region: running in London and somewhere else, with a tested plan to shift traffic. That roughly doubles your bill and your complexity, which is why most teams do not have it. The banks that went dark on Tuesday are a reminder of what that savings costs.

This happened before

The London region has been here before. On July 19, 2022, during the hottest day ever recorded in the UK, cooling failed in a building hosting europe-west2-a. Google's engineers could not get temperatures down, so they deliberately powered down part of the zone to save the hardware. That decision terminated about 35 percent of the virtual machines in that zone.

Different zone, different cause, same lesson. A zone is a unit of failure. It can go dark from heat, from power, from a bad config push, from a backhoe through a fiber bundle. The cause changes. The shape of the failure does not.

Audit your own blast radius

"We run multi-AZ" is a claim. Here is how to check whether it is true.

# Where do your VMs actually live? Not where you think. Where they are.
gcloud compute instances list --format="table(name,zone.basename())"

# Are your Kubernetes nodes spread across zones, or quietly all in one?
kubectl get nodes -L topology.kubernetes.io/zone

Then ask the questions no dashboard answers for you:

Which zone holds your database primary right now? Your managed database console will tell you. Write it down.
If that zone went dark, how long until a replica gets promoted, and have you ever watched it happen? If the answer is "the docs say it is automatic," you have not tested it.
Which zone do your NAT gateways, load balancers, and file stores live in? These are the shared plumbing. They have zones too.
During the last incident, could you have created new capacity? If your control plane calls were failing, your runbook's step one was fiction.

The cheapest version of this audit is a game day: pick a zone, pretend it is gone, and walk through what breaks. The 2022 incident's official workaround was "relaunch your workloads in another region." That advice only helps if you know, before the bad day, that you can.

The question

Draw your blast radius on paper. Pick the single thing, a zone, a database primary, a region, whose failure at 7am would ruin your day. If you cannot name it, that is the audit, and it is overdue.

Do you know, right now, which zone your database primary lives in? When did you last check?

I kept approving Claude Code commands I didn't understand, so I built clear-caption

Kashif Manzer — Tue, 20 Oct 2026 21:18:54 +0000

I approve dozens of commands from Claude Code every day.

Most I skim. Some I don't really understand, especially the long, complex ones.

I click yes anyway and hope nothing breaks.

Like everyone else.

So I built clear-caption.

Now, before anything runs, Claude tells me in plain English what it is about to do. A command, a file edit, doesn't matter. One quiet line above the call, in words I can actually read.

The real command stays on screen, unchanged. Nothing gets hidden or rewritten. The caption just sits above it.

What it looks like

Before, a normal session looked like this:

● Bash(find . -name "*.tmp" -exec rm {} \;)

● Bash(git reset --hard HEAD)

● Update(math.js)

Three calls. One deletes files by pattern, one throws away local work, one edits code for a reason you have to guess from the diff.

With clear-caption, the same session looks like this:

● clear-caption: Delete all .tmp files

● Bash(find . -name "*.tmp" -exec rm {} \;)

● clear-caption: ⚠ Throw away all local changes

● Bash(git reset --hard HEAD)

● clear-caption: Fix clamp: Math.min should cap at hi, not lo.

● Update(math.js)

That is the whole idea. Reviewing a step takes a glance instead of a decode.

What gets a caption

Every shell command and every subagent call gets one. It is the call's own description, shown right above its row.

Every file edit gets one too. Edit and Write have no description of their own, so the mod asks Claude to write one sentence on why the file is changing, and shows it above the diff.

Subagents' own calls stay quiet. A busy subagent running twenty commands should not flood your screen with captions, so only the top level gets them.

Captions are also their own lines, separate from the tool rows. If you use a skin or another mod that redraws those rows, the captions still show.

The warning is the part I wanted most

Some commands can wipe out your work in one keystroke. Those get a ⚠ in the caption, even when the description sounds harmless.

The warning is based on the real command, not on what Claude says about it. It triggers on things like:

Recursive deletes (rm -r) outside temp folders
Commands that throw away git work, like git reset --hard, git clean -f, git branch -D, or dropping and clearing stashes
Force pushes
Deletes by search (find ... -delete)
sudo, piping curl into a shell, or chmod -R

Deleting files only under /tmp or a scratchpad folder is not flagged. Cleanup in a temp folder is normal work, not a risk.

It is a short list of common spellings, matched as patterns. It is not a guarantee, and I won't pretend otherwise. It catches the stuff people actually type, which is where the accidents live.

Turn it up or down

/caption all     every caption (default)
/caption risky   only ⚠ captions
/caption off     no captions
/caption         show the current setting

Your choice is saved, so it carries across sessions.

If you only want the safety net and none of the narration, /caption risky is a good place to start.

The honest limits

The caption is Claude's own words. It can be vague, and it can be wrong. That is exactly why the real command and the real diff always stay on screen. The caption is a reading aid, not a replacement for the call itself.

On testing: edit captions landed on 57 of 57 edits across Opus, Sonnet, and Haiku. The average cost on the Opus test tasks was the same with captions on or off, for about 3% more output tokens.

It is built on Claude Code's mod hooks, which are early access. Hooks can change between releases, so things may shift. It is tested on Claude Code 2.1.296.

How it works, briefly

Two hooks do the work. One asks Claude for a one-line description on Edit and Write calls, then strips that description out before the call is checked, so the edit itself runs exactly as it would have. The other holds each call until its arguments arrive, logs the caption, and passes the call on unchanged.

Install

Paste this at the Claude Code prompt:

/plugin install clear-caption --marketplace KashifManzer/clear-caption

Answer y to add the marketplace, pick a scope, and you are done.

It is free and open source under the MIT license. The repo is here: https://github.com/KashifManzer/clear-caption

If it saves you a squint, a star helps other people find it. Bug reports, ideas, and pull requests are all welcome too.

One question before you go.

Have you ever approved something you didn't fully understand?

How a DNS race condition took DynamoDB offline in us-east-1

Kashif Manzer — Mon, 19 Oct 2026 16:12:33 +0000

How a DNS race condition took DynamoDB offline in us-east-1

Sunday night, 11:48 PM Pacific. One of the largest databases on Earth is running perfectly. Every server is up. Every disk is fine. And nobody can reach it, because its name no longer has an address.

This happened in October 2025. DynamoDB in us-east-1 did not crash, did not slow down, did not lose data. It vanished from DNS. Every system that tried to connect, customer apps and AWS's own internal services alike, got the same answer: the name dynamodb.us-east-1.amazonaws.com resolves to nothing.

AWS later published a full summary of the event, and it is one of the most instructive postmortems I have ever read. Not because the bug was exotic, but because it shows how the cloud actually holds together: layers of automation, each depending on the layer below, until one quiet race condition pulls the thread and half the region unravels.

Why a database needs hundreds of thousands of DNS records

First, the part that surprised me. DynamoDB maintains hundreds of thousands of DNS records. Not as a directory for humans. As a scaling primitive.

DynamoDB runs a very large, heterogeneous fleet of load balancers in every region. When capacity is added, when hardware fails, when traffic needs to shift, the way the service steers clients is DNS. Automation constantly rewrites those records: add the new load balancers, drop the dead ones, rebalance the weights. DNS is not the phone book here. It is the steering wheel.

That automation is split into two independent components, deliberately, for availability. The first is the DNS Planner. It watches the health and capacity of the load balancers and periodically produces a new DNS plan for each endpoint: a set of load balancers plus weights. One regional plan is shared across endpoint variants, the public regional endpoint, a FIPS endpoint, an IPv6 endpoint, account-specific endpoints, which keeps capacity management simple.

The second component is the DNS Enactor. It takes plans and applies them as changes in Route 53. It is designed to have minimal dependencies so it can recover the system in any scenario, and it runs redundantly and fully independently in three availability zones. Each Enactor picks up new plans and applies them using Route 53 transactions, so an endpoint always gets a consistent plan even when several Enactors are working at once.

Before an Enactor starts applying a plan, it does one check: is my plan newer than the one already applied? One check, at the start. Remember that detail. It is the whole story.

The race

Right before the outage, one Enactor hit unusually high delays. It kept getting blocked trying to apply its plan to several endpoints and had to retry, slowly working its way through the list.

While it was stuck, two things happened. The Planner kept running and produced many newer generations of plans. Then a second Enactor picked up one of the newer plans and applied it rapidly across all endpoints.

When the second Enactor finished, it ran the plan clean-up process. Clean-up deletes plans that are significantly older than the one just applied. Housekeeping. Reasonable.

But at that exact moment, the first, delayed Enactor finally applied its much older plan to the regional DynamoDB endpoint, overwriting the newer plan. The freshness check it had done at the start was stale by now, after all those delays, so nothing stopped it.

Then the second Enactor's clean-up ran and deleted that older plan, because it was many generations behind the plan it had just applied. Deleting the plan removed every IP address for the regional endpoint. And because the plan that was deleted was the active one, the system was left in an inconsistent state where no subsequent plan update could be applied by any Enactor. The automation that was supposed to repair itself could not. A human had to step in.

From the outside, this is what it looked like:

import socket
# What every client, human or machine, effectively saw that night:
socket.getaddrinfo("dynamodb.us-east-1.amazonaws.com", 443)
# socket.gaierror: [Errno -2] Name or service not known

Not a timeout. Not a 500. The name did not resolve. The database was right there, healthy, unreachable.

Then AWS forgot how to launch servers

Here is where it gets properly strange. The DynamoDB DNS failure was fixed by 2:25 AM. But EC2 could not launch new instances until 1:50 PM, more than fourteen hours after the first failure. Why? Because inside AWS, EC2's own machinery depends on DynamoDB.

Deep inside EC2 there is a subsystem called the DropletWorkflow Manager. It manages the physical servers, which AWS calls droplets, that host EC2 instances. Each manager holds a lease on the droplets it manages, renewed by state checks every few minutes. Those state checks depend on DynamoDB.

At 11:48 PM, the checks started failing. Running instances were fine, but over the next two and a half hours, leases across the fleet quietly timed out. And a droplet without an active lease is not considered a candidate for new launches. So EC2 started refusing new instance launches with "insufficient capacity" errors. There was plenty of capacity. The system just could not prove to itself that any server was allowed to take work.

When DynamoDB recovered at 2:25 AM, the lease manager tried to re-establish leases across the entire fleet at once. Too much work arrived too fast, the queues choked, and the work kept timing out before it could complete. The system had entered what AWS called congestive collapse: too busy failing to recover to actually recover. Engineers throttled incoming work and selectively restarted the lease manager hosts at 4:14 AM. By 5:28 AM, every droplet had a lease again.

But new instances still had no network. A second subsystem, Network Manager, propagates network configuration to new instances, and it was sitting on a huge backlog of delayed state changes. Propagation times did not return to normal until 10:36 AM. Full EC2 recovery landed at 1:50 PM.

The blast radius kept widening from there. Lambda function creation and updates broke, and an internal SQS polling subsystem failed and had to be restored by hand at 4:40 AM. ECS, EKS, and Fargate could not launch containers or scale clusters until 2:20 PM. Amazon Connect callers heard busy tones and dead air. IAM user sign-ins to the console failed. Redshift query processing, which reads and writes through DynamoDB endpoints, broke, and because Redshift resolves IAM user groups through an IAM API pinned to us-east-1, Redshift customers in every region lost the ability to run queries with IAM credentials for over an hour.

My favorite detail: the AWS Support Center failed over to another region exactly as designed, and still blocked legitimate users. The failover bypass was built to kick in when the account metadata subsystem returned unsuccessful responses. That night it returned invalid responses instead. The designers had predicted one failure mode. Reality chose the adjacent one.

The shape of the fix

AWS disabled the DNS Planner and Enactor automation worldwide and said it would fix the race condition and add protections before turning it back on. The protections are the interesting part for the rest of us, because the pattern applies to any system that applies versioned plans. In sketch form:

def apply_dns_plan(plan, state, route53):
    # The freshness check must be re-done at apply time,
    # not once at the start of a slow walk over endpoints.
    if plan.generation <= state.active_generation:
        return "skip: stale plan"

    route53.replace_record(plan.endpoint, plan.ips)
    state.active_generation = plan.generation


def cleanup_old_plans(plans, state):
    for plan in plans:
        # Never delete the plan that is currently serving traffic,
        # no matter how old it looks.
        if plan.generation == state.active_generation:
            continue
        if plan.generation < state.active_generation - KEEP_GENERATIONS:
            delete(plan)

Two guards. Never apply a plan older than what is live, checked at the moment of applying. Never delete the plan that is live. Either guard alone would have prevented the outage. The original system had a version of the first guard, but it checked once at the beginning of a long, retried operation, and time made liars of it.

What I take from this

Three things stuck with me.

First, your dependencies have dependencies. Nobody who launched an EC2 instance that night thought about DynamoDB DNS records. Nobody running Redshift in another region thought their queries depended on an IAM API in Virginia. The dependency graph of a real system is always deeper than the diagram on the wall.

Second, automation that cannot repair itself is automation with a single point of failure wearing a costume. The DNS system was designed for resilience against many operational issues, and it met an issue that left it in a state its own recovery logic could not process. If your self-healing has states it cannot handle, you do not have self-healing. You have a pager with extra steps.

Third, error messages lie during cascades. "Insufficient capacity" meant "we lost our leases." Health checks failed on healthy machines because network state had not caught up. When everything is failing at once, read the errors as symptoms, not diagnoses.

AWS's summary is public and worth reading in full if you run anything on the cloud. It is a rare, honest look at the machinery underneath the dashboard.

What is the deepest hidden dependency you have found in your own stack, the one that would surprise everyone on your team?

Sources: AWS "Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region" (aws.amazon.com/message/101925/); press coverage at the time listed Venmo, Snapchat, Fortnite, and Amazon's own Alexa among affected services.

Your pipeline failed but your YAML was fine: how a Gitaly timeout broke GitLab CI on Friday

Kashif Manzer — Sun, 18 Oct 2026 16:10:56 +0000

Your pipeline failed but your YAML was fine: how a Gitaly timeout broke GitLab CI on Friday

Friday afternoon. You push a commit, the pipeline kicks off, and instead of green checks you get this:

"CI configuration fetch from Gitaly timed out. This may indicate Gitaly service slowness or an outage."

Your first instinct is to open your .gitlab-ci.yml and hunt for the typo. Stop. There is no typo. Your config was never read at all.

What actually happened

On October 16, GitLab.com had a rough afternoon. At 11:33 AM UTC, GitLab posted that CI/CD was failing for some customers with exactly that error, and that the UI was slow too. They applied a temporary mitigation at 12:28 PM UTC and marked it resolved at 1:21 PM UTC. One hour and forty-eight minutes.

Then, later the same day, a second incident: 503s across the website, the API, Git operations, both registries, Pages, and the hosted runners on Linux, Windows, and macOS, from 4:10 PM UTC to 5:24 PM UTC.

Two incidents, one afternoon. The first one is the interesting one, because the error message tells you exactly where the system is coupled.

Why creating a pipeline reads your repo

Here is the part most people never think about. Your .gitlab-ci.yml lives in your repository. When you push and a pipeline is created, GitLab has to go read your config out of the repo, merge it with any includes, and only then can it build the list of jobs. There is no warm copy sitting next to the scheduler. Every pipeline starts with a read.

All Git reads in GitLab go through a service called Gitaly. Think of it as the front desk for every repository: instead of each part of GitLab opening raw Git data directly, they call Gitaly over RPC and Gitaly does the reading.

So the chain for every single pipeline looks like this:

push -> pipeline created -> fetch config from the repo via Gitaly -> parse and merge -> schedule jobs

That fetch step is synchronous. It sits on the critical path, and it has a timeout. When Gitaly got slow on Friday, the fetch hit the timeout, and pipeline creation failed before a single job was scheduled. Nothing ran. Nothing could run. The config was fine; the reader was sick.

That is also why the UI was slow at the same time. The same Gitaly layer serves the reads behind project pages, file views, and merge requests. One slow layer, many symptoms.

How to read this error

This is the practical skill the incident teaches. Not every red pipeline means the same thing, and the fix depends on which kind of red you got.

If your YAML has a syntax error, GitLab tells you your YAML has a syntax error. It names the file and the line. That is a config problem, and the fix is in your editor.

"CI configuration fetch from Gitaly timed out" is a different sentence. It says: I could not fetch. The failure is in the reading, not the writing. Debugging your YAML against this error is like proofreading a letter the post office lost.

So the first move is not your editor. It is the status page. If GitLab is having a Gitaly moment, no amount of YAML polishing will turn that pipeline green.

What to do while you wait

You still want that pipeline to run. The instinct is to hammer the retry button, or re-push the same commit five times. That instinct makes the problem worse.

Every retry is another config fetch through the same struggling layer. Thousands of developers retrying at once is a thundering herd aimed directly at the thing that is already down. (I wrote about why retries need jitter a couple of weeks ago: https://dev.to/kashif_manzer/why-your-retries-need-jitter-the-thundering-herd-explained-3eje. This is exactly the scenario it is for.)

The calmer version: retry with backoff, and let the error message decide whether retrying even makes sense. Here is a small wrapper I keep around for triggering pipelines from scripts:

import random
import time

import requests

API = "https://gitlab.com/api/v4"
HEADERS = {"PRIVATE-TOKEN": "your-token-here"}

def create_pipeline(project_id, ref="main", tries=5):
    delay = 2  # seconds
    for attempt in range(1, tries + 1):
        r = requests.post(
            f"{API}/projects/{project_id}/pipeline",
            headers=HEADERS,
            json={"ref": ref},
            timeout=30,
        )
        if r.ok:
            return r.json()
        # Only retry fetch-side timeouts. Anything else fails fast.
        if "gitaly" not in r.text.lower() or attempt == tries:
            r.raise_for_status()
        wait = delay + random.uniform(0, delay)
        print(f"attempt {attempt}: config fetch timed out, backing off {wait:.1f}s")
        time.sleep(wait)
        delay *= 2

The logic is simple. If the failure mentions the config fetch timing out, wait and try again with a growing, jittered delay. If it fails for any other reason, fail fast, because retrying a genuinely broken config is just burning API calls.

Two more habits that pay off during incidents like this:

Check the status page before you touch your code. Five seconds on the GitLab status page saves twenty minutes of phantom debugging.
Keep your includes lean. Every include is another file GitLab has to fetch and merge before your pipeline exists. A config that pulls in six included files gives the fetch step six files to read before a single job is scheduled.

The design lesson

Zoom out and the incident is a textbook coupling story. A config file feels static, like something the system should just know. But GitLab treats it as dynamic: fetched fresh on every pipeline, through a network service, on the critical path, with a timeout.

Dynamic fetching buys you real things. Edit the YAML, push, and the next pipeline picks it up with no cache to invalidate and no stale config to chase. The price is a dependency: your pipeline cannot exist faster than the slowest read in its setup path.

Every system has a version of this. A service that reads its feature flags from a remote store on every request. A deploy that pulls its manifest from an API before it starts. The question is always the same: is this read on the critical path, and what happens when it is slow instead of down? Slow is the cruel one. Down fails fast. Slow burns through timeouts, holds connections open, and takes the UI down with it.

Friday's error message was honest about all of this. It named the layer, named the timeout, and told you it might be slowness rather than an outage. Most error messages are not that generous. When one is, believe it.

What is the fetch-on-the-critical-path in your own system, the read everything quietly depends on? Have you ever watched it get slow?

How the DNS root rotated its master key on October 11 without breaking the internet

Kashif Manzer — Fri, 16 Oct 2026 16:19:58 +0000

How the DNS root rotated its master key on October 11 without breaking the internet

On October 11, 2026, the most trusted key on the internet changed hands. No deploy went out. No status page turned red. Nobody got paged. The DNS root zone simply stopped signing its list of keys with KSK-2017 and started signing with KSK-2024.

It was only the second time this has ever happened. The first was in 2018.

In Malaysia that weekend, some websites stopped loading for users on Maxis DNS. Local press tied the reports to the rollover. Maxis has not confirmed a network-wide cause, so treat that link as reported, not proven. But the failure shape fits exactly, and the shape is the interesting part: when a resolver misses this kind of change, nothing looks broken except everything.

The one key you cannot look up

DNSSEC lets a resolver check digital signatures on DNS answers, so it knows the answer is authentic and unmodified. Trust forms a chain. To believe example.com's keys, your resolver checks the signature from .com. To believe .com's keys, it checks the signature from the root. Each parent vouches for its child with a DS record, which is just a fingerprint of the child's key.

The root has no parent. So every validating resolver ships with one key it believes without proof: the root key-signing key, kept as a trust anchor in its software or configuration.

The division of labor is neat. The KSK signs only one thing: the root zone's DNSKEY record set, which is the published list of the root's public keys. That list contains the zone-signing key, the ZSK. The ZSK signs everything else, including the DS records for top-level domains like .com. So the KSK is the anchor for the anchor list. Lose it, and the chain has no starting point. Not one domain fails. Every domain fails.

Changing a lock while the building is full

You cannot email every resolver operator on earth and ask them to update a config on the same Sunday. Firewalls, home routers, containers, and appliances run resolvers whose owners never chose their DNS behavior and rarely check it. So ICANN ran the rollover in slow motion. The full schedule, from IANA's trust anchor page:

October 27, 2016: KSK-2017 generated.
October 11, 2018: KSK-2017 takes over signing the root. First rollover ever done.
April 27, 2023: KSK-2023 generated. Then abandoned, superseded before it ever signed anything. Someone made a master key for the internet and threw it away.
April 26, 2024: KSK-2024 generated.
January 11, 2025: KSK-2024 published in the root DNSKEY set, alongside KSK-2017. Nothing trusts it yet. It just sits there, its presence signed by the old key.
February 10, 2025: resolvers following RFC 5011 can finish adopting it.
October 11, 2026: the switch. KSK-2024 signs the DNSKEY set. KSK-2017 stops.
January 11, 2027: KSK-2017 published with the REVOKE bit set.
March 22, 2027: KSK-2017 removed from the root zone.

The quiet hero here is RFC 5011, automated trust-anchor updates. Your resolver watches the root DNSKEY set. One day it notices a new key in there, and the new key's presence is signed by the old key it already trusts. It does not trust it immediately. It waits. For 30 days it must keep seeing the new key, consistently, in normally signed responses. Only then does it promote the key to a trust anchor on its own. No config push, no restart, no human.

The 30-day hold-down is the whole security model in one number. It exists so a brief spoof or glitch cannot plant a fake anchor. But read the assumption hiding inside it: the resolver must be up and querying regularly through the transition. A validator powered off for a month, a VM snapshot resumed after the window, a container image that restores stale state on every deploy, a trust-anchor file on a read-only filesystem where updates fail silently. Any of them can still trust only KSK-2017 on the morning of the switch. Automation is not magic. It is a scheduled job with preconditions.

The question you are not allowed to ask

This is my favorite part of the story. Suppose you run a fleet of resolvers and want to know which ones trust the new key. There is no DNS query for "list your trust anchors." So RFC 8509 encodes the question inside a domain name.

Try these two lookups:

dig root-key-sentinel-is-ta-38696.dnstest.dev
dig root-key-sentinel-not-ta-38696.dnstest.dev

The authoritative server answers both names. But a validating resolver that trusts key tag 38696 will fail validation on the second name and return SERVFAIL to you. The resolver's own trust state leaks out through which answers survive. It is an interrogation disguised as a lookup.

How to read your result:

is-ta answers, not-ta returns SERVFAIL: your resolver trusts KSK-2024. You are fine.
Both names answer: inconclusive. Your resolver either does not validate DNSSEC or does not speak the sentinel protocol. Most home setups land here.
is-ta returns SERVFAIL: your resolver validates but does not trust the new key. Fix it before January, when the old key gets revoked.

What "not ready" looks like

It does not look like an error page. A validating resolver that only trusts KSK-2017 can no longer verify the root's key list, which means it cannot verify anything below it. Every lookup returns SERVFAIL. The sites are up. Your connection is fine. The names just stop resolving, all of them, at once.

That is why the scattered reports after October 11 are worth noting even though none of them is a confirmed smoking gun. Users in Malaysia reported .my sites failing to load on Maxis DNS between October 11 and 13. Router vendor forums had users describing lookups failing with DNSSEC enabled and recovering the moment they switched it off. No broad outage materialized, which is the good news. The bad news is the shape: this failure is silent and total, and it lands on exactly the devices nobody inventories.

Check the root yourself

I pulled the live root DNSKEY set today, October 16, and computed the key tags from it. Five days after the rollover, three keys are published:

8763, the zone-signing key, doing the daily signing work.
20326, KSK-2017: still published, no longer signing, revocation scheduled for January 2027.
38696, KSK-2024: the new signer.

You can do the same check from any machine with Python. DNS-over-HTTPS avoids depending on your local resolver's mood, and the key tag follows RFC 4034's recipe. One detail that bit me: the tag is computed over the full DNSKEY RDATA in wire format, not just the key bytes. My first attempt hashed only the public key and produced nonsense tags. Adding the four header bytes (flags, protocol, algorithm) fixed it.

import base64, json, struct, urllib.request

req = urllib.request.Request(
    "https://cloudflare-dns.com/dns-query?name=.&type=DNSKEY",
    headers={"Accept": "application/dns-json"},
)
data = json.load(urllib.request.urlopen(req, timeout=20))

def key_tag(rdata):
    flags, proto, algo, b64 = rdata.split()
    wire = struct.pack("!HBB", int(flags), int(proto), int(algo))
    wire += base64.b64decode(b64 + "=" * (-len(b64) % 4))
    total = 0
    for i, byte in enumerate(wire):
        total += byte << (8 if i % 2 == 0 else 0)
    total += total >> 16
    return total & 0xFFFF

for ans in data["Answer"]:
    flags = ans["data"].split()[0]
    role = "ZSK" if flags == "256" else "KSK"
    print(f"{role:>3}  key tag {key_tag(ans['data'])}")

Real output, October 16, 2026:

ZSK  key tag 8763
KSK  key tag 20326
KSK  key tag 38696

The numbers going in

The headline telemetry looked strong. ICANN reported that more than 95 percent of resolvers signaling their trust anchors had adopted KSK-2024. Verisign's measurements in March 2025 put adoption at 91.3 percent of monitored resolvers.

Two numbers complicate the picture. About 3.5 percent of reporting resolvers still carried KSK-2010, a key revoked back in 2019. Seven years after its funeral, a dead master key is still sitting in trust stores on real machines. That is how long stale trust state survives.

And APNIC's Geoff Huston measured from the user side rather than the root side. In April 2026, his sentinel-based tests found under 20 percent of users behind DNSSEC-validating resolvers showing KSK-2024 as trusted, which he called, quote, "certainly not the result that we had anticipated." Different populations, different methods, noisy data. But it is a reminder that the view from the root and the reality at the edge can disagree, and the edge is where your users live.

The actual lesson

Strip away the cryptography and the root KSK rollover is a distributed state problem. One organization publishes a key. Millions of independently operated resolvers have to observe it, keep that observation across restarts, upgrades, snapshots, and config management, and then the entire world switches which key it depends on. There is no central dashboard that can say everyone is ready, because DNS does not work that way. The 21-month runway, the 30-day hold-down, the sentinel interrogation protocol, the staged revocation: every piece of the design is an admission that trust, at internet scale, is a coordination problem wearing a cryptography costume.

Run the two sentinel queries against your own resolver and post what you got. I am genuinely curious about the ratio of clean passes to inconclusives out there.