The new machine with the old map: how GitHub routed writes to servers that no longer existed

On October 7, 2026, GitHub broke in a strange, specific way. You could read almost anything. You could not save almost anything.

Push a commit, open a pull request, file an issue, start an Actions run, sign in: those are all writes, actions that save or change data. Reads, looking at code and pages that already existed, were largely unaffected. That split is the first clue, and it is worth sitting with before you look for a cause. When reads work and writes do not, the data itself is probably fine. The road that new data travels is not.

That is the mystery this article works through. It ends with a small guard, in Python, that you can steal for your own routing config. The guard does one job: it refuses to use a map that points to servers that no longer exist.

Clue one: only the writing road was broken

GitHub had two outages that day with the same cause, totalling about 19 minutes. The first ran from 15:06 to 15:16 UTC, about 10 minutes. The second ran from 16:52 to 17:01 UTC, about 9 minutes.

In the first window, 5.6 percent of all web and API requests failed, peaking at 8.9 percent, and more than 12,000 organizations saw failed requests. Up to 26 percent of Git authentication requests failed, and 17 percent of Actions workflow runs failed with infrastructure errors. In the second window, 6.2 percent of web and API requests failed, peaking at 9.3 percent, and more than 11,000 organizations were affected. Up to 25 percent of Git authentication requests failed, and 19 percent of Actions workflow runs failed with infrastructure errors.

No databases lost data, according to GitHub, and operations retried after recovery succeeded. So this was not a story about lost work. It was a story about work that could not find its destination.

GitHub's own status report, posted October 7, 2026, names the affected areas in those windows: Git operations, pull requests, issues, GitHub Actions, the API, webhooks, sign-in and authentication, GitHub Apps and OAuth, code scanning, Codespaces, and Copilot. That list looks random until you ask what those things share. They all need to save something new.

Clue two: a new machine with an old map

To understand the cause, you need three plain ideas.

A database server is a machine that stores data and answers questions about it. A routing config is a list that says which database servers should receive new and updated data. Think of it as a map: it tells each write where to go.

A quorum is a group of machines that must agree before a change counts. A leader is the machine in that group whose word the others accept for now. Leader election is just the process of choosing which machine leads. If a machine with a wrong map becomes leader, its wrong map becomes the map everyone follows.

GitHub relies on an internal service that keeps track of which database servers should receive new and updated data. While replacing machines that run this service as part of regular maintenance, one of the new machines started with an outdated copy of its records. That copy still listed database servers GitHub had already retired. Retired here means taken out of service, gone, not answering anymore.

When this node came online, it incorrectly modified routing configuration with the outdated records, routing write requests to non-existent database servers. Writes were sent to machines that were not there. Reads did not need that road in the same way, so they mostly kept working.

Each outage ended when one of GitHub's established machines, with the correct configuration, became the leader of the quorum and sent the correct configuration. The good map replaced the bad map. That is why the first outage was only about 10 minutes long. The system healed itself, once the right leader was in charge.

Clue three: the pause that was not honoured

Here is the part that should worry you more than the first outage. After the first outage, GitHub's engineers identified the new machine as the source and paused automated maintenance to stop it from being leader again. Then, at 16:52 UTC, an automated maintenance task that did not honour the pause put it back in charge, and the outage happened again.

Read that slowly. The team did the right thing after the first failure. They found the bad machine and they paused the automation around it. But the pause was not honoured by every piece of automation. One task did not know about the pause, or did not check it, and it undid the fix.

GitHub shut that machine down at 17:02 UTC and has since removed it. GitHub also says the automation now honours the pause, and that this failure to honour it was a new regression, a bug introduced by an update to its Puppet infrastructure. Puppet is a tool for managing machine configuration automatically, which is exactly the kind of tool that can quietly reintroduce a problem while nobody is looking.

This is the pattern to remember. The first failure was a stale map. The second failure was a safety control that only some of the system respected. A pause that lives in one place, and is ignored by another, is not a pause. It is a hope.

The guard: never trust a map you have not checked

GitHub says it is now doing several things to prevent a repeat: clearing retired database servers from the service's records before any new machine is added, keeping new machines from being elected leader until their records have been checked, rejecting configuration that points to servers that do not exist and keeping the last known-good settings instead, giving responders a faster way to take a misbehaving machine out of service, and adding alerts so changes like this are spotted within seconds.

You can borrow the middle idea, the one about rejecting a bad map, in a few lines. The rule is simple. Before you apply a new routing config, check every destination against the live inventory, the list of servers that actually exist right now. If even one destination is missing, reject the whole config and keep the last known-good one. Do not apply the good half and skip the bad half. A half applied map is still a wrong map.

def apply_routing_config(new_config, live_servers, last_good_config):
    """Apply new_config only if every destination really exists.

    new_config: a list of server names the new map points writes to.
    live_servers: a set of server names that exist right now.
    last_good_config: the last config we know worked.
    """
    unknown = [server for server in new_config if server not in live_servers]

    if unknown:
        print(f"Refusing bad routing config. Unknown servers: {unknown}")
        return last_good_config

    return new_config

That is the whole guard. The rest is discipline around it. Check the map at the point where it is applied, not only at the point where it is written. A config file can be correct when it is saved and stale by the time it is used, which is exactly what happened at GitHub: the records were outdated when the new machine started.

A boundary test makes the rule concrete. Test three cases: a config with no unknown servers, which should pass; a config with one unknown server among many good ones, which should fail; and a config where every server is retired, which should also fail. The one-unknown case is the important one. It is tempting to let it through, because most of the map looks fine. That temptation is the bug.

live = {"db-a", "db-b", "db-c"}
last_good = ["db-a", "db-b"]

assert apply_routing_config(["db-a", "db-b"], live, last_good) == ["db-a", "db-b"]
assert apply_routing_config(["db-a", "db-gone"], live, last_good) == last_good
assert apply_routing_config(["db-gone", "db-also-gone"], live, last_good) == last_good

Two more checks belong next to this guard, and they come straight from GitHub's own list. First, do not let a new machine become leader until its records have been checked. A new member of a quorum should prove its map is fresh before it is allowed to lead. Second, make a pause a single, checked thing, not a note in one tool. If a maintenance task can run without asking whether it is paused, it will, on the worst day, run at the worst time.

The question

You probably have a map like this somewhere: a config, a service discovery list, a set of connection strings, a deploy script that names servers. When was it last checked against what actually exists?

Find the newest machine in your system, the one added most recently, and ask a blunt question about it. Does it hold a copy of anything that was written before something was retired? If you cannot answer that in a minute, that is the audit, and it is worth doing this week.

When did you last verify that every destination in your routing config still exists?