Four domains and one wrong type: the 54 minutes Control D's paid DNS went dark

On a Sunday afternoon in October, paying customers of Control D lost the internet. Not some of it. All of it, on every device, in every location, for about 54 minutes.

Their phones showed no connection. Their laptops showed no connection. Even phones on mobile data, using a persistent Control D profile, looked dead. To most of the people affected, it felt like the internet itself had gone down.

It had not. The free service kept working the whole time. No query was answered wrongly, and no customer data was touched. What stopped was much smaller and much stranger: four domain names, saved with one wrong data type, in a configuration file that every paid DNS server read at the same moment.

This is a story about a type mismatch. It is also a story about the two design choices that turned a type mismatch into a global outage, and the three checks that would have stopped it.

First, what DNS actually does for you

Almost everything you do online starts with a DNS lookup. DNS is the directory of the internet. You type a name like example.com, and a DNS resolver turns that name into an address your computer can connect to. If the resolver stops answering, nothing loads. Your browser cannot even start the conversation.

Control D runs that resolver as a service. Paying customers point their devices at it, and it answers lookups while applying their filtering rules: block this category, allow that service. The filtering rules live in profiles, and profiles contain services. A service is a ready-made entry for something like a streaming platform, backed by a list of domains that Control D maintains. Each domain in a service can carry an optional override, such as "allow" or "block", that changes how that one domain is treated.

That override is the whole story.

The bug that waited 25 days

On 16 September 2026, Control D released an API update. It added validation for the optional per-domain override in services. As part of that update, the API began saving the override as an integer, a whole number, instead of as text.

The DNS software that reads those services expects the override as a string, as text. A number is not a string. To the reader, the saved value was unreadable.

But nothing broke on 16 September, because the new code only ran when a domain was saved with an explicit override. Ordinary additions to a service never touched it. Services were edited in the following weeks without incident. The bug sat in production for 25 days, waiting for someone to save a domain with an explicit override for the first time.

That first save happened at 16:40 UTC on Sunday 11 October. A member of the team made a routine change in the admin tools to resolve a customer request, adding four domains to one service with an explicit "allow" override. The API stored the four overrides as numbers and told every DNS server for paid plans to reload that service.

Two minutes later, at 16:42, no DNS queries from paying customers were being answered anywhere.

Why every server fell at once

There were two separate failures here, and it matters to keep them apart, because each one has its own fix.

The first failure was the type mismatch itself. The API wrote a number. The DNS software expected text. A writer and a reader disagreed about the format of a shared value, and nobody had a test that checked the writer's output against the reader's expectations. Control D said so plainly in its report: "Our tests did not check that data saved by our API stayed in the format our DNS software reads."

The second failure was what the DNS software did when it met the unreadable value. It stopped. It did not log the bad entry and move on to the rest of the service. It stopped the whole process. And because a service's domain list is shared configuration, delivered to every DNS server for paid plans at the same moment, all of them stopped at once.

The failover system made it worse in the way failovers do when the fault is everywhere at once. As locations went offline, a safeguard kicked in that moves traffic to healthy locations. But every location had the same problem, so there was nowhere healthy to send it.

Then the automatic restarts made it worse still. The DNS software restarted on its own, as designed. But it was also designed not to start if it could not read all of its configuration, so that it would never serve a half-loaded filtering policy. Every restart therefore failed on the same four stored numbers, over and over, until a person corrected them.

Think about that for a moment. The safeguard that was supposed to protect customers from a half-loaded policy is what kept the service down. The system chose to serve nothing rather than serve something incomplete, and the incomplete thing was four values out of thousands.

The timeline, in numbers

The report Control D published on 20 October gives the whole sequence:

• 16:40 UTC: the routine change is saved. Every DNS server for paid plans stops.
• 16:42: no paid DNS queries are answered anywhere.
• 16:46 to 16:47: the first automated alerts fire, but they report individual servers and locations failing. None of them says that DNS for paying customers is down.
• 16:51: failures across multiple locations are reported internally, and the on-call engineer is paged.
• 16:55: on-call acknowledges and starts triage.
• 17:05 to 17:16: the outage is acknowledged publicly on Reddit and Discord.
• 17:16: the cause is traced to the 16:40 change and the September API update.
• 17:30: the affected service is saved again with the four overrides removed, so no number is stored.
• 17:31 to 17:35: DNS starts returning, and is fully restored at every location by 17:35.

Fifty-four minutes from the save to full restoration. Thirty-six minutes from the first alert to the cause being found. And the fix itself took five minutes once the cause was known: remove four values, let the servers restart cleanly.

There is a second timeline hidden inside this one. The alerting gap is its own lesson. The system knew servers were failing within six minutes. But it reported the failures one server at a time, and nobody's alert said the one sentence that mattered: "DNS for paying customers is down." Control D now pages on-call when answered DNS queries for paying customers drop to zero in any region for more than two minutes. That single alert would have paged someone within two to four minutes on 11 October.

What this looks like in your own code

You probably do not run a DNS service. But you almost certainly have a writer and a reader that share a format, and a process that loads configuration at startup. Here is the pattern, in Go, with the two mistakes and their fixes side by side.

First, the reader that stops on one bad entry. This is the shape of the Control D failure:

func loadServices(path string) ([]Service, error) {
    data, err := os.ReadFile(path)
    if err != nil {
        return nil, err
    }
    var services []Service
    if err := json.Unmarshal(data, &services); err != nil {
        return nil, err // one bad entry kills the whole load
    }
    return services, nil
}

One unreadable entry, and the caller gets nothing. If this runs at startup, the process refuses to start. If it runs on a config reload, the process stops. The fix is to read entry by entry, set aside the ones you cannot read, and log them loudly:

func loadServices(path string) ([]Service, error) {
    data, err := os.ReadFile(path)
    if err != nil {
        return nil, err
    }
    var raw []json.RawMessage
    if err := json.Unmarshal(data, &raw); err != nil {
        return nil, err
    }
    services := make([]Service, 0, len(raw))
    for i, entry := range raw {
        var svc Service
        if err := json.Unmarshal(entry, &svc); err != nil {
            log.Printf("skipping unreadable service entry %d: %v", i, err)
            continue
        }
        services = append(services, svc)
    }
    return services, nil
}

The difference is a design decision, not a trick. The first version treats configuration as all or nothing. The second treats each entry as independent, so one bad entry costs you one service instead of the whole system. That is exactly the fix Control D shipped: an entry it cannot read is now set aside and logged, during reloads and at startup, instead of stopping the software.

Second, the writer. The bug was not only in the reader. The API wrote a number where the reader expected text, and nothing checked the writer's output. The cheapest test in the world catches this: save through the real code path, then read the stored value back with the real reader, and compare the format. In Python, the shape of that check is:

def test_override_round_trip():
    saved = api.save_service_override(domain="example.com", override="allow")
    stored = store.read_raw(saved["id"])
    assert isinstance(stored["override"], str), (
        f"override stored as {type(stored['override']).__name__}, "
        "reader expects str"
    )
    assert reader.load_service(saved["id"]).domains[0].override == "allow"

That test runs the writer and the reader against each other, which is the point. A unit test of the writer alone would have passed, because the writer did exactly what its new code told it to do. The failure lived in the gap between two components, so the test has to live in that gap too. Control D's own report says that tests of this shape, run against the code that was live on 11 October, reproduce the failure.

Third, the rollout. Control D rolls out software updates gradually, a few servers at a time. Configuration changes were not staged at all: the bad config reached every paid server at the same moment. The fix in progress is to roll out configuration changes gradually too, the way software updates already are. The same principle applies at any scale. If a change can break every instance at once, it should reach a small set first, with a check that it loaded cleanly, before it reaches the rest. A canary for config is worth as much as a canary for code, and it is usually easier to build.

The lesson that is easy to miss

It is tempting to file this one under "type error" and move on. That misses the interesting part. The type error was a small, ordinary bug, the kind any team ships. What made it an outage was everything around it: a shared config with no staging, a reader that stopped instead of skipping, a startup guard that turned a bad value into a restart loop, and alerts that described symptoms instead of the service-level fact.

Each of those is a choice you can make differently this week, in a system you already run:

• Validate what you write with the reader that will read it, not with a separate schema that can drift.
• Never let one bad entry kill a whole load. Skip it, log it loudly, and keep serving the rest.
• Stage configuration changes the way you stage code changes.
• Alert on the customer-facing fact, not on the individual server. "Queries answered dropped to zero" is an alert. "Server 14 is down" is a footnote.

The four domains were removed at 17:30, and by 17:35 everything was back. The fix took five minutes. Finding it took thirty-six. The bug had been live for twenty-five days. Every one of those numbers is a reminder that the expensive part of an incident is almost never the fix. It is the time between "something is wrong" and "we know what it is", and the distance between the change that caused it and the system that felt it.

What is the shared format in your system that a writer and a reader could disagree about, and is there a test that runs them against each other?