How a DNS race condition took DynamoDB offline in us-east-1

Sunday night, 11:48 PM Pacific. One of the largest databases on Earth is running perfectly. Every server is up. Every disk is fine. And nobody can reach it, because its name no longer has an address.

This happened in October 2025. DynamoDB in us-east-1 did not crash, did not slow down, did not lose data. It vanished from DNS. Every system that tried to connect, customer apps and AWS's own internal services alike, got the same answer: the name dynamodb.us-east-1.amazonaws.com resolves to nothing.

AWS later published a full summary of the event, and it is one of the most instructive postmortems I have ever read. Not because the bug was exotic, but because it shows how the cloud actually holds together: layers of automation, each depending on the layer below, until one quiet race condition pulls the thread and half the region unravels.

Why a database needs hundreds of thousands of DNS records

First, the part that surprised me. DynamoDB maintains hundreds of thousands of DNS records. Not as a directory for humans. As a scaling primitive.

DynamoDB runs a very large, heterogeneous fleet of load balancers in every region. When capacity is added, when hardware fails, when traffic needs to shift, the way the service steers clients is DNS. Automation constantly rewrites those records: add the new load balancers, drop the dead ones, rebalance the weights. DNS is not the phone book here. It is the steering wheel.

That automation is split into two independent components, deliberately, for availability. The first is the DNS Planner. It watches the health and capacity of the load balancers and periodically produces a new DNS plan for each endpoint: a set of load balancers plus weights. One regional plan is shared across endpoint variants, the public regional endpoint, a FIPS endpoint, an IPv6 endpoint, account-specific endpoints, which keeps capacity management simple.

The second component is the DNS Enactor. It takes plans and applies them as changes in Route 53. It is designed to have minimal dependencies so it can recover the system in any scenario, and it runs redundantly and fully independently in three availability zones. Each Enactor picks up new plans and applies them using Route 53 transactions, so an endpoint always gets a consistent plan even when several Enactors are working at once.

Before an Enactor starts applying a plan, it does one check: is my plan newer than the one already applied? One check, at the start. Remember that detail. It is the whole story.

The race

Right before the outage, one Enactor hit unusually high delays. It kept getting blocked trying to apply its plan to several endpoints and had to retry, slowly working its way through the list.

While it was stuck, two things happened. The Planner kept running and produced many newer generations of plans. Then a second Enactor picked up one of the newer plans and applied it rapidly across all endpoints.

When the second Enactor finished, it ran the plan clean-up process. Clean-up deletes plans that are significantly older than the one just applied. Housekeeping. Reasonable.

But at that exact moment, the first, delayed Enactor finally applied its much older plan to the regional DynamoDB endpoint, overwriting the newer plan. The freshness check it had done at the start was stale by now, after all those delays, so nothing stopped it.

Then the second Enactor's clean-up ran and deleted that older plan, because it was many generations behind the plan it had just applied. Deleting the plan removed every IP address for the regional endpoint. And because the plan that was deleted was the active one, the system was left in an inconsistent state where no subsequent plan update could be applied by any Enactor. The automation that was supposed to repair itself could not. A human had to step in.

From the outside, this is what it looked like:

import socket
# What every client, human or machine, effectively saw that night:
socket.getaddrinfo("dynamodb.us-east-1.amazonaws.com", 443)
# socket.gaierror: [Errno -2] Name or service not known

Not a timeout. Not a 500. The name did not resolve. The database was right there, healthy, unreachable.

Then AWS forgot how to launch servers

Here is where it gets properly strange. The DynamoDB DNS failure was fixed by 2:25 AM. But EC2 could not launch new instances until 1:50 PM, more than fourteen hours after the first failure. Why? Because inside AWS, EC2's own machinery depends on DynamoDB.

Deep inside EC2 there is a subsystem called the DropletWorkflow Manager. It manages the physical servers, which AWS calls droplets, that host EC2 instances. Each manager holds a lease on the droplets it manages, renewed by state checks every few minutes. Those state checks depend on DynamoDB.

At 11:48 PM, the checks started failing. Running instances were fine, but over the next two and a half hours, leases across the fleet quietly timed out. And a droplet without an active lease is not considered a candidate for new launches. So EC2 started refusing new instance launches with "insufficient capacity" errors. There was plenty of capacity. The system just could not prove to itself that any server was allowed to take work.

When DynamoDB recovered at 2:25 AM, the lease manager tried to re-establish leases across the entire fleet at once. Too much work arrived too fast, the queues choked, and the work kept timing out before it could complete. The system had entered what AWS called congestive collapse: too busy failing to recover to actually recover. Engineers throttled incoming work and selectively restarted the lease manager hosts at 4:14 AM. By 5:28 AM, every droplet had a lease again.

But new instances still had no network. A second subsystem, Network Manager, propagates network configuration to new instances, and it was sitting on a huge backlog of delayed state changes. Propagation times did not return to normal until 10:36 AM. Full EC2 recovery landed at 1:50 PM.

The blast radius kept widening from there. Lambda function creation and updates broke, and an internal SQS polling subsystem failed and had to be restored by hand at 4:40 AM. ECS, EKS, and Fargate could not launch containers or scale clusters until 2:20 PM. Amazon Connect callers heard busy tones and dead air. IAM user sign-ins to the console failed. Redshift query processing, which reads and writes through DynamoDB endpoints, broke, and because Redshift resolves IAM user groups through an IAM API pinned to us-east-1, Redshift customers in every region lost the ability to run queries with IAM credentials for over an hour.

My favorite detail: the AWS Support Center failed over to another region exactly as designed, and still blocked legitimate users. The failover bypass was built to kick in when the account metadata subsystem returned unsuccessful responses. That night it returned invalid responses instead. The designers had predicted one failure mode. Reality chose the adjacent one.

The shape of the fix

AWS disabled the DNS Planner and Enactor automation worldwide and said it would fix the race condition and add protections before turning it back on. The protections are the interesting part for the rest of us, because the pattern applies to any system that applies versioned plans. In sketch form:

def apply_dns_plan(plan, state, route53):
    # The freshness check must be re-done at apply time,
    # not once at the start of a slow walk over endpoints.
    if plan.generation <= state.active_generation:
        return "skip: stale plan"

    route53.replace_record(plan.endpoint, plan.ips)
    state.active_generation = plan.generation


def cleanup_old_plans(plans, state):
    for plan in plans:
        # Never delete the plan that is currently serving traffic,
        # no matter how old it looks.
        if plan.generation == state.active_generation:
            continue
        if plan.generation < state.active_generation - KEEP_GENERATIONS:
            delete(plan)

Two guards. Never apply a plan older than what is live, checked at the moment of applying. Never delete the plan that is live. Either guard alone would have prevented the outage. The original system had a version of the first guard, but it checked once at the beginning of a long, retried operation, and time made liars of it.

What I take from this

Three things stuck with me.

First, your dependencies have dependencies. Nobody who launched an EC2 instance that night thought about DynamoDB DNS records. Nobody running Redshift in another region thought their queries depended on an IAM API in Virginia. The dependency graph of a real system is always deeper than the diagram on the wall.

Second, automation that cannot repair itself is automation with a single point of failure wearing a costume. The DNS system was designed for resilience against many operational issues, and it met an issue that left it in a state its own recovery logic could not process. If your self-healing has states it cannot handle, you do not have self-healing. You have a pager with extra steps.

Third, error messages lie during cascades. "Insufficient capacity" meant "we lost our leases." Health checks failed on healthy machines because network state had not caught up. When everything is failing at once, read the errors as symptoms, not diagnoses.

AWS's summary is public and worth reading in full if you run anything on the cloud. It is a rare, honest look at the machinery underneath the dashboard.

What is the deepest hidden dependency you have found in your own stack, the one that would surprise everyone on your team?

---

Sources: AWS "Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region" (aws.amazon.com/message/101925/); press coverage at the time listed Venmo, Snapchat, Fortnite, and Amazon's own Alexa among affected services.