Four instances of the core service. Two physically separate data centers. And a single operating-system defect degraded the whole thing across an entire state anyway.
That's the one-line version of Pennsylvania's statewide Next-Generation 911 (NG911) disruption last July, and if it doesn't make you a little uneasy about your own "highly available" setup, it should. This is the failure mode that redundancy diagrams are quietly built to ignore, and it has a name: common-mode failure.
No attacker. State officials traced it to "a defect in an operating system" and said it was "not believed to be" a cyberattack. Just a bug -- in a component the redundant copies apparently shared.
Here's what happened, why four copies didn't save it, and then the part you can actually use: how to check whether your own redundancy is independent or just replicated.
The setup
When someone dials 911 on a modern system, the call doesn't go straight to the local center. It hits Next Generation Core Services (NGCS) -- the components that ingest the call plus whatever location data the originating network provides, resolve the location, and route to the correct PSAP (Public Safety Answering Point, i.e. the 911 center). Pennsylvania runs four NGCS instances across two data centers, operated by a single vendor under a statewide contract.
On July 11, around 2:00 p.m., calls started failing intermittently: dropped connections, calls arriving with no ANI/ALI (callback number / location), long quiet gaps then bursts. Roughly 85 minutes in, the state pushed a Wireless Emergency Alert telling people to use county non-emergency lines. A majority of calls kept getting through, and no statewide failure count was ever published. Restored around midnight.
Eleven days later: "a defect in an operating system," not a cyberattack. No component named, no trigger confirmed. (The trigger is genuinely muddy, worth flagging: on day one officials said it wasn't a pushed update; later reporting attributed to the same agency said a routine vendor update triggered the defect. The public record never reconciled the two.)
Why four copies didn't help
The mental model: redundancy gives you availability only when the redundant units fail independently. Four instances behind a load balancer protect you against one box dying, one rack losing power, one data center flooding -- uncorrelated, mostly physical failures.
They do almost nothing against a fault in something all four share: the same OS build, the same config, the same dependency, the same change pushed to all of them in the same window. When the failure is in the shared layer, "four instances" just means the flaw is present in four places. That's a common-mode failure -- correlated failure across supposedly independent units because they have a common cause.
NIST SP 800-160 says it plainly: redundancy is susceptible to common-mode failure, and the mitigation it points to is diversity -- making the copies genuinely different -- not simply more identical copies.
A second data center survives losing a place. It does not, by itself, survive losing a piece of logic that every place is running.
Careful about what's proven: the public record doesn't establish an identical defect on every node. A shared config, a shared update path, or a synchronization/backpressure mechanism could produce the same correlated result. Doesn't change the lesson. If the failure domains aren't independent, the instance count is close to irrelevant.
The part you can use: is your redundancy actually independent?
Stop counting instances and start looking for shared fate. A starting triage for your own stack:
# 1) Version parity -- if every "redundant" node runs the identical build,
# a bad build is a common-mode event, not an isolated one.
for host in core-a core-b core-c core-d; do
echo -n "$host: "; ssh "$host" 'cat /etc/app/VERSION'
done
# All four identical? Then a bad release has a blast radius of "everything."
# 2) Shared config origin -- one bad template/secret/flag hits all of them.
for host in core-a core-b core-c core-d; do
echo -n "$host: "; ssh "$host" 'sha256sum /etc/app/config.yaml'
done
# Same hash everywhere = same fate when that config is wrong.
The parts a diff won't answer, and that matter more:
Change-pipeline questions:
- Does one deploy push the same artifact to every instance at once,
or is the rollout staggered / canaried / ringed?
- Is rollback per-instance, or all-or-nothing?
- Does a config or feature-flag change apply everywhere in one shot?
- Do all instances share one upstream (same DB, DNS, auth service,
control plane)?
If the honest answers are "same build, same config source, one-shot deploy, shared control plane," you don't have four independent systems. You have one system with a replication factor of four: great for load, close to worthless for the failure that matters here. The fix isn't more copies. It's independence -- staggered rollouts, canaries, config diversity, an instance or a data center deliberately kept a version behind, so a bad change can't take all of them in the same instant.
The operational and contract translation
A few things fall out of this that aren't code.
Runbooks and escalation. The failure was intermittent, which is the nasty kind, because you can't see the requests you're not getting. On the 911 side that's calls; on yours it's whatever your users silently can't complete. Intermittent upstream degradation hides in normal variance until someone connects the dots, so the win is a fast, no-blame path for whoever's on point to say "something upstream is wrong, escalating" before they're certain.
Ownership boundary. Pennsylvania's 911 plan draws the vendor's security responsibility at the demarcation equipment at each center; past that line it's a mix of local ownership. Know where your vendor's responsibility ends and yours begins before an incident, because "whose problem is this" is the most expensive question to answer live.
Notification SLAs. If you run or buy anything life-safety-adjacent, know the floor. As of April 15, 2025, FCC rules require covered 911 and originating providers to notify affected PSAPs as soon as possible and no later than 30 minutes after discovering a 911-affecting outage, with material updates as they develop (originating providers owe a first follow-up within two hours). APCO argued for fifteen. Your contract can demand more than the regulatory minimum -- put the disclosure timeline in it.
The transparency gap
Months on, the only public technical explanation is still "a defect in an operating system." A fuller root-cause analysis exists -- the vendor presented a summary to the state's 911 Advisory Board in September 2025 -- but the substance isn't public, even though the same vendor runs statewide 911 cores in eight other states with a direct interest in knowing whether they're exposed. For a life-safety system, "we found a bug and fixed it" is a thin postmortem, and the opacity is the thing that keeps everyone else from learning.
Takeaways
- Redundancy buys availability only across independent failure domains. Shared build / config / pipeline / dependency = shared fate, regardless of instance count.
- Audit your blast radius: version parity, config origin, rollout strategy, shared upstreams. If one bad change can hit everything at once, that's your common-mode exposure.
- Diversity (staggered versions, canaries, ringed config) is the mitigation NIST actually points to, not more identical copies.
- Intermittent degradation is operationally worse than a clean outage. Build the "escalate before you're sure" path.
- Put incident-disclosure timelines in the contract. The regulatory floor is a floor.
You don't need an attacker to lose a critical system. You need one flaw in a shared component and a redundancy model that assumed an independence it never had.
How do you keep your "redundant" instances from sharing fate -- deliberate version skew, ringed deploys, something else? Curious what's actually working in production.
Top comments (1)
The triage I'd add after version parity and config hash: run the same audit on whatever decides which copy to believe. Four diverse instances still sit behind one load balancer, one health-check rule, one config saying the canary gets 5%. Diversify the workers all you want and the adjudicator is still a single build with a single bug surface. Diversity doesn't remove common-mode, it moves it up a layer to whatever votes on the diverse outputs.
Diversity also buys less independence than the instance count implies. Knight and Leveson tested exactly this in '86 with N-version programming: independently written versions still failed together, because hard inputs are hard for everyone regardless of who wrote the code. Different teams, different languages, correlated failures on the same edge cases. Version skew saves you from one bad release taking all four at once. It does almost nothing when the trigger is a malformed input each implementation mishandles in its own way.
The version that's bitten hardest in my experience isn't infrastructure at all. Four genuinely independent instances that all apply the same acceptance policy, same threshold, same "reject if X", collapse to one instance for any input crafted against that rule. Spread them across two clouds and three OS builds and a single request still trips all of them identically, because the thing all four shared was the decision. The substrate underneath was diverse and it didn't matter. Worth folding into the blast-radius audit: which judgments are identical across the fleet, tracked separately from which binaries are.