Every morning, the coffee brews, the phone charges and the lights come on. None of it feels like infrastructure. Each one sits at the end of a long chain of power, water, shipping and people, and we only notice the chain when it breaks.
When it does break, we go looking for a cause: a storm, a strike, a bad deploy. That instinct is natural, but stopping at the trigger leaves much of the failure unexplained.
I recently worked through the Foundations of Disruptive Systems module from WorldQuant University, and one idea stayed with me. A trigger starts the disruption. The system's design helps determine how far it spreads and how hard recovery becomes.
Developers already half-know this. A useful postmortem traces how a config change exposed a missing safeguard and how that allowed an outage to spread. This post takes that habit outside software, to power grids, supply chains and coffee farms. The lessons travel straight back to the systems we build.
The shape of it
```plain text
trigger (the storm, the fire, the bad config file)
- enabling conditions (tight coupling, no buffers, no redundancy)
- feedback loops (failures that feed further failures)
- missing safeguards (what could have stopped the cascade) = a cascade, and someone downstream who pays for it ```
A system is a set of connected parts working towards a purpose. A part can fail without much happening. Whether a failure stays local or spreads depends on the design around it: how tightly the parts are connected, whether buffers absorb the shock, and whether another path exists when one route fails.
1. Single points of failure are design outcomes
A single point of failure is a component whose breakdown can stop the whole system. The component isn't necessarily fragile. The system simply has no alternative to it.
In March 2021, the Ever Given ran aground in the Suez Canal and blocked it for six days. The blockage disrupted trade and delayed shipments. The grounding was the trigger. Dependence on one narrow route helped spread the disruption through supply chains.
This is familiar territory for developers. One region, one database primary, one DNS provider, one maintainer of a tiny package that half the internet installs. Each choice made sense when things were normal. Each one turns into the whole system's problem on a bad day.
2. Efficiency and resilience pull against each other
Just-in-time supply chains keep inventory low and schedule deliveries to match demand. The savings are real, but reducing reserve stock can leave less room to absorb a disruption.
Resilience comes from three design mechanisms:
| Mechanism | Physical world | Software |
|---|---|---|
| Buffers | Reserve stock of medicine or fuel | Queues, caches, rate-limit headroom |
| Redundancy | Three suppliers for a critical part | Replicas, multi-region, fallback providers |
| Slack | Empty surge beds in a hospital | Spare capacity, on-call time that isn't already booked |
Each one costs money: storage, duplicate contracts, capacity that sits idle most of the time. That makes the trade-off a decision about who carries the risk. Removing a buffer doesn't remove the risk. It moves it downstream, to customers, communities and workers who have less room to absorb it. COVID-19 made that visible when lean supply chains left hospitals short of protective equipment.
Resilience also tends to be cheaper than recovery. A 2019 study by the US National Institute of Building Sciences estimated that federal hazard-mitigation grants save about $6 for every $1 spent (Natural Hazard Mitigation Saves). The useful question isn't "does resilience cost money?" It does. The question is whether it costs less than the cascade it prevents.
3. Texas, 2021: how the cold became a wider crisis
In February 2021, extreme cold hit Texas, and millions of people lost power for days. The headline was simple: cold weather caused a blackout. The FERC and NERC review tells a more useful story.
Freezing gas wells and equipment cut fuel to power plants, so generation fell. The grid shed load to stay up, and some of those rolling outages cut power to gas facilities. That reduced the gas supply further, so generation fell further. That is a reinforcing feedback loop: each failure made the next one worse.
The cold exposed weaknesses in how the gas and power systems were prepared and connected:
- Much of the gas infrastructure wasn't weatherised for extreme cold.
- Disruptions to gas production and processing left some power plants short of fuel.
- Gas facilities needed electricity, while gas-fired power plants needed a steady fuel supply. That dependence made recovery harder.
These weaknesses gave the disruption several ways to spread. Addressing them could have reduced the scale and duration of the outages. The final report made 28 recommendations, including changes to winterisation standards and coordination between gas and electricity systems.
4. Map the cause as a chain, not a culprit
A headline like "Cold weather caused the Texas blackout" leaves out the conditions that made the blackout so severe. A causal chain map helps identify where people could have reduced the damage. It separates four elements:
| Element | What it is | In Texas |
|---|---|---|
| Trigger | The proximate shock. Visible, easy to blame, rarely enough on its own. | Extreme cold |
| Enabling conditions | The design choices that let the trigger spread. | No weatherisation, thin buffers, two-way coupling |
| Feedback loops | How the first failures amplify or dampen. | Less gas → less power → less gas |
| Missing safeguards | What could have interrupted the cascade. | Winterisation standards, backup fuel, protecting gas sites from load shedding |
The software version is easy to find. In July 2024, a faulty CrowdStrike content update crashed Windows machines around the world (root cause analysis). The bad file was the trigger. The enabling conditions were the ones worth reading about: a failure in content validation, an error the sensor could not safely handle, and a content update distributed broadly without an initial canary rollout to a small group.
A useful trick: rewrite any failure headline as a condition. Under what conditions does [trigger] cause [outcome]? Then ask which conditions people could have changed and which safeguards could have limited the damage.
5. Watch the margins before they run out
Lagging indicators tell you something failed: outages, incidents, losses. Leading indicators tell you the margins are shrinking: falling reserve margins, a growing maintenance backlog, near misses that luck absorbed.
Texas had leading indicators for years, including deferred winterisation and earlier cold snaps that came close. Systems near a nonlinear threshold look fine right up to the point where a small push produces an outsized result. The point of watching leading indicators is to notice you're near the edge before you find it.
If you run production systems, you already have useful signals. Rising saturation and queue depth can warn that capacity is running short. Error-budget burn measures how quickly you are using your allowance for unreliability. It records problems already happening and can warn that you are on course to miss a reliability target. Near misses also deserve attention before a safeguard fails.
6. Draw the stakeholder boundary wider
Cascades reach people who never signed the contract. A flood at one supplier's factory directly hits that supplier and its buyer. Indirectly, it reaches consumers who never knew the supplier existed, retailers relying on just-in-time restocking, and workers whose shifts disappear.
A power–interest matrix sorts stakeholders by how much influence they have and how much they have at stake. Two zones matter most:
- High stakes, low power: workers, nearby residents and small retailers. They feel the consequences and rarely get a say in the design.
- High stakes, high power: buyers, regulators and platform owners. They are the ones who can add buffers, redundancy and slack.
The first group often only becomes visible in a crisis. During COVID-19, warehouse staff, delivery drivers and carers moved from "background" to "critical" almost overnight. A system that designs out slack also designs out the people who would have absorbed the surge.
A simple way to start: draw one ring for the people directly affected, then a second ring for the people affected by their response. Ask "who depends on this actor?" and "who supplies this actor?" at each step.
7. Test accountability with leverage, foreseeability and capacity
Direct contracts are one way to draw an accountability boundary. They do not settle every question about who bears the consequences. Three questions help test that boundary:
- Leverage: can we shape upstream conditions through price, contract terms or volume?
- Foreseeability: is the risk documented, or has it happened before somewhere similar?
- Capacity: do we have the money, knowledge and access to act?
The course uses coffee as its example. Climate shocks have hit smallholder coffee farmers hard, while the brands buying the beans hold contracts with exporters, not farmers. When all three answers are "yes", "our contracts end at the exporter" isn't a neutral description of the system. It's a design choice about who carries the risk. Price premiums, long-term contracts and investment in drought-resistant varieties aren't charity. They protect the brand's own supply.
The same questions apply to software. If your platform makes a decision that breaks your users' users, ask the three questions before you decide whose problem it is.
Watching a cascade happen
Tables explain the parts. It helps to watch them interact, so here's a small generative sketch called Load Bearing. You can run it in your browser. About 1,100 points link to their nearest neighbours. Each carries one unit of load, and most have only a little spare capacity, while a few have a lot. One point near the centre fails. Each failed point hands its load to its neighbours, slightly amplified, and any neighbour pushed past capacity fails next. This is an illustrative model, not a forecast of a real grid or supply chain. Its results follow the load-sharing rules and capacity limits chosen for the sketch.
Three settings shape the result, and each one maps to an idea from this post:
- Coupling is how many neighbours each point links to. In this model, those links give a failure paths along which to spread. The effect of adding links depends on the load-sharing rules and available capacity.
- Slack is the average spare capacity. Less slack means a smaller push tips a point over.
- Feedback is how much the load grows each time it's passed on. Above 1, the loop reinforces itself.
The defaults sit just below a tipping point, so the outcome is nonlinear in the sense of section 5. On seed 12345, halving the coupling from 4 to 2 links cuts the failures from 508 to 22. Trimming slack from 0.6 to 0.4 takes them to 1,061 of 1,100. The trigger is identical every time. Only the architecture changes.
The sage rings are worth a second look. They mark points that took extra load and held: the buffers that worked, which never make the headline.
In the interactive version, try this:
- Watch the baseline play out. A readout counts failures round by round. Pause and Step let you follow the front one round at a time, and you can see it feed itself.
- Pick a "What if…" scenario. Each one keeps the same seed and the same shock, and changes one design choice: cut the buffers, add slack, tighten or loosen coupling, or amplify or damp feedback.
- Click anywhere on the artwork to move the shock. The same network fails very differently depending on where it's hit, which is the single-point-of-failure question from section 1.
- Try other seeds to see how often a system that looks fine is one push from the edge.
On seed 12345, the scenarios go from 16 failures (add slack) to 1,061 (cut the buffers), all from the same trigger.
Using this on your own systems
Next time something breaks, or before it does:
- Name the trigger, then trace what let the disruption spread.
- List the enabling conditions: coupling, missing buffers, missing redundancy, no slack.
- Look for feedback loops where a failure feeds itself, such as retries that pile onto an overloaded service.
- Name the missing safeguards and who could add them.
- Track leading indicators, not just incidents.
- Draw two rings of stakeholders, and check who's in the high-stakes, low-power corner.
- Run the three accountability questions before deciding something is "not our problem".
The next time you read "X caused Y", what was designed in a way that made Y possible, and who could have designed it differently?
Top comments (0)