What AWS, Cloudflare and GitHub can teach you about failures that clean architecture diagrams leave out.
You can draw a second database in 3 seconds. Explaining what happens when both copies contain writes the other one has never seen takes a little longer.
That gap is where system design starts getting interesting for me. The diagram looks reassuring, everyone agrees that redundancy is sensible, and then someone asks how the system actually comes back. Suddenly, the arrows need a much better explanation.
Before my next system design interview, I want some practice with that uncomfortable middle. Public incident reports are lovely material for it because they show the assumptions that disappeared under real conditions, along with the decisions engineers made afterward.
I picked three reports from AWS, Cloudflare and GitHub. These are historical incidents, and their dates matter. We’re reading them for the reasoning they make visible, rather than treating them as breaking news or claiming these companies asked an interview question.
Bring one notebook page. You do not need to memorize every service name to get something useful from this.
Read for the missing assumption
My preferred starting point is a five-line note. As you read, record what changed, what assumption stopped holding, how the failure spread, what complicated recovery, and what evidence could show users were okay again.
That last line is easy to skip because a report usually tells us how the incident ended. In an interview, though, we have to explain what we would observe before declaring our own system healthy.
Keep reported facts separate from your proposed safeguards. If you think a version check or a rollout gate would help, write it as a design idea and explain its limits. Knowing the ending of an incident does not make the response obvious to the people living through it.
1. AWS shows why an old operation can become dangerous later
During the October 19–20, 2025 DynamoDB incident in Northern Virginia, a delayed DNS updater applied an older plan after another updater had installed a newer one. Cleanup then deleted the older plan, which was now active. The regional endpoint lost its IP addresses, and the resulting inconsistent state needed manual intervention.
The freshness check had happened before the delay, so it no longer protected the eventual update. Follow that sequence in AWS’s post-event summary.
My takeaway is to examine the gap between checking permission to change state and actually changing it. Redundant workers still need a safe way to disagree about which operation is current.
Try a smaller version in your notebook
Imagine a fictional settings service with two workers. One prepares revision 12 and pauses. Another commits revision 13. When the first resumes, where does your design reject its outdated update?
Put that decision at the storage boundary, using an atomic conditional write against the expected revision. A separate read followed by an unconditional write leaves another gap for a competing worker. Then consider cleanup separately, because a version can be old without being safe to delete if something still uses it.
You now have a concrete concurrency discussion to practice, including what a rejected worker should reread before retrying.
2. Cloudflare makes an ordinary internal file worth questioning
On November 18, 2025, a database permissions change caused duplicate entries in a Bot Management feature file. The enlarged file spread across Cloudflare’s network and exceeded a limit in the software consuming it, causing failures. Recovery included stopping the bad file’s generation and distribution, inserting a known good file and restarting the core proxy.
Start with the cause and recovery sections of Cloudflare’s incident report. This was not a cyberattack.
I take away a useful boundary question. What prevents internally generated data from carrying the same mistake to every instance?
Give your imaginary service a bad configuration
For a separate practice scenario, imagine a recommendation API receiving a generated rules file. Write down what the consumer validates before activation, including schema, maximum size and supported version. Decide what happens when validation fails.
Keeping the last accepted version may preserve service, but stale rules can have consequences too. State which rules may become stale and which require the service to stop or degrade. Then choose a limited rollout group and a user-facing error signal that would halt expansion.
That is a much more useful answer than adding a box labelled validation and hoping the diagram takes care of the details.
3. GitHub separates a repaired connection from a recovered service
GitHub’s October 21, 2018 network interruption lasted 43 seconds. It triggered failover, while databases in different locations accumulated writes missing from the other side. Switching back safely became difficult, and applications also struggled with the changed latency. Service degradation lasted 24 hours and 11 minutes.
Read the timeline in GitHub’s post-incident analysis, especially the decisions made to preserve data. The report describes degraded service, not a complete shutdown of every GitHub feature.
My takeaway is that a working network link does not settle which data is authoritative or make the application ready to use it.
Make recovery prove something
In a fictional two-region order service, suppose a customer received confirmation just before connectivity failed. Which durable record lets you prove that order still exists after failover?
Choose whether your design prioritizes accepting new orders or pausing writes when authority is uncertain. Explain the customer consequence either way. Then name what prevents the old writer from continuing after a new writer takes over.
A leadership decision needs enforcement where writes happen. Merely telling a disconnected process that it has lost leadership is a fairly optimistic recovery plan.
Turn your notes into a decision you can defend
This is where I’d stop reading for a moment. Collecting three more links is tempting, but the useful output is one decision you can explain under pressure.
Use the worksheet as a reading aid, then build a small scenario of your own. Keep it narrow enough that you can trace an individual request instead of narrating the whole internet.
For example, imagine a fictional receipt service whose email provider is unavailable. Your first response might be to retry failed sends, which is a reasonable beginning. Now make that answer specific.
Set a retry limit and explain what happens after it is reached, so unsuccessful work has an owner and a next step.
Consider a timeout after the provider has accepted the email. Explain whether a repeated request can create a duplicate and what evidence could resolve the uncertainty.
Limit recovery traffic so the accumulated backlog does not overwhelm the provider when it returns.
Choose a recovery check that includes the age of the oldest pending receipt, because successful new requests can hide unfinished older work.
Those choices form a connected explanation. You have described the failure, the uncertainty it creates, the response, and the evidence you need afterward.
Notice how a generic plan to add retries becomes a discussion about ownership, duplicate effects and backlog recovery. You can get there without claiming you have operated a service at any particular company’s scale.
Give your safeguard an awkward follow-up
Every safeguard has a dependency or a cost, so choose one and press on it. If recovery needs an administrator to flip a switch, can that administrator still sign in during the incident? If your fallback serves older data, how old can it get before the response becomes unacceptable?
I prefer writing down the condition that would change my decision. For example, a stale product description and a stale account permission deserve very different answers. Naming that difference shows what your fallback actually guarantees.
You are allowed to leave an open question in the notebook. Mark what you need to learn next instead of filling the gap with a reassuring technology name.
A 20 min practice session you can actually finish
After an initial read of one report, try this routine with the report available for reference. It is a practice suggestion, not a validated assessment or a claim that twenty minutes covers distributed systems.
Spend the first five minutes filling the worksheet with short notes. Mark anything you inferred rather than found in the report.
Use the next five minutes to redraw the failure path with only the components needed to explain it. Include the state that changes, not just the machines.
Spend five minutes changing one assumption in your own scenario. A delayed worker resumes, a dependency stays unavailable, or a successful response gets lost. Choose a response and name its cost.
Use the final five minutes to explain your decision aloud and write a concrete recovery check. If you get stuck, record the exact uncertainty for your next session.
Later, pick a fresh practice question, choose one of its dependencies and apply the same method. The goal is to carry the reasoning into an unfamiliar design rather than reproduce the incident’s architecture.
Save the completed page with your notes. Next time you revisit it, cover your answer and see whether you can still explain why that particular safeguard belongs there.
Bring the reasoning back to your diagram
I still want a clear architecture diagram. I also want every important arrow to survive a conversation about delays, conflicting state and recovery.
That is why these reports belong in my preparation pile. They give us something concrete to think with, and the worksheet turns that reading into a habit we can repeat.
Before opening another resource, choose one box in your own design and explain how you would know its users had recovered. If the answer gets uncomfortable, you have found a useful place to practice.
Top comments (0)