On November 18, 2025, at 11:20 UTC, a small automated update caused a huge problem. A configuration file became larger than the software could handle, and that tiny mistake ended up taking down around 20% of the internet for more than three hours.
Major services like X(Twitter), ChatGPT, Spotify, League of Legends, and even banking systems went offline. The estimated damage was $5 to $15 billion per hour.
It wasn’t caused by hackers. It wasn’t caused by a hardware failure. It was caused by a simple design mistake in how one system handled a large file.
What Happened: A Quick Timeline
A bot-mitigation system (used to block bad traffic) updated its configuration file. This time the file got too big. When the software tried to load it, it crashed.
Because many other systems depended on it, they crashed too.
The delay in realizing the global scale of the problem highlights a major challenge in internet-scale operations. Here’s the breakdown:
- 11:20 UTC: The bad config file is loaded. The system crashes.
- 11:48 UTC: The company publicly confirms the issue.
- 13:09 UTC: Engineers finally find the root cause.
- 14:42 UTC: A fix is deployed. Outage ends after 3 hours and 22 minutes.
Since all 330+ data centers were running the exact same software, they all failed in exactly the same way. A global outage happened instantly.
Why It Failed: Same Design, Same Problem Everywhere
This company uses a global network where every data center runs the same software and configuration. Normally, this makes operations smooth and simple. But when there’s a bug, that same sameness becomes a huge weakness.
How Their Network Usually Works
In a traditional setup, a user's request from Tokyo would travel to a server in New York and back. Cloudflare use a system called Anycast. Instead of sending your request (for example, from Tokyo) all the way to a server in New York, Anycast sends it to the closest data center automatically.
This is great for speed, and the network can usually route around failures.
The Real Issue: A Chain Reaction
Every user request goes through a series of steps:
- It reaches the nearest router.
- DDoS protection filters the traffic.
- The bot mitigation system checks if the user is a bot.
- A load balancer chooses a server.
- Firewall and security checks run.
- The request goes to the website’s server.
Because the bot system is used early in the process, many other systems depend on it. When it crashed, everything that relied on it crashed too. This is called tight coupling, meaning one failure spreads to many systems.
The Biggest Weakness: Everything Was Identical
Every data center was running:
- the same version of the software
- the same configuration
- the same bad file
So every location broke at the same time. There was no backup version or different setup to save the situation.
What We Learned: Build Systems That Fail Safely
The oversized config file only caused a crash under real production load, which shows that not every problem can be caught in testing.
Here are the key lessons:
1. Validate Configurations Properly
Config files should be checked for:
- size limits
- correct format
- safe defaults
And alerts should fire when something gets too big or unusual.
2. Remove Tight Coupling Between Services
If one service fails, others should keep working with:
- cached data
- default settings
- reduced functionality
Running with limited protection is still better than having everything go offline.
3. Understand Which Systems Are “Too Important to Fail”
Some systems support many others. These must be:
- reviewed more carefully
- tested more often
- protected with extra safeguards
4. Improve Detection and Response Time
The outage lasted long because it took time to see what was wrong. Systems need:
- better monitoring
- faster alerts
- clear dashboards
- strong escalation paths
The quicker teams understand the failure, the quicker they can fix it.
Conclusion🎉
One oversized file should not be able to take down a large part of the internet—but it did. This incident showed that even world-class infrastructure can break from basic design oversights.
In large systems, it’s not enough to plan for success. You must plan for failure, limit how far it spreads, and make recovery fast.
Even small mistakes can cause big problems at global scale.
Subscribe to My Newsletter:
If you're ready to subscribe, simply click the link below:
Stay updated with my latest and most interesting articles by following me.
If this guide has been helpful to you and your team please share it with others!
Top comments (0)