What worked, what nearly broke, and why our disaster recovery plan had a 47-minute blind spot
1. Tuesday Morning, 9:47 AM
It was a normal Tuesday. The kind of Tuesday that doesn't earn a mention in any retro.
Our developer portal was humming along at its usual baseline: roughly 2,000 concurrent users, following the traffic pattern we'd watched for two years running. A gradual ramp through the morning, a lunchtime dip, an afternoon plateau, a slow fade into evening. Nothing about the dashboards suggested this day would be different.
Then a product announcement we'd quietly shipped the week before got picked up by a tech influencer with a sizable following. The post took off. Within minutes, our traffic graph stopped looking like a graph and started looking like a wall.
Two thousand concurrent users became eighty thousand in eleven minutes. Request rate: forty times baseline, sustained.
Auto-scaling triggers are fired. Alarms lit up the on-call channel. That morning, the on-call engineer, let's refer to her as Maya, stared at her dashboard as it displayed a pattern entirely unfamiliar to her.
Here's the twist, though: the portal didn't crash. What nearly went down was everything around it.
This isn't a hero story, and it isn't really a victory lap, even though it might read like one at first. It's a postmortem wearing a victory lap's clothing. The spike didn't expose a single point of failure. It exposed five assumptions we didn't know we were making and a 47-minute window where our dashboards were green while a chunk of our users were stuck staring at a spinner.
2. The Portal: Context Before Chaos
Understanding the actual stakes involved is essential prior to examining the chronological sequence.
The portal in question is a customer-facing developer hub: documentation, API status pages, SDK downloads, and a support ticket submission flow. It's not a marketing site. It's infrastructure that roughly 12,000 paying customers rely on to do their jobs. When it's slow, support tickets pile up. When it's down, SLA penalties start accruing and churn risk goes up with every passing minute.
Going into that Tuesday, we felt good about our architecture. We had load balancers in front of everything. Auto-scaling groups configured and tested. A CDN fronting static assets. Database reads replicas to spread query load. On paper, this is a textbook setup. We had checked every box on the standard "are we prepared" checklist.
Six months earlier, we'd even run a formal capacity planning exercise. We tested the portal up to 10x our normal baseline traffic, watched it hold steady, declared the exercise a success, and filed the runbook away.
What we didn't fully appreciate at the time: load testing simulates traffic volume. It does not simulate traffic behavior. Those turned out to be very different things, and the gap between them is most of what this post is about.
3. The Spike: Minute-by-Minute
Here's how the morning actually unfolded, reconstructed from logs, dashboards, and Maya's own notes from the incident channel.
Reading that table top to bottom, the pattern is hard to miss: the application layer itself never really blinked. What buckled, one piece at a time, was the supporting infrastructure around it: the connection pooler, the rate limiter, and the logging pipeline. Each failure was independently survivable. Stacked together in a 24-minute window, they formed something closer to a slow-motion cascade than a single outage.
The core insight that shaped everything we did afterward: the portal application survived the spike. The systems we'd built to protect and observe it nearly didn't.
4. What Worked: Three Systems That Earned Their Keep
It would be dishonest to frame this purely as a string of failures. A few design decisions paid for themselves that morning, even if not perfectly.
CDN edge caching (partially)
Static assets documentation images, CSS, JavaScript bundles were served entirely from the edge and never touched our origin servers. That's exactly what a CDN is for, and it worked as designed.
Where it fell short: our API documentation pages render OpenAPI specs server-side, which makes them feel static to a user but behave dynamically under the hood. Every new visitor triggered a fresh server-side render. The cache hit ratio collapse from 94% to 61% wasn't a CDN failure; it was a content classification failure. We'd told ourselves these pages were "static" when they were really pseudo-dynamic content wearing a static costume.
The lesson: a caching strategy needs to distinguish genuinely static assets from content that merely looks static to a human but gets regenerated on every request.
Horizontal pod autoscaling
Our Kubernetes compute layer scaled from 8 pods to 47 in about six minutes. This is the part of the story that looks the most like a clean win, and largely it was. CPU-based scaling metrics, pre-warmed container images, and pods with no dependency on shared external state meant new capacity came online fast and cleanly.
The lesson here generalizes well beyond this incident: scaling speed matters more than scaling ceiling. A system capable of 100x capacity is worthless if it takes twenty minutes to materialize and your spike peaks in eleven.
Circuit breaker patterns
When the database connection pool saturated, the portal didn't fall over; it degraded gracefully. Non-critical features like usage analytics and the recommendation engine auto-disabled under load, while core functions authentication, documentation access, support ticket submission kept working.
The lesson: not every feature deserves equal protection. Designing explicitly for graceful degradation, rather than chasing uniform availability across every feature, gave us a portal that stayed useful even while parts of it were quietly switched off.
5. What Nearly Broke: Five Hidden Assumptions
The systems that worked are reassuring. The systems that nearly didn't are where the real lessons live.
*Assumption 1: “Increased queries won't overload a single database node.” *
The red replicas themselves scaled without complaint. The bottleneck turned out to be the connection pooler (PgBouncer) sitting in front of them. Every new pod that spun up opened new database connections. The pooler maxed out its connection limit, which meant new pods couldn't fully initialize, which meant traffic queued, which is what produced the first visible error spike.
The fix: connection pool sizing has to scale in lockstep with computers, not just with database capacity. A database that can handle the load is irrelevant if the gatekeeper in front of it can't.
Assumption 2: "Rate limiting protects us"
Our token bucket rate limiter was tuned for gradual traffic ramps, the kind of growth pattern we'd actually tested against. A 40x burst drained the bucket almost instantly, and the limiter started returning 429 errors to legitimate users. The system meant to protect us from abuse ended up throttling our own real traffic. In practice, our system ended up generating its own traffic overload, creating a failure pattern of its own making.
The fix: burst allowances and sliding-window algorithms designed explicitly for viral, spiky traffic patterns, not just sustained growth.
Assumption 3: "Logs are cheap"
Log volume rose 60x, not 40x, because users facing errors retried their requests, which generated even more log lines than the raw traffic increase alone would account for. The log aggregation pipeline choked under that volume, and we lost roughly 12 minutes of observability during the exact window we needed it most.
The fix: severity-based log sampling and gating, plus separating high-volume metrics pipelines from lower-volume, higher-fidelity logging pipelines so one can't take down the other.
Assumption 4: "Our runbook covers this"
Our existing runbook said, in effect: scale the database if CPU exceeds 80%. During the incident, database CPU sat at a comfortable 45%. The connection pooler, meanwhile, was pegged at 100%. We were watching the wrong gauge. The metric we'd built our response around wasn't the metric that actually mattered in this failure mode.
The fix: runbooks need to map symptoms to root causes, not just static thresholds to static actions. A runbook built around predefined thresholds remains effective only when the same system component continues to be the primary constraint.
Assumption 5: Peak-traffic simulations guarantee production readiness.
Our load testing simulated a gradual ramp to 10x baseline over 30 minutes. Reality delivered an instant 40x shock in under 11. Systems under gradual stress and systems under sudden shock are not the same systems, even when the underlying code is identical the failure modes that show up are fundamentally different.
The fix: capacity testing needs to simulate burst patterns, not just raw scale. Plan for viral moments, not only predictable peak events like Black Friday.
6. The 47-Minute Blind Spot
Here's the part of the story that took the longest to fully understand, and arguably mattered the most.
Our metrics were available throughout the incident. Our dashboards were green for most of it. Our alerts were firing on the things we'd configured them to fire on. By every internal signal we had, the system looked like it was holding up reasonably well.
What we couldn't see: a user in Sydney was sitting through 8-second page loads. Our own monitoring showed a 95th-percentile latency of 400 milliseconds because that monitoring originated from our us-east-1 region. The CDN edge node serving Sydney traffic was genuinely overloaded, but nothing in our origin-based metrics reflected that, because we were never measuring from the edge in the first place.
The realization that came out of this, stated as plainly as we could put it: we were measuring system health, not user experience. Those are not the same thing, and conflating them cost us 47 minutes of blindness to real degradation.
We completely redesigned our observability strategy. The first step was implementing real user monitoring across all customer regions, rather than limiting visibility to the locations of our servers. Second, we deployed synthetic checks from São Paulo, Mumbai, Sydney, and Frankfurt, not just our home region in Virginia. Third, we rewrote our service-level indicators to measure what humans actually felt: the gap between clicking a link and interacting with the page, not merely how fast our origin server responded. Server response time had become a vanity metric. User-perceived latency was the truth we had been avoiding.
7. The Aftermath: What Changed in Our Capacity Planning
The shift in how we think about capacity planning is easiest to show as a before-and-after.
Concretely, our new capacity planning practice now includes quarterly burst tests that simulate 50x traffic arriving in under five minutes, full dependency mapping so every downstream service has a documented burst tolerance, explicit cost modeling for what a 40x traffic event actually costs to absorb, and a clearer business conversation about the difference between a portal that survives a spike and one that thrives through it. Surviving means core functions hold. Thriving means everything keeps working. They cost different amounts of money, and someone with budget authority should get to choose which one you're building toward.
8. A Diagnostic Framework for Your Own Portal
If you're reading this wondering how your own systems would have fared, these are the five questions we now ask ourselves on a recurring basis and that we'd suggest any team running customer-facing infrastructure ask too.
What does "40x traffic" actually look like for your system specifically? Have you identified the most likely source of a traffic surge such as widespread social sharing, a major product announcement, or users migrating from a competing platform during an outage? The shape of the spike matters as much as its size.
What's your first failure point, not your last? Most teams can name the component that fails under extreme load. Far fewer can name the one that fails first, and that's the one that actually defines your real ceiling.
Are you testing for burst, or just for scale? A clean 10x gradual ramp tells you very little about how your system behaves under a 40x instant shock; they are different tests measuring different failure modes.
Are you monitoring user experience, or just system health? Green dashboards are reassuring, but they don't tell you whether a real person in another region is staring at a spinner.
What does "surviving" cost compared to "thriving," and is that cost actually budgeted? Forty times normal capacity isn't free, and the decision about how much to spend on it belongs to the business, not just to whoever's holding the pager.
9. The Humility of Surviving
There's a particular kind of pride that comes from watching your system absorb a 40x spike without going down. We felt it that morning. It's a real, earned feeling, and we don't want to undersell it.
But surviving wasn't actually the goal. Understanding why we survived and how close several of our supporting systems came to not surviving is what made the day useful rather than just lucky.
The traffic spike was, in a strange way, a gift. It exposed our blind spots without producing a customer-facing outage that would have shown up in a churn report. The next spike might not be so forgiving, and there's no guarantee the next failure mode looks anything like this one.
So the challenge we'd leave with anyone reading this: don't wait for your own 40x moment to find out where your assumptions live. Audit your capacity planning this week. Ask the five questions above. Find your first failure point before a tech influencer finds it for you.
The best capacity planning doesn't actually prevent failure. It ensures you learn from a controlled failure before an uncontrolled one teaches you the same lesson at a much higher cost.
Further Reading
For teams looking to go deeper on the ideas touched on here, a few starting points worth exploring: the capacity planning chapter of Google's SRE book, the Reliability Pillar within AWS's Well-Architected Framework, Cloudflare's public writeups on absorbing massive DDoS traffic at the edge, and Netflix's body of work on chaos engineering as a discipline.
And one more, less glamorous resource worth checking: your own runbook. Open it up. Look at the timestamp on the last edit. If it predates your last major incident, it's probably already out of date.
Top comments (0)