On August 17, GitHub went down for 7 hours and 47 minutes. Web and API error rates hit ~20%. Archive and raw content downloads hit ~50%. Issues, PRs, Actions, Copilot, SAML/OIDC — all degraded or dead.
The root cause, on paper, sounds small: one sidecar pod ran out of headroom. That's it. That's the whole initial failure.
What turned it into an 8-hour global incident is the part every engineer needs to sit with, because you have this exact bug in your system right now and you don't know it.
The 30-second failure that mattered
One Istio sidecar pod in GitHub's Central US datacenter hit its concurrency ceiling and failed to autoscale.
Why didn't it scale? Because the autoscaler was watching the host service's metrics — CPU, memory, request count on the main container. It had no idea the sidecar sitting next to it, proxying every single request, had its own independent concurrency limit that it was about to slam into.
This is the classic sidecar blind spot: you scale the thing you're measuring, not the thing that's actually saturated. If you're running Istio, Envoy, Linkerd, anything with a proxy sidecar, and your autoscaling policy only looks at app-container metrics — stop reading this and go check right now. This isn't a GitHub-specific bug. It's a shape of bug.
How a saturated pod becomes a global outage
Here's the cascade, and it's worth tracing step by step because each link is individually reasonable:
- The sidecar saturates and starts shedding connections onto the HAProxy layer behind it.
- Four HAProxy nodes absorb that overflow and blow through their own flow limits, one after another.
- Those HAProxy nodes front the gateway auth path. Auth latency climbs.
- Every client watching that auth path sees slow responses, decides they're failures, and retries.
And here's the sentence that should be tattooed on every backend engineer's forearm: the auth path now gets more traffic than it did before it broke.
That's not a metaphor. The system generates more load in its failure state than in its healthy state. This is a positive feedback loop, and positive feedback loops in distributed systems don't resolve themselves — they resolve when someone finds the loop and physically breaks it.
The twist: it wasn't GitHub's retries that kept it down
GitHub fixed the underlying capacity issue by early afternoon. The outage kept running for another five hours anyway.
The reason: a latent retry bug in VS Code. Somewhere in VS Code's GitHub integration, a retry policy on a slow endpoint had no adequate backoff, no jitter, no ceiling that mattered at this scale. Multiply that by every open VS Code window on the planet quietly retrying against a degraded endpoint, and you've turned a client feature into, in the words of the postmortem writeup, "a load generator running on millions of laptops."
Nobody at GitHub wrote that code. Nobody at GitHub could patch it in production. It shipped in a client months earlier, sat dormant because the endpoint was never slow enough to trigger it, and then one bad afternoon turned it into the thing keeping the outage alive.
This is the part that should actually scare you. Your retry storm's ammunition isn't in your codebase. It's in every SDK, CLI, and IDE extension your users installed, running code you don't control, that you cannot hotfix, that has been silently correct for years because your service has never been slow enough to expose it.
Fixing it required an uncomfortable move
To break the loop, GitHub had to pause all four saturated HAProxy nodes at the same time.
Not one by one. Not the worst one first, watch, then the next. Simultaneously. The postmortem is explicit that staggering the pause "would have kept the saturation alive" — because traffic drained from a paused node would just pile onto the still-live ones, saturating those in turn. You'd be playing whack-a-mole against your own infrastructure.
That's a genuinely uncomfortable operational call to make live, during an incident, with dashboards on fire: take everything down at once, on purpose, to stop it from staying down by accident.
What GitHub is actually doing about it
Buried in the corporate-speak remediation list are two items that matter more than the rest combined:
- Consistent retry limits, budgets, and variable timeouts across service interactions — i.e., stop treating retry policy as a per-team, per-client afterthought and make it a platform-level contract.
- System isolation and removal of shared dependencies — i.e., stop letting one saturated sidecar's blast radius include your auth gateway.
Everything else — 3 million CPU cores, 120 petabytes of storage — is buying headroom. Headroom is not the fix. Monthly commits went from 1.4 billion in April to 2.9 billion in August; Azure went from 12% to 58% of platform load in the same window. You cannot out-provision a positive feedback loop. You can only break the loop.
The lesson for the rest of us
If your service has clients you don't control — a public API, an SDK, a CLI, a VS Code extension, a mobile app you can't force-update — your retry storm is already deployed. It's just never been triggered.
Three things worth checking this week:
- Does your autoscaler see what your sidecar sees? If proxy concurrency isn't in your scaling signal, you have GitHub's exact blind spot.
- Do your clients retry with jittered exponential backoff and a hard ceiling, or did someone write
for i in range(5): retry()in 2023 and nobody's looked since? - Can you pause traffic to a saturated component instantly and simultaneously, or does your runbook assume you have time to be gradual? During a feedback loop, gradual is how you stay down longer.
GitHub's outage wasn't caused by too little capacity. It was caused by a failure mode that manufactures its own load the harder you fight it. That's a much scarier class of bug than "we need more servers" — and it's sitting in more codebases than anyone wants to admit.
Top comments (0)