At 9:40am, login started failing for every customer. Our auth provider's status page showed green. Our own dashboard showed errors. For two hours we debugged our side — token validation, cache layers, session stores — because "vendor pages are often wrong" and ours was the one we could actually touch.
The provider was down. It was down for three hours. Our status page said nothing for the first two, because — and this is the sentence I now hate — "it wasn't our outage."
Here's the runbook we wrote after, so your version of that Tuesday goes better.
1. Classify in 10 minutes: yours, theirs, or both.
Open the vendor status page AND your error timeline side by side. Vendor degraded + your errors started in the same window = theirs (probably). Vendor green + your errors rising = yours. Both degraded but your errors predate their window = both, and you fix yours first. Write down the timestamps — they settle the blame conversation in the retro before it starts.
The one cross-check that saves you an hour of debugging: a canary request through a second path (different network, different resolver, manual test with a fresh account). One verification, then stop debugging your own stack.
2. Degrade deliberately, out loud.
Every critical vendor needs a pre-decided answer to: what do we do while it's down? Three postures:
- Wait: honest, fine for short outages, dangerous for silent ones.
- Degrade with a feature flag: queue the writes, serve cached reads, turn off the one feature that touches the vendor. Customers prefer a working 80% to a broken 100%.
- Fail over: only real if the second provider was configured in peacetime. An untested failover is a second outage with extra steps.
We had none of the three pre-decided. We picked "wait" by not deciding — the default that feels like a choice.
3. Communicate before customers ask.
Two hours of vendor outage costs less trust than two hours of silence. What works:
- First update inside 15 minutes, even if it says nothing new: "We're aware login is failing due to an outage at our auth provider. Next update in 30 minutes." Naming the vendor buys credibility; "some users may experience" burns it.
- Update on a clock — every 30–60 minutes whether or not anything changed. Silence reads as "nobody is working on it."
- The all-clear names the cause and one lesson. That sentence is what keeps the trust you just spent.
4. Harvest the dependency list — the only part worth keeping.
Within a week, write down: every feature that broke and the vendor behind it (you just discovered this map by fire), which dependencies deserve a real plan (degrade mode, failover, or a signed "we accept this risk"), which SLA clauses you never logged (ten minutes of note-taking per outage turns into real credits), and the canary alert you're adding so your monitoring speaks before the vendor's status page does.
That list is the free audit the outage paid for. Our fill-in-the-blanks runbook is here:
→ https://hive80-lab.github.io/ops-notes/vendor-outage-runbook.html
If your incidents need the surrounding layer — severity matrix, comms templates, the first-30-minutes card — that's the Ops Starter Kit ($14), and Vol. 2 ($27) adds the DR plan and evidence log for the review after. The Automation Starter Pack ($19) turns the canary checks and update clocks into workflows that run themselves. Launch week: 30% off with code HIVE-LAUNCH30 at https://hive80lab.gumroad.com
Free starting point: The First 30 Minutes — the one-page quick-start for any outage: https://hive80lab.gumroad.com/l/first-30-minutes
Top comments (0)