DEV Community

Hive80-lab
Hive80-lab

Posted on

When a dependency breaks, a small team really has two options

#management #incident #business #productivity

When a dependency breaks, a small team really has two options: the reduced service it chose in advance, or a full outage it chose by default. At 2am nobody designs a fallback — they improvise a risky one, or they freeze the whole product. Teams that survive vendor outages with a banner wrote one page on a calm afternoon: if this goes down, we serve X, customers hear it from us, and one named switch flips it.

You can read the full note on the HIVE80lab ops-notes site — here is the working version.

The mode ladder: one per tier-1 dependency, chosen in advance 

MODE 0  FULL          everything works
MODE 1  STALE READS   last-known data + "may be N minutes old" banner
MODE 2  QUEUE+ACK     accept the work, queue it, confirm with a timestamp
MODE 3  READ-ONLY     viewing works; anything that writes is off
MODE 4  DOWN          honest full outage; status page carries the load
Enter fullscreen mode Exit fullscreen mode

One written row per dependency: address-validation vendor down → MODE 2 → queue shipments, validate on recovery → switch: FEATURE_ADDR_QUEUE=on → owner: Dana. "We'd figure something out" is not a mode. The pre-choice exists so the first five minutes are execution, not design.

The kill-switch rule

A degraded mode you cannot reach is a rumor. Every written mode needs a switch that flips in under five minutes (config or flag, not a deploy), has been pulled at least once in a drill, has a named owner and a back-out, and degrades one dependency — not the product.

The conversation, pre-written

Silent degradation is discovered by customers, and discovery without explanation reads as betrayal. Stale reads get a banner: "Reports may be up to 15 minutes old — your data is safe and syncing." Queue mode gets a receipt: "Received 14:22, will process when our partner recovers; you'll get a confirmation either way." Recovery gets counts: "All 214 queued orders processed by 16:40; three failed and were refunded." That last sentence is what turns a degraded hour into a trust deposit.

The five traps

  1. The 2am design — the ladder written during the incident is a wish, not a plan.
  2. The never-pulled switch — the flag from six months ago that fails on first real pull. Drill one switch a month.
  3. Silent degradation — customers find the stale data before you mention it.
  4. Queue without a receipt — no ack, no bound, no drain estimate is just a slower outage.
  5. No way back — read-only nobody owns exiting becomes permanent read-only.

The worked example

A nine-person logistics SaaS lost three hours and 31 support conversations to an address-validation vendor outage — checkout included, because validation sat on the critical path and nobody had decided it didn't have to be. A half-day later they had the ladder, a tested config flag, and a banner. The vendor's next outage: 4-minute flip, 214 receipted shipments, 3 support conversations — two of them forwarding the same receipt.

The metrics

  • 100% of tier-1 dependencies have a written mode + switch + owner (one screen, quarterly review)
  • one kill-switch drill per month; every switch pulled successfully each quarter
  • time-to-degrade under 10 minutes, measured in drills, not first tried in incidents
  • zero silent degradations; 100% of queued work gets a receipt, a bound, and a recovery report

From the HIVE80lab ops-notes rail — free incident templates, runbooks, and checklists for teams under fifty. The First 30 Minutes quick-start is free; the Ops Starter Kit ($14) and Vol. 2 ($27) carry the full versions.

Top comments (0)