On a Tuesday in March, a five-person SaaS team opened their cloud invoice and found it had tripled. Nobody had launched anything. Nobody had scaled anything. The last deploy was a form fix. Finance asked engineering for "an explanation by Friday" — and the honest answer was: nobody knew.
Here's what actually happened, and the runbook that now keeps it from happening twice.
What the invoice said: "everything is a bit more."
That's the worst kind of spike. A single runaway instance is easy; a broad 3x across compute, egress, and logs is a system symptom. The team's first instinct was to start deleting things — "surely we don't need that dev cluster" — which is exactly how you delete revenue during a growth spike.
The triage that worked: top movers first.
Every provider has a cost explorer. Sort by absolute dollar change vs. last month, not percentage — a 900% jump on a $2 line item is noise next to a 30% jump on compute. Their top three movers:
- Egress + NAT gateway — up $900. A "temporary" data export job from three months ago was still scheduled nightly, pushing 400GB through a NAT gateway that charges per GB for the privilege of reaching the internet.
- Logs — up $600. A chatty microservice had gone from INFO to DEBUG during an incident in February. Retention was set to "default" (forever). The logs cost more than the instance they came from.
- Retry storms — up $400. A slow dependency made every caller retry ×3; retries made it slower; the multiplier ran silently for five weeks.
Total diagnosis time: 30 minutes, once they stopped guessing.
The three fixes, in order:
1. Kill the cause, don't cut the muscle. The export job died in one crontab -e. Log retention went from forever to 30 days — one CLI call, applied account-wide. The retry storm needed a real fix (timeouts + circuit breaker), which took a day but stopped the bleed at the source. None of these required deleting production capacity.
2. Add the guardrail that would have caught it. One alert: forecast-based budget alert ("on pace to exceed $X this month"), not just a hard cap — forecasts alert you while you can still change course. Plus provider anomaly detection on daily spend. Free, one notification per day, to a channel people read.
3. Make the review boring and scheduled. Five minutes of daily-spend review, once a week — as a script, not a meeting. The teams that never get surprise bills aren't smarter; they just look every week instead of when finance escalates.
The uncomfortable part: the bill was an engineering output. Chatty services, unbounded logs, and retry storms are code decisions, not finance mysteries. The spike made it into the retro as a line item: "our architecture had opinions about money, and nobody was listening."
If this is your week: the triage path (top movers → start date → fix → guardrail) is free:
→ https://hive80-lab.github.io/ops-notes/cloud-cost-spike-runbook.html
The checks that keep this from recurring — the weekly spend review, the retention sweep, the ephemeral-environment shutdown — are exactly what the Automation Starter Pack ($19) turns into pick-first workflows that run themselves. If your spikes come with outages, the Ops Starter Kit ($14) covers incident response, and Vol. 2 ($27) adds the DR plan. Launch week: 30% off with code HIVE-LAUNCH30 at https://hive80lab.gumroad.com
Free starting point: The First 30 Minutes — the one-page quick-start for any outage: https://hive80lab.gumroad.com/l/first-30-minutes
Top comments (0)