A single misconfigured autoscaling policy cost us $9,200 before anyone looked at the billing console. The invoice was correct. The failure was that cost is an incident, and we were treating it as an accounting report. Here's the daily check that fixed it — five minutes, once a day, by whoever opens the ops laptop first.
The five-minute daily cost check
- Yesterday's spend vs the 30-day same-weekday average — day-over-day hides weekends; week-over-week hides the spike pattern you actually care about. Threshold: >40% above average = investigate today, not in the invoice meeting.
- Top 5 services by cost, not by line item count — sorted by absolute dollars. Ninety percent of surprises live in three services.
- The delta leaders — which service grew the most in dollars, not percent. A service going from $0.30 to $3.00 is noise; one going $400 to $900 is the fire.
- One grep for known budget-killers — NAT gateway data processed, cross-AZ transfer, forgotten load balancers, snapshot accumulation. These four cause most of the silent bleed.
- A one-line note in the ops channel — "Yesterday $212 vs avg $140: ECS up, expected, scale-down at 18:00." The note is the audit trail; when the question comes in three months, you have dates.
The alerts you should have anyway
The daily check catches what alerts miss, but set the alarms too: an absolute daily spend threshold at 2x your normal day, and a forecast alert (month-to-date pace vs budget). Both are native in every major cloud's billing tooling; both are off by default; both take ten minutes to enable. Do them now, before this article, before coffee.
Culture is the multiplier
The check only works if a cost anomaly is treated like a 2am page: not shameful, not deferred, someone's name on it until closed. The first month we ran it, we caught four issues — a dev environment someone forgot to schedule down ($700/month), an over-provisioned Redis node, a logging pipeline writing twice, and the autoscaling policy that started this story. Total recovered: $2,100/month. Five minutes a day buys a part-time engineer's salary.
Every drill in this post ships in the Ops Starter Kit ($29). All five kits in one download: Ops Mega Bundle ($49). Or let the always-on ops desk run them for you.
Top comments (0)