DEV Community

Cover image for My Monitoring Missed the Breach. My Credit Card Caught It.
Dhruv Malaviya
Dhruv Malaviya

Posted on

My Monitoring Missed the Breach. My Credit Card Caught It.

A leaked key spun up instances in a region my dashboards didn't watch; the invoice screamed first. Now I treat spend as a security sensor — per-minute resolution, a prepaid fuse, and a nightly audit on Krova Cloud.

The leak was a CI variable I'd forgotten existed, wired to a cloud API key with far too much confidence. Over one weekend, someone's script used it to spin up instances in a region I'd never deployed to.

My dashboards stayed green — they were watching the resources I knew about. Nobody's monitoring watches a region they forgot exists. The first thing that screamed was my credit card statement, Monday morning, looking like a typo.

Spend is the only metric that can't lie by omission
CPU, latency, error rates — they measure what you chose to point them at. Money aggregates everything, including what you never thought to watch. An attacker can stay quiet to avoid your alerts. They cannot consume resources without touching your bill. Every leaked key, runaway loop and forgotten box has one shared property: it costs money per minute it exists.

The invoice isn't accounting. It's the widest-coverage security sensor you own — and the most ignored. The catch is resolution and caps: a monthly invoice is a smoke detector that mails you a letter after the fire. I needed the signal during the fire, and a ceiling on the fire's size.

The setup on Krova
Three properties make spend usable as a control on Krova Cloud:

  1. Per-minute resolution. Metered by the minute, no rounding up. High-resolution billing = high-resolution signal; a loop shows up within the hour, not at month-end.
  2. A literal fuse. Billing is prepaid credit, and when a space's balance hits zero its Cubes are automatically powered off — no data loss. A leaked key physically cannot spend past the credit I chose in advance. Dollar-capped blast radius.
  3. Hourly cost on every Cube. Each Cube reports its own costPerHour, so the audit is arithmetic, not estimation.

The nightly audit — expected list vs. actual list, plus burn rate:

#!/bin/bash
# spend-audit.sh — reality vs. budget, every night
set -euo pipefail
krova context use prod

# what's actually running, and what it burns
krova list --json | jq -r '.[]
  | select(.state=="running")
  | "\(.name)\t\(.resources.vcpu)vcpu\t$\(.costPerHour)/h"'

# total burn rate vs. plan
krova list --json | jq '[.[]
  | select(.state=="running") | .costPerHour] | add // 0'
Enter fullscreen mode Exit fullscreen mode
0 2 * * *   spend-audit.sh | diff - expected.txt && true || alert "reality ≠ budget"
0 * * * *   burn-rate-check.sh   # alert if hourly spend exceeds plan + 20%
Enter fullscreen mode Exit fullscreen mode

A stranger's box can't hide when the whole estate fits in one jq. And for in-guest anomalies, the complement is event-driven: Krova's webhooks include resource.alert.cpu/memory/disk — spend catches the economic anomaly, those catch the hot process. Tripwire plus camera.

Sizing the fuse
Prepaid means choosing your ceiling on purpose:

  • Sandbox space gets exactly one experiment's worth of credit. If it melts down, the fuse trips before the surprise does.
  • Prod gets normal load plus headroom — sized like a breaker: above normal, below pain. Misjudge it and the fuse is also an availability incident, so this is a staffing decision, not a slider.

The honest part

  • The bill says something is wrong, not what. Sensor first, forensics second: when spend twitches, the running-list and logs find the limb.
  • Cheap attacks hide under the noise. A patient $0.02/hour adversary sits inside your variance. Spend-catching nails the realistic failures — leaks, loops, zombies — not a careful human. For that you still need walls: no public IP by default, own kernel per Cube, default-deny inbound. The bill is a layer, not the castle.
  • Ops still yours. The audit tells you a box exists; patching, backups and drills remain your job, same as any self-hosted estate.

The cheapest SOC tool you already own
We file the invoice under "finance" instead of "security," then get surprised by it. Move it. Read it daily, alert on its rate of change, cap what it can reach.

My monitoring is good now. But the sensor I'd least want to lose is the one my accountant installed by accident.

Top comments (0)