DEV Community

Cover image for Catching a Cost Spike the Same Day It Happens: Inside Our Z-Score Anomaly Detector
Rick Wise
Rick Wise

Posted on Originally published at cloudcostwise.io

Catching a Cost Spike the Same Day It Happens: Inside Our Z-Score Anomaly Detector

#aws #finops #observability #python

AWS Cost Explorer's own data can lag up to 24 hours behind real usage, and nobody's on call for "the bill." A forgotten load test, a runaway Auto Scaling event, a script that didn't tear down its own infrastructure — by the time a human opens the console, whatever ran already ran.

CloudWise's anomaly detector is built to close that gap: a daily scheduled scan, not a dashboard you have to remember to check. Here's the actual statistics behind it — the real thresholds from lambdas/anomaly_detector/handler.py — and the exact test case, already in our shipped suite, that proves it catches a spike.

The mechanic: z-score, not a percentage

The naive version of this feature checks "is today more than X% above yesterday?" That breaks immediately: a service that costs $2/day naturally swings 200% day to day out of pure noise, while a service that costs $4,000/day moving 15% is a real four-figure problem. Percentage-of-yesterday has no sense of a service's normal variance.

The detector instead computes a z-score per AWS service, per account, every day:

# lambdas/anomaly_detector/handler.py
SPIKE_THRESHOLD_STD_DEV = 2.0   # standard deviations from the mean
MIN_ABSOLUTE_CHANGE_USD = 5.0   # minimum dollar change to trigger, at all
MIN_DATA_POINTS = 7             # minimum days of history required

mean = statistics.mean(historical)
std_dev = statistics.stdev(historical) if len(historical) > 1 else 0
change = today_cost - mean
z_score = (today_cost - mean) / std_dev if std_dev > 0 else 0

if z_score >= SPIKE_THRESHOLD_STD_DEV:
    # ...flag it
Enter fullscreen mode Exit fullscreen mode

historical is the trailing daily costs for that service, excluding today. If a service doesn't have at least 7 days of history yet, it's skipped — no history means no baseline, and a false "anomaly" on day one of a brand-new resource is worse than useless. And before z-score even runs, there's a $5 absolute-change floor: if today's cost moved by less than five dollars, the detector doesn't care how many standard deviations that represents. A service that normally costs $0.02/day moving to $0.11/day is a 450% swing, a huge z-score, and completely irrelevant to anyone's bill. The dollar floor is what keeps a statistically-driven detector from paging you about noise.

Severity isn't binary

Once something clears the 2.0 z-score bar, it gets bucketed:

SEVERITY_THRESHOLDS = {
    'CRITICAL': 4.0,
    'HIGH': 3.0,
    'MEDIUM': 2.0,
    'LOW': 1.5,
}
Enter fullscreen mode Exit fullscreen mode

(Yes, LOW is below the 2.0 trigger threshold in the table — that band exists for a second call site that widens the net for manual/on-demand scans. The scheduled daily job only ever emits MEDIUM and up, because the scheduled job's trigger is 2.0.) The severities aren't decoration — they're what a Shield-tier customer's Slack alert leads with, and what determines whether the message reads as "worth a look this week" or "look now."

The test that proves it

Rather than a hypothetical dollar figure, here's the exact fixture from our own shipped test suite (lambdas/anomaly_detector/test_handler.py::test_detects_anomaly_with_high_z_score) — real, committed, running in CI on every change to this file:

An EC2 line item costs $9–$11/day for seven straight days: $10, $11, $9, $10.5, $9.5, $10, $10. Mean $10.00, standard deviation $0.65. Then it jumps to $100 in a day.

change     = 100 - 10        = $90
z_score    = (100 - 10) / 0.65 = 139.4
percentage = 90 / 10 * 100     = 900%
Enter fullscreen mode Exit fullscreen mode

A z-score of 139.4 isn't a borderline call by any measure — the test asserts the detector flags it as CRITICAL or HIGH severity, and it does, deterministically, every run. The absolute dollars here are small on purpose: it's a unit test, not a customer account. The statistical shape — a service ten times its normal cost, standard deviations off its own baseline — is exactly what the 2.0 threshold exists to catch on day one, whether the line item is $10 or $10,000.

Why this is Shield-tier, and why it's read-only

The detector runs once a day via EventBridge, over the last 30 days of each Shield-tier account's cost data, and writes findings to an alerts table before dispatching Slack and email notifications. It doesn't touch anything in your AWS account — no remediation, no API calls beyond reading Cost Explorer data CloudWise already ingests for the dashboard. That's deliberate: detecting a spike and deciding what to do about it are different problems, and this one only does the first.

The mechanic is boring on purpose — mean, standard deviation, a threshold, a dollar floor. Boring is what you want from something that's deciding whether to interrupt your weekend.

Top comments (0)