AWS Cost Anomaly Detection Script: A Bash Alert Checklist

Originally published on kuryzhev.cloud

---

Our AWS bill jumped by $2,000 overnight last quarter, and Cost Explorer only surfaced it a full day later — well after the on-call engineer had already moved on to other fires. That delay is exactly why we ended up writing our own AWS cost anomaly detection script instead of waiting on native tooling. It's not glamorous, it's a bash script running on a cron job, but it catches spikes hours before AWS's own anomaly detection gets around to telling you.

Why this checklist

AWS Budgets and AWS Cost Anomaly Detection exist, and they're free. But alert latency on both can run up to 24 hours, and neither integrates natively with Slack in a way anyone actually reads. Budgets alerts show up as emails that get filtered into a folder nobody checks. So teams — us included — end up building a cron-based checker on top of the Cost Explorer API and posting straight to a Slack channel.

The problem is that a bad cost-monitoring script gives you false confidence in either direction. Too noisy, and people mute the channel within a week — classic alert fatigue. Too quiet, and it misses the exact spike you built it for, which is worse than having nothing at all because now you *trust* it. This checklist is aimed at closing both failure modes before you ship the script to production.

Scope-wise, this is for teams already comfortable with aws-cli and the Cost Explorer API — not a pitch for building a full FinOps platform. If you're managing dozens of linked accounts with complex allocation tags, you probably want a paid tool. If you're a small-to-mid platform team that just needs "tell me if yesterday's spend looks weird," this is the right level of investment.

The checklist (numbered)

Go through these before you trust the script with a production Slack channel. We tested against aws-cli/2.15.30 Python/3.11.8 — Cost Explorer subcommands require CLI 2.0+, so check your version first with aws --version.

1. IAM scoped to ce:GetCostAndUsage only. Don't attach full Billing access. A scoped policy limits blast radius if the credentials leak.
2. No root or long-lived admin creds. Use an IAM role with a narrow trust policy, ideally assumed by a Lambda execution role once you graduate off cron.
3. Compare trailing 7-day average vs. yesterday's actual spend — not month-to-date. Month-to-date totals trend upward every single day after the 1st, which guarantees false positives.
4. Pull an 8-day window so you have 7 days of baseline plus yesterday, rather than querying twice and doubling your API cost.
5. Threshold on percentage AND absolute dollar delta, whichever is smaller. A 25% jump on a $10/day account is noise; a 25% jump on a $5,000/day account is real money.
6. Use --granularity DAILY deliberately. Cost Explorer doesn't support HOURLY granularity at all — don't waste time trying.
7. Account for the ~24h billing lag. "Today's spike" alerts are really detecting yesterday's spend. Say so explicitly in the Slack message.
8. Pin all date math to UTC. AWS billing data is UTC-native; if your team is UTC+3, "yesterday" boundaries shift and you can double-count or skip a day entirely.
9. Deduplicate alerts with a state file or DynamoDB item. Without this, the same anomaly fires every cron run until spend drops — we've seen this flood a channel every 15 minutes.
10. Store the Slack webhook in SSM Parameter Store as SecureString, retrieved via aws ssm get-parameter --with-decryption. Never hardcode it or drop it in a .env file.
11. Use Slack Block Kit, not plain text, for anything with more than one data point — plain text walls of numbers get ignored.
12. Add curl --retry 3 --retry-delay 5 on the webhook POST to survive transient 429/500 responses from Slack.
13. Use distinct exit codes — 0 for no anomaly, 1 for anomaly sent, 2 for script error — so cron monitoring (healthcheck.io, Nagios) can tell a real failure from a quiet day.
14. Log every run with a timestamp, rotated, even on the "no anomaly" path. You'll need this history the first time someone asks "did the script even run last Tuesday?"
15. Build a test mode with mocked JSON instead of hitting the real API on every CI run — Cost Explorer charges per request.
16. Break down cost by service in the Slack fields (EC2, S3, data transfer) so the on-call person doesn't have to open the console to start investigating.
17. Watch the 40,000-character Slack payload limit. A verbose per-service breakdown across many linked accounts can get silently truncated.
18. Link the runbook directly in the alert. "Go check the console" is not a runbook — put the actual triage steps behind a link.

Here's the actual script we run, wired to the logic above. It compares yesterday's UnblendedCost against a trailing 7-day average and posts to Slack only when the delta crosses either threshold:

#!/usr/bin/env bash
# cost-anomaly.sh — checks yesterday's AWS spend against a 7-day trailing average
# and posts a Slack alert if the delta exceeds threshold.
set -euo pipefail

THRESHOLD_PCT=25          # alert if spend jumps >25%
THRESHOLD_ABS=50          # OR jumps by more than $50 absolute
STATE_FILE="/var/lib/cost-anomaly/last_alert.json"
SLACK_WEBHOOK=$(aws ssm get-parameter \
  --name "/costmonitor/slack_webhook" \
  --with-decryption --query "Parameter.Value" --output text)

YESTERDAY=$(date -u -d "yesterday" +%Y-%m-%d)
WEEK_AGO=$(date -u -d "8 days ago" +%Y-%m-%d)
TODAY=$(date -u +%Y-%m-%d)

# Pull daily cost data for the trailing 8 days (baseline + yesterday)
RAW=$(aws ce get-cost-and-usage \
  --time-period Start="$WEEK_AGO",End="$TODAY" \
  --granularity DAILY \
  --metrics "UnblendedCost" \
  --output json)

# Guard against empty ResultsByTime (new/zero-usage accounts)
if [[ $(echo "$RAW" | jq '.ResultsByTime | length') -eq 0 ]]; then
  echo "No cost data returned — exiting quietly." >&2
  exit 2
fi

# Extract yesterday's cost and the average of the previous 7 days
YDAY_COST=$(echo "$RAW" | jq -r --arg d "$YESTERDAY" \
  '.ResultsByTime[] | select(.TimePeriod.Start==$d) | .Total.UnblendedCost.Amount')
AVG_COST=$(echo "$RAW" | jq -r --arg d "$YESTERDAY" \
  '[.ResultsByTime[] | select(.TimePeriod.Start!=$d) | .Total.UnblendedCost.Amount | tonumber] | add/length')

DELTA_ABS=$(echo "$YDAY_COST - $AVG_COST" | bc -l)
DELTA_PCT=$(echo "($YDAY_COST - $AVG_COST) / $AVG_COST * 100" | bc -l)

# Round for readability in Slack message
DELTA_PCT_R=$(printf "%.1f" "$DELTA_PCT")
YDAY_COST_R=$(printf "%.2f" "$YDAY_COST")

is_anomaly=false
if (( $(echo "$DELTA_PCT > $THRESHOLD_PCT" | bc -l) )) || \
   (( $(echo "$DELTA_ABS > $THRESHOLD_ABS" | bc -l) )); then
  is_anomaly=true
fi

if [[ "$is_anomaly" == false ]]; then
  echo "No anomaly: yesterday=\$${YDAY_COST_R}, delta=${DELTA_PCT_R}%"
  exit 0
fi

# Dedup: skip if we already alerted for this date
LAST_ALERTED=$(jq -r '.last_date // ""' "$STATE_FILE" 2>/dev/null || echo "")
if [[ "$LAST_ALERTED" == "$YESTERDAY" ]]; then
  echo "Already alerted for $YESTERDAY, skipping."
  exit 0
fi

curl --retry 3 --retry-delay 5 -sf -X POST "$SLACK_WEBHOOK" \
  -H "Content-Type: application/json" \
  -d "{\"text\": \"🚨 AWS cost anomaly: \$${YDAY_COST_R} on $YESTERDAY (+${DELTA_PCT_R}% vs 7-day avg)\"}"

echo "{\"last_date\": \"$YESTERDAY\"}" > "$STATE_FILE"
exit 1

The cron entry we run on the bastion box, logging both the anomaly and quiet-day cases:

0 9 * * * /opt/scripts/cost-anomaly.sh >> /var/log/cost-anomaly.log 2>&1

And the Block Kit payload it actually sends, versus the log lines you'll see when it correctly skips a duplicate alert or a quiet day:

{
  "text": "🚨 AWS cost anomaly detected",
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*🚨 AWS Cost Anomaly — 2024-05-14*\n*Yesterday's spend:* $412.87\n*7-day avg:* $298.11\n*Delta:* +38.5% ($114.76)\n*Threshold:* 25% / $50"
      }
    },
    {
      "type": "context",
      "elements": [
        {
          "type": "mrkdwn",
          "text": "Account: `123456789012` | Runbook: <https://wiki.internal/cost-runbook|link>"
        }
      ]
    }
  ]
}

# Corresponding cron log lines (for troubleshooting silent failures):
2024-05-14T09:00:03Z Already alerted for 2024-05-13, skipping.
2024-05-15T09:00:02Z No anomaly: yesterday=$301.44, delta=1.1%

Commonly missed items

These are the ones that pass code review clean and then break in week two of production. First: forgetting --filter scoping in a multi-account org. Without it, the script pulls org-wide costs even when only one linked account is relevant, which inflates false positives every single day because you're comparing apples to a whole fruit basket.

Second: comparing raw dollar totals without normalizing for weekday/weekend patterns. If batch jobs only run Monday through Friday, every Monday morning looks like a spike compared to a quiet Sunday baseline — and your channel gets trained to ignore Monday alerts specifically, which is the opposite of what you want.

Third — and this one bit us directly — not handling jq parsing failures when Cost Explorer returns an empty ResultsByTime array. New accounts or freshly linked sub-accounts with zero usage will trigger jq: error (at <stdin>:0): Cannot index null and crash the whole script with a stack trace nobody reads at 9am. Guard for it explicitly, as in the script above.

Fourth: assuming Cost Explorer is near-real-time. It isn't — it lags roughly 24 hours. "Today's spike" language in your Slack alert is misleading; say "yesterday's spend" instead, or someone will go looking for a live spike that already ended.

Fifth, and honestly the dumbest one we've made twice: hardcoding the Slack webhook or channel per environment instead of parameterizing it. We once had a prod cost spike fire quietly into a staging channel that three people had muted six months earlier. Nobody noticed for four days.

Automation ideas

Once the bash version has proven the alerting logic — meaning it's run for a month without a false positive flood or a missed spike — it's worth graduating it off a cron job on a bastion host. Package it as a Lambda function triggered by EventBridge on a rate(1 day) schedule. This removes the single point of failure of "the bastion host is down" and gives you native CloudWatch logging for free.

Next, stop comparing only two data points. Store historical daily costs in DynamoDB or S3 as CSV/Parquet, and build a proper rolling baseline — 7, 14, and 30-day moving averages — so seasonal spend patterns don't trigger false alarms every quarter-end.

Add a --dry-run flag and wire a CI job (GitHub Actions works fine) that mocks the aws ce get-cost-and-usage output and asserts on the anomaly decision logic. This catches regressions in the threshold math before they hit prod, and it's the cheapest insurance you can buy for a script that gates financial alerts.

For teams with a real on-call rotation, extend the Slack payload with an interactive "Acknowledge" button using Slack Bolt or a simple webhook plus API Gateway. That gives you a paper trail of which anomalies were actually investigated versus ignored — useful data when you're arguing for more FinOps headcount.

Finally, once the bash version has earned trust, feed the same signal into AWS's own Cost Anomaly Detection API as a second opinion. Don't replace the script — run both in parallel for a while and compare false-positive rates before deciding which one owns the pager. We've written more about wiring alert layers correctly in our AWS automation posts, and it's the same discipline whether you're watching cost, drift, or infrastructure health.

None of this is exotic engineering. It's a bash script, an IAM policy, and a webhook — but an AWS cost anomaly detection script built with these guardrails will catch a real spike hours before Cost Explorer's own dashboard updates, and that's the whole point.

For the underlying API details and rate limits, the AWS Cost Explorer GetCostAndUsage API reference is worth bookmarking, and Slack's incoming webhooks documentation covers the payload limits and retry behavior referenced above.

Related

• More AWS automation and cost-monitoring patterns
• Bash scripting practices for production infra tooling
• Monitoring and alerting checklists beyond Slack webhooks