AWS Cost Anomaly Alerts: Route to Owners with Tag Context

Originally published on kuryzhev.cloud

---

The scenario

A shared finance mailbox gets an email titled "Cost anomaly detected," and nobody knows whose problem it is. This is the usual failure with default AWS cost anomaly alerts: detection works, but the message goes to a person who cannot act on it. By the time someone finds the owning team, the spend may have run for another day.

The goal here is narrower than "more alerts." Each anomaly should reach the team that owns the spend, with enough context to start investigating immediately. That context includes the service, account, usage type, dollar impact, and a link to the details. AWS Cost Anomaly Detection already does the statistical work. What you build is the routing layer around it.

The design has four parts:

• Monitors scoped by service and by a cost allocation tag.
• An SNS topic that Cost Anomaly Detection can publish to.
• A subscription with a sensible impact threshold.
• A small Lambda that adds ownership context and forwards the alert to the right channel.

The documented SNS payload carries the following fields:

• the anomaly ID and the monitor ARN
• impact figures
• root causes: service, region, linked account, and usage type

The payload does not carry the tags of individual resources. Ownership therefore has to come from which monitor fired or from your own mapping. Verify the current payload shape in the Cost Anomaly Detection documentation before relying on specific fields.

Prerequisites

Cost Anomaly Detection depends on Cost Explorer, so Cost Explorer must be enabled in the account. If you use AWS Organizations, create monitors in the management account to cover linked accounts. Cost Explorer and anomaly APIs are served from us-east-1, so set that region for the CLI calls.

Your tagging has to be ready too. Pick one tag key that identifies ownership, for example team, and activate it as a cost allocation tag in the Billing console.

Watch out for the activation delay: activation can take up to 24 hours to appear in cost data. By default, the tag only applies going forward. You can request a backfill of historical data for activated tags, so check the current backfill limits in the Billing documentation. A tag monitor created before there is tagged cost history will have little to evaluate.

You also need permissions to create the pieces:

• ce:CreateAnomalyMonitor and ce:CreateAnomalySubscription
• sns:CreateTopic and sns:SetTopicAttributes
• Lambda and IAM permissions for the router function

Have a chat webhook URL ready and a short list mapping each owner to a channel. The example below posts a {"text": ...} body, which matches Slack incoming webhooks. Other tools expect their own payload format. For Microsoft Teams, legacy Office 365 connectors have been retired in favor of Workflows.

If you prefer a managed chat integration, Amazon Q Developer in chat applications (formerly AWS Chatbot) can subscribe to an SNS topic directly. Check current docs for how it renders Cost Anomaly messages. A custom Lambda is what lets you add your own ownership fields.

Step 1: Create monitors that match how you assign ownership

A monitor defines what is watched. Start with a monitor on the AWS service dimension, which catches surprises nobody tagged. Accounts are typically limited to one AWS services monitor, and many already have a default one. Run aws ce get-anomaly-monitors first and reuse an existing monitor if present.

Then add one cost allocation tag monitor per team value. A separate monitor per team means the alert's monitorArn tells you exactly whose tag the spend fell under.

# Service-level monitor: catches spikes in any AWS service (skip if one exists)
aws ce create-anomaly-monitor --region us-east-1 --anomaly-monitor '{
  "MonitorName": "aws-services",
  "MonitorType": "DIMENSIONAL",
  "MonitorDimension": "SERVICE"
}'

# One tag monitor per team, so monitorArn in the alert identifies the owner
for team in payments data platform; do
  aws ce create-anomaly-monitor --region us-east-1 \
    --query MonitorArn --output text --anomaly-monitor "{
    \"MonitorName\": \"team-${team}\",
    \"MonitorType\": \"CUSTOM\",
    \"MonitorSpecification\": {
      \"Tags\": {\"Key\": \"team\", \"Values\": [\"${team}\"], \"MatchOptions\": [\"EQUALS\"]}
    }
  }"
done

Each call returns a MonitorArn. Save them all, because they become keys in the owner map. Monitor counts are subject to service quotas, so verify the current limits if you have many teams. The service monitor will also fire for untagged spend, which makes it a useful catch-all owned by your platform or FinOps group.

Step 2: Create an SNS topic Cost Anomaly Detection can publish to

Email subscribers can receive daily or weekly summaries, but SNS subscribers only receive individual (immediate) alerts. That is what you want for routing.

The topic needs a resource policy that allows the costalerts.amazonaws.com service principal to publish. Without it, the subscription is created but alerts never arrive. If the topic uses SSE-KMS encryption, the key policy must also allow that principal to use the key.

aws sns create-topic --name cost-anomaly-alerts --region us-east-1

# policy.json: allow the Cost Anomaly Detection service to publish
cat > policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowCostAnomalyPublish",
    "Effect": "Allow",
    "Principal": {"Service": "costalerts.amazonaws.com"},
    "Action": "SNS:Publish",
    "Resource": "arn:aws:sns:us-east-1:111122223333:cost-anomaly-alerts",
    "Condition": {"StringEquals": {"aws:SourceAccount": "111122223333"}}
  }]
}
EOF

aws sns set-topic-attributes --region us-east-1 \
  --topic-arn arn:aws:sns:us-east-1:111122223333:cost-anomaly-alerts \
  --attribute-name Policy --attribute-value file://policy.json

Replace the account ID and ARN with your own. The aws:SourceAccount condition is a good-practice guard against cross-account misuse. Confirm the recommended policy in the current docs.

Step 3: Attach a subscription with a meaningful threshold

A subscription links monitors to subscribers and sets the alert threshold. Set it too low and every small fluctuation pages someone, and people then mute the channel. The 75 USD absolute total impact below is only an illustrative starting value. Tune it once you see your own anomaly history.

aws ce create-anomaly-subscription --region us-east-1 --anomaly-subscription '{
  "SubscriptionName": "route-to-lambda",
  "MonitorArnList": ["<SERVICE_MONITOR_ARN>", "<TEAM_PAYMENTS_MONITOR_ARN>",
                     "<TEAM_DATA_MONITOR_ARN>", "<TEAM_PLATFORM_MONITOR_ARN>"],
  "Subscribers": [{
    "Type": "SNS",
    "Address": "arn:aws:sns:us-east-1:111122223333:cost-anomaly-alerts"
  }],
  "Frequency": "IMMEDIATE",
  "ThresholdExpression": {
    "Dimensions": {
      "Key": "ANOMALY_TOTAL_IMPACT_ABSOLUTE",
      "MatchOptions": ["GREATER_THAN_OR_EQUAL"],
      "Values": ["75"]
    }
  }
}'

Older examples use a plain Threshold field, which is deprecated. ThresholdExpression is the current form and also supports percentage-based impact. The two fields cannot be combined, so check the API reference for your CLI version.

Step 4: Enrich and route with a small Lambda

Subscribe a Lambda function to the SNS topic. It reads the anomaly, looks up an owner from a mapping you maintain, and posts a readable message.

The mapping is passed as an environment variable. Its keys are either monitor ARNs (for the per-team tag monitors) or linked account IDs (for spend caught by the service monitor). In a larger setup, keep it in SSM Parameter Store or DynamoDB instead.

import json, os, urllib.request

# Keys: monitor ARNs or linked account IDs, e.g.
# {"arn:aws:ce::111122223333:anomalymonitor/EXAMPLE": {"team": "payments", "webhook": "https://..."},
#  "444455556666": {"team": "data", "webhook": "https://..."}}
OWNERS = json.loads(os.environ["OWNER_MAP"])
DEFAULT = {"team": "finops", "webhook": os.environ["DEFAULT_WEBHOOK"]}

def handler(event, context):
    for record in event["Records"]:
        a = json.loads(record["Sns"]["Message"])
        causes = a.get("rootCauses") or [{}]
        top = causes[0]
        owner = (OWNERS.get(a.get("monitorArn"))
                 or OWNERS.get(top.get("linkedAccount"))
                 or DEFAULT)  # fall back to FinOps
        text = (
            f"Cost anomaly for {owner['team']}\n"
            f"Impact: ${a['impact']['totalImpact']:.2f} "
            f"(expected ${a['impact']['totalExpectedSpend']:.2f})\n"
            f"Service: {top.get('service')} | Region: {top.get('region')} | "
            f"Usage type: {top.get('usageType')}\n"
            f"Details: {a.get('anomalyDetailsLink')}"
        )
        req = urllib.request.Request(
            owner["webhook"],
            data=json.dumps({"text": text}).encode(),
            headers={"Content-Type": "application/json"},
        )
        # Let errors raise: Lambda retries failed async invocations
        urllib.request.urlopen(req, timeout=5)

Allow SNS to invoke the function with aws lambda add-permission, then subscribe the function to the topic. SNS invokes Lambda asynchronously, so a failed webhook call is retried by Lambda's async retry mechanism, not by SNS. Configure an on-failure destination or dead-letter queue so alerts that exhaust their retries are not lost.

Gotcha: the payload's root causes may already include a linked account name. If you resolve extra account details or account tags through AWS Organizations, note that only the management account (or a delegated administrator) can call those APIs. A function in a member account will get access denied unless you provide a cross-account role.

Verify and test

You cannot force a real anomaly on demand, so test in two layers.

First, publish a synthetic message shaped like the documented payload to your topic. This confirms the Lambda parses it, finds an owner, and reaches the webhook. Trim the sample to the fields your code reads, and use a real monitor ARN from your map to exercise that lookup path.

aws sns publish --region us-east-1 \
  --topic-arn arn:aws:sns:us-east-1:111122223333:cost-anomaly-alerts \
  --message '{"anomalyDetailsLink":"https://console.aws.amazon.com/cost-management/home#/anomaly-detection",
    "monitorArn":"<TEAM_PAYMENTS_MONITOR_ARN>",
    "impact":{"totalImpact":240.5,"totalExpectedSpend":60.0},
    "rootCauses":[{"service":"Amazon EC2","region":"eu-west-1",
      "linkedAccount":"111122223333","usageType":"BoxUsage:m5.4xlarge"}]}'

# Check the function logs for parse errors or webhook failures
aws logs tail /aws/lambda/cost-anomaly-router --since 10m --region us-east-1

Second, confirm the real path:

• Run aws ce get-anomalies with a --date-interval to see whether the monitors have produced history.
• Check the Cost Anomaly Detection console for subscription status.
• Watch for the first genuine alert.

If the topic receives nothing while the console shows anomalies, recheck the SNS topic policy (and the KMS key policy, if the topic is encrypted). That is a frequent cause and the first thing to rule out. A related failure is a tag monitor that never fires. This happens when the tag was activated recently or when the values do not match exactly, including case.

Also confirm the fallback works by testing an account ID and monitor ARN that are missing from the owner map. The alert should land in the FinOps channel, not disappear.

Where this leaves you

Routing is only as good as the ownership data behind it. Treat the owner map and the tag policy as living configuration, and update the map whenever you add a team monitor or a new account. Revisit the threshold after a month of real alerts. Keep the catch-all channel staffed so unowned spend gets triaged instead of ignored.

For more AWS operational patterns, browse the rest of kuryzhev.cloud. Re-check payload fields and API limits against the official documentation, since both can change between releases.

Related

• SLO burn-rate alerts explained through a traffic spike
• PostgreSQL VACUUM Monitoring: Stop Table Bloat Before It Hurts
• Lambda Cold Start in Python: Diagnose and Cut Init Time