Utkarsh Rastogi

Posted on Jul 13

Tracking AWS KMS Costs by Service — When Cost Explorer Can't Help

#aws #serverless #cloudtrail #costoptimization

Why KMS Costs Are Hard to Track

Every AWS account that uses encryption is paying for KMS. If you're using S3 with SSE-KMS, EBS encrypted volumes, RDS encryption, Lambda environment variables, DynamoDB with KMS keys — you're making KMS API calls. Every Encrypt, Decrypt, and GenerateDataKey call costs $0.03 per 10,000 (after a free tier of 20,000 requests/month).

That sounds small. But it adds up fast when your services make millions of these calls per day without you realizing it. AWS-managed keys don't have the $1/month storage fee that customer-managed keys do — so the entire cost is purely from API calls. And those calls are invisible unless you go looking.

The real problem isn't the cost itself — it's that you can't see WHERE it's coming from.

AWS Cost Explorer shows you one line item: "KMS: $X." You can see total API request charges, but you can't break it down by key, by caller, or by which service generated the calls. There's no way to tell if it's your S3 buckets, your EBS volumes, your Lambda functions, or your RDS instances.

And here's the catch: AWS-managed KMS keys (aws/s3, aws/ebs, aws/rds) cannot be tagged. This is confirmed in AWS docs — "You cannot tag AWS managed keys, AWS owned keys, or KMS keys in other AWS accounts." These keys are shared across all services of that type in your account and region. So even if you're good at tagging everything else, KMS keys remain a black box.

What Makes This Different from Cost Explorer or CloudWatch

So what already exists and why didn't it work?

Cost Explorer shows aggregate KMS spend per account. Can't break down by key, by caller, by service, or by resource. You know you spent $X but not why.

CloudWatch Metrics — KMS doesn't publish per-key call count metrics. There's no built-in dashboard that shows "this key got called 5M times today by this role."

AWS Cost Anomaly Detection is great for catching sudden account-wide spikes. But it doesn't tell you which specific S3 bucket or Lambda function is the culprit. It works at the billing level, not the API call level.

Tagging works for customer-managed keys. Doesn't work for aws-managed keys. If most of your KMS cost comes from aws/s3 or aws/ebs keys (which is common), tags won't help.

CloudTrail manually — yes, CloudTrail logs every KMS call with full context. But have you tried querying CloudTrail logs with Athena to find who called what key how many times? It's doable but painful. You'd need to set up partitions, write Presto queries, and do it regularly. I tried this first and gave up after the third time.

This project automates all of that. Real-time processing, automatic aggregation, daily email with the answer.

The Core Idea

CloudTrail logs contain everything you need. Each log entry tells you:

Which KMS key was used (even aws-managed keys)
Which IAM role made the call
The encryption context (S3 bucket ARN, EBS volume ID, table name)
The API action (Encrypt, Decrypt, GenerateDataKey)
Timestamp

From one record you can map: aws/s3 key + data-lake-role + arn:aws:s3:::my-bucket/file.json = "this bucket is generating KMS calls through this role."

The problem is volume. A busy account generates millions of these per day. You can't grep through them. You need something that processes them as they arrive, counts the calls, and gives you a summary.

How the System Works

Real-time Processing

CloudTrail delivers KMS events to CloudWatch Logs. A subscription filter pushes these events to a Lambda function in real-time (within seconds of the API call).

The Lambda (Aggregator) does:

Decode the CloudWatch Logs payload (base64 + gzip)
Parse each CloudTrail record
Extract: which key, which caller, which action, which resource, what hour
Group identical calls (same key + same caller + same action + same hour)
Save an atomic counter to DynamoDB using UpdateItem ADD

This means DynamoDB always has the current count. No batch jobs, no waiting until end of day.

Spike Detection

The Aggregator also watches for unusual volume. After each save, DynamoDB returns the new hourly total. If it crosses a configured threshold (e.g., 500 calls in one hour for a single key+caller), it sends an alert immediately via SES.

This catches runaway services the same day they're deployed, not next month on the bill.

To prevent alert fatigue, there's a 24-hour cooldown. If the same key+caller triggers a spike, you only get one alert per day. The cooldown state is stored in DynamoDB with TTL so it cleans itself up.

Daily Report

Every morning at 8 AM UTC, a second Lambda (Reporter) runs:

Queries DynamoDB for yesterday's data using a GSI (efficient, no table scan)
Builds stats: total calls, cost, per-key breakdown, top callers
Compares with the previous day (day-over-day change, who increased the most)
Calculates hourly heatmap (which hour was busiest)
Groups costs by AWS service (S3 vs Lambda vs DynamoDB vs EBS)
Calls Bedrock Nova Lite for AI-powered recommendations
Sends an HTML email with a CSV attachment via SES

The AI recommendations are short and specific. They reference actual keys and callers from your data. If your cost is low, it just says "usage is low, no action needed" instead of giving generic advice.

Weekly Summary

Every Monday, the system also sends a weekly summary:

7-day daily breakdown (calls and cost per day)
Peak day highlighted
Week-over-week comparison (this week vs last week, percentage change)

This helps you see trends that don't show up in a single day's report.

Cost Threshold Alert

If the daily cost exceeds a configured threshold, a separate alert fires. This is different from the spike alert (spike = one key going crazy, cost alert = total daily spend too high).

What the Daily Email Contains

The email has:

Header: day name, date, region, time range
Summary cards: total calls, daily cost, projected monthly cost, number of keys used
Day-over-day comparison: yesterday vs today, percentage change, who moved the most
By service: pills showing S3: X calls, Lambda: Y calls, DynamoDB: Z calls
Top actions: breakdown of Encrypt vs Decrypt vs GenerateDataKey (tells you if it's read-heavy or write-heavy)
Peak hour: busiest hour of the day
Top keys table: key name, total calls, cost, top caller for each key
AI recommendations: 2-3 specific actions with expected savings
CSV attachment: full data for anyone who wants to dig deeper in Excel

It's designed to be scannable in 30 seconds. You open the email, see the numbers, check if anything spiked, and move on.

Common Culprits This System Finds

From my experience and what the reports surface:

S3 buckets without Bucket Keys — This is the biggest one. Without Bucket Keys, every PutObject and GetObject makes its own KMS call. A bucket with 1M objects accessed daily = 1M KMS calls = $3/day just for that bucket. Enable Bucket Keys and it drops to ~100 calls/day regardless of object count.

Lambda functions decrypting environment variables — Every cold start decrypts encrypted env vars. High-concurrency Lambdas with short execution times can generate thousands of Decrypt calls per hour.

DynamoDB with SSE-KMS — If you chose "AWS managed key" or "Customer managed key" for encryption, every read/write makes a KMS call. Switching to "AWS owned key" (the default for new tables) eliminates this entirely — and it's free.

High-IOPS EBS volumes — Every I/O operation on an encrypted EBS volume involves KMS. High-IOPS workloads (io2, gp3 with high IOPS) can generate millions of calls. Not much you can fix here except being aware of the cost.

Secrets Manager rotation — Frequent secret access or rotation generates Decrypt calls. Usually not expensive but shows up in the report.

Architecture

CloudTrail (KMS data events)
      |
      v
CloudWatch Logs
      |
      v
Subscription Filter (kms.amazonaws.com only)
      |
      v
Aggregator Lambda (real-time)
  - Counts per key/caller/hour
  - Saves to DynamoDB (atomic counters)
  - Spike detection with 24h cooldown
  - Alerts via SES
      |
      v
DynamoDB (90-day TTL, date-index GSI)
      |
      v (daily 8 AM via EventBridge)
Reporter Lambda
  - Queries yesterday's data (GSI, no scan)
  - Day-over-day comparison
  - Bedrock AI recommendations
  - HTML email + CSV via SES
  - Cost threshold alerts
  - Weekly summary (Mondays)
      |
      v
SES (all emails) + CloudWatch Dashboard

Tech Decisions and Tradeoffs

CloudWatch subscription filter vs Athena queries — Real-time processing means I get spike alerts immediately. Athena would require scheduled queries and adds cost for scanning.

DynamoDB with atomic counters vs raw event storage — Storing every CloudTrail record would cost a fortune in storage. Aggregating per key/caller/hour means the table stays small (thousands of records vs millions).

GSI for daily queries — The Reporter needs all records for one day. Without a GSI, you'd scan the entire table. The reportDate GSI makes this a single efficient Query.

SES over SNS — SNS adds an unsubscribe footer to every email. SES gives full control over formatting. Both Lambdas use SES directly.

Content-hash deploys — Lambda code is zipped with an MD5 hash in the S3 key. Same code = same hash = CloudFormation skips the update. Prevents unnecessary cold starts when only infra changes.

24-hour spike cooldown — Without cooldown, a sustained spike generates an alert every few minutes. The cooldown uses DynamoDB records with TTL so they auto-delete.

Self-call filtering — The system itself makes KMS calls (DynamoDB encryption, Lambda env var decryption). These are filtered out at both the aggregator level (skip saving) and reporter level (skip displaying).

What It Costs to Run

Component	Monthly Cost	Why
CloudTrail KMS data events	~$60	$2 per 100K events. Scales with your KMS call volume
S3 (CloudTrail log storage)	~$2	14-day lifecycle, auto-deletes old logs
Lambda - Aggregator	~$2	Runs per CloudTrail batch, 256MB, ~60ms per invocation
Lambda - Reporter	~$0	Runs once daily, 256MB, ~3s per invocation
DynamoDB (on-demand)	~$1	Pay per read/write. 90-day TTL auto-deletes old records
Bedrock Nova Lite	~$0	1 call/day, ~500 input tokens, ~150 output tokens
SES	~$0	Free tier covers 62K emails/month
CloudWatch Dashboard	$3	Fixed cost per dashboard
CloudWatch Logs	~$1	7-day retention on Lambda logs
Total	~$69/month

The biggest variable is CloudTrail. If your account makes 100M KMS calls/month, CloudTrail data events cost ~$60. If you make 10M calls, it's ~$6. The rest stays constant regardless of volume.

Time to Build

Infrastructure (8 CloudFormation stacks): designed and deployed in a day
Aggregator Lambda: the parsing and DynamoDB logic took a few hours
Reporter Lambda: email formatting, AI integration, weekly summary took longer
Testing and edge cases: caller identification, reserved keywords, self-call filtering

What You Get for $69/month

Real-time spike alerts (catch runaway services same day)
Daily cost attribution (know exactly which service uses which key)
AI recommendations (specific, actionable fixes)
Weekly trends (is cost going up or down)
Cost threshold alerts (get notified before the bill surprises you)
CSV data for auditing or sharing with your team

The ROI

The system finds savings you can't find any other way. Common fixes:

Enable S3 Bucket Keys: can save $10-50/day per bucket
Switch DynamoDB from SSE-KMS to AWS-owned: saves the entire DynamoDB KMS cost
Identify unused encrypted resources still generating calls

If your KMS bill is over $200/month, this system will likely pay for itself in the first week by identifying one fix. If your bill is under $50/month, it's probably not worth running.

AWS Free Tier

Some of these services have free tier:

Lambda: 1M requests/month free (more than enough)
DynamoDB: 25 read/write capacity units free
SES: 62K emails/month free
CloudWatch: 10 custom metrics free

In practice, the only real cost is CloudTrail data events. Everything else is either free tier or under $5/month.

Deploy and Test

git clone <repo>
cd ai-kms-sentinel
cp .env.example .env
# Edit: SENDER_EMAIL, RECIPIENT_EMAIL

./deploy.sh all           # 8 CloudFormation stacks
./test-kms.sh all         # Generate test KMS calls
# Wait 10-15 minutes for CloudTrail
./test-report.sh status   # Check if data arrived
./test-report.sh today    # Trigger report

One command to deploy. One command to destroy. Everything is infrastructure-as-code.

When You Don't Need This

If your KMS bill is under $10/month — not worth the CloudTrail cost
If you only use customer-managed keys with proper tags — Cost Explorer allocation tags work fine
If you have a single service using KMS — you already know where the cost is

This system is for accounts where multiple services share aws-managed keys and you can't tell from the bill who's responsible for what.

What's Next

Slack integration for spike alerts and multi-account aggregation are on the list. But for now, daily email + real-time spikes covers 90% of what I need.

Do You Even Have This Problem?

Before building anything, check if KMS is actually costing you money. Run this:

aws ce get-cost-and-usage \
  --time-period Start=2026-06-01,End=2026-07-01 \
  --granularity MONTHLY \
  --metrics BlendedCost \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["AWS Key Management Service"]}}' \
  --query 'ResultsByTime[0].Total.BlendedCost.Amount' \
  --output text

If it returns under $10, you probably don't need this. If it returns $50+ and you don't know why, keep reading.

You can also check in the AWS Console: Billing > Cost Explorer > filter by Service = "AWS Key Management Service" > group by Usage Type. You'll see "KMS-Requests" as one big number with no further breakdown.

S3 Bucket Keys: The #1 Fix

If you take nothing else from this post, remember this.

When you upload an object to an encrypted S3 bucket (SSE-KMS), S3 normally makes a separate KMS GenerateDataKey call for every single object. Upload 1 million objects per day = 1 million KMS calls = $3/day just for that one bucket. I didn't believe this until I saw it in the data.

S3 Bucket Keys fix this. When enabled, S3 generates a bucket-level key from KMS and reuses it for multiple objects. Instead of 1 call per object, it makes roughly 1 call per few minutes regardless of how many objects you upload.

1M objects/day without Bucket Keys: 1M KMS calls ($3/day).
1M objects/day with Bucket Keys: ~100 KMS calls ($0.0003/day).

That's not a typo. 99.99% reduction.

To enable it:

aws s3api put-bucket-encryption \
  --bucket your-bucket-name \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "aws:kms"
      },
      "BucketKeyEnabled": true
    }]
  }'

Or in the console: S3 > Bucket > Properties > Default encryption > Edit > Enable Bucket Key.

This only affects new objects. Existing ones still use the old key style. But for high-write buckets, the savings start on day one.

The daily report identifies these buckets automatically — when it sees aws/s3 getting millions of calls from one role, the AI recommendation tells you which bucket to fix.

Before and After

Without this system:

You get the monthly bill. It says KMS: $X. You open Cost Explorer, see one line item, shrug, and move on. Maybe you spend 30 minutes in CloudTrail once, get overwhelmed by the volume, and give up. Repeat next month.

With this system:

8 AM email: "aws/s3 key: 12M calls from data-lake-role, resource: data-lake-raw bucket."

You know exactly what's happening. Enable Bucket Keys on that bucket. Next day's report shows calls dropped by 99%. Done. Move on to the next biggest key.

The difference is attribution. "KMS costs $58/day" is not actionable. "Your data-lake-raw bucket is generating 12M calls because Bucket Keys are not enabled" — that you can fix in 30 seconds.

Limitations

Being honest about what this won't do:

AWS-owned keys (like DynamoDB's default encryption) don't show up in CloudTrail and don't cost you anything. This only tracks aws-managed and customer-managed keys.
Cross-account — each AWS account needs its own deployment. No aggregation across accounts yet.
Cross-region — deploys in one region. Multi-region needs separate deployments or a multi-region trail.
Historical data — starts tracking from deployment. Can't tell you what happened last month.
No auto-fix — tells you what to fix but doesn't do it for you.
Per-resource granularity — attributes cost at key+caller level. If one role accesses 10 buckets, you see the total for that role (though encryption context often reveals the specific bucket).

DEV Community