You cannot optimize a bill you cannot attribute. Before touching a single memory setting or reserved-capacity purchase, get to the point where you can name the feature, service, or team behind every line over about 1% of your spend. On AWS that means three things: the Cost and Usage Report queried in Athena, tags applied at the IaC layer rather than by hand, and a separate plan for the costs that tags will never explain — NAT Gateway, log ingestion, and cross-AZ traffic.
A commenter on an earlier post about surprise serverless bills made a point I keep coming back to: it is hard to control costs when you do not know what is actually charging you. That is the whole problem. Most cost-optimization advice starts at step two. This post is step one.
Why doesn't the billing console answer the question?
The billing console and Cost Explorer group by service, and services are not how you think about your system. Cost Explorer will happily tell you that NAT Gateway cost $180 last month. It will not tell you that $150 of that was one image-processing function pulling source files from a third-party API through a VPC.
The mismatch is structural. Your mental model is "the export feature costs too much." AWS's model is "EC2-Other, usage type NatGateway-Bytes." Nothing in the default tooling bridges those two, and every optimization decision you actually care about — kill this feature, cache this call, move this workload — lives on the feature side of the gap.
There is also a granularity trap. Cost Explorer's default daily, service-level view is free; resource-level granularity is a paid opt-in for some services and still does not exist for others. So the console can tell you that something spiked, and roughly when, but rarely which resource.
Takeaway: Cost Explorer is a smoke alarm, not a diagnostic tool — it tells you something is burning, never what.
What does "visible" actually mean?
It helps to be precise about the level of attribution you are buying, because each one costs more effort than the last.
| Level | What you can answer | What it takes | Good enough for |
|---|---|---|---|
| Service | "Which AWS service jumped?" | Nothing — it's the default console | Bills under a few hundred a month |
| Resource | "Which function, table, or log group?" | Cost and Usage Report + Athena | Most teams, most of the time |
| Feature / team | "Which product area owns this?" | Enforced tags + a mapping | Multi-team or multi-tenant products |
| Per-customer | "Is this account profitable?" | Request-level tracing joined to cost | Usage-based pricing models |
Most teams need level two and think they need level four. Get resource-level attribution working end to end before you build anything fancier, because levels three and four are both built on top of it.
Takeaway: pick the shallowest attribution level that answers the decisions you actually make, and stop there.
How do I get resource-level cost data?
The Cost and Usage Report (CUR) is the only AWS billing source that contains resource IDs and tag columns for a broad set of services. You enable delivery to an S3 bucket, wait for the first daily drop, and query it with Athena. As of mid-2026 AWS offers this both as the classic CUR and through Data Exports; either lands Parquet in S3 and both work with the query below.
The one thing to get right on day one is partitioning, because CUR files are large and Athena bills you for bytes scanned. Always filter on the partition columns.
SELECT
line_item_product_code AS service,
line_item_usage_type AS usage_type,
line_item_resource_id AS resource,
resource_tags_user_service AS owner,
ROUND(SUM(line_item_unblended_cost), 2) AS cost
FROM cur_db.cur_table
WHERE year = '2026'
AND month = '7'
AND line_item_line_item_type = 'Usage'
GROUP BY 1, 2, 3, 4
HAVING SUM(line_item_unblended_cost) > 1
ORDER BY cost DESC
LIMIT 50;
Two notes. Column names come from the Glue crawler's lowercase-underscore convention, so lineItem/LineItemType becomes line_item_line_item_type — check your own table schema before assuming. And the HAVING clause is doing real work: it collapses a few thousand rows of noise into the twenty lines you will actually act on.
Run that once a month and read the top twenty rows out loud. It is the single highest-leverage hour in cloud cost work, and it needs no product beyond Athena.
Takeaway: if your billing data has no resource IDs in it, you are not doing cost analysis, you are doing cost astrology.
How do I make tagging survive contact with reality?
Tags are how resource IDs become feature names, and hand-applied tags always decay. The fix is to apply them at the layer that creates resources. In Terraform, provider-level default tags cover everything the provider creates:
provider "aws" {
region = "us-east-1"
default_tags {
tags = {
service = "billing-api"
environment = "prod"
owner = "platform"
managed_by = "terraform"
}
}
}
Three caveats worth knowing before you rely on this. Not every AWS resource type supports tags, so coverage will never be 100%. Resources created implicitly by other resources — an ENI a Lambda attaches to a VPC, for instance — often do not inherit them. And critically, cost allocation tags are not retroactive: after you create a tag key you must activate it in the Billing console, and it only appears in CUR data from that point forward. Activate the keys the day you invent them.
For the gap that remains, an organization-level tag policy plus a scheduled Config rule that flags untagged resources is the difference between "we tag things" and "we can trust the tag column."
Takeaway: a tag applied by a human is a tag that will be missing on the resource that matters.
What about the costs no tag will ever explain?
Three line items resist attribution structurally, and they are disproportionately the ones that spike.
NAT Gateway bills per hour and per GB processed, and the CUR line has no notion of which function sent the bytes. To attribute it you need VPC Flow Logs, then aggregate bytes by source ENI and map ENIs back to workloads. Flow logs themselves cost money to ingest, so turn them on for a diagnostic window rather than permanently.
CloudWatch Logs bills on ingested volume, and the culprit is almost always one chatty log group left at debug level. You do not need CUR for this one — CloudWatch publishes IncomingBytes per log group:
#!/usr/bin/env bash
set -euo pipefail
END=$(date -u +%Y-%m-%dT%H:%M:%SZ)
START=$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
|| date -u -v-7d +%Y-%m-%dT%H:%M:%SZ)
aws logs describe-log-groups --query 'logGroups[].logGroupName' --output text \
| tr '\t' '\n' \
| while read -r lg; do
bytes=$(aws cloudwatch get-metric-statistics \
--namespace AWS/Logs --metric-name IncomingBytes \
--dimensions Name=LogGroupName,Value="$lg" \
--start-time "$START" --end-time "$END" \
--period 86400 --statistics Sum \
--query 'sum(Datapoints[].Sum)' --output text)
awk -v b="${bytes:-0}" -v n="$lg" \
'BEGIN { printf "%10.2f GB %s\n", b/1073741824, n }'
done | sort -rn | head -20
Cross-AZ traffic is the quietest of the three, because nothing in your code says "this call crossed an availability zone." It shows up as a data-transfer usage type with no resource ID, and the only real diagnostic is reading your architecture for chatty cross-zone paths.
Takeaway: budget a day for the unattributable line items specifically — they will not fall out of any dashboard you build.
When is a cost tool worth paying for?
Once attribution works, tooling buys you time, not new information. Infracost is the one that changes behavior earliest, because it estimates the cost delta of a Terraform change in the pull request, before the resource exists. Vantage is the managed option if you want per-resource cost views and Kubernetes allocation without building the Athena layer yourself. CloudZero is aimed squarely at the harder problem of cost-per-customer allocation for usage-priced products, which is real work to build in-house.
The honest limitations: Infracost estimates list prices and cannot know your negotiated discounts or actual usage volume; Vantage adds a percentage-of-spend cost that only pays off above a certain bill size; CloudZero requires you to define an allocation model, and a wrong model produces confident, wrong numbers. And all three read the same CUR you can query yourself for the price of an Athena scan.
Takeaway: buy a cost tool to save engineering hours, never to discover data you have not bothered to turn on.
FAQ
Why is my AWS bill higher than my Lambda costs?
Because Lambda compute is usually the smallest meter in the request path. NAT Gateway hours, CloudWatch Logs ingestion, API Gateway per-request charges, and data transfer routinely add up to more than the GB-seconds you were watching.
How do I find which CloudWatch log group is costing the most?
Query the IncomingBytes metric in the AWS/Logs namespace with a LogGroupName dimension, summed over the last week, and sort descending. Ingestion volume, not stored volume, is what drives the CloudWatch Logs line on your bill.
Do AWS cost allocation tags work retroactively?
No. A tag key must be activated in the Billing console, and it only populates cost data from the activation date forward. Historical months keep whatever tag columns were active at the time.
Bottom line
If your monthly bill is small, the free path is enough: turn on the Cost and Usage Report, query the top fifty resources in Athena once a month, and keep a log-group ranking script in your repo. If you run multiple teams or product areas, invest in provider-level default tags and a policy that fails builds on missing owners, because feature-level attribution is impossible to backfill. Reach for a paid cost platform only when the engineering hours spent maintaining your own attribution pipeline visibly exceed its price. Optimization work done before this foundation exists is guesswork that happens to be expensive.
Top comments (0)