DEV Community

Cover image for The CloudWatch to S3 Tables Mirror Is Not an Archive. It Expires With Your Log Group.
Polliog for AWS Community Builders

Posted on

The CloudWatch to S3 Tables Mirror Is Not an Archive. It Expires With Your Log Group.

As of June 2026, CloudWatch Logs can mirror a log group into S3 Tables in Apache Iceberg format, queryable with standard SQL in Athena and any Iceberg-compatible engine. The copy in S3 Tables, and the Iceberg maintenance that keeps it performant, carry no additional storage or compaction charge. You pay only for the queries.

The natural reading of that sentence is: pay CloudWatch's expensive ingestion once, then keep a cheap long-term copy in S3 for historical SQL analytics. That reading is wrong, and building a retention strategy on it loses the history you think you are keeping.

The coupling nobody reads

From the AWS documentation, stated plainly and easy to skip: set a retention period on the CloudWatch Logs log group, and the S3 Tables integration applies the same retention to the Iceberg data. The two are coupled. The S3 Tables copy is not an independent archive. It expires with the log group, and deleting the log group or a log stream deletes the Iceberg data with it.

This inverts the mental model most people bring to it. The S3 Tables copy is not downstream storage you control separately. It is a SQL-queryable view over exactly the same retention window as the log group. Set 30 days on the log group and you have 30 days of Iceberg data. Not 30 days hot plus an S3 tail going back a year. Thirty days, everywhere.

If your plan was short CloudWatch retention for cost with a long Iceberg history for compliance or trend analysis, that plan does not exist in this feature. Retention is one number and it governs both copies.

What the mirror actually is

Read correctly, the feature is not tiered storage. It is a second query interface over the same data, and that framing tells you exactly when it earns its place.

CloudWatch Logs Insights and Athena solve different problems. Insights is built for recent operational work: what happened in the last hour, why is this alarm firing, structured fields within seconds of delivery. It charges per GB scanned per query, and its query language is its own.

Athena over the Iceberg copy is built for retrospective analytical work: multi-week aggregations, joins across datasets, correlating access logs with CloudTrail in a single query, chargeback attribution by requester and operation. It is standard SQL, it is partitioned by day for partition pruning, and it charges roughly $5 per TB scanned.

The mirror gives you both interfaces over one ingestion. You are not choosing between operational and analytical access. You are paying to ingest once and reading through whichever engine fits the question. That is the actual value, and it is a real one. It is just not the value of an archive.

The pricing that makes it attractive, and the pricing that bites

The reason this feature reads as a free win is that the expensive parts of running Iceberg yourself are waived.

Stand up your own Iceberg log lake and you pay for S3 Tables storage, per-object monitoring, and compaction. Concretely, from the S3 Tables pricing page, us-east-1: storage runs $0.0265 per GB per month for the first 50 TB. Object monitoring is $0.025 per 1,000 objects. Compaction is $0.002 per 1,000 objects plus $0.005 per GB processed. A 1 TB table taking 30,000 new 5 MB files a month lands around $28 per month all in, and the compaction share of that grows with write churn, which for log ingestion is relentless.

With the CloudWatch mirror, all of that is zero. Storage, monitoring, compaction, snapshot management: included. What you actually pay is two things.

First, the CloudWatch Logs ingestion you were already paying, unchanged. The mirror does not add an ingestion charge, but it does not reduce your existing one either. This is not a way to lower a CloudWatch bill. It is a way to get analytical access without standing up a pipeline.

Second, the Athena query charges. At $5 per TB scanned this is where an unwary team gets hurt, and the trap is not the Athena line item. It is the S3 GET charges underneath it. A dataset of a million small files scans for $5 in Athena and simultaneously costs $0.40 in GET requests, and if you run that query a hundred times a day the GET charges compound to over a thousand dollars a month independent of compression. The managed compaction on the mirror works in your favor here, since AWS is merging small files for you, but the discipline still holds: bound every query on the partition column, and do not point a dashboard that refreshes every minute at a full table scan.

Getting it running

The integration is configured once per account per Region, then every server access log delivery in that account and Region flows to S3 Tables automatically. The observabilityadmin command group is recent, so update to the latest AWS CLI first or the subcommand will not resolve. Enable it on an existing delivery with the observability admin API:

aws observabilityadmin create-s3-table-integration \
  --role-arn arn:aws:iam::123456789012:role/CWLogsS3TableIntegrationRole \
  --encryption '{"SseAlgorithm":"AES256"}' \
  --region us-east-1
Enter fullscreen mode Exit fullscreen mode

Encryption is required. Use AES256 for SSE-S3, or pass aws:kms with a KmsKeyArn to hold the table data under your own key. If the log group is KMS-encrypted, the systemtables.cloudwatch.amazonaws.com and maintenance.s3tables.amazonaws.com service principals need access to that key, which is the step most first setups miss.

The data lands in a managed table bucket named aws-cloudwatch, under the logs namespace. For S3 server access logs the fully qualified name is:

"s3tablescatalog/aws-cloudwatch"."logs"."amazon_s3__server_access"
Enter fullscreen mode Exit fullscreen mode

The table is partitioned by day on cwl__timestamp, the moment the record was created, which is always at or after the event's own request_time. That distinction is not pedantic, it is a correctness requirement for every query you write.

The query pattern that is a correctness issue, not an optimization

Partition pruning on this table has a sharp edge, and getting it wrong silently drops rows rather than throwing an error.

You filter your data on request_time because that is the business timestamp you care about. But the table is partitioned on cwl__timestamp. To prune partitions you must also bound cwl__timestamp, and here is the rule: put a lower bound on cwl__timestamp that matches your window, and never an upper bound.

WHERE request_time    > current_timestamp - interval '30' day
  AND cwl__timestamp  > current_timestamp - interval '30' day
Enter fullscreen mode Exit fullscreen mode

The lower bound is safe because cwl__timestamp is always at or after request_time, so it cannot drop a row that is inside your request_time window. An upper bound is not safe. Records delivered late get a cwl__timestamp after their request_time, and capping the partition column throws them away. You get a query that looks correct, runs fast, and quietly under-reports. Lower bound to prune, never upper bound, is the pattern to internalize.

When to enable it, and when not to

Enable it if you have retrospective analytical questions over your recent log window that Logs Insights answers awkwardly: chargeback by requester and operation, week-over-week pattern comparison, joins between access logs and CloudTrail. You get SQL over data you are already ingesting, with the operational cost of running Iceberg absorbed by AWS.

Do not enable it expecting a cheaper CloudWatch bill, because your ingestion charge is untouched. Do not enable it as a compliance archive, because it inherits the log group's retention and is not independent. And if genuine long-term retention is the requirement, the mirror is the wrong tool. Deliver to an S3 general purpose bucket in Parquet, or route through Amazon Data Firehose, which is the standard managed path for this pattern, then set your own lifecycle policy and own a copy whose lifetime you actually control.

The one-line test: the S3 Tables mirror is a query interface, not a storage tier. Reach for it when the question is analytical and the window is recent. Reach for something else when the requirement is that the data outlives the log group.

Top comments (0)