DEV Community

Cover image for AWS CDK 100 Drill Exercises #009: CloudWatch Logs S3 Archive — Comparing 3 Log Archive Patterns in One Repository
AWS Community Builders profile image TOMOAKI ishihara
TOMOAKI ishihara for AWS Community Builders

Posted on

AWS CDK 100 Drill Exercises #009: CloudWatch Logs S3 Archive — Comparing 3 Log Archive Patterns in One Repository

#aws #cdk #100drillexercises #cloudwatchlogs

Level 200

Introduction

This is the 9th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

Leave CloudWatch Logs alone and anything past its retention period simply disappears. When you need to keep logs around for audits or incident investigation, most teams eventually land on "archive it to S3" — but there's more than one way to get there.

This time we implement three different patterns for archiving CloudWatch Logs to S3 as five independent CDK stacks, and compare the latency, cost, and operational-simplicity trade-offs with concrete code.

What you'll learn in this article

  • Near-real-time log archiving with Kinesis Data Firehose (via a subscription filter)
  • Tiered storage-class transitions with S3 lifecycle rules (Standard → IA → Glacier IR → Deep Archive)
  • How to bolt archiving onto an existing CloudWatch Logs log group after the fact
  • Scheduled batch export with the CloudWatch Logs Export Task API (and how to work around the one-concurrent-execution limit)
  • Decoding the CWL payload in a Lambda subscription filter and writing it to S3 in a custom format
  • How the required IAM trust relationships and bucket policies differ between patterns for the CloudWatch Logs service

📁 Code repository: GitHub

Architecture Overview

Architecture Overview

Pattern Approach Stacks
A: Firehose Subscription filter → Kinesis Data Firehose → S3 Basic / Lifecycle / Existing (3 stacks)
B: Export Task EventBridge Scheduler → Lambda → Export Task API → S3 Export (1 stack)
C: Direct Lambda write Subscription filter → Lambda → S3 Lambda (1 stack)

Why three patterns

Trait Pattern A (Firehose) Pattern B (Export Task) Pattern C (Lambda)
Delivery latency Near real-time (seconds to minutes) Scheduled (tolerates up to ~12h of delay) Near real-time
Cost at high volume Moderate (Firehose + S3) Low (Lambda + S3 only) Moderate (scales with Lambda invocation count)
Output format Raw CWL JSON lines, GZIP-compressed Raw CWL format Custom JSON (fully controllable)
Operational simplicity High (managed Firehose) High (managed Export API) Low (Lambda code needs maintenance)
Custom transformation Limited (depends on Firehose's data transformation feature) None Fully controllable inside Lambda

In short: A if you need near-real-time delivery, B if a cost-optimized batch job is good enough, C if you want to define the output format yourself.

Data Flow 

Pattern A (Stacks 1-3)
CloudWatch Log Group
  → Subscription filter (SubscriptionFilter + FirehoseDestination, role = CwlToFirehoseRole)
  → Kinesis Data Firehose (GZIP compression)
  → S3 archive bucket
  * Stack 1 (Basic) shares 5 log groups and uses dynamic partitioning on owner/logGroup
    to route each log group to its own S3 prefix. Stack 2 (Lifecycle) uses a single log
    group with a plain date prefix and no dynamic partitioning.

Pattern B (Stack 4)
EventBridge Scheduler
  → Lambda (calls logs:CreateExportTask for the previous day's logs)
  → CloudWatch Logs Export Task API
  → S3 archive bucket (bucket policy allows writes from logs.amazonaws.com)

Pattern C (Stack 5)
CloudWatch Log Group
  → Subscription filter (LambdaDestination, CDK manages invoke permissions automatically)
  → Lambda (decodes the gzip+base64 CWL payload and writes JSON to S3)
  → S3 archive bucket
Enter fullscreen mode Exit fullscreen mode

Key Components

Component Design point
S3 archive bucket SSE-S3 encryption, versioning enabled, public access blocked, SSL enforced. Provisioned independently per stack (5 buckets total)
CwlToFirehoseRole Assumed by logs.amazonaws.com. Scoped to log groups via an aws:SourceArn condition
FirehoseRole Assumed by firehose.amazonaws.com. Grants the standard Firehose→S3 permission set (PutObject, GetObject, ListBucket, multipart upload actions) scoped to the archive bucket
Dynamic partitioning (Stack 1) jq extracts owner/logGroup to route 5 shared log groups into separate S3 prefixes. CloudWatchLogProcessor can't be used because it strips those fields; a jq fallback is required to handle CONTROL_MESSAGE records
Tiered lifecycle rules (Stack 2) Standard → IA (30 days) → Glacier IR (90 days) → Deep Archive (365 days) → expiration (7 years)
Importing an existing log group (Stack 3) Referenced via LogGroup.fromLogGroupName(). No new AWS::Logs::LogGroup resource is created
Export Task Lambda (Stack 4) A concurrency guard that checks for running/pending tasks before calling CreateExportTask
Direct Lambda write (Stack 5) LambdaDestination automatically manages invoke permissions. The payload is decoded and saved as custom JSON

Implementation Highlights

1. Pattern A — Kinesis Data Firehose subscription

For CloudWatch Logs to deliver to Firehose, it needs an explicit IAM role. We split responsibilities across two roles.

// CwlToFirehoseRole: assumed by logs.amazonaws.com. Only the trust policy is defined here
const cwlToFirehoseRole = new iam.Role(this, 'CwlToFirehoseRole', {
    assumedBy: new iam.ServicePrincipal('logs.amazonaws.com', {
        conditions: {
            StringLike: {
                'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*`,
            },
        },
    }),
});

// L2 SubscriptionFilter + FirehoseDestination(role: cwlToFirehoseRole)
const filterPattern = logGroupParams.filterPattern ?? defaultLogGroupArchiveConfig.filterPattern;
new logs.SubscriptionFilter(this, 'CwlSubscriptionFilter', {
    logGroup: this.logGroup,
    destination: new logs_destinations.FirehoseDestination(this.deliveryStream, {
        role: cwlToFirehoseRole,
    }),
    filterPattern: filterPattern
        ? logs.FilterPattern.literal(filterPattern)
        : logs.FilterPattern.allEvents(),
});
Enter fullscreen mode Exit fullscreen mode

Why not attach firehose:PutRecord/PutRecordBatch permissions directly on the role?
Inside bind(), aws-logs-destinations's FirehoseDestination calls deliveryStream.grantPutRecords(role) on the role you pass in (or one it generates), automatically granting the necessary permissions. If you also add your own inline policy to the role, you end up with two AWS::IAM::Policy resources granting the same permission. The trick is to define only the trust policy yourself (scoped by SourceArn) and let the L2 construct handle the permission grant.

I used to write this with the L1 CfnSubscriptionFilter plus a hand-rolled inline policy, on the assumption that "the L2 SubscriptionFilter can't accept a roleArn for Firehose." That assumption was wrong. In the current aws-cdk-lib (2.261.0 in this repository), FirehoseDestination wires the role's ARN all the way through to CfnSubscriptionFilter for you, so the L2 construct alone is sufficient.

The aws:SourceArn condition scopes who can assume this role down to "log groups within this account/region." Combined with scoping the access policy side to just this specific Firehose delivery stream, permissions are minimized in two independent layers.

1a. Per-log-group dynamic partitioning (Stack 1 – Basic)

Stack 1 creates 5 log groups that share the same Firehose delivery stream. Without partitioning, events from all five would end up mixed together in the same S3 object, so dynamic partitioning routes each log group to its own prefix.

const s3Destination = new firehose.S3Bucket(archiveBucket, {
    dataOutputPrefix:
        'AWSLogs/!{partitionKeyFromQuery:owner}/CWLogGroup/!{partitionKeyFromQuery:logGroup}/!{timestamp:yyyy/MM/dd/HH}/',
    dynamicPartitioning: { enabled: true },
    bufferingInterval: cdk.Duration.seconds(60), // dynamic partitioning requires >= 60s
    bufferingSize: cdk.Size.mebibytes(64),        // and >= 64MiB
    processors: [
        new firehose.DecompressionProcessor({
            compressionFormat: firehose.DecompressionProcessorCompressionFormat.GZIP,
        }),
        firehose.MetadataExtractionProcessor.jq16({
            owner: '(if (.owner // "") == "" then "controlmessages" else .owner end)',
            logGroup:
                '(if (.logGroup // "") == "" then "controlmessages" else (.logGroup | ltrimstr("/")) end)',
        }),
        new firehose.AppendDelimiterToRecordProcessor(),
    ],
});
Enter fullscreen mode Exit fullscreen mode

I hit two snags here.

First: don't add the CloudWatchLogProcessor (message extraction). Firehose has a "message extraction" feature that pulls just the contents of message out of a decompressed CWL record — but enabling it wipes out owner/logGroup/logStream entirely, leaving only the raw contents of message. With those gone, the downstream MetadataExtractionProcessor fails with DynamicPartitioning.MetadataExtractionFailed when it tries to reference .owner/.logGroup via jq. Right after DecompressionProcessor, the record still has owner/logGroup at the top level (see Fig. 1 of Firehose's "message extraction" documentation), so jq can read them directly without a message extraction processor.

Second: handling the CONTROL_MESSAGE records CloudWatch Logs periodically sends as a health check. These records have owner/logGroup set to the empty string "", so a naive .logGroup | ltrimstr("/") returns the empty string as-is, which fails with partitionKeys values must not be null or empty. You need a jq if/then/else to supply a fallback value ("controlmessages"). It's tempting to reach for // (the alternative operator) here, but // only substitutes on null/false, not on an empty string.

The trade-off is that each record written to S3 is the entire CWL envelope (with a logEvents array that may contain multiple events), not a flat one-row-per-log-event line. Flattening (CloudWatchLogProcessor) is incompatible with metadata-based partitioning. If you need individual messages, UNNEST/json_extract the logEvents array in something like Athena.

2. Tiered S3 lifecycle (Stack 2 – Lifecycle)

For long-term archives, you can optimize cost by moving objects to progressively cheaper storage classes as access frequency drops.

archiveBucket.addLifecycleRule({
    id: `MoveToIAAfter${moveToIaAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.INFREQUENT_ACCESS, transitionAfter: cdk.Duration.days(moveToIaAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `MoveToGlacierAfter${moveToGlacierAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.GLACIER_INSTANT_RETRIEVAL, transitionAfter: cdk.Duration.days(moveToGlacierAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `MoveToDeepArchiveAfter${moveToDeepArchiveAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.DEEP_ARCHIVE, transitionAfter: cdk.Duration.days(moveToDeepArchiveAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `ExpireCurrentObjectsAfter${expireAfterDays}Days`,
    enabled: true,
    expiration: cdk.Duration.days(expireAfterDays),
});
Enter fullscreen mode Exit fullscreen mode

Non-current versions (older objects overwritten while versioning is enabled) get their own IA transition and expiration rule, and incomplete multipart uploads are aborted after 7 days. Rather than bundling everything into one giant lifecycle configuration, each transition gets its own addLifecycleRule so parameters like moveToIaAfterDays can be tuned per environment independently.

3. Attaching to an existing log group (Stack 3 – Existing)

Sometimes you don't want to create a new log group — you want to bolt archiving onto a log group an application is already writing to.

const existingLogGroup = logs.LogGroup.fromLogGroupName(
    this,
    'ExistingLogGroup',
    existingLogGroupParams.logGroupName,
);
Enter fullscreen mode Exit fullscreen mode

fromLogGroupName() only references the log group in CDK — it doesn't create an AWS::Logs::LogGroup resource. Since you can pass this existing log group's ARN (existingLogGroup.logGroupArn) straight into the trust policy's aws:SourceArn condition, the subscription filter and IAM role wiring can reuse the exact same code as Stacks 1 and 2.

4. Pattern B — Scheduled Export Task

The CloudWatch Logs Export Task API has a one concurrent execution per account limit. Before submitting a new task, the Lambda checks for running or pending tasks to avoid errors.

# export-task/index.py (core logic)
def lambda_handler(event, context):
    logs_client = boto3.client("logs")

    # Skip if a task is already running or pending
    running = logs_client.describe_export_tasks(statusCode="RUNNING")
    pending = logs_client.describe_export_tasks(statusCode="PENDING")
    if running["exportTasks"] or pending["exportTasks"]:
        print("An export task is already running or pending. Skipping.")
        return {"status": "skipped", "reason": "task_in_progress"}

    # Export the previous day's (UTC) logs
    yesterday = datetime.datetime.utcnow() - datetime.timedelta(days=1)
    from_time = int(yesterday.replace(hour=0, minute=0, second=0, microsecond=0).timestamp() * 1000)
    to_time = int(yesterday.replace(hour=23, minute=59, second=59, microsecond=999999).timestamp() * 1000)

    response = logs_client.create_export_task(
        taskName=f"export-{yesterday.strftime('%Y-%m-%d')}",
        logGroupName=LOG_GROUP_NAME,
        fromTime=from_time,
        to=to_time,
        destination=S3_BUCKET_NAME,
        destinationPrefix=f"{S3_PREFIX}/{yesterday.strftime('%Y/%m/%d')}",
    )
    return {"status": "created", "taskId": response["taskId"]}
Enter fullscreen mode Exit fullscreen mode

The key detail is checking PENDING in addition to RUNNING. If a scheduled task is merely sitting in the queue and you submit the next one anyway, you get a LimitExceededException.

The S3 bucket side needs two resource policies allowing writes from the CloudWatch Logs service principal.

// 1. Permission to check the bucket ACL before exporting
archiveBucket.addToResourcePolicy(
    new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        principals: [new iam.ServicePrincipal('logs.amazonaws.com')],
        actions: ['s3:GetBucketAcl'],
        resources: [archiveBucket.bucketArn],
        conditions: {
            ArnLike: { 'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*` },
        },
    }),
);

// 2. Permission to write the exported objects
archiveBucket.addToResourcePolicy(
    new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        principals: [new iam.ServicePrincipal('logs.amazonaws.com')],
        actions: ['s3:PutObject'],
        resources: [archiveBucket.arnForObjects('*')],
        conditions: {
            StringEquals: { 's3:x-amz-acl': 'bucket-owner-full-control' },
            ArnLike: { 'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*` },
        },
    }),
);
Enter fullscreen mode Exit fullscreen mode

The condition pinning s3:x-amz-acl to bucket-owner-full-control is explicitly required by CloudWatch Logs' official documentation. Omit it and the export task fails with a permissions error.

5. Pattern C — Subscription filter → Lambda → S3

With LambdaDestination, CDK automatically manages the resource policy (Lambda::Permission) that lets logs.amazonaws.com invoke your Lambda.

new logs.SubscriptionFilter(this, 'CwlSubscriptionFilter', {
    logGroup: this.logGroup,
    destination: new logDestinations.LambdaDestination(this.archiveFunction),
    filterPattern: filterPattern
        ? logs.FilterPattern.literal(filterPattern)
        : logs.FilterPattern.allEvents(),
});
Enter fullscreen mode Exit fullscreen mode

On the Lambda side, the code decodes the CWL compressed payload and writes it to S3 in a JSON format it defines itself, per invocation.

# cwl-to-s3/index.py (core logic)
def lambda_handler(event, context):
    compressed = base64.b64decode(event["awslogs"]["data"])
    payload = json.loads(gzip.decompress(compressed).decode("utf-8"))

    if payload.get("messageType") == "CONTROL_MESSAGE":
        # Health-check message CloudWatch Logs sends, e.g. when a subscription starts
        return {"status": "skipped", "reason": "control_message"}

    now = datetime.datetime.utcnow()
    safe_stream = payload["logStream"].replace("/", "_").replace("$", "").replace("[", "").replace("]", "")
    s3_key = f"{S3_PREFIX}/{now.strftime('%Y/%m/%d/%H')}/{safe_stream}_{uuid.uuid4().hex[:8]}.json"

    record = {
        "logGroup": payload["logGroup"],
        "logStream": payload["logStream"],
        "owner": payload.get("owner"),
        "exportedAt": now.isoformat() + "Z",
        "events": payload["logEvents"],
    }
    s3_client.put_object(Bucket=S3_BUCKET_NAME, Key=s3_key, Body=json.dumps(record, ensure_ascii=False).encode("utf-8"))
    return {"status": "ok", "processed": len(payload["logEvents"]), "s3Key": s3_key}
Enter fullscreen mode Exit fullscreen mode

Note the early skip for payloads with messageType == "CONTROL_MESSAGE". CloudWatch Logs sends these health-check control messages right after a subscription starts (among other times), and they have no logEvents key at all — without this check, the invocation fails with a KeyError.

Both patterns A and C deliver via a subscription filter for near-immediate delivery, but with A you can let Firehose handle the transformation, while with C you write the Lambda code yourself and get complete control over the output format in exchange.

Deploy & Verify 

export PROJECT=your-project
export ENV=dev

npm run bootstrap -w workspaces/cloudwatch-logs-s3-archive   # first time only
npm run stage:deploy:all -w workspaces/cloudwatch-logs-s3-archive
Enter fullscreen mode Exit fullscreen mode

To deploy individual stacks, use the per-stack stage:deploy:* scripts (each scoped to a cdk deploy stack selector like **/*Basic).

npm run stage:deploy:basic -w workspaces/cloudwatch-logs-s3-archive
npm run stage:deploy:lifecycle -w workspaces/cloudwatch-logs-s3-archive
npm run stage:deploy:export -w workspaces/cloudwatch-logs-s3-archive
Enter fullscreen mode Exit fullscreen mode 
# Pattern A Stack 1 (Basic): write test data into 5 log groups (/<project>/<env>/basic-*)
# (given the dynamic-partitioning buffering requirements of >= 60s / >= 64MiB, use the
#  repo's helper script rather than a one-off put-log-events call)
./write-test-logs.sh --project <project> --env <env>

# After ~60s, output appears partitioned under AWSLogs/<owner>/CWLogGroup/<logGroup>/...
aws s3 ls s3://<archive-bucket>/AWSLogs/ --recursive

# Pattern A Stack 2 (Lifecycle): single log group, no dynamic partitioning
./write-test-logs-lifecycle.sh --project <project> --env <env>
aws s3 ls s3://<archive-bucket>/ --recursive

# Pattern B: manually invoke the export Lambda
aws lambda invoke --function-name <project>-<env>-cwl-export-task --payload '{}' response.json
cat response.json

# Pattern C: write a test log entry (the subscription filter invokes the Lambda immediately)
aws logs put-log-events \
  --log-group-name /<project>/<env>/app-lambda \
  --log-stream-name test-stream \
  --log-events timestamp=$(date +%s000),message="hello lambda"
aws s3 ls s3://<archive-bucket>/subscriptions/ --recursive
Enter fullscreen mode Exit fullscreen mode

Cost Estimate

💰 Rough monthly estimate (Tokyo region, 1 GB of log data per day)

Service Applies to Rough monthly cost
Kinesis Data Firehose A ~$0.90 (at $0.03/GB)
Lambda B, C Effectively negligible at low invocation counts
S3 (Standard) All ~$0.75 (at $0.025/GB-month)
CloudWatch Logs (ingestion) All ~$0.76/GB (charged regardless of which archive pattern you use, and not included in the total below)
EventBridge B $1.00 per million events (negligible at minimal frequency)

Rough total for Pattern A's archiving cost alone: about $2/month at 1 GB/day of log volume. CloudWatch Logs ingestion itself (~$23/month at 1 GB/day) is a separate cost you pay upfront, before any archive pattern runs

Running all five stacks simultaneously at production-scale traffic for comparison testing will cost more than the above. Don't forget to clean up afterward with npm run stage:destroy:all -w workspaces/cloudwatch-logs-s3-archive.

Summary

What we learned from this pattern:

  1. Pattern A (Firehose): Minimal code, well-suited for near-real-time archiving. Delivery from CloudWatch Logs requires FirehoseDestination (L2) plus two IAM roles with distinct responsibilities
  2. Pattern B (Export Task): The lowest cost for batch use cases, but limited to one concurrent execution per account. A guard checking for running/pending tasks on the Lambda side is essential
  3. Pattern C (Lambda): Full control over the output format, at the cost of ongoing Lambda code maintenance, including handling control messages
  4. IAM trust relationships differ by service: For Firehose, FirehoseDestination grants permission to the trusted role via grantPutRecords; for Export Task, an S3 bucket policy with an s3:x-amz-acl condition is required; for LambdaDestination, CDK auto-generates the invoke permission — the same "delivery from CloudWatch Logs" concept wires up differently every time
  5. Split tiered lifecycle rules by transition: rather than one giant rule, separate rules make the transitionAfter thresholds easy to tune independently as environment parameters
  6. fromLogGroupName() is enough for bolting archiving onto an existing log group: pass the existing ARN into the trust policy directly and reuse the same wiring, with no new resource created
  7. Firehose dynamic partitioning is incompatible with message extraction: CloudWatchLogProcessor strips owner/logGroup, so drop it if you're using jq to derive partition keys from those fields. You also need to account for CloudWatch Logs' CONTROL_MESSAGE records (where owner/logGroup are empty strings) with a jq if/then/else fallback (// doesn't catch empty strings)

References

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

Top comments (0)