DEV Community: TOMOAKI ishihara

AWS CDK 100 Drill Exercises #013: CloudFront S3 Static Website — Serving a Private Bucket over HTTPS with OAC and Cross-Region WAF

TOMOAKI ishihara — Sat, 29 Aug 2026 07:27:51 +0000

Introduction

This is the 13th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

Last time (#012), we covered publishing a site using S3 static website hosting alone. That setup had two clear limitations: no HTTPS support, and the bucket itself sitting open to Principal * (albeit IP-restricted).

This time we rebuild the same static site behind Amazon CloudFront, with a completely private S3 bucket. The bucket can never be read directly from the outside again, traffic is served as HTTPS over TLS 1.3, and you can optionally add defense-in-depth with WAFv2.

What you'll learn in this article

How to serve from CloudFront without ever making the S3 bucket public, using Origin Access Control (OAC)
Applying security headers like CSP and HSTS via a ResponseHeadersPolicy, and hiding the Server header
A design that rewrites 403/404 to index.html for SPA client-side routing, while showing genuine 5xx failures with their real status codes intact
How to handle the constraint that CLOUDFRONT-scoped WAFv2 is pinned to us-east-1 using crossRegionReferences
A two-stage IP allow-list design: "before managed rule evaluation" and "after managed rule evaluation"

📁 Code repository: GitHub

Architecture Overview

Trait	Benefit
Private bucket + OAC	Unlike S3 website hosting, the bucket itself is never publicly reachable. Only CloudFront's OAC ID can read it
Cross-region WAF stack	The Web ACL and its related resources live in a stack pinned to us-east-1, with only the ARN passed to the main stack via `crossRegionReferences: true`
Error mapping that doesn't hide real failures	`403`/`404` go to the SPA's `index.html`; `500`-`504` keep their original status codes and show a dedicated error page
Single-flag toggles	`enableWaf` and `geoRestrictionCountries` can each be turned on/off with one optional parameter

Data Flow

Viewer
  │  HTTPS (min TLS 1.3), optional geo-restriction
  ▼
CloudFront Distribution
  ├─ WAFv2 Web ACL (optional, us-east-1): managed rule groups + IP allow-list → default action: block
  ├─ ResponseHeadersPolicy: applies CSP + security headers to every response
  ├─ Default behavior "/*"
  │     └─ S3 origin (OAC) ───────────────────► WebsiteBucket (private)
  │
  └─ Error handling
        ├─ 403 / 404  → /index.html, HTTP 200, TTL 5 min
        └─ 500-504    → /error.html, original status code preserved, TTL 1 min

Implementation Highlights

1. Private bucket + OAC — the polar opposite of "who can read this"

In the previous S3 website hosting version, the bucket was (conditionally) open to Principal: *. This time it's the exact opposite: the bucket itself is completely private.

const websiteBucket = createAccountRegionalBucket({
  scope: this, id: 'WebsiteBucket',
  // no websiteIndexDocument or similar settings at all
});

const distribution = new cloudfront.Distribution(this, 'WebsiteDistribution', {
  defaultBehavior: {
    origin: cloudfrontOrigins.S3BucketOrigin.withOriginAccessControl(websiteBucket),
    viewerProtocolPolicy: cloudfront.ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
  },
});

S3BucketOrigin.withOriginAccessControl() handles both creating the OAC and adding a bucket policy scoped to the distribution's ARN in a single line. Nobody can read the bucket on its own — the only relationship that grants read access is a resource-level one with "this specific CloudFront distribution."

2. Bulk security headers via `ResponseHeadersPolicy`

CloudFront's ResponseHeadersPolicy can inject response headers regardless of how the origin is implemented. On top of CSP, HSTS, and X-Frame-Options, we overwrite the Server header with an empty string to hide server information.

securityHeadersBehavior: {
  contentSecurityPolicy: {
    contentSecurityPolicy: "default-src 'self'; script-src 'self'; ...",
    override: true,
  },
  strictTransportSecurity: {
    accessControlMaxAge: cdk.Duration.days(365),
    includeSubdomains: true,
    preload: true,
    override: true,
  },
  // ...
},
customHeadersBehavior: {
  customHeaders: [{ header: 'server', value: '', override: true }],
},

Even though this setup only serves static HTML, the CSP explicitly blocking anything beyond self-hosted scripts/styles — and not allowing unsafe-inline — sets a useful design precedent for whenever a JS framework gets layered on top later.

3. Gentle with the SPA, honest about real failures

CloudFront's custom error responses rewrite both 403 and 404 to index.html + 200. This ensures that when a path that doesn't exist as an S3 object (an SPA client-side route) is accessed directly, control is correctly handed off to the app's own routing.

Genuine origin/edge failures like 500-504, on the other hand, keep their original status code and only swap in error.html for the body.

errorResponses: [
  { httpStatus: 403, responseHttpStatus: 200, responsePagePath: '/index.html', ttl: cdk.Duration.minutes(5) },
  { httpStatus: 404, responseHttpStatus: 200, responsePagePath: '/index.html', ttl: cdk.Duration.minutes(5) },
  // 5xx are genuine failures, not SPA routing concerns, so keep the status code as-is
  // and only replace the body, so origin error details aren't leaked
  ...[500, 502, 503, 504].map((httpStatus) => ({
    httpStatus, responseHttpStatus: httpStatus, responsePagePath: '/error.html', ttl: cdk.Duration.minutes(1),
  })),
],

You'll often see implementations that just turn everything into a 200 to paper over errors, but doing that buries genuine failures under a 200 from a monitoring perspective. Here, 403/404 are treated as SPA-specific, while 5xx stays honest — the two roles are kept separate.

4. CLOUDFRONT-scoped WAFv2 is pinned to us-east-1 — wire it with `crossRegionReferences`

When you specify scope: 'CLOUDFRONT' for a WAFv2 Web ACL, that Web ACL can only be created in us-east-1, regardless of which region the distribution itself lives in. Try creating it elsewhere and you get rejected with The scope is not valid.

In this reference implementation, all WAF-related resources are factored out into a separate stack (CloudfrontWafStack) that always deploys to us-east-1. Only its ARN is passed to the main stack, using crossRegionReferences: true set on both stacks.

const wafStack = new CloudfrontWafStack(this, pascalCase(`${props.project}Waf`), {
  env: { account: props.params.accountId, region: 'us-east-1' },
  crossRegionReferences: true,
  enableWaf: props.params.enableWaf,
  allowedIpsAfterRules: props.allowedIps,
});

const mainStack = new CloudfrontS3StaticWebsiteStack(this, pascalCase(`${props.project}Main`), {
  webAclArn: wafStack.webAclArn,
  crossRegionReferences: true,
});
mainStack.addStackDependency(wafStack);

Even when enableWaf: false (or unset), CloudfrontWafStack itself is still deployed — it just doesn't create a Web ACL, and webAclArn becomes an empty string, so the distribution deploys without a webAclId. By designing this as "don't create the Web ACL" rather than "remove the whole stack when WAF isn't needed," toggling enableWaf on and off later never breaks the cross-region reference wiring itself.

5. Allow-listing before or after the managed rules

The Web ACL's defaultAction is block. That means nothing gets through unless there's an explicit Allow rule. This stack has a two-stage IP allow list.

AllowSpecificIPsBeforeRules (priority 1, created only when configured): bypasses evaluation of the AWS managed rule groups (Common/KnownBadInputs/AdminProtection/IpReputation/AnonymousIp) entirely
AllowSpecificIPsAfterRules (priority 100, always created): evaluated after the managed rule groups. When no IPs are specified, it defaults to allowing the entire IPv4/IPv6 address space (the managed rules still apply)

addresses: props.allowedIpsAfterRules
  ? props.allowedIpsAfterRules.map(ip => `${ip}/32`)
  : ['0.0.0.0/1', '128.0.0.0/1'], // WAF rejects /0, so the full IPv4 range is split into two /1s

In the default wiring, the deploy operator's IP is passed as allowedIpsAfterRules. That means even allow-listed IPs first get inspected for known attack patterns by rules like Core/KnownBadInputs before being let through. Use allowedIpsBeforeRules only for the exceptional case where you want to bypass managed-rule false positives during active testing.

6. WAF logs go directly to S3, with sensitive headers masked

To deliver WAF logs directly to S3, the bucket name must start with aws-waf-logs- (otherwise creating the CfnLoggingConfiguration itself is rejected). On top of that, authorization and cookie headers are masked before being logged.

redactedFields: [
  { singleHeader: { Name: 'authorization' } },
  { singleHeader: { Name: 'cookie' } },
],

Deploy & Verify

export PROJECT=your-project
export ENV=dev

npm run bootstrap   # first time only
# Also bootstrap us-east-1 once — the WAF stack deploys there and crossRegionReferences
# needs that region bootstrapped as well:
#   npx cdk bootstrap aws://<account-id>/us-east-1 --profile $PROJECT-$ENV

npm run stage:deploy:all

curl https://<output WebsiteDistributionDomainName>/

The reference dev config ships with enableWaf: true and geoRestrictionCountries: ['JP'], so a straight deploy gives you a WAF-protected, Japan-only distribution. Adjust parameters/dev-params.ts to change either.

Access from an IP not in the WAF's allow list, and WAF returns a 403 before the request ever reaches CloudFront's own error page.

Cost Estimate

💰 Rough monthly estimate (Tokyo region, low traffic)

Service	Usage	Rough monthly cost
CloudFront	Low request volume, minimal data transfer	~$1-2
AWS WAF	Web ACL ($5) + 6 rules ($1 each), low request volume	~$11
S3 (2 buckets)	A few MB of content + access/WAF logs	Under $0.10

Total (WAF enabled): ~$12-14/month
Total (enableWaf: false): ~$1-2/month

WAF is billed per Web ACL, per rule, and per evaluated request, making it the dominant cost factor at low traffic volumes. For pure demo use where IP/geo restriction isn't needed, enableWaf: false significantly lowers the cost.

Summary

What we learned from this pattern:

Origin Access Control lets you serve from CloudFront without ever making the S3 bucket public. The bucket alone is unreadable by anyone — the polar opposite of last time's setup
ResponseHeadersPolicy lets you apply security headers in bulk, regardless of the origin's implementation. Even for a static site, designing the CSP to explicitly deny anything beyond self-hosted resources holds up well for future expansion
Don't conflate error responses for SPA routing rewrites with visibility into genuine failures. 403/404 become 200; 5xx keeps its status code — both are "show a friendly page," but they mean very different things
CLOUDFRONT-scoped WAFv2 is pinned to us-east-1. This constraint applies regardless of which region the distribution itself lives in, and it never goes away. Separating the feature on/off toggle from the region-constraint wiring via crossRegionReferences: true makes future changes more resilient
defaultAction: block pairs naturally with a design where even allow-listed IPs still get inspected for attack patterns once. Allowing before managed-rule evaluation should stay the exception, not the norm

References

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS CDK 100 Drill Exercises #012: S3 Static Web Site — Getting a "Public" Bucket Policy Past Block Public Access

TOMOAKI ishihara — Sat, 29 Aug 2026 07:27:43 +0000

Introduction

This is the 12th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

This time we build the simplest way to publish a static site on AWS — S3 static website hosting — with no CDN, no WAF, and no custom domain. It looks like a plain, unremarkable pattern, but implementing it surfaces one genuinely interesting question.

"A bucket policy that allows s3:GetObject to Principal: * should count as 'public' — so why does deploying it against a bucket with BlockPublicAccess.BLOCK_ALL enabled succeed?"

The short answer: S3's Block Public Access doesn't decide public vs. private purely based on whether Principal is *. This time we verify that behavior with actual CDK code.

What you'll learn in this article

The basics of S3 static website hosting (websiteIndexDocument/websiteErrorDocument)
How the aws:SourceIp condition exempts a "public" policy from S3's Block Public Access evaluation
Why enforceSSL can't be enabled, because the S3 website endpoint doesn't support HTTPS
How to auto-detect the deploy operator's own global IP so the site is viewable immediately after deployment
The limits of this pattern — what you can and can't do without CloudFront

📁 Code repository: GitHub

Architecture Overview

Trait	Benefit
No CloudFront, no WAF, no ACM certificate	The fastest, cheapest way to get a static site online. A solid baseline before moving on to `cloudfront-s3-static-website`, covered next time
An IP allow list that coexists with Block Public Access	Lets you actually verify that a `Principal: *` policy scoped with an `aws:SourceIp` condition is exempted from S3's "public" determination
Automatic detection of the operator's IP	Detects the deploying machine's own IP via `curl`, so the site is viewable right after deployment with no manual configuration

Data Flow

Browser
  │  HTTP (the S3 website endpoint doesn't support HTTPS)
  ▼
S3 bucket website endpoint (<bucket>.s3-website-<region>.amazonaws.com)
  │  Evaluated by bucket policy: source IP must match allowedIps/allowedIpv6s
  ▼
index.html / error.html

Every request is anonymous. The website endpoint doesn't support SigV4 authentication, so the only access control available is the bucket policy's aws:SourceIp condition.

Implementation Highlights

1. `websiteIndexDocument` alone doesn't finish the job — the tug-of-war with Block Public Access

Just enabling S3 static website hosting is only a few lines of CDK code.

new s3.Bucket(this, 'WebsiteBucket', {
  websiteIndexDocument: 'index.html',
  websiteErrorDocument: 'error.html',
});

But that alone leaves nobody able to view the site. New S3 buckets have public access blocked by default (at the account level), and this reference implementation also sets blockPublicAccess: BLOCK_ALL on the bucket explicitly (via the repo's shared bucket helper) — so every anonymous s3:GetObject gets denied. Simply turning Block Public Access off, though, risks unintentionally opening the bucket to the entire world.

Instead, the stack keeps blockPublicAccess: BLOCK_ALL in place and adds only a bucket policy scoped down by aws:SourceIp.

websiteBucket.addToResourcePolicy(new iam.PolicyStatement({
  effect: iam.Effect.ALLOW,
  principals: [new iam.AnyPrincipal()],
  actions: ['s3:GetObject'],
  resources: [websiteBucket.arnForObjects('*')],
  conditions: {
    IpAddress: {
      'aws:SourceIp': allowedIps.map(ip => `${ip}/32`),
    },
  },
}));

At first glance, this policy — which includes Principal: * — looks like it should get the PutBucketPolicy call itself rejected by Block Public Access's BlockPublicPolicy: true. In practice, though, the deployment succeeds.

Why: when S3's Block Public Access decides "is this policy public?", it looks at more than just whether Principal is * — it also inspects the Condition clause. To be judged non-public, a statement has to be scoped to a fixed value of one of a specific set of keys: aws:SourceIp, aws:SourceArn, aws:SourceVpc, aws:SourceVpce, aws:SourceOwner, aws:SourceAccount, aws:PrincipalOrgID, s3:DataAccessPointArn, s3:DataAccessPointAccount, or aws:userid. A Principal: * policy whose reachable callers are pinned down by an aws:SourceIp CIDR condition therefore isn't treated as "public." (One caveat: an aws:SourceIp range broader than /8 for IPv4 — e.g. 0.0.0.0/1 — is still considered public. The /32 host addresses this stack uses are well within bounds.)

This behavior also comes up frequently in the context of SCPs and IAM Access Analyzer, but actually deploying an IP-restricted bucket policy against a Block-Public-Access-enabled bucket in CDK and confirming it firsthand deepens the understanding considerably.

⚠️ Note: if you omit both allowedIps and allowedIpv6s, no bucket policy is added at all. blockPublicAccess: BLOCK_ALL still applies, so the result is a site that's "unreachable by anyone," not "publicly open." This stack has no "allow everyone if unspecified" fallback.

2. `enforceSSL: false` — a setting that looks wrong at first glance

Every other bucket in this repository follows a baseline policy of enforceSSL: true, rejecting HTTP connections. WebsiteBucket is the one exception, set to enforceSSL: false.

const bucket = new s3.Bucket(scope, id, {
  enforceSSL: false, // the website endpoint only supports HTTP
  websiteIndexDocument: 'index.html',
  websiteErrorDocument: 'error.html',
  // ...
});

The S3 static website hosting endpoint (*.s3-website-<region>.amazonaws.com) doesn't support HTTPS at all, to begin with. Set enforceSSL: true here, and the bucket policy would reject the very (plaintext HTTP) requests the website endpoint actually receives, breaking the site entirely.

In other words, "I want this over HTTPS" isn't something this bucket can solve on its own — it's solved by putting CloudFront in front of it. That's exactly the bridge to cloudfront-s3-static-website, covered in the next article.

3. Auto-detecting the deploy operator's IP so the site is instantly viewable

bin/s3-static-web-site.ts fetches the deploying machine's own global IP via curl and feeds it straight into the bucket policy's allow list.

const allowedIps = parseIpListEnv(process.env.ALLOWED_IPS) ?? [getMyGlobalIp()];
const allowedIpv6s = parseIpListEnv(process.env.ALLOWED_IPV6S) ?? (() => {
  const myIpv6 = getMyGlobalIpv6();
  return myIpv6 ? [myIpv6] : undefined;
})();

IPv6 detection (getMyGlobalIpv6()) returns undefined quietly when it can't be determined (e.g. in a devcontainer/CI environment with no IPv6 connectivity), while IPv4 detection (getMyGlobalIp()) throws an exception on failure. That means deploying from a fully offline machine fails unless you explicitly set the ALLOWED_IPS environment variable. The same override applies when the machine you're deploying from differs from the machine whose browser will view the site (e.g. deploying from a devcontainer but browsing from the host PC) — use ALLOWED_IPS/ALLOWED_IPV6S explicitly in that case too.

ALLOWED_IPS=203.0.113.10,203.0.113.20 \
ALLOWED_IPV6S=2001:db8::1 \
npm run stage:deploy:all

Deploy & Verify

export PROJECT=your-project
export ENV=dev

npm run bootstrap   # first time only
npm run stage:deploy:all

curl http://<output WebsiteBucketUrl>/

Access from an IP not on the allow list, and the S3 website endpoint returns an HTTP 403 (Access Denied) page.

Cost Estimate

💰 Rough monthly estimate (Tokyo region, low traffic)

Service	Usage	Rough monthly cost
S3 storage	A few MB of static content	Under $0.01
S3 requests	Low-volume GET	Under $0.01
S3 server access logs	Small volume	Under $0.01

Total: under $0.05/month

With no CloudFront, no WAF, and no compute in the picture at all, the cost is essentially just S3 storage and request charges.

Summary

What we learned from this pattern:

S3's Block Public Access evaluation doesn't decide public/private purely by whether Principal is *. Statements scoped by a specific set of condition keys that includes aws:SourceIp are exempted from that "public" determination
The S3 static website hosting endpoint is HTTP-only. Enabling enforceSSL breaks the site entirely — going HTTPS requires an additional layer such as CloudFront
The "fallback when unspecified" behavior isn't consistent across helpers. IPv6 detection quietly returns undefined on failure, while IPv4 detection throws. When using helper functions, design your error handling around this asymmetry rather than assuming both behave the same
An IP allow list is not a substitute for authentication. aws:SourceIp is only evaluated per request, so while it's useful as a restriction for demos or personal use, it's insufficient as access control tied to a specific individual

Next time, we'll evolve this same workspace into CloudFront + WAF + OAC to add HTTPS and a private origin, in cloudfront-s3-static-website.

References

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS CDK 100 Drill Exercises #011: A CodePipeline for CloudFront & S3 — and a self-reference bug

TOMOAKI ishihara — Sat, 29 Aug 2026 07:12:17 +0000

Introduction

This is the 11th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

This time, the pattern itself is a plain one: a CodePipeline that pulls a static site from CodeCommit, builds it with CodeBuild, uploads it to S3, and invalidates CloudFront. Nothing exotic. What made this one worth writing up is what happened when I actually wired the test suite up and ran it for the first time: the synthesized CloudFormation template came back with

Template is undeployable, these resources have a dependency cycle: ... -> PipelineXXXX -> PipelineXXXX

The pipeline was referencing itself.

What you'll learn in this article

A CodePipeline with no hand-written pipeline or build role — each pipeline action grants itself only the resource-scoped permissions it needs (the invoked Lambdas still carry their own execution-role policies)
Why "upload" and "delete stale files" are worth splitting into two separate pipeline stages instead of one aws s3 sync --delete
The CodePipeline continuation-token pattern for polling a long-running operation (CloudFront invalidation) without blocking a Lambda invocation
How a CDK token used inside a pipeline's own stage configuration can create a genuine CloudFormation self-reference cycle — and how to avoid it
Two more deploy-time landmines that only show up once you actually run Template.fromStack(): an S3 bucket name that isn't lowercase, and a notification rule with zero targets

📁 Code repository: GitHub

Architecture Overview

Feature	Benefit
No custom pipeline/build IAM roles	Every action grants itself exactly the resource-scoped permissions it needs when bound to the pipeline
Upload/cleanup split across stages	`S3DeployAction` uploads; a dedicated Lambda removes what's no longer in the build
Continuation-token invalidation	CloudFront invalidation status is polled via CodePipeline's own re-invocation mechanism, not a blocking wait
Opt-in approval + notifications	A single `approvalTopicArn` parameter turns on both a Manual Approval stage and pipeline-result notifications

Data Flow

CodeCommit (push to branch)
    │  EventBridge rule triggers the pipeline
    ▼
CodePipeline
  ├─ Source              : CodeCommitSourceAction → SourceOutput artifact
  ├─ Build (optional)    : CodeBuildAction (buildspec.yml) → BuildOutput artifact   ← enabled by envParams.enableBuild
  ├─ Approval (optional) : ManualApprovalAction, notifies approvalTopicArn
  ├─ Deploy              : S3DeployAction → upload the build to the target bucket
  ├─ Sync                : Lambda → delete target-bucket objects not present in the build
  └─ InvalidateCache     : Lambda → CloudFront CreateInvalidation, polled via continuation token

The Build stage is opt-in (envParams.enableBuild, off by default in the reference config). When it's disabled, the source artifact is deployed to S3 directly and the Deploy/Sync stages consume SourceOutput instead of BuildOutput.

Implementation Highlights

1. Let CDK grant IAM permissions per action — don't hand-roll a pipeline role

Every CodePipeline action construct (CodeCommitSourceAction, CodeBuildAction, S3DeployAction, LambdaInvokeAction) grants itself the exact, resource-scoped permissions it needs the moment it's bound to a stage. So the pipeline is created with no custom role at all:

const pipeline = new codepipeline.Pipeline(this, 'Pipeline', {
  pipelineName: pipelineName,
  restartExecutionOnUpdate: true,
  artifactBucket: artifactBucket,
  // no `role` — CDK creates one and grants each action's permissions as stages are added
});

This reference implementation originally had a hand-written pipeline role with artifactBucket.grantReadWrite(), repository.grantRead(), and three addToPolicy() calls including codedeploy:* and codestar-notifications:* on Resource: '*'. None of those wildcard grants were actually needed — nothing in the pipeline used CodeDeploy, and CodeStarNotifications doesn't check permissions on the pipeline's own role at all. Once I traced through what each action construct grants on bind() (CodeBuildAction scopes codebuild:BatchGetBuilds/StartBuild/StopBuild to the project ARN, S3DeployAction calls bucket.grantWrite() on the deploy target and grantRead() on the artifact bucket, LambdaInvokeAction calls lambda.grantInvoke(), and the Pipeline construct itself calls artifactBucket.grantReadWrite(this.role) internally), the entire hand-written pipeline role and its policies could just be deleted.

The two Lambdas the pipeline invokes are a separate matter: their execution roles still get explicit, resource-scoped policies in the stack (s3:ListBucket/GetObject/PutObject/DeleteObject on the target bucket for the sync Lambda; cloudfront:CreateInvalidation/GetInvalidation and sns:Publish for the invalidation Lambda), plus an artifactBucket.grantRead() so the sync Lambda can read the input artifact. LambdaInvokeAction only grants the pipeline role permission to invoke the function — it doesn't touch the function's own role.

2. Split "upload" from "cleanup" instead of one `aws s3 sync --delete`

// Deploy stage: upload/overwrite only
new codepipeline_action.S3DeployAction({
  actionName: 'S3_Deploy',
  bucket: cdk.aws_s3.Bucket.fromBucketName(this, 'TargetBucket', props.envParams.deploymentTargetBucketName),
  input: buildOutput,
});

// Sync stage: a separate Lambda removes stale objects afterward
new codepipeline_action.LambdaInvokeAction({
  actionName: 'Lambda_S3_Sync',
  lambda: s3SyncLambda,
  inputs: [buildOutput],
  userParameters: { DEST_BUCKET_NAME: props.envParams.deploymentTargetBucketName },
});

S3DeployAction is a managed action, so the common "upload the build" case needs no custom code at all. Deletion — which requires diffing the new build's file list against what's already in the bucket — is isolated in its own Lambda, testable and replaceable independently of the upload step.

3. The continuation-token pattern for CloudFront invalidation

CloudFront invalidations can take minutes to complete. Instead of a Lambda that polls in a loop until it's done (risking a timeout), the invalidation Lambda reports a continuationToken back to CodePipeline (via put_job_success_result), and CodePipeline re-invokes it until it reports success:

def lambda_handler(event, context):
    job_id = event['CodePipeline.job']['id']
    job_data = event['CodePipeline.job']['data']

    if 'continuationToken' in job_data:
        # Re-invoked: check on the invalidation we created last time
        continuation_token = json.loads(job_data['continuationToken'])
        invalidation_id = continuation_token['InvalidationId']
        status = monitor_invalidation_state(distribution_id, invalidation_id)
        if status != 'Completed':
            continue_job_later(job_id, invalidation_id)  # ask CodePipeline to call us again
        else:
            put_job_success(job_id)
    else:
        # First invocation: create the invalidation and ask to be re-invoked
        invalidation_id = create_invalidation(distribution_id)
        continue_job_later(job_id, invalidation_id)

Each invocation is short-lived; CodePipeline itself handles the "come back later and check again" scheduling. On completion (or failure) the Lambda also publishes a short status message to a dedicated SNS topic the stack always creates — that notification is independent of the opt-in approvalTopicArn feature below.

4. The bug: a pipeline that referenced itself

The InvalidateCache Lambda's userParameters originally included the pipeline's own name, read straight off the construct:

// Before — looks harmless
userParameters: {
  "PIPELINE_NAME": pipeline.pipelineName,
  "DISTRIBUTION_ID": props.envParams.cloudfrontDistributionId,
}

pipeline.pipelineName is a CDK token that resolves to { "Ref": "PipelineXXXX" }. And this userParameters blob is itself embedded inside one of the Pipeline resource's own Stages[].Actions[].Configuration properties — because the InvalidateCache action is a stage of that very pipeline. The rendered template ends up with the Pipeline resource containing a Ref to itself, which Template.fromStack()'s cyclic-dependency checker correctly rejects:

Template is undeployable, these resources have a dependency cycle: ... -> PipelineXXXX -> PipelineXXXX

I never hit this while the stack only existed in bin/, because nothing ever called Template.fromStack() on it. The moment I wrote a real snapshot test, it surfaced immediately. The fix is to compute the name as a plain string before constructing the Pipeline, and reuse that instead of the token:

// Fixed — a real string, not a Ref back to the Pipeline resource
const pipelineName = `${props.project}-${props.environment}-pipeline`;

const pipeline = new codepipeline.Pipeline(this, 'Pipeline', {
  pipelineName: pipelineName,
  // ...
});

// later, in the InvalidateCache stage:
userParameters: {
  "PIPELINE_NAME": pipelineName, // plain string, no cycle
  "DISTRIBUTION_ID": props.envParams.cloudfrontDistributionId,
}

The general lesson: any token that resolves to Ref/Fn::GetAtt on a construct is safe to use in most places, but if it ends up embedded inside that same construct's own properties (because the reference lives inside an action/stage that belongs to that construct), CloudFormation sees a resource pointing at itself. If you already have the plain value on hand before construction, prefer it.

5. Two more bugs that only a real `Template.fromStack()` run would catch

An S3 bucket name that wasn't guaranteed lowercase.

// Before
bucketName: `${props.project}-${props.environment}-artifact-bucket`,

S3 bucket names must be lowercase. A props.project of "TestProject" produced InvalidBucketNameValue at synth time. The fix, matching the convention already used elsewhere in this repo:

bucketName: `${props.project}-${props.environment}-artifact-bucket`.toLowerCase(),

A NotificationRule that could get zero targets.

The pipeline notification rule is only meaningful when an approval/notification topic is configured. The original code always created it, with an empty targets: [] when no topic was set:

// Before — deploys fine when a topic is set, fails when it isn't
new codestar_notification.NotificationRule(this, 'PipelineNotificationRule', {
  // ...
  targets: props.envParams.approvalTopicArn
    ? [sns.Topic.fromTopicArn(this, 'NotificationTopic', props.envParams.approvalTopicArn)]
    : [],
});

AWS::CodeStarNotifications::NotificationRule requires at least one target — an empty list deploys to a CloudFormation error, not a synth-time one, which makes it even easier to miss in local testing. The fix is to skip creating the resource entirely when there's no topic to notify:

if (props.envParams.approvalTopicArn) {
  new codestar_notification.NotificationRule(this, 'PipelineNotificationRule', {
    // ...
    targets: [sns.Topic.fromTopicArn(this, 'NotificationTopic', props.envParams.approvalTopicArn)],
  });
}

The same if also gates the Manual Approval stage, so a single optional parameter (approvalTopicArn) turns the entire "gate deployments and notify someone" feature on or off per environment.

Deploy & Verify

# The npm scripts read $PROJECT / $ENV and target the AWS named profile ${PROJECT}-${ENV}
export PROJECT=your-project
export ENV=dev

npm run bootstrap   # first time only
npm run deploy:all

# Push to the configured branch to trigger the pipeline, then check its status
aws codepipeline get-pipeline-state --name <project>-<env>-pipeline

Cost Estimate

💰 Rough monthly estimate (Tokyo region, ~20 pipeline executions/month)

Service	Usage	Rough monthly cost
CodePipeline (V2 pricing)	~20 runs	~$0.40
CodeBuild (`BUILD_GENERAL1_SMALL`, ~3 min/build) — only when `enableBuild`	~20 builds	~$0.30
Lambda (Sync + Invalidate)	Light invocations	~$0.05
S3 artifact bucket	A few MB	< $0.05
CloudWatch Logs (2 log groups, or 3 with the build stage; 1-week retention)	Light volume	< $0.05

Total: roughly $1/month with the build stage enabled, ~$0.55 without

Cost scales with pipeline executions, not idle time — nothing runs between deploys.

Summary

What we learned from this pattern:

Let CDK's action constructs grant their own IAM permissions. A hand-written pipeline role tends to accumulate wildcard grants that nothing in the pipeline actually uses
Split "upload" and "cleanup" across stages when a managed action (S3DeployAction) covers the common case and only the edge case (stale-object deletion) needs custom code
The continuation-token pattern lets a Lambda-backed pipeline action represent a long-running operation without blocking on it
A token that resolves to a construct's own Ref can create a self-reference if it ends up embedded inside that construct's own properties — compute plain values ahead of time when they're going to be reused inside the same construct's actions
Template.fromStack() is the fastest way to catch deploy-time-only failures (bucket naming rules, resources with mandatory non-empty properties, dependency cycles) — none of the three bugs in this article showed up until the test suite actually synthesized and validated the template
Guard optional child resources with the same condition as the feature they belong to — a NotificationRule with zero targets and a Manual Approval stage that's never wired to a topic are two symptoms of the same missing guard

References

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS CDK 100 Drill Exercises #009: CloudWatch Logs S3 Archive — Comparing 3 Log Archive Patterns in One Repository

TOMOAKI ishihara — Mon, 17 Aug 2026 17:24:58 +0000

Introduction

This is the 9th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

Leave CloudWatch Logs alone and anything past its retention period simply disappears. When you need to keep logs around for audits or incident investigation, most teams eventually land on "archive it to S3" — but there's more than one way to get there.

This time we implement three different patterns for archiving CloudWatch Logs to S3 as five independent CDK stacks, and compare the latency, cost, and operational-simplicity trade-offs with concrete code.

What you'll learn in this article

Near-real-time log archiving with Kinesis Data Firehose (via a subscription filter)
Tiered storage-class transitions with S3 lifecycle rules (Standard → IA → Glacier IR → Deep Archive)
How to bolt archiving onto an existing CloudWatch Logs log group after the fact
Scheduled batch export with the CloudWatch Logs Export Task API (and how to work around the one-concurrent-execution limit)
Decoding the CWL payload in a Lambda subscription filter and writing it to S3 in a custom format
How the required IAM trust relationships and bucket policies differ between patterns for the CloudWatch Logs service

📁 Code repository: GitHub

Architecture Overview

Pattern	Approach	Stacks
A: Firehose	Subscription filter → Kinesis Data Firehose → S3	Basic / Lifecycle / Existing (3 stacks)
B: Export Task	EventBridge Scheduler → Lambda → Export Task API → S3	Export (1 stack)
C: Direct Lambda write	Subscription filter → Lambda → S3	Lambda (1 stack)

Why three patterns

Trait	Pattern A (Firehose)	Pattern B (Export Task)	Pattern C (Lambda)
Delivery latency	Near real-time (seconds to minutes)	Scheduled (tolerates up to ~12h of delay)	Near real-time
Cost at high volume	Moderate (Firehose + S3)	Low (Lambda + S3 only)	Moderate (scales with Lambda invocation count)
Output format	Raw CWL JSON lines, GZIP-compressed	Raw CWL format	Custom JSON (fully controllable)
Operational simplicity	High (managed Firehose)	High (managed Export API)	Low (Lambda code needs maintenance)
Custom transformation	Limited (depends on Firehose's data transformation feature)	None	Fully controllable inside Lambda

In short: A if you need near-real-time delivery, B if a cost-optimized batch job is good enough, C if you want to define the output format yourself.

Data Flow

Pattern A (Stacks 1-3)
CloudWatch Log Group
  → Subscription filter (SubscriptionFilter + FirehoseDestination, role = CwlToFirehoseRole)
  → Kinesis Data Firehose (GZIP compression)
  → S3 archive bucket
  * Stack 1 (Basic) shares 5 log groups and uses dynamic partitioning on owner/logGroup
    to route each log group to its own S3 prefix. Stack 2 (Lifecycle) uses a single log
    group with a plain date prefix and no dynamic partitioning.

Pattern B (Stack 4)
EventBridge Scheduler
  → Lambda (calls logs:CreateExportTask for the previous day's logs)
  → CloudWatch Logs Export Task API
  → S3 archive bucket (bucket policy allows writes from logs.amazonaws.com)

Pattern C (Stack 5)
CloudWatch Log Group
  → Subscription filter (LambdaDestination, CDK manages invoke permissions automatically)
  → Lambda (decodes the gzip+base64 CWL payload and writes JSON to S3)
  → S3 archive bucket

Key Components

Component	Design point
S3 archive bucket	SSE-S3 encryption, versioning enabled, public access blocked, SSL enforced. Provisioned independently per stack (5 buckets total)
CwlToFirehoseRole	Assumed by `logs.amazonaws.com`. Scoped to log groups via an `aws:SourceArn` condition
FirehoseRole	Assumed by `firehose.amazonaws.com`. Grants the standard Firehose→S3 permission set (`PutObject`, `GetObject`, `ListBucket`, multipart upload actions) scoped to the archive bucket
Dynamic partitioning (Stack 1)	jq extracts `owner`/`logGroup` to route 5 shared log groups into separate S3 prefixes. `CloudWatchLogProcessor` can't be used because it strips those fields; a jq fallback is required to handle `CONTROL_MESSAGE` records
Tiered lifecycle rules (Stack 2)	Standard → IA (30 days) → Glacier IR (90 days) → Deep Archive (365 days) → expiration (7 years)
Importing an existing log group (Stack 3)	Referenced via `LogGroup.fromLogGroupName()`. No new `AWS::Logs::LogGroup` resource is created
Export Task Lambda (Stack 4)	A concurrency guard that checks for running/pending tasks before calling `CreateExportTask`
Direct Lambda write (Stack 5)	`LambdaDestination` automatically manages invoke permissions. The payload is decoded and saved as custom JSON

Implementation Highlights

1. Pattern A — Kinesis Data Firehose subscription

For CloudWatch Logs to deliver to Firehose, it needs an explicit IAM role. We split responsibilities across two roles.

// CwlToFirehoseRole: assumed by logs.amazonaws.com. Only the trust policy is defined here
const cwlToFirehoseRole = new iam.Role(this, 'CwlToFirehoseRole', {
    assumedBy: new iam.ServicePrincipal('logs.amazonaws.com', {
        conditions: {
            StringLike: {
                'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*`,
            },
        },
    }),
});

// L2 SubscriptionFilter + FirehoseDestination(role: cwlToFirehoseRole)
const filterPattern = logGroupParams.filterPattern ?? defaultLogGroupArchiveConfig.filterPattern;
new logs.SubscriptionFilter(this, 'CwlSubscriptionFilter', {
    logGroup: this.logGroup,
    destination: new logs_destinations.FirehoseDestination(this.deliveryStream, {
        role: cwlToFirehoseRole,
    }),
    filterPattern: filterPattern
        ? logs.FilterPattern.literal(filterPattern)
        : logs.FilterPattern.allEvents(),
});

Why not attach firehose:PutRecord/PutRecordBatch permissions directly on the role?
Inside bind(), aws-logs-destinations's FirehoseDestination calls deliveryStream.grantPutRecords(role) on the role you pass in (or one it generates), automatically granting the necessary permissions. If you also add your own inline policy to the role, you end up with two AWS::IAM::Policy resources granting the same permission. The trick is to define only the trust policy yourself (scoped by SourceArn) and let the L2 construct handle the permission grant.

I used to write this with the L1 CfnSubscriptionFilter plus a hand-rolled inline policy, on the assumption that "the L2 SubscriptionFilter can't accept a roleArn for Firehose." That assumption was wrong. In the current aws-cdk-lib (2.261.0 in this repository), FirehoseDestination wires the role's ARN all the way through to CfnSubscriptionFilter for you, so the L2 construct alone is sufficient.

The aws:SourceArn condition scopes who can assume this role down to "log groups within this account/region." Combined with scoping the access policy side to just this specific Firehose delivery stream, permissions are minimized in two independent layers.

1a. Per-log-group dynamic partitioning (Stack 1 – Basic)

Stack 1 creates 5 log groups that share the same Firehose delivery stream. Without partitioning, events from all five would end up mixed together in the same S3 object, so dynamic partitioning routes each log group to its own prefix.

const s3Destination = new firehose.S3Bucket(archiveBucket, {
    dataOutputPrefix:
        'AWSLogs/!{partitionKeyFromQuery:owner}/CWLogGroup/!{partitionKeyFromQuery:logGroup}/!{timestamp:yyyy/MM/dd/HH}/',
    dynamicPartitioning: { enabled: true },
    bufferingInterval: cdk.Duration.seconds(60), // dynamic partitioning requires >= 60s
    bufferingSize: cdk.Size.mebibytes(64),        // and >= 64MiB
    processors: [
        new firehose.DecompressionProcessor({
            compressionFormat: firehose.DecompressionProcessorCompressionFormat.GZIP,
        }),
        firehose.MetadataExtractionProcessor.jq16({
            owner: '(if (.owner // "") == "" then "controlmessages" else .owner end)',
            logGroup:
                '(if (.logGroup // "") == "" then "controlmessages" else (.logGroup | ltrimstr("/")) end)',
        }),
        new firehose.AppendDelimiterToRecordProcessor(),
    ],
});

I hit two snags here.

First: don't add the CloudWatchLogProcessor (message extraction). Firehose has a "message extraction" feature that pulls just the contents of message out of a decompressed CWL record — but enabling it wipes out owner/logGroup/logStream entirely, leaving only the raw contents of message. With those gone, the downstream MetadataExtractionProcessor fails with DynamicPartitioning.MetadataExtractionFailed when it tries to reference .owner/.logGroup via jq. Right after DecompressionProcessor, the record still has owner/logGroup at the top level (see Fig. 1 of Firehose's "message extraction" documentation), so jq can read them directly without a message extraction processor.

Second: handling the CONTROL_MESSAGE records CloudWatch Logs periodically sends as a health check. These records have owner/logGroup set to the empty string "", so a naive .logGroup | ltrimstr("/") returns the empty string as-is, which fails with partitionKeys values must not be null or empty. You need a jq if/then/else to supply a fallback value ("controlmessages"). It's tempting to reach for // (the alternative operator) here, but // only substitutes on null/false, not on an empty string.

The trade-off is that each record written to S3 is the entire CWL envelope (with a logEvents array that may contain multiple events), not a flat one-row-per-log-event line. Flattening (CloudWatchLogProcessor) is incompatible with metadata-based partitioning. If you need individual messages, UNNEST/json_extract the logEvents array in something like Athena.

2. Tiered S3 lifecycle (Stack 2 – Lifecycle)

For long-term archives, you can optimize cost by moving objects to progressively cheaper storage classes as access frequency drops.

archiveBucket.addLifecycleRule({
    id: `MoveToIAAfter${moveToIaAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.INFREQUENT_ACCESS, transitionAfter: cdk.Duration.days(moveToIaAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `MoveToGlacierAfter${moveToGlacierAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.GLACIER_INSTANT_RETRIEVAL, transitionAfter: cdk.Duration.days(moveToGlacierAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `MoveToDeepArchiveAfter${moveToDeepArchiveAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.DEEP_ARCHIVE, transitionAfter: cdk.Duration.days(moveToDeepArchiveAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `ExpireCurrentObjectsAfter${expireAfterDays}Days`,
    enabled: true,
    expiration: cdk.Duration.days(expireAfterDays),
});

Non-current versions (older objects overwritten while versioning is enabled) get their own IA transition and expiration rule, and incomplete multipart uploads are aborted after 7 days. Rather than bundling everything into one giant lifecycle configuration, each transition gets its own addLifecycleRule so parameters like moveToIaAfterDays can be tuned per environment independently.

3. Attaching to an existing log group (Stack 3 – Existing)

Sometimes you don't want to create a new log group — you want to bolt archiving onto a log group an application is already writing to.

const existingLogGroup = logs.LogGroup.fromLogGroupName(
    this,
    'ExistingLogGroup',
    existingLogGroupParams.logGroupName,
);

fromLogGroupName() only references the log group in CDK — it doesn't create an AWS::Logs::LogGroup resource. Since you can pass this existing log group's ARN (existingLogGroup.logGroupArn) straight into the trust policy's aws:SourceArn condition, the subscription filter and IAM role wiring can reuse the exact same code as Stacks 1 and 2.

4. Pattern B — Scheduled Export Task

The CloudWatch Logs Export Task API has a one concurrent execution per account limit. Before submitting a new task, the Lambda checks for running or pending tasks to avoid errors.

# export-task/index.py (core logic)
def lambda_handler(event, context):
    logs_client = boto3.client("logs")

    # Skip if a task is already running or pending
    running = logs_client.describe_export_tasks(statusCode="RUNNING")
    pending = logs_client.describe_export_tasks(statusCode="PENDING")
    if running["exportTasks"] or pending["exportTasks"]:
        print("An export task is already running or pending. Skipping.")
        return {"status": "skipped", "reason": "task_in_progress"}

    # Export the previous day's (UTC) logs
    yesterday = datetime.datetime.utcnow() - datetime.timedelta(days=1)
    from_time = int(yesterday.replace(hour=0, minute=0, second=0, microsecond=0).timestamp() * 1000)
    to_time = int(yesterday.replace(hour=23, minute=59, second=59, microsecond=999999).timestamp() * 1000)

    response = logs_client.create_export_task(
        taskName=f"export-{yesterday.strftime('%Y-%m-%d')}",
        logGroupName=LOG_GROUP_NAME,
        fromTime=from_time,
        to=to_time,
        destination=S3_BUCKET_NAME,
        destinationPrefix=f"{S3_PREFIX}/{yesterday.strftime('%Y/%m/%d')}",
    )
    return {"status": "created", "taskId": response["taskId"]}

The key detail is checking PENDING in addition to RUNNING. If a scheduled task is merely sitting in the queue and you submit the next one anyway, you get a LimitExceededException.

The S3 bucket side needs two resource policies allowing writes from the CloudWatch Logs service principal.

// 1. Permission to check the bucket ACL before exporting
archiveBucket.addToResourcePolicy(
    new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        principals: [new iam.ServicePrincipal('logs.amazonaws.com')],
        actions: ['s3:GetBucketAcl'],
        resources: [archiveBucket.bucketArn],
        conditions: {
            ArnLike: { 'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*` },
        },
    }),
);

// 2. Permission to write the exported objects
archiveBucket.addToResourcePolicy(
    new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        principals: [new iam.ServicePrincipal('logs.amazonaws.com')],
        actions: ['s3:PutObject'],
        resources: [archiveBucket.arnForObjects('*')],
        conditions: {
            StringEquals: { 's3:x-amz-acl': 'bucket-owner-full-control' },
            ArnLike: { 'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*` },
        },
    }),
);

The condition pinning s3:x-amz-acl to bucket-owner-full-control is explicitly required by CloudWatch Logs' official documentation. Omit it and the export task fails with a permissions error.

5. Pattern C — Subscription filter → Lambda → S3

With LambdaDestination, CDK automatically manages the resource policy (Lambda::Permission) that lets logs.amazonaws.com invoke your Lambda.

new logs.SubscriptionFilter(this, 'CwlSubscriptionFilter', {
    logGroup: this.logGroup,
    destination: new logDestinations.LambdaDestination(this.archiveFunction),
    filterPattern: filterPattern
        ? logs.FilterPattern.literal(filterPattern)
        : logs.FilterPattern.allEvents(),
});

On the Lambda side, the code decodes the CWL compressed payload and writes it to S3 in a JSON format it defines itself, per invocation.

# cwl-to-s3/index.py (core logic)
def lambda_handler(event, context):
    compressed = base64.b64decode(event["awslogs"]["data"])
    payload = json.loads(gzip.decompress(compressed).decode("utf-8"))

    if payload.get("messageType") == "CONTROL_MESSAGE":
        # Health-check message CloudWatch Logs sends, e.g. when a subscription starts
        return {"status": "skipped", "reason": "control_message"}

    now = datetime.datetime.utcnow()
    safe_stream = payload["logStream"].replace("/", "_").replace("$", "").replace("[", "").replace("]", "")
    s3_key = f"{S3_PREFIX}/{now.strftime('%Y/%m/%d/%H')}/{safe_stream}_{uuid.uuid4().hex[:8]}.json"

    record = {
        "logGroup": payload["logGroup"],
        "logStream": payload["logStream"],
        "owner": payload.get("owner"),
        "exportedAt": now.isoformat() + "Z",
        "events": payload["logEvents"],
    }
    s3_client.put_object(Bucket=S3_BUCKET_NAME, Key=s3_key, Body=json.dumps(record, ensure_ascii=False).encode("utf-8"))
    return {"status": "ok", "processed": len(payload["logEvents"]), "s3Key": s3_key}

Note the early skip for payloads with messageType == "CONTROL_MESSAGE". CloudWatch Logs sends these health-check control messages right after a subscription starts (among other times), and they have no logEvents key at all — without this check, the invocation fails with a KeyError.

Both patterns A and C deliver via a subscription filter for near-immediate delivery, but with A you can let Firehose handle the transformation, while with C you write the Lambda code yourself and get complete control over the output format in exchange.

Deploy & Verify

export PROJECT=your-project
export ENV=dev

npm run bootstrap -w workspaces/cloudwatch-logs-s3-archive   # first time only
npm run stage:deploy:all -w workspaces/cloudwatch-logs-s3-archive

To deploy individual stacks, use the per-stack stage:deploy:* scripts (each scoped to a cdk deploy stack selector like **/*Basic).

npm run stage:deploy:basic -w workspaces/cloudwatch-logs-s3-archive
npm run stage:deploy:lifecycle -w workspaces/cloudwatch-logs-s3-archive
npm run stage:deploy:export -w workspaces/cloudwatch-logs-s3-archive

# Pattern A Stack 1 (Basic): write test data into 5 log groups (/<project>/<env>/basic-*)
# (given the dynamic-partitioning buffering requirements of >= 60s / >= 64MiB, use the
#  repo's helper script rather than a one-off put-log-events call)
./write-test-logs.sh --project <project> --env <env>

# After ~60s, output appears partitioned under AWSLogs/<owner>/CWLogGroup/<logGroup>/...
aws s3 ls s3://<archive-bucket>/AWSLogs/ --recursive

# Pattern A Stack 2 (Lifecycle): single log group, no dynamic partitioning
./write-test-logs-lifecycle.sh --project <project> --env <env>
aws s3 ls s3://<archive-bucket>/ --recursive

# Pattern B: manually invoke the export Lambda
aws lambda invoke --function-name <project>-<env>-cwl-export-task --payload '{}' response.json
cat response.json

# Pattern C: write a test log entry (the subscription filter invokes the Lambda immediately)
aws logs put-log-events \
  --log-group-name /<project>/<env>/app-lambda \
  --log-stream-name test-stream \
  --log-events timestamp=$(date +%s000),message="hello lambda"
aws s3 ls s3://<archive-bucket>/subscriptions/ --recursive

Cost Estimate

💰 Rough monthly estimate (Tokyo region, 1 GB of log data per day)

Service	Applies to	Rough monthly cost
Kinesis Data Firehose	A	~$0.90 (at $0.03/GB)
Lambda	B, C	Effectively negligible at low invocation counts
S3 (Standard)	All	~$0.75 (at $0.025/GB-month)
CloudWatch Logs (ingestion)	All	~$0.76/GB (charged regardless of which archive pattern you use, and not included in the total below)
EventBridge	B	$1.00 per million events (negligible at minimal frequency)

Rough total for Pattern A's archiving cost alone: about $2/month at 1 GB/day of log volume. CloudWatch Logs ingestion itself (~$23/month at 1 GB/day) is a separate cost you pay upfront, before any archive pattern runs

Running all five stacks simultaneously at production-scale traffic for comparison testing will cost more than the above. Don't forget to clean up afterward with npm run stage:destroy:all -w workspaces/cloudwatch-logs-s3-archive.

Summary

What we learned from this pattern:

Pattern A (Firehose): Minimal code, well-suited for near-real-time archiving. Delivery from CloudWatch Logs requires FirehoseDestination (L2) plus two IAM roles with distinct responsibilities
Pattern B (Export Task): The lowest cost for batch use cases, but limited to one concurrent execution per account. A guard checking for running/pending tasks on the Lambda side is essential
Pattern C (Lambda): Full control over the output format, at the cost of ongoing Lambda code maintenance, including handling control messages
IAM trust relationships differ by service: For Firehose, FirehoseDestination grants permission to the trusted role via grantPutRecords; for Export Task, an S3 bucket policy with an s3:x-amz-acl condition is required; for LambdaDestination, CDK auto-generates the invoke permission — the same "delivery from CloudWatch Logs" concept wires up differently every time
Split tiered lifecycle rules by transition: rather than one giant rule, separate rules make the transitionAfter thresholds easy to tune independently as environment parameters
fromLogGroupName() is enough for bolting archiving onto an existing log group: pass the existing ARN into the trust policy directly and reuse the same wiring, with no new resource created
Firehose dynamic partitioning is incompatible with message extraction: CloudWatchLogProcessor strips owner/logGroup, so drop it if you're using jq to derive partition keys from those fields. You also need to account for CloudWatch Logs' CONTROL_MESSAGE records (where owner/logGroup are empty strings) with a jq if/then/else fallback (// doesn't catch empty strings)

References

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS CDK 100 Drill Exercises #010: Budget Alerts & Cost Anomaly Detection — 5 FinOps Alerting Patterns in CDK

TOMOAKI ishihara — Mon, 17 Aug 2026 15:24:33 +0000

Introduction

This is the 10th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

By the time you notice, AWS costs have often already piled up. You catch the anomaly when the invoice arrives, or you open the console and think "wait, I shouldn't have spent this much this month." AWS gives you several ways to prevent this, but they're actually quite different in nature. Do you want to be told when you cross a known threshold? Do you want to catch unusual activity even while still within budget? Or do you want weekly usage automatically shared with your team? The right mechanism depends on which of these you're actually after.

This time we implement five different patterns for AWS cost alerting as five independent CDK stacks, and compare each one's detection trigger, notification path, and setup complexity with concrete code.

What you'll learn in this article

How to express AWS Budgets threshold notifications as a data-driven { type, thresholdPercent }[] array instead of hardcoding them
The frequency: IMMEDIATE constraint required for SNS delivery in AWS Cost Anomaly Detection, and how to hand-assemble thresholdExpression as a JSON string
How AWS Budgets, Cost Anomaly Detection, and CloudWatch alarms each require SNS publish permissions in their own distinct way (different SourceAccount/SourceArn conditions per service principal)
The hard limit that AWS-managed Cost Anomaly Detection monitors are effectively capped at one or two per account — a constraint I only discovered by actually deploying, and the redesign needed to share one monitor across multiple stacks
The trap where AWS Chatbot's guardrailPolicies defaults to AdministratorAccess when left unset — and why an empty array [] doesn't prevent this (CDK synth strips empty arrays out of the template)
How to use Step Functions' new query language JSONata to format Cost Explorer API results straight into an SNS message with zero Lambda functions
A string-escaping trap that spans three layers — JavaScript → JSON → JSONata (\n happens to work, \s silently breaks)
Six reusable constructs factored out into common/constructs/cost/ for reuse across the whole repository

📁 Code repository: GitHub

Architecture Overview

Pattern	Approach	Stack
A: Budgets	Threshold-based budget alert → SNS → Email	Budget (1 stack)
B: Anomaly Detection	ML-based anomaly detection → SNS → Email	Anomaly (1 stack)
C: Unified	Budgets + Anomaly Detection consolidated into one SNS topic, optionally Slack	Unified (1 stack)
D: Billing Alarm	Classic CloudWatch `EstimatedCharges` alarm → SNS → Email	BillingAlarm (1 stack, pinned to us-east-1)
E: Cost Digest	Scheduled Step Functions → SNS → Slack/Teams	CostDigest (1 stack)

Why five patterns

Trait	A (Budgets)	B (Anomaly Detection)	C (Unified)	D (Billing Alarm)	E (Cost Digest)
Trigger	Reactive — known threshold	Reactive — deviation from historical trend (ML)	Both, reactive	Reactive — cumulative spend threshold	Proactive — scheduled (no threshold needed)
Best suited for	"Tell me when I cross $X/month"	"I want to notice unusual activity even within budget"	Consolidating everything into one channel	The oldest safety net, independent of Cost Explorer	"Share weekly spend with the team"
Notification channel	SNS + Email	SNS + Email	SNS + Email + optional Slack	SNS + Email	SNS + Email + optional Slack/Teams

In short: A to stop at a known ceiling, B to catch unexpected movement, C to consolidate both into one place, D as the last-resort insurance for when you can't even trust Cost Explorer, E for a proactive recurring report.

Data Flow

Pattern A (Stack 1)
CfnBudget (account-wide + service filter)
  → SNS topic (budgets.amazonaws.com allowed to Publish)
  → Email

Pattern B (Stack 2)
CfnAnomalyMonitor (DIMENSIONAL / SERVICE)
  → CfnAnomalySubscription (frequency: IMMEDIATE)
  → SNS topic (costalerts.amazonaws.com allowed to Publish)
  → Email

Pattern C (Stack 3)
CfnBudget + CfnAnomalySubscription (attaches to Stack 2's monitor, creates no new monitor)
  → SNS topic (both service principals allowed)
  → Email + AWS Chatbot → Slack (optional)

Pattern D (Stack 4, pinned to us-east-1)
CloudWatch alarm (AWS/Billing EstimatedCharges)
  → SNS topic (cloudwatch.amazonaws.com allowed to Publish)
  → Email

Pattern E (Stack 5)
EventBridge Scheduler (cron)
  → Step Functions (JSONata): GetCostAndUsage → PublishCostDigest
  → SNS topic (Publish from its own IAM role, no service policy needed)
  → Email + AWS Chatbot → Slack / Microsoft Teams (either or both)

Key Components

Component	Design point
`CostAlertTopic` (shared construct)	SNS topic + resource policy driven by `allowBudgetsPublish`/`allowCostAnomalyDetectionPublish`/`allowCloudWatchAlarmPublish` flags
`CostBudget` (shared construct)	A `CfnBudget` wrapper that dynamically builds `notificationsWithSubscribers` from `BudgetNotificationRule[]`
`CostAnomalyDetection` (shared construct)	`CfnAnomalyMonitor` + `CfnAnomalySubscription`, encapsulating the `thresholdExpression` JSON assembly
`BillingAlarm` (shared construct)	`AWS/Billing EstimatedCharges` metric + alarm. Assumes it's pinned to us-east-1
`SafeSlackChannelConfiguration` / `SafeMicrosoftTeamsChannelConfiguration` (shared constructs)	Safe Chatbot wrappers that fall back to `ReadOnlyAccess` when `guardrailPolicies` isn't specified
Step Functions (Stack 5)	`QueryLanguage.JSONATA`, `tracingEnabled: true`, full execution logging to CloudWatch Logs

Implementation Highlights

1. Data-driven budget notification rules

Rather than hardcoding "80%, 100%, forecasted 100%" into the stack, notification rules are stored as a plain array in the environment parameters.

// common/types/cost.ts
export interface BudgetNotificationRule {
    readonly type: 'ACTUAL' | 'FORECASTED';
    readonly thresholdPercent: number;
}

// parameters/dev-params.ts
budget: {
    amount: 10,
    notifications: [
        { type: 'FORECASTED', thresholdPercent: 100 },
        { type: 'ACTUAL', thresholdPercent: 80 },
        { type: 'ACTUAL', thresholdPercent: 100 },
        { type: 'ACTUAL', thresholdPercent: 200 }, // escalation for runaway costs
    ],
},

This lets any environment add or remove thresholds without touching the stack code. common/constructs/cost/budget.ts just .map()s over the array.

const notificationRules = props.notifications ?? defaultBudgetNotifications;
const notificationsWithSubscribers = notificationRules.map((rule) => ({
    notification: {
        notificationType: rule.type,
        comparisonOperator: 'GREATER_THAN',
        threshold: rule.thresholdPercent,
        thresholdType: 'PERCENTAGE',
    },
    subscribers,
}));

2. SNS topic policies aren't optional — and each service does it differently

Both AWS Budgets and AWS Cost Anomaly Detection publish to SNS as service principals rather than by assuming an IAM role. CloudWatch alarm actions likewise need explicit permission. Without it, notifications fail silently — the configuration looks correct, but nothing ever arrives.

// common/constructs/cost/cost-alert-topic.ts (simplified)
if (props.allowBudgetsPublish) {
    topic.addToResourcePolicy(new iam.PolicyStatement({
        principals: [new iam.ServicePrincipal('budgets.amazonaws.com')],
        actions: ['sns:Publish'],
        conditions: {
            StringEquals: { 'aws:SourceAccount': account },
            ArnLike: { 'aws:SourceArn': `arn:${partition}:budgets::${account}:*` },
        },
    }));
}
if (props.allowCostAnomalyDetectionPublish) {
    // costalerts.amazonaws.com, condition is aws:SourceAccount only (no SourceArn scoping needed)
}
if (props.allowCloudWatchAlarmPublish) {
    topic.grantPublish(new iam.ServicePrincipal('cloudwatch.amazonaws.com'));
}

All three need "permission to publish to SNS," yet the strictness of the conditions is all over the place. Budgets requires both SourceAccount and SourceArn; Cost Anomaly Detection requires only SourceAccount (even AWS's own official samples don't specify SourceArn); and CloudWatch's cloudwatch-actions.SnsAction doesn't even do the kind of automatic grant other alarm actions do — you have to call grantPublish explicitly.

Don't add a customer-managed KMS key to these topics. The troubleshooting documentation for both Budgets and Cost Anomaly Detection explicitly calls out topic encryption as a classic cause of silently-failed notifications, because the service principal would also need kms:GenerateDataKey*/kms:Decrypt granted in the key policy. Here we enforce enforceSSL: true for transport encryption only, leaving at-rest encryption at SNS's default managed encryption.

3. Cost Anomaly Detection's `thresholdExpression`

CfnAnomalySubscription.thresholdExpression is typed as a plain string in aws-cdk-lib. CDK doesn't have a typed representation of Cost Explorer's Expression grammar, so you have to assemble it yourself and JSON.stringify() it.

thresholdExpression: JSON.stringify({
    And: [
        {
            Dimensions: {
                Key: 'ANOMALY_TOTAL_IMPACT_PERCENTAGE',
                MatchOptions: ['GREATER_THAN_OR_EQUAL'],
                Values: [String(thresholdPercentage)],
            },
        },
        {
            Dimensions: {
                Key: 'ANOMALY_TOTAL_IMPACT_ABSOLUTE',
                MatchOptions: ['GREATER_THAN_OR_EQUAL'],
                Values: [String(thresholdAbsoluteUsd)],
            },
        },
    ],
}),

ANDing "percentage relative to expected spend" with "absolute dollar amount" filters out noise like "10x the usual cost, but the absolute amount is only $3."

SNS delivery requires frequency: IMMEDIATE. AWS Cost Anomaly Detection only supports SNS subscribers on IMMEDIATE subscriptions — DAILY/WEEKLY are email-only. If you also want a daily email summary, add a second CfnAnomalySubscription (frequency: DAILY, EMAIL subscriber) pointing at the same monitor.

4. Stack 3 doesn't create its own anomaly monitor — AWS's "one per account" limit

This is a trap I only hit after actually deploying, once the writing was already done. Deploy Stack 3 with Stack 2 already deployed, and it fails reliably with this error:

CREATE_FAILED | AWS::CE::AnomalyMonitor | ...
Resource handler returned message: "null" (HandlerErrorCode: AlreadyExists)

At first I thought "the monitor names are different, so why?" — but the cause wasn't the name, it was an AWS-imposed limit. The CreateAnomalyMonitor API reference itself doesn't mention this constraint; the source is a blog post from the AWS Cloud Financial Management team.

"You can create one AWS services managed monitor plus one additional AWS managed monitor (linked account, cost allocation tag, or cost category) per management account."

"AWS managed monitors for linked accounts, cost allocation tags, and cost categories can only be created in management accounts."

— Extending AWS managed monitors in AWS Cost Anomaly Detection (AWS Cloud Financial Management blog)

Since Stack 2 has already created the account's one and only SERVICE-dimension monitor, Stack 3 trying to create a second one on the same dimension always hits AlreadyExists. Switching monitorDimension to LINKED_ACCOUNT doesn't help either — that runs into the separate constraint of only being creatable in an AWS Organizations management account, which fails in most test accounts anyway.

The fix is simple: don't have Stack 3 create a new monitor — attach a second CfnAnomalySubscription to Stack 2's existing monitor instead. Attaching multiple subscriptions to one monitor is a configuration AWS officially supports.

// common/constructs/cost/anomaly-detection.ts
if (props.existingMonitorArn) {
    this.monitorArn = props.existingMonitorArn;
} else {
    this.monitor = new ce.CfnAnomalyMonitor(this, 'Monitor', { /* ... */ });
    this.monitorArn = this.monitor.attrMonitorArn;
}

this.subscription = new ce.CfnAnomalySubscription(this, 'Subscription', {
    monitorArnList: [this.monitorArn],
    // ...
});

On the Stage side, Stack 2 is instantiated before Stack 3, and anomalyStack.monitorArn is simply passed into Stack 3's props. CDK converts this into a real CloudFormation cross-stack export/import, so there's no need for a hand-written Fn::ImportValue.

const anomalyStack = new BudgetsCostAnomalyDetectionAnomalyStack(this, ..., { ... });

new BudgetsCostAnomalyDetectionUnifiedStack(this, ..., {
    ...commonStackProps,
    anomalyMonitorArn: anomalyStack.monitorArn,
    // ...
});

I'll admit it stings a bit that Stack 3 ended up depending on Stack 2, despite the whole premise being "five independent patterns." But since this is a hard constraint on AWS's side, I decided it's more honest to make the dependency explicit in both code and documentation rather than hide it.

Once the deploy went through, I noticed something else. Asking myself "what's actually different between drillexercises-dev-service-anomaly-subscription (Stack 2) and drillexercises-dev-unified-anomaly-subscription (Stack 3)," the answer turned out to be: they're watching exactly the same thing. Both are attached to the same monitor, and both read thresholdPercentage/thresholdAbsoluteUsd from the same params.anomalyDetection, so by default the thresholds are identical too. That means deploying both Stack 2 and Stack 3 results in two notifications for the same single anomaly — an email via Stack 2, plus an email (and optionally Slack) via Stack 3.

To address this, I made it possible to set a stricter threshold on Stack 3 specifically.

// lib/types/anomaly-params.ts
export interface AnomalyDetectionParams extends AnomalyThreshold {
    readonly monitorDimension?: 'SERVICE' | 'LINKED_ACCOUNT';
    // Escalation threshold for Stack 3. Falls back to the base value if unset
    readonly unifiedEscalation?: AnomalyThreshold;
}

// lib/stacks/budgets-cost-anomaly-detection-unified-stack.ts
const anomalyThreshold = anomalyParams.unifiedEscalation ?? anomalyParams;
new CostAnomalyDetection(this, 'UnifiedAnomalyDetection', {
    existingMonitorArn: props.anomalyMonitorArn,
    thresholdPercentage: anomalyThreshold.thresholdPercentage,
    thresholdAbsoluteUsd: anomalyThreshold.thresholdAbsoluteUsd,
    // ...
});

In dev-params.ts, the base threshold is "20%+ actual cost AND $5+ absolute," while Stack 3's escalation threshold is "50%+ actual cost AND $20+ absolute." This demonstrates a severity split: Stack 2 casts a wide net via email, while Stack 3 (the unified/Slack channel) only receives the more serious anomalies.

To be honest, though, this doesn't eliminate the duplication itself. An anomaly large enough to satisfy both thresholds still triggers two notifications. In real-world usage, the straightforward choice is to use only one of Stack 2 or Stack 3 for anomaly detection. Think of unifiedEscalation purely as a demonstration that "a single monitor can feed multiple severity-tiered subscriptions."

5. AWS Chatbot's guardrail policy default is `AdministratorAccess` (for both Slack and Teams)

This is the biggest "hurts if you don't know it" gotcha of this whole exercise. Both SlackChannelConfigurationProps.guardrailPolicies and CfnMicrosoftTeamsChannelConfigurationProps.guardrailPolicies apply the AWS-managed AdministratorAccess policy by default when left unspecified.

"So can I disable it by passing an empty array []?" — that's the trap. CDK's synth step strips empty list properties out of the synthesized CloudFormation template entirely, so from the API's perspective the property is simply unset, and the AdministratorAccess default kicks in anyway. I only noticed this by actually inspecting cdk synth output.

// common/constructs/cost/safe-slack-channel.ts
guardrailPolicies:
    props.guardrailPolicies && props.guardrailPolicies.length > 0
        ? props.guardrailPolicies
        : [iam.ManagedPolicy.fromAwsManagedPolicyName('ReadOnlyAccess')],

Since Microsoft Teams has no CDK L2 construct at all, SafeMicrosoftTeamsChannelConfiguration also auto-generates the minimal "notification only" IAM role (cloudwatch:Describe*/Get*/List* only) used by AWS's own sample Chatbot policy.

6. Reusable cost constructs (`common/constructs/cost/`)

The pieces above aren't specific to this workspace, so they're factored out into infrastructure/common/constructs/cost/ for reuse across the whole repository.

Construct	Wraps	Used by
`CostAlertTopic`	SNS Topic + conditional resource policy + email subscription	Stack 1, 2, 3, 4
`CostBudget`	`CfnBudget` driven by `BudgetNotificationRule[]`	Stack 1 (x2), Stack 3
`CostAnomalyDetection`	`CfnAnomalyMonitor` + `CfnAnomalySubscription` (skips monitor creation when `existingMonitorArn` is given)	Stack 2, Stack 3
`BillingAlarm`	`AWS/Billing EstimatedCharges` alarm	Stack 4
`SafeSlackChannelConfiguration`	Slack configuration with safe guardrail defaults	Stack 3, Stack 5
`SafeMicrosoftTeamsChannelConfiguration`	Teams configuration with a least-privilege role plus safe guardrails	Stack 5

Stack 5's Step Functions/JSONata logic itself isn't factored out, since the message formatting is tightly coupled to this workspace's specific content; only the Chatbot delivery portion uses the safe wrappers above.

7. The cost digest's JSONata state machine (Stack 5)

Stack 5 generalizes a "post a weekly cost summary to Teams" CloudFormation template originally hand-written for Microsoft Teams so it also supports Slack — the biggest challenge of this exercise. The state machine consists of exactly two states, both written as raw ASL via sfn.CustomState (since there's no typed CDK task for the Cost Explorer or SNS AWS-SDK integrations).

const getCostAndUsage = new sfn.CustomState(this, 'GetCostAndUsage', {
    stateJson: {
        Type: 'Task',
        Resource: 'arn:aws:states:::aws-sdk:costexplorer:getCostAndUsage',
        Arguments: { /* Granularity, Metrics, TimePeriod (JSONata), GroupBy, Filter */ },
        Assign: { AngryThreshold: angryThresholdUsd, AccountId: cdk.Aws.ACCOUNT_ID, /* ... */ },
        Output: { Start: '{% ... %}', CostSum: '{% ... %}', CostSorted: '{% ... %}' },
    },
});
const publishCostDigest = new sfn.CustomState(this, 'PublishCostDigest', {
    stateJson: {
        Type: 'Task',
        Resource: 'arn:aws:states:::sns:publish',
        Arguments: { Message: { /* title/description, JSONata templates */ }, TopicArn: topic.topicArn },
    },
});
getCostAndUsage.next(publishCostDigest);

new sfn.StateMachine(this, 'CostDigestStateMachine', {
    definitionBody: sfn.DefinitionBody.fromChainable(getCostAndUsage),
    queryLanguage: sfn.QueryLanguage.JSONATA,
    tracingEnabled: true,
    logs: { destination: logGroup, level: sfn.LogLevel.ALL, includeExecutionData: true },
});

Next/End aren't hand-written anywhere. Just chaining with .next() is enough — CDK's state graph composition attaches them automatically (Next: "PublishCostDigest" on the first state, End: true on the last).

A JS string-escaping trap worth knowing about. The description message concatenates JSONata string literals for line breaks ("\n") and a regex for stripping the AWS/Amazon prefix off service names ($replace(/^(AWS|Amazon)\s*/, "")). Write either escape as a single backslash inside a TypeScript template literal and JavaScript resolves it immediately at the string-literal level — \n silently becomes a real newline character, and \s (not a recognized JS escape) silently loses its backslash entirely, leaving a bare s that turns the regex into s* and quietly breaks the service-name cleanup. The fix for both is the same: write \\n and \\s (double backslash) in the TypeScript source, so what actually reaches JSONata — after this string round-trips through JSON as part of the synthesized ASL — is the two-character escape sequence \n/\s, not a literal control character or a mangled regex. I found this not just by reading the JSONata spec, but by actually inspecting the synthesized cdk synth output before writing tests.

The EventBridge Scheduler's StepFunctionsStartExecution target (aws-scheduler-targets) automatically generates and attaches the scheduler's own execution role — unlike the original CloudFormation template this was ported from, there's no need to hand-write the states:StartExecution IAM statement.

Also worth noting: the title/description JSONata expressions aren't one-off inline strings — they come from two locale-specific builder functions, buildCostDigestTitleExpression/buildCostDigestDescriptionExpression. Flipping params.costDigest.locale ('ja' | 'en', default 'en') is all it takes to switch to the Japanese digest message (e.g. "😱 Costs are spiking" / "😊 Costs are steady" becomes "😱 コストが跳ね上がっています" / "😊 コストは落ち着いています").

Deploy & Verify

export PROJECT=your-project
export ENV=dev

npm run bootstrap -w workspaces/budgets-cost-anomaly-detection   # first time only
npm run synth -w workspaces/budgets-cost-anomaly-detection
npm run deploy:all -w workspaces/budgets-cost-anomaly-detection

Before deploying, replace the placeholder email address in parameters/dev-params.ts with a real one (SNS/Budgets email subscriptions require confirmation). To use Slack/Teams delivery, uncomment notification.slack/notification.teams.

# Pattern A: check the budget configuration (real billing data updates only a few times a day, so no instant trigger)
aws budgets describe-budgets --account-id <account-id>

# Pattern B: check the anomaly monitor/subscription (baseline establishment takes 24h+)
aws ce get-anomaly-monitors
aws ce get-anomaly-subscriptions

# Pattern C: check the topic's subscriptions
aws sns list-subscriptions-by-topic --topic-arn <finops-alert-topic-arn>

# Pattern D: check the alarm and its data (this stack only exists in us-east-1)
aws cloudwatch describe-alarms --alarm-names <project>-<env>-estimated-charges --region us-east-1

# Pattern E: run on demand without waiting for the schedule
aws stepfunctions start-execution --state-machine-arn <cost-digest-state-machine-arn>

Cost Estimate

💰 Rough monthly estimate (Tokyo region, Stack 4 only in us-east-1)

Service	Applies to	Rough monthly cost
AWS Budgets	A, C	First 2 budgets free; ~$0.02/day per budget after that
AWS Cost Anomaly Detection	B, C	No additional charge
Amazon SNS	All	First 1M requests within the free tier
AWS Chatbot	C, E (optional)	No additional charge
CloudWatch alarm (billing)	D	~$0.10/alarm/month
Step Functions (Standard)	E	First 4,000 state transitions/month free; negligible at once daily
EventBridge Scheduler	E	First 14M invocations/month free

Rough total: effectively $0-2/month at typical alert volumes.

Summary

What we learned from this pattern:

Pattern A (Budgets): Best for a known, fixed cost ceiling. Notifications fail silently if the SNS topic policy isn't set explicitly
Pattern B (Anomaly Detection): Best for catching unexpected spend even within budget. SNS delivery requires IMMEDIATE frequency, and thresholdExpression is a hand-assembled JSON string
Pattern C (Unified): Consolidating into one alert channel is more operationally realistic than managing a separate topic per signal — but since AWS caps AWS-managed Cost Anomaly Detection monitors at effectively one or two per account, Stack 3 doesn't create its own monitor and instead attaches an extra subscription to Stack 2's monitor via a real cross-stack reference. unifiedEscalation lets you split by severity by raising Stack 3's threshold, but as long as both stacks are deployed, the duplicate notification itself never goes away — so in real usage, pick either Stack 2 or Stack 3 for anomaly detection, not both
Pattern D (Billing Alarm): The simplest safety net, but has two gotchas CDK can't express: the manual "Receive Billing Alerts" opt-in, and being pinned to us-east-1
Pattern E (Cost Digest): A proactive, scheduled digest complements the other four patterns well. sfn.CustomState lets you write raw ASL/JSONata and skip Lambda entirely, but watch the string escaping at the JS → JSON → JSONata boundaries
AWS Chatbot's AdministratorAccess guardrail default applies to both Slack and Teams, and an empty array doesn't disable it — always pin an explicit, least-privilege guardrail
Wiring that repeats across stacks is worth factoring out into a construct like common/constructs/cost/ once, rather than copy-pasting

References

AWS Budgets User Guide
Creating an Amazon SNS topic for budget notifications
AWS Cost Anomaly Detection
Extending AWS managed monitors in AWS Cost Anomaly Detection (the source for the "one SERVICE monitor per account" limit behind Stack 3's design)
Create a billing alarm to monitor your estimated AWS charges
Setting up AWS Chatbot with Microsoft Teams
Variables and JSONata in AWS Step Functions
Amazon EventBridge Scheduler

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS CDK 100 Drill Exercises #009: CloudWatch Logs S3 Archive — Comparing 3 Log Archive Patterns in One Repository

TOMOAKI ishihara — Mon, 17 Aug 2026 15:24:21 +0000

Introduction

This is the 9th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

What you'll learn in this article

Near-real-time log archiving with Kinesis Data Firehose (via a subscription filter)
Tiered storage-class transitions with S3 lifecycle rules (Standard → IA → Glacier IR → Deep Archive)
How to bolt archiving onto an existing CloudWatch Logs log group after the fact
Scheduled batch export with the CloudWatch Logs Export Task API (and how to work around the one-concurrent-execution limit)
Decoding the CWL payload in a Lambda subscription filter and writing it to S3 in a custom format
How the required IAM trust relationships and bucket policies differ between patterns for the CloudWatch Logs service

📁 Code repository: GitHub

Architecture Overview

Pattern	Approach	Stacks
A: Firehose	Subscription filter → Kinesis Data Firehose → S3	Basic / Lifecycle / Existing (3 stacks)
B: Export Task	EventBridge Scheduler → Lambda → Export Task API → S3	Export (1 stack)
C: Direct Lambda write	Subscription filter → Lambda → S3	Lambda (1 stack)

Why three patterns

Trait	Pattern A (Firehose)	Pattern B (Export Task)	Pattern C (Lambda)
Delivery latency	Near real-time (seconds to minutes)	Scheduled (tolerates up to ~12h of delay)	Near real-time
Cost at high volume	Moderate (Firehose + S3)	Low (Lambda + S3 only)	Moderate (scales with Lambda invocation count)
Output format	Raw CWL JSON lines, GZIP-compressed	Raw CWL format	Custom JSON (fully controllable)
Operational simplicity	High (managed Firehose)	High (managed Export API)	Low (Lambda code needs maintenance)
Custom transformation	Limited (depends on Firehose's data transformation feature)	None	Fully controllable inside Lambda

In short: A if you need near-real-time delivery, B if a cost-optimized batch job is good enough, C if you want to define the output format yourself.

Data Flow

Pattern A (Stacks 1-3)
CloudWatch Log Group
  → Subscription filter (SubscriptionFilter + FirehoseDestination, role = CwlToFirehoseRole)
  → Kinesis Data Firehose (GZIP compression)
  → S3 archive bucket
  * Stack 1 (Basic) shares 5 log groups and uses dynamic partitioning on owner/logGroup
    to route each log group to its own S3 prefix. Stack 2 (Lifecycle) uses a single log
    group with a plain date prefix and no dynamic partitioning.

Pattern B (Stack 4)
EventBridge Scheduler
  → Lambda (calls logs:CreateExportTask for the previous day's logs)
  → CloudWatch Logs Export Task API
  → S3 archive bucket (bucket policy allows writes from logs.amazonaws.com)

Pattern C (Stack 5)
CloudWatch Log Group
  → Subscription filter (LambdaDestination, CDK manages invoke permissions automatically)
  → Lambda (decodes the gzip+base64 CWL payload and writes JSON to S3)
  → S3 archive bucket

Key Components

Component	Design point
S3 archive bucket	SSE-S3 encryption, versioning enabled, public access blocked, SSL enforced. Provisioned independently per stack (5 buckets total)
CwlToFirehoseRole	Assumed by `logs.amazonaws.com`. Scoped to log groups via an `aws:SourceArn` condition
FirehoseRole	Assumed by `firehose.amazonaws.com`. Grants the standard Firehose→S3 permission set (`PutObject`, `GetObject`, `ListBucket`, multipart upload actions) scoped to the archive bucket
Dynamic partitioning (Stack 1)	jq extracts `owner`/`logGroup` to route 5 shared log groups into separate S3 prefixes. `CloudWatchLogProcessor` can't be used because it strips those fields; a jq fallback is required to handle `CONTROL_MESSAGE` records
Tiered lifecycle rules (Stack 2)	Standard → IA (30 days) → Glacier IR (90 days) → Deep Archive (365 days) → expiration (7 years)
Importing an existing log group (Stack 3)	Referenced via `LogGroup.fromLogGroupName()`. No new `AWS::Logs::LogGroup` resource is created
Export Task Lambda (Stack 4)	A concurrency guard that checks for running/pending tasks before calling `CreateExportTask`
Direct Lambda write (Stack 5)	`LambdaDestination` automatically manages invoke permissions. The payload is decoded and saved as custom JSON

Implementation Highlights

1. Pattern A — Kinesis Data Firehose subscription

For CloudWatch Logs to deliver to Firehose, it needs an explicit IAM role. We split responsibilities across two roles.

// CwlToFirehoseRole: assumed by logs.amazonaws.com. Only the trust policy is defined here
const cwlToFirehoseRole = new iam.Role(this, 'CwlToFirehoseRole', {
    assumedBy: new iam.ServicePrincipal('logs.amazonaws.com', {
        conditions: {
            StringLike: {
                'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*`,
            },
        },
    }),
});

// L2 SubscriptionFilter + FirehoseDestination(role: cwlToFirehoseRole)
const filterPattern = logGroupParams.filterPattern ?? defaultLogGroupArchiveConfig.filterPattern;
new logs.SubscriptionFilter(this, 'CwlSubscriptionFilter', {
    logGroup: this.logGroup,
    destination: new logs_destinations.FirehoseDestination(this.deliveryStream, {
        role: cwlToFirehoseRole,
    }),
    filterPattern: filterPattern
        ? logs.FilterPattern.literal(filterPattern)
        : logs.FilterPattern.allEvents(),
});

Why not attach firehose:PutRecord/PutRecordBatch permissions directly on the role?
Inside bind(), aws-logs-destinations's FirehoseDestination calls deliveryStream.grantPutRecords(role) on the role you pass in (or one it generates), automatically granting the necessary permissions. If you also add your own inline policy to the role, you end up with two AWS::IAM::Policy resources granting the same permission. The trick is to define only the trust policy yourself (scoped by SourceArn) and let the L2 construct handle the permission grant.

I used to write this with the L1 CfnSubscriptionFilter plus a hand-rolled inline policy, on the assumption that "the L2 SubscriptionFilter can't accept a roleArn for Firehose." That assumption was wrong. In the current aws-cdk-lib (2.261.0 in this repository), FirehoseDestination wires the role's ARN all the way through to CfnSubscriptionFilter for you, so the L2 construct alone is sufficient.

1a. Per-log-group dynamic partitioning (Stack 1 – Basic)

const s3Destination = new firehose.S3Bucket(archiveBucket, {
    dataOutputPrefix:
        'AWSLogs/!{partitionKeyFromQuery:owner}/CWLogGroup/!{partitionKeyFromQuery:logGroup}/!{timestamp:yyyy/MM/dd/HH}/',
    dynamicPartitioning: { enabled: true },
    bufferingInterval: cdk.Duration.seconds(60), // dynamic partitioning requires >= 60s
    bufferingSize: cdk.Size.mebibytes(64),        // and >= 64MiB
    processors: [
        new firehose.DecompressionProcessor({
            compressionFormat: firehose.DecompressionProcessorCompressionFormat.GZIP,
        }),
        firehose.MetadataExtractionProcessor.jq16({
            owner: '(if (.owner // "") == "" then "controlmessages" else .owner end)',
            logGroup:
                '(if (.logGroup // "") == "" then "controlmessages" else (.logGroup | ltrimstr("/")) end)',
        }),
        new firehose.AppendDelimiterToRecordProcessor(),
    ],
});

I hit two snags here.

2. Tiered S3 lifecycle (Stack 2 – Lifecycle)

For long-term archives, you can optimize cost by moving objects to progressively cheaper storage classes as access frequency drops.

archiveBucket.addLifecycleRule({
    id: `MoveToIAAfter${moveToIaAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.INFREQUENT_ACCESS, transitionAfter: cdk.Duration.days(moveToIaAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `MoveToGlacierAfter${moveToGlacierAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.GLACIER_INSTANT_RETRIEVAL, transitionAfter: cdk.Duration.days(moveToGlacierAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `MoveToDeepArchiveAfter${moveToDeepArchiveAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.DEEP_ARCHIVE, transitionAfter: cdk.Duration.days(moveToDeepArchiveAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `ExpireCurrentObjectsAfter${expireAfterDays}Days`,
    enabled: true,
    expiration: cdk.Duration.days(expireAfterDays),
});

3. Attaching to an existing log group (Stack 3 – Existing)

Sometimes you don't want to create a new log group — you want to bolt archiving onto a log group an application is already writing to.

const existingLogGroup = logs.LogGroup.fromLogGroupName(
    this,
    'ExistingLogGroup',
    existingLogGroupParams.logGroupName,
);

4. Pattern B — Scheduled Export Task

The CloudWatch Logs Export Task API has a one concurrent execution per account limit. Before submitting a new task, the Lambda checks for running or pending tasks to avoid errors.

# export-task/index.py (core logic)
def lambda_handler(event, context):
    logs_client = boto3.client("logs")

    # Skip if a task is already running or pending
    running = logs_client.describe_export_tasks(statusCode="RUNNING")
    pending = logs_client.describe_export_tasks(statusCode="PENDING")
    if running["exportTasks"] or pending["exportTasks"]:
        print("An export task is already running or pending. Skipping.")
        return {"status": "skipped", "reason": "task_in_progress"}

    # Export the previous day's (UTC) logs
    yesterday = datetime.datetime.utcnow() - datetime.timedelta(days=1)
    from_time = int(yesterday.replace(hour=0, minute=0, second=0, microsecond=0).timestamp() * 1000)
    to_time = int(yesterday.replace(hour=23, minute=59, second=59, microsecond=999999).timestamp() * 1000)

    response = logs_client.create_export_task(
        taskName=f"export-{yesterday.strftime('%Y-%m-%d')}",
        logGroupName=LOG_GROUP_NAME,
        fromTime=from_time,
        to=to_time,
        destination=S3_BUCKET_NAME,
        destinationPrefix=f"{S3_PREFIX}/{yesterday.strftime('%Y/%m/%d')}",
    )
    return {"status": "created", "taskId": response["taskId"]}

The key detail is checking PENDING in addition to RUNNING. If a scheduled task is merely sitting in the queue and you submit the next one anyway, you get a LimitExceededException.

The S3 bucket side needs two resource policies allowing writes from the CloudWatch Logs service principal.

// 1. Permission to check the bucket ACL before exporting
archiveBucket.addToResourcePolicy(
    new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        principals: [new iam.ServicePrincipal('logs.amazonaws.com')],
        actions: ['s3:GetBucketAcl'],
        resources: [archiveBucket.bucketArn],
        conditions: {
            ArnLike: { 'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*` },
        },
    }),
);

// 2. Permission to write the exported objects
archiveBucket.addToResourcePolicy(
    new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        principals: [new iam.ServicePrincipal('logs.amazonaws.com')],
        actions: ['s3:PutObject'],
        resources: [archiveBucket.arnForObjects('*')],
        conditions: {
            StringEquals: { 's3:x-amz-acl': 'bucket-owner-full-control' },
            ArnLike: { 'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*` },
        },
    }),
);

The condition pinning s3:x-amz-acl to bucket-owner-full-control is explicitly required by CloudWatch Logs' official documentation. Omit it and the export task fails with a permissions error.

5. Pattern C — Subscription filter → Lambda → S3

With LambdaDestination, CDK automatically manages the resource policy (Lambda::Permission) that lets logs.amazonaws.com invoke your Lambda.

new logs.SubscriptionFilter(this, 'CwlSubscriptionFilter', {
    logGroup: this.logGroup,
    destination: new logDestinations.LambdaDestination(this.archiveFunction),
    filterPattern: filterPattern
        ? logs.FilterPattern.literal(filterPattern)
        : logs.FilterPattern.allEvents(),
});

On the Lambda side, the code decodes the CWL compressed payload and writes it to S3 in a JSON format it defines itself, per invocation.

# cwl-to-s3/index.py (core logic)
def lambda_handler(event, context):
    compressed = base64.b64decode(event["awslogs"]["data"])
    payload = json.loads(gzip.decompress(compressed).decode("utf-8"))

    if payload.get("messageType") == "CONTROL_MESSAGE":
        # Health-check message CloudWatch Logs sends, e.g. when a subscription starts
        return {"status": "skipped", "reason": "control_message"}

    now = datetime.datetime.utcnow()
    safe_stream = payload["logStream"].replace("/", "_").replace("$", "").replace("[", "").replace("]", "")
    s3_key = f"{S3_PREFIX}/{now.strftime('%Y/%m/%d/%H')}/{safe_stream}_{uuid.uuid4().hex[:8]}.json"

    record = {
        "logGroup": payload["logGroup"],
        "logStream": payload["logStream"],
        "owner": payload.get("owner"),
        "exportedAt": now.isoformat() + "Z",
        "events": payload["logEvents"],
    }
    s3_client.put_object(Bucket=S3_BUCKET_NAME, Key=s3_key, Body=json.dumps(record, ensure_ascii=False).encode("utf-8"))
    return {"status": "ok", "processed": len(payload["logEvents"]), "s3Key": s3_key}

Deploy & Verify

export PROJECT=your-project
export ENV=dev

npm run bootstrap -w workspaces/cloudwatch-logs-s3-archive   # first time only
npm run stage:deploy:all -w workspaces/cloudwatch-logs-s3-archive

To deploy individual stacks, use the per-stack stage:deploy:* scripts (each scoped to a cdk deploy stack selector like **/*Basic).

npm run stage:deploy:basic -w workspaces/cloudwatch-logs-s3-archive
npm run stage:deploy:lifecycle -w workspaces/cloudwatch-logs-s3-archive
npm run stage:deploy:export -w workspaces/cloudwatch-logs-s3-archive

# Pattern A Stack 1 (Basic): write test data into 5 log groups (/<project>/<env>/basic-*)
# (given the dynamic-partitioning buffering requirements of >= 60s / >= 64MiB, use the
#  repo's helper script rather than a one-off put-log-events call)
./write-test-logs.sh --project <project> --env <env>

# After ~60s, output appears partitioned under AWSLogs/<owner>/CWLogGroup/<logGroup>/...
aws s3 ls s3://<archive-bucket>/AWSLogs/ --recursive

# Pattern A Stack 2 (Lifecycle): single log group, no dynamic partitioning
./write-test-logs-lifecycle.sh --project <project> --env <env>
aws s3 ls s3://<archive-bucket>/ --recursive

# Pattern B: manually invoke the export Lambda
aws lambda invoke --function-name <project>-<env>-cwl-export-task --payload '{}' response.json
cat response.json

# Pattern C: write a test log entry (the subscription filter invokes the Lambda immediately)
aws logs put-log-events \
  --log-group-name /<project>/<env>/app-lambda \
  --log-stream-name test-stream \
  --log-events timestamp=$(date +%s000),message="hello lambda"
aws s3 ls s3://<archive-bucket>/subscriptions/ --recursive

Cost Estimate

💰 Rough monthly estimate (Tokyo region, 1 GB of log data per day)

Service	Applies to	Rough monthly cost
Kinesis Data Firehose	A	~$0.90 (at $0.03/GB)
Lambda	B, C	Effectively negligible at low invocation counts
S3 (Standard)	All	~$0.75 (at $0.025/GB-month)
CloudWatch Logs (ingestion)	All	~$0.76/GB (charged regardless of which archive pattern you use, and not included in the total below)
EventBridge	B	$1.00 per million events (negligible at minimal frequency)

Running all five stacks simultaneously at production-scale traffic for comparison testing will cost more than the above. Don't forget to clean up afterward with npm run stage:destroy:all -w workspaces/cloudwatch-logs-s3-archive.

Summary

What we learned from this pattern:

Pattern A (Firehose): Minimal code, well-suited for near-real-time archiving. Delivery from CloudWatch Logs requires FirehoseDestination (L2) plus two IAM roles with distinct responsibilities
Pattern B (Export Task): The lowest cost for batch use cases, but limited to one concurrent execution per account. A guard checking for running/pending tasks on the Lambda side is essential
Pattern C (Lambda): Full control over the output format, at the cost of ongoing Lambda code maintenance, including handling control messages
IAM trust relationships differ by service: For Firehose, FirehoseDestination grants permission to the trusted role via grantPutRecords; for Export Task, an S3 bucket policy with an s3:x-amz-acl condition is required; for LambdaDestination, CDK auto-generates the invoke permission — the same "delivery from CloudWatch Logs" concept wires up differently every time
Split tiered lifecycle rules by transition: rather than one giant rule, separate rules make the transitionAfter thresholds easy to tune independently as environment parameters
fromLogGroupName() is enough for bolting archiving onto an existing log group: pass the existing ARN into the trust policy directly and reuse the same wiring, with no new resource created
Firehose dynamic partitioning is incompatible with message extraction: CloudWatchLogProcessor strips owner/logGroup, so drop it if you're using jq to derive partition keys from those fields. You also need to account for CloudWatch Logs' CONTROL_MESSAGE records (where owner/logGroup are empty strings) with a jq if/then/else fallback (// doesn't catch empty strings)

References

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS CDK 100 Drill Exercises #008: CloudFront VPC Origin — Reaching an Internal ALB Directly, and Designing an Incident-Response Escape Hatch

TOMOAKI ishihara — Tue, 28 Jul 2026 15:19:43 +0000

Introduction

This is the 8th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

This time, we place two very different origins behind a single Amazon CloudFront distribution: a private S3 bucket serving a static site, and a private, internal ALB that CloudFront can reach only through a VPC Origin. No public IP, no NAT Gateway — CloudFront reaches the ALB directly in a private subnet.

And we don't stop at a "working architecture" — we also build in an incident-response mechanism based on a real AWS outage.

To keep this sample simple, the CloudFront⇔ALB communication uses HTTP (port 80) only. In production, you'd want to attach an ACM certificate to the ALB, add an HTTPS listener, and switch CloudFront's origin protocol policy to HTTPS_ONLY (or match-viewer).

What you'll learn in this article

How to serve content without ever making an origin public, using VPC Origin + Origin Access Control (OAC)
Automatic failover from a dynamic origin to a static one via Origin Group
How to allow-list IP addresses with AWS WAF
Recording denied requests to CloudWatch Logs via CloudFront Function + standard logging v2
How to deal with the constraint that some CloudFront APIs and metrics are pinned to us-east-1
A way of thinking about designing "what to do when it breaks" as a CDK parameter, assuming failure up front

📁 Code repository: GitHub

Architecture Overview

Feature	Benefit
VPC Origin	CloudFront reaches a private ALB directly, with no public IP and no NAT Gateway
Origin Group failover	Shows a static fallback page instead of a raw 502/503 when the ALB is unhealthy
Edge IP allow-listing	Disallowed IPs are rejected at the edge before ever reaching the ALB or S3
Geo restriction	Restricts delivery to only the explicitly allowed countries
Incident-response escape hatch	Routes around a VPC Origin-side outage with just a parameter switch — no code changes

Data Flow

Viewer
  │  HTTPS (min TLS 1.3, geo-restricted to JP/US/GB/CA/AU/NZ/IE)
  ▼
CloudFront distribution
  ├─ WAF (Web ACL, always active): managed rules + rejects IPs not on the allow list
  ├─ CloudFront Function (optional, disabled by default): rejects IPs not on the allow list
  │
  ├─ Default behavior "/*"
  │     └─ S3 origin (OAC) ────────► WebsiteBucket (private)
  │
  └─ Behavior "/alb/*"
      └─ Origin Group
          ├─ Primary: VPC Origin ──► Internal ALB (private subnet)
          └─ Fallback (403/404/5xx): S3 origin (OAC) ─► ErrorBucket (private)

The ALB's security group allows inbound traffic only on port 80 from the CloudFront-managed prefix list. CloudFront itself is reachable from the internet, but the ALB is not directly accessible.

Key Components

Component	Design Points
ALB (internal)	`internetFacing: false`, placed in a private-isolated subnet. The VPC Origin's target
ALB (Public, created only when `publicAlbFailover` is enabled)	`internetFacing: true`, public subnet. A dedicated security group also allows only the CloudFront-managed prefix list
WebsiteBucket / ErrorBucket	Private S3 buckets, readable only by CloudFront via OAC
CloudFront distribution	Min TLS 1.3, geo restriction, dedicated access log bucket
CloudfrontWafStack (pinned to us-east-1)	Managed rules (Core/KnownBadInputs) + IP allow list (before/after rules selectable), defaultAction is block, logs delivered directly to S3
CloudFront Function	Viewer IP allow list (optional, created only when `allowedCloudFunctionIps` is set), evaluated on `viewer-request`
CloudfrontLogDeliveryStack (pinned to us-east-1, created only when the CloudFront Function is used)	Delivers denied-request logs to CloudWatch Logs
CloudfrontMonitoringStack (pinned to us-east-1)	5xx error rate alarm + SNS notification

Implementation Highlights

1. An internal ALB reachable only from CloudFront

The ALB's security group opens port 80 only to the CloudFront-managed prefix list.

albSecurityGroup.addIngressRule(
  ec2.Peer.prefixList(props.cloudfrontManagedPrefixList),
  ec2.Port.tcp(80),
  'Allow inbound HTTP traffic from CloudFront managed prefix list'
);

const alb = new elbv2.ApplicationLoadBalancer(this, 'Alb', {
  vpc: this.vpc.vpc,
  internetFacing: false,
  securityGroup: albSecurityGroup,
  vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
});

The prefix list ID is region-specific. Check it beforehand.

aws ec2 describe-managed-prefix-lists \
  --filters "Name=prefix-list-name,Values=com.amazonaws.global.cloudfront.origin-facing"

2. Automatic failover with VPC Origin and Origin Group

CloudFront reaches the internal ALB directly via a VPC Origin. If the ALB stops responding, CloudFront automatically fails over to the static page in S3.

const originGroup = new cloudfront_origins.OriginGroup({
  primaryOrigin: cloudfront_origins.VpcOrigin.withApplicationLoadBalancer(alb, {
    httpPort: 80,
  }),
  fallbackOrigin: cloudfront_origins.S3BucketOrigin.withOriginAccessControl(errorBucket),
  fallbackStatusCodes: [403, 404, 500, 502, 503, 504],
});

distribution.addBehavior('/alb/*', originGroup, {
  viewerProtocolPolicy: cloudfront.ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
  // Without this, the default CACHING_OPTIMIZED policy applies (1-day default TTL). The ALB's
  // fixed responses carry no Cache-Control header at all, so once a successful response gets
  // cached at the edge, it keeps being served for up to a day regardless of the origin's or
  // Origin Group's actual state.
  cachePolicy: cloudfront.CachePolicy.CACHING_DISABLED,
});

Without this explicit cache policy, you'll run into a confusing situation while testing failover: "the origin has switched, but the browser/edge keeps serving a stale cached success response, so nothing looks different." /alb/* exists to show you the ALB's live response, so disabling caching outright is the straightforward choice here.

3. IP allow-listing and managed rules with WAF

Edge access control is handled first by a WAFv2 Web ACL attached to the whole distribution (CloudfrontWafStack). Since defaultAction is block, the effective behavior is that everything except allow-listed IPs is rejected.

const wafAcl = new wafv2.CfnWebACL(this, 'WafAcl', {
  scope: 'CLOUDFRONT',
  defaultAction: { block: {} },
  rules: [
    // Allow before managed rules (only created if IPs are actually specified. priority 1)
    { name: 'AllowSpecificIPsBeforeRules', priority: 1, action: { allow: {} }, ... },
    { name: 'CoreRuleSet', priority: 2, overrideAction: { none: {} }, /* AWSManagedRulesCommonRuleSet */ },
    { name: 'KnownBadInputsRuleSet', priority: 3, overrideAction: { none: {} }, /* AWSManagedRulesKnownBadInputsRuleSet */ },
    // Allow after managed rules (enabled by default. priority 100)
    { name: 'AllowSpecificIPsAfterRules', priority: 100, action: { allow: {} }, /* IPSetReferenceStatement */ },
  ],
});

In the current wiring, the deploying operator's IP (getMyGlobalIp()) is passed as allowedIpsAfterRules, and is allowed after the managed rules are evaluated (priority 100). In other words, even an allow-listed IP is first checked against the Core/KnownBadInputs managed rules for known attack patterns before being allowed through. The allowedIpsBeforeRules option (allow before rule evaluation) is there for cases where you want to verify behavior before the WAF rules take effect.

WAF logs are delivered directly to an S3 bucket whose name must carry the aws-waf-logs- prefix (CfnLoggingConfiguration). The authorization and cookie headers are masked via redactedFields before being recorded.

4. Edge control via CloudFront Function, and incident investigation via standard logging v2

Separately from WAF, a CloudFront Function-based allow list is also implemented for cases where you want finer, per-behavior control. When allowedCloudFunctionIps is set, every viewer request is evaluated and IPs not on the list are rejected. Denied requests are recorded as custom data via cf.logCustomData().

function handler(event) {
  var request = event.request;
  var allowedIps = ['203.0.113.10']; // baked in at synth time
  var clientIp = event.viewer.ip; // viewer IP accessor for the JS_2_0 runtime
  if (!allowedIps.includes(clientIp)) {
    cf.logCustomData(JSON.stringify({ clientIp: clientIp, allowedIps: allowedIps }));
    return { statusCode: 403, statusDescription: 'Forbidden', body: 'Access denied' };
  }
  return request;
}

The CloudFront Function is positioned as an optional feature you enable as needed.

5. The incident-response escape hatch (`publicAlbFailover`)

This is the point I most want to get across in this article.

On July 16, 2026, AWS CloudFront experienced a multi-hour outage in which 5xx errors spiked for customers using VPC Origins. The root cause was not in customers' own stacks — it was in the VPC Origin connectivity layer itself.

We identified the root cause of the issue as an internal constraint on the fleet that manages connections to private VPC origins. When this constraint was reached, the system responsible for distributing routing configuration to our network processors failed to load the updated configuration data correctly, affecting routing of VPC Origin connections.

AWS's guidance at the time was: "customers who are able to should temporarily switch away from the VPC Origin connection, and switch back once it's resolved."

The problem is that trying to do this by manually rewriting CloudFormation while the incident is ongoing makes both the decision-making and the work slower. So this stack prepares the switch ahead of time, as a single publicAlbFailover parameter.

// The internal ALB (for VPC Origin) is always created. A separate public ALB is created only
// when publicAlbFailover is enabled.
const publicAlb = publicAlbFailoverEnabled
  ? new elbv2.ApplicationLoadBalancer(this, 'PublicAlb', {
      internetFacing: true,
      securityGroup: publicAlbSecurityGroup,
      vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
    })
  : undefined;

// The VPC Origin used as the primaryOrigin in normal mode
const vpcOriginAlb = cloudfront_origins.VpcOrigin.withApplicationLoadBalancer(alb, {
  httpPort: 80,
  protocolPolicy: cloudfront.OriginProtocolPolicy.HTTP_ONLY,
});

// Escape hatch enabled: reach the separate public ALB as a plain public HTTP origin
const publicAlbOrigin = publicAlbFailoverEnabled
  ? new cloudfront_origins.HttpOrigin(publicAlb!.loadBalancerDnsName, {
      httpPort: 80,
      protocolPolicy: cloudfront.OriginProtocolPolicy.HTTP_ONLY,
    })
  : undefined;

// The fallback is always the S3 error page; only the primary swaps to the public ALB when enabled
const originGroup = new cloudfront_origins.OriginGroup({
  primaryOrigin: publicAlbFailoverEnabled ? publicAlbOrigin! : vpcOriginAlb,
  fallbackOrigin: cloudfront_origins.S3BucketOrigin.withOriginAccessControl(errorBucket),
  fallbackStatusCodes: [403, 404, 500, 502, 503, 504],
});

There are 4 design points here.

Traffic still only ever flows through CloudFront. Enabling this makes the newly-created public ALB internet-facing, but its security group — just like the internal ALB's — still only allows the CloudFront-managed prefix list, so it's never directly reachable by clients. The client-facing URL doesn't change either.
Setting publicAlbFailover.enabled: true makes cloudfrontManagedPrefixList a required field. Without it, there'd be no way to restrict the public ALB to CloudFront only, so it throws at synth time.
The prefix list alone isn't enough. The CloudFront managed prefix list is a shared IP range used by every CloudFront customer's distribution, so if someone else's CloudFront distribution pointed at this public ALB's DNS name as an origin, it would still pass the security group check. So, following AWS's official recommended procedure, we generate a secret value at synth time and send it as a CloudFront origin custom header, with the ALB listener rejecting (via its default action, 403) any request that doesn't carry that header.
It switches over in "minutes, not instantly." We originally tried a design where keeping the VPC Origin bound as the Origin Group's fallback would avoid recreating it. But CDK auto-derives an origin's logical ID from its "position" — which includes whether it's the OriginGroup's primary or fallback role — so simply swapping that role changes the VPC Origin's own logical ID, and CloudFormation ends up deleting and recreating it anyway (up to ~15 minutes). (I cover this CDK logical-ID auto-numbering mechanism in depth in my own article, "AWS CDK Logical ID Deep Dive: How Adding One CloudFront Origin Broke My Entire AWS CDK Deployment".) You can pin the logical ID with overrideLogicalId to avoid that, but then you trade it for a different constraint: CloudFront's UpdateVpcOrigin API rejects any in-place property change to the origin while it's still associated with a distribution. This sample favors simplicity and accepts that the VPC Origin gets recreated.

If you need genuinely zero-downtime, instant switching in production, consider the CloudFront continuous deployment approach (a staging distribution with header/weight-based gradual traffic shifting) or CloudFront Functions + KeyValueStore-based weighted routing, both covered in Migrate Amazon CloudFront public origins to private VPC origins. Both keep both origins alive at all times and shift traffic gradually by ratio — a step up in complexity from this article's Origin Group swap, but the right choice when you truly need zero-downtime switching.

Using it is simple.

// parameters/dev-params.ts
publicAlbFailover: {
    enabled: true, // false → true
},

npm run deploy:all

Once AWS resolves the issue, just set enabled: false and redeploy.

6. 5xx error rate alarm (also pinned to us-east-1)

CloudFront's request/error metrics (the AWS/CloudFront namespace) are only published to us-east-1, regardless of the distribution's own region. A CloudWatch Alarm can only evaluate a metric in its own region, so this alarm is also broken out into its own stack, CloudfrontMonitoringStack, deployed to us-east-1.

const alarm = new cloudwatch.Alarm(this, 'CloudFront5xxErrorRateAlarm', {
  metric: new cloudwatch.Metric({
    namespace: 'AWS/CloudFront',
    metricName: '5xxErrorRate',
    dimensionsMap: { DistributionId: props.distributionId },
    period: cdk.Duration.minutes(5),
    statistic: cloudwatch.Stats.AVERAGE,
  }),
  threshold: 5, // 5%
  evaluationPeriods: 3,
  datapointsToAlarm: 3, // sustained for 15 minutes before treating it as a real incident
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
});

This alarm watches exactly the metric AWS itself called out during the 2026-07-16 outage (the distribution's overall 5xx error rate). In other words, it's designed to plug directly into an operational flow: "alarm fires" → "check the AWS Health Dashboard" → "if it's a VPC Origin-side problem, enable publicAlbFailover."

Deploy & Verify

export PROJECT=your-project
export ENV=dev

npm run bootstrap   # first time only
npm run deploy:all

# Served from S3
curl https://<distribution-domain>/

# Served from the ALB via the Origin Group
curl https://<distribution-domain>/alb/

Cost Estimate

💰 Rough monthly estimate (Tokyo region, low traffic)

Service	Usage	Rough monthly cost
Application Load Balancer	Always on	~$25.00
CloudFront	Low request volume	~$1–2
AWS WAF	Web ACL + 3 rules + low request volume	~$8.00
S3 (3 buckets)	A few MB of static content + logs	< $0.10
CloudWatch Alarm + SNS	5xx error rate alarm notifications	< $0.10

Total: roughly $23–25/month

The ALB is billed hourly regardless of traffic. Don't forget to run npm run destroy:all to clean up after you're done experimenting.

If cdk destroy / npm run destroy:all gets stuck — a non-empty S3 bucket, a leftover ENI tied to the VPC Origin, or some other resource CloudFormation refuses to delete on its own — delstack, an OSS tool by AWS DevTools Hero k.goto (@365_step_tech), is worth having on hand. It forcibly cleans up the kind of resources that commonly block stack deletion.

Summary

What we learned from this pattern:

VPC Origin: CloudFront can reach a fully private ALB directly, with no public IP and no NAT Gateway
Origin Group: Automatic failover from a dynamic origin to a static one when a server error occurs
Defense in depth at the edge: Geo restriction, WAF, and CloudFront Function give you access control before traffic ever reaches the origin
CloudFront's us-east-1 constraint: Both the log delivery API and metrics are pinned to us-east-1, regardless of the distribution's own region
Bake incident response into code: Given a real outage, prepare "how to route around it" as a parameter ahead of time rather than as manual work
The prefix list isn't a silver bullet: Since it's a shared IP range across every customer, an internet-facing origin should also verify a secret custom header
Be explicit about cache policy on dynamic paths: Leaving the default CACHING_OPTIMIZED in place means a successful response from an origin that never sends Cache-Control gets cached for up to a day, masking your failover test results
CDK's origin logical-ID auto-numbering also depends on "role": Just swapping which origin is primary vs. fallback changes that origin's logical ID, so it can be recreated on every switch. Genuinely zero-downtime switching needs continuous deployment or CloudFront Functions + KeyValueStore-based weighted routing

References

Amazon CloudFront VPC origins
Using origin groups for failover
AWS CDK Logical ID Deep Dive: How Adding One CloudFront Origin Broke My Entire AWS CDK Deployment — my own deep dive into how CDK's origin logical-ID auto-numbering works
Migrate Amazon CloudFront public origins to private VPC origins
Using the managed cache policies (CachingOptimized / CachingDisabled)
Restrict access to Application Load Balancers
AWS WAF developer guide
CloudFront standard logging v2
Monitoring CloudFront distributions with CloudWatch metrics
AWS Health Dashboard
X: Amazon CloudFront Issue Resolved
delstack — an OSS tool by AWS DevTools Hero k.goto (@365_step_tech) for force-deleting CloudFormation stacks stuck on resources like non-empty S3 buckets or leftover ENIs

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS or Amazon Quiz — How Well Do You Know AWS Service Prefixes?

TOMOAKI ishihara — Thu, 23 Jul 2026 15:16:57 +0000

Introduction

AWS (Amazon Web Services) offers 200+ services these days. That's a lot of names to keep straight.

And there's one thing that trips up everyone, beginners included. You get used to just saying "Route 53" out loud, and then when you're writing documentation you suddenly blank on the official name: is it "AWS Route 53" or "Amazon Route 53"?

Right?

AWS or Amazon

I figured turning that everyday confusion into a game could be fun, so I looked around for something similar. There are quizzes that show a service icon and ask you to name it, and quizzes that describe a service and ask you to pick the name — but nothing focused purely on "is it AWS or Amazon?"
Naming the full service correctly is hard for beginners, but just picking the prefix felt like something anyone could enjoy.

These days, if you have an idea, AI makes it quick to build. So I sketched out the rules for the game right away.

By the way, the distinction between the two prefixes is explained in plenty of places, but roughly:

Amazon: standalone services that work on their own
AWS: utility services typically used by other services

What I built

Here's what I ended up with. It's playable entirely in the browser.
It's hosted on GitHub Pages, and the source code is here.

Designing the game

I wanted something simple: a two-choice format you can breeze through. So I built it to pull 20 random questions from a dataset of AWS service info.
Beyond just picking an answer, each question shows a short explanation of the service and a link to the official docs, so you can actually learn something along the way.

Choosing where to host it

Since the game is just plain JavaScript and a single HTML file, GitHub Pages was the obvious choice for hosting.

Improving the UI

In the first version, the "Next" button appeared at the bottom of the screen after you answered. That worked, but it was too far from the answer choices, and on mobile it made moving to the next question clunky.

So I compared a few design options.

I also had a rank displayed based on your score, but it wasn't obvious whether a given rank was good or bad, so I simplified it to a plain evaluation instead.

Wrapping up

That's the story behind building "AWS or Amazon Quiz" and what I paid attention to along the way.
If you play it and catch yourself thinking "wait, I thought that one was Amazon" — that's a sign the naming conventions are starting to stick.

If you enjoyed it, a star on GitHub would mean a lot.
Bug reports and corrections (like "this service's description is wrong") are welcome via Issues or comments — the service descriptions were generated by AI since there are so many of them, so a few may not be entirely accurate.

The problem of putting Transit Gateway subnets at the "end" of a CIDR — existing tools couldn't solve it, so I built one myself

TOMOAKI ishihara — Sun, 19 Jul 2026 13:01:50 +0000

Introduction

How do you calculate CIDRs when designing VPC subnets? Do you work through them one by one with an online subnet calculator?

Or maybe you reach for the AWS Subnet Calculator, or the Excel-based subnet calculator spreadsheet referenced in AWS docs > Amazon EKS > Amazon VPC CNI?

Both are handy tools, but I ran into a wall: I needed the subnet reserved for Transit Gateway to sit at the very end of the VPC's CIDR block, and neither tool made that easy.

Here's the background. At one organization I worked with, on-premises connectivity was built using AWS Direct Connect and Transit Gateway. Whenever a new AWS account was issued, the VPC and its Transit Gateway subnet came pre-provisioned as part of the standard setup. On top of that, the Transit Gateway subnet was always carved out from the tail end of the VPC's CIDR block by design.

So when I needed to lay out subnets for a business application, I needed a tool that could compute the Transit Gateway subnet from the end of the CIDR range. None of the existing tools did that, so I built my own.

The tool is live on GitHub Pages, and the source is on GitHub.

Features

Variable VPC CIDR — enter any valid CIDR block (e.g. 10.0.0.0/22)
Multi-tier subnet design — add, remove, and configure tiers with custom subnet masks
Drag-and-drop reordering — reorder tiers by dragging them into place
Dual AZ support — automatically allocates subnets across ap-northeast-1a and ap-northeast-1c
TGW tail allocation — tiers named "TGW" get carved out from the end of the VPC address space (toggle on/off)
Free address space detection — surfaces every leftover range: alignment gaps, the space between forward-allocated and TGW subnets, and trailing space
AWS reserved address aware — shows the usable host count after subtracting the 5 AWS-reserved addresses per subnet
Japanese/English language switching — toggle the UI language with a button; auto-detects the browser's language on load
Dark mode support — automatically adapts to the system color scheme

How the CIDR calculation works

There's no server and no database behind this tool — the entire calculation runs as bitwise arithmetic in the browser. The process breaks down into four steps.

1. Turn the CIDR string into a 32-bit integer

First, an input string like 10.0.0.0/22 gets converted into a 32-bit integer — a network address plus a total address count — so it can be manipulated with bitwise operations.

function parseVPC(cidr) {
  var m = cidr.trim().match(/^(\d+\.\d+\.\d+\.\d+)\/(\d+)$/);
  if (!m) return null;
  var parts = m[1].split('.').map(Number);
  if (parts.some(function(p) { return p < 0 || p > 255; })) return null;
  var prefix = parseInt(m[2]);
  if (prefix < 8 || prefix > 28) return null;
  var base = ((parts[0]<<24)|(parts[1]<<16)|(parts[2]<<8)|parts[3]) >>> 0;
  var total = Math.pow(2, 32 - prefix);
  var network = (base & (~((1 << (32 - prefix)) - 1) >>> 0)) >>> 0;
  return { base: network, prefix: prefix, total: total, end: network + total - 1 };
}

Two things are worth calling out:

parts[0]<<24 | parts[1]<<16 | parts[2]<<8 | parts[3] packs the four octets into a single 32-bit integer.
The >>> 0 matters. JavaScript's bitwise operators treat numbers as signed 32-bit integers, so without it, any first octet of 128 or higher would come out negative.

~((1 << (32 - prefix)) - 1) then builds the subnet mask, and base & mask rounds down to the network address. So even if you type 10.0.0.5/22, with a non-zero host part, it automatically snaps to 10.0.0.0/22.

2. Forward tiers: round up to the alignment boundary

Regular tiers like Private-App and Public get packed from the front of the VPC, AZ by AZ. The key constraint here is alignment: a /26 subnet can only start on a 64-address boundary (x.x.x.0, x.x.x.64, x.x.x.128, and so on).

var fwdCursor = vpc.base;
for (var ni = 0; ni < normalTiers.length; ni++) {
  var ntier = normalTiers[ni];
  var sz = Math.pow(2, 32 - ntier.mask);
  for (var az = 0; az < 2; az++) {
    var aligned = Math.ceil(fwdCursor / sz) * sz;
    var end = aligned + sz - 1;
    if (end > vpc.end) { overflow = true; break; }
    allocRows.push({ kind:'subnet', tier: ntier.name, az: AZS[az], azIdx: az,
      cidr: ipToStr(aligned) + '/' + ntier.mask, first: ipToStr(aligned), last: ipToStr(end),
      total: sz, usable: sz-5, tail: false, startNum: aligned, endNum: end });
    fwdCursor = end + 1;
  }
  if (overflow) break;
}

The line that does the real work is Math.ceil(fwdCursor / sz) * sz: it rounds the current cursor position up to the nearest multiple of the subnet size. That means even when one tier is packed right up against the end of the previous one, the result always snaps to a valid CIDR boundary. If the allocation would run past the end of the VPC, an overflow flag stops the calculation.

3. TGW tiers: round down to the alignment boundary, from the end

Any tier whose name contains "TGW" gets allocated in reverse, starting from the end of the VPC. This is the feature that made me build the tool in the first place.

function floorAligned(addr, size) { return Math.floor(addr / size) * size; }

var bwdCursor = vpc.end;
var tgwBuilt = [];
var tgwRev = tgwTiers.slice().reverse();
for (var gi = 0; gi < tgwRev.length; gi++) {
  var gtier = tgwRev[gi];
  var gsz = Math.pow(2, 32 - gtier.mask);
  for (var gaz = 1; gaz >= 0; gaz--) {
    var galigned = floorAligned(bwdCursor - gsz + 1, gsz);
    var gend = galigned + gsz - 1;
    if (galigned < vpc.base) { overflow = true; break; }
    tgwBuilt.unshift({ kind:'subnet', tier: gtier.name, az: AZS[gaz], azIdx: gaz,
      cidr: ipToStr(galigned) + '/' + gtier.mask, first: ipToStr(galigned), last: ipToStr(gend),
      total: gsz, usable: gsz-5, tail: true, startNum: galigned, endNum: gend });
    bwdCursor = galigned - 1;
  }
  if (overflow) break;
}

This mirrors the forward-allocation logic. bwdCursor - gsz + 1 gives a candidate starting address, and floorAligned rounds it down to a valid boundary — the mirror image of rounding up. The TGW tier list is reversed before processing, and each result is pushed onto the front of the array with unshift, so on screen the tiers still appear in their original order even though they're being filled in from the tail.

4. Detecting free address space

Once forward and tail allocations are done, all subnets get sorted by starting address, and any gap between two adjacent subnets gets recorded as free space.

var allSubnets = allocRows.concat(tgwBuilt).sort(function(a,b) { return a.startNum - b.startNum; });

var gaps = [];
var cursor = vpc.base;
for (var si = 0; si < allSubnets.length; si++) {
  var s = allSubnets[si];
  if (s.startNum > cursor) gaps.push({ kind:'free', startNum: cursor, endNum: s.startNum-1 });
  cursor = s.endNum + 1;
}
if (cursor <= vpc.end) gaps.push({ kind:'free', startNum: cursor, endNum: vpc.end });

This is the classic "find the gaps between intervals" algorithm. A cursor walks forward from the start of the VPC; whenever there's a gap between the cursor and the next subnet, it gets recorded, and if the cursor never reaches the end of the VPC, the leftover range at the end is recorded too. The odd leftover addresses caused by alignment — the space between the last forward-packed subnet and the TGW subnets, for example — get caught by this same mechanism, with nothing falling through the cracks.

Put together, two simple algorithms — rounding up or down to an alignment boundary, and finding gaps between intervals — are enough to handle a fairly involved requirement (multiple tiers, multiple AZs, tail allocation) in a few dozen lines of serverless code.

If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in My GitHub Repository.

CTF Event Report: Security-JAWS 10th Anniversary Day 2 — All 27 AWS Security Challenges Solved

TOMOAKI ishihara — Mon, 18 May 2026 13:13:37 +0000

Introduction

I participated in the CTF held on Day 2 of "Security-JAWS DAYS ~10th Anniversary Event~", organized by Security-JAWS, a Japanese AWS user community focused on cloud security.

The CTF was themed around a fictional SaaS company called "TechVault", and the scenario had us conducting a penetration investigation — starting from their employee portal and ultimately uncovering evidence of fraudulent transactions. It was an exceptionally well-crafted CTF with a cohesive narrative running through all challenges.

Event Overview

Duration: 13:00–17:00 / 4 hours
Total challenges: 27
- Tutorial: 6 / 290 pt
- Mainline: 12 / 2,300 pt
- Bonus: 5 / 1,400 pt
- Advanced: 2 / 700 pt
- Blue Team: 1 / 300 pt
- Finale: 1 / 600 pt
Setting: AWS environment of a fictional SaaS company "TechVault"

The story begins with an intrusion investigation of TechVault's portal service and culminates in gathering evidence of someone's fraudulent transactions. The level of polish in the scenario design was remarkable.

Results

Challenges solved: 27 out of 27
Score: 5,310 pt (max: ~5,590–5,650 pt)
Time to complete: 2 hours 48 minutes 36 seconds
Final ranking: 12th out of 125 participants
What went well:
- Using knowledge and CLI tools I don't normally touch in daily work, and working through them hands-on gave me a much deeper understanding of each technique.
What I'd improve:
- I should have set up my environment beforehand. I normally use devContainers, so my host machine only had the minimum: AWS CLI and Python. Docker, OpenSSL, Boto3, and similar tools were missing, which cost me more time than necessary.

Challenge Structure

Tutorial (290 pt)

A step-by-step introduction to web reconnaissance and AWS CLI basics.

ID	Title	Description
T1	Web Recon · robots.txt	Discover hidden paths from `robots.txt` Disallow directives
T2	The Unlocked Warehouse · Public S3	Retrieve files directly from a publicly exposed S3 bucket
T3	Behind the Page · HTML Source	Investigate credentials buried in HTML comments
T4	First Steps with curl	Check information embedded in HTTP response headers
T5	Leaked Config · .env File	Find a `.env` file mistakenly placed in the web root
T6	First Steps with AWS CLI	Use the key found in `.env` to run `sts get-caller-identity`

The T5→T6 flow was clever. You grab a key from .env and immediately use it with the AWS CLI — a hands-on demonstration of how a web vulnerability becomes an AWS entry point.

Mainline (2,300 pt)

The core of the CTF: an attack chain that follows the path of intrusion → privilege escalation → evidence collection, starting from Stage 0.

ID	Title	Description
Stage 0	The Forgotten Debug Mode	Extract AWS keys from debug output left in an auth API error response
Stage 1A	Flip the Bucket	Find a file hidden under a `.hidden/` prefix in an S3 bucket
Stage 1B	Who Am I?	Read IAM policy metadata to understand the compromised user's permissions
Stage 1C	The Past Never Disappears	Recover AWS keys left in a Git repository's commit history
Stage 1D	Ask the AI	Prompt injection against an AI assistant embedded in the employee portal
Stage 2A	The Deleted File	Recover a deleted file using S3 object versioning
Stage 2B	The Permission Map	Use `sts:AssumeRole` to pivot laterally into the DataAnalystRole
Stage 2D	The Function's Secret	Retrieve sensitive data stored in Lambda environment variables
Stage 2E	The Parameter Labyrinth	Navigate SSM Parameter Store paths to collect secrets
Stage 2G	The AI's Permissions	Extract S3 data via an over-privileged Bedrock agent
Stage 3A	The Vault Key	Retrieve the ZIP decryption password from Secrets Manager
Final	Consolidate the Evidence	Decrypt the evidence file using all collected information to expose the CEO's fraud

The Bedrock agent challenge (Stage 2G) was fresh. The agent was configured with direct S3 access, so data from a bucket I couldn't read directly could be pulled out simply by asking the agent "show me the project metadata." It drove home how important permission design is when integrating AI into your stack.

Bonus / Advanced / Blue Team / Finale (3,000 pt)

Additional challenges branching off the mainline, each requiring deeper technical knowledge.

ID	Title	Category	Description
Stage 2C	The Server's Shadow	bonus	Flag stored in EC2 instance tags
Stage 2F	Find It Automatically	bonus	Scan all branches for secrets using `gitleaks`
Stage 3B	The Invisible Voice	bonus	SSRF to IMDSv1 to steal EC2 role temporary credentials
Stage 3C	The False Face	advanced	Self-declare `custom:role=admin` during Cognito sign-up
Stage 3D	The Truth Inside the Image	bonus	Recover files deleted by `RUN rm` from Docker image layers
Stage 3E	The Neighbor's Vault	advanced	Read another tenant's data via wildcard permissions on S3 Vectors
Stage 4	Follow the Trail	blueteam	Identify attacker operation timestamps from CloudTrail logs
Stage 5	Suspicious Activity	finale	Decrypt CTO complicity evidence by tracing late-night activity in CloudTrail
Stage 5B	Combined Attack Surface	bonus	Call an internal API by combining intelligence from Stage 3D and Stage 3E

Stage 3D was by far the most time-consuming for me — because Docker wasn't installed in my CTF environment. Instead of using docker history (which would have shown it in seconds), I had to query the ECR API directly to fetch the image manifest, download each layer, and extract the tarballs manually. Painful on the clock, but I ended up with a much deeper understanding of how Docker image layers actually work.

Stage 4 and Stage 5 involved parsing large CloudTrail log files with jq to reconstruct the attacker's footsteps — a great taste of what SOC/incident response work feels like. Stage 5 in particular required chaining multiple steps: find the suspicious late-night (JST) operations in the logs, track down the Secrets Manager path they pointed to, and decrypt the encrypted evidence file with OpenSSL. OpenSSL wasn't available either, so I ended up implementing the decryption in Python.

The Full Attack Chain

Each challenge looks independent, but they're all connected as a single story.

Obtain AWS keys from debug API response (Stage 0)
        ↓
IAM recon reveals an AssumeRole-able role (Stage 1B)
        ↓
Pivot laterally into DataAnalystRole (Stage 2B)
        ↓
┌─────────────────────────────────────────────────────┐
│  Collect intelligence across multiple parallel paths │
│  · EC2 tags (Stage 2C)                              │
│  · S3 versioning — recover deleted files (Stage 2A) │
│  · Lambda environment variables (Stage 2D)          │
│  · SSM Parameter Store (Stage 2E)                   │
│  · SSRF → IMDSv1 (Stage 3B)                        │
│  · ECR Docker layer analysis (Stage 3D)             │
│  · S3 Vectors cross-tenant leak (Stage 3E)          │
└─────────────────────────────────────────────────────┘
        ↓
Retrieve password from Secrets Manager (Stage 3A)
        ↓
Decrypt ZIP to obtain CEO fraud evidence (Final)
        ↓
Trace CTO complicity via CloudTrail (Stage 4 → 5)

Each individual vulnerability might look limited in isolation, but chaining them together produces a critical breach. Stage 5B is the perfect example: an internal API only reachable by combining intelligence gathered from two separate advanced stages.

Key Takeaways and Mitigations

Information Leakage (Debug, Headers, etc.)

Debug output that's convenient during development can leak AWS keys if left enabled in production. HTML comments, response headers, and robots.txt are all reconnaissance vectors attackers regularly check.

Mitigations:

Disable debug mode in production environments.
Remove unnecessary response headers like X-Powered-By.
Never place secrets in front-end source code.

S3 Misconfiguration

Three distinct S3 issues appeared: public access enabled, a .hidden/ prefix used as security-by-obscurity, and deleted files recoverable via versioning. All three stem from treating S3 like a traditional filesystem.

Mitigations:

Enable Block Public Access on all buckets.
Prefixes are not access controls.
If versioning is enabled, also design lifecycle policies to expire delete markers and old versions.

Secrets in Git History

Even after deleting a .env file and committing the removal, git log -p surfaces it instantly. Tools like gitleaks can scan every branch and every commit in seconds.

Mitigations:

Integrate git-secrets or gitleaks as a pre-commit hook.
If a secret was already committed, rewrite history with git filter-repo and rotate the key immediately.

Prompt Injection

A single sentence — "ignore previous instructions" — was enough to extract the contents of the system prompt. Using the system prompt as a "hidden" information store is not a security boundary.

Mitigations:

Never put sensitive information in system prompts.
Validate both inputs and outputs. Make the boundary between user input and system instructions explicit.

Overly Broad IAM Permissions and AssumeRole

Having sts:AssumeRole allows switching to a different role. In this CTF, flags were embedded in IAM policy descriptions and EC2 tags for challenge purposes — but in the real world, metadata fields are an underappreciated place for sensitive data to accumulate.

Mitigations:

Apply the principle of least privilege rigorously.
When granting sts:AssumeRole, restrict the target resources.
Use Condition keys in trust policies to restrict callers.

Poor Secret Management

Three storage locations appeared: Lambda environment variables, SSM Parameter Store, and Secrets Manager. Even Secrets Manager provides no protection if the IAM permissions granting GetSecretValue are too broad.

Mitigations:

Manage secrets in Secrets Manager.
Scope the GetSecretValue resource policy to the specific secret ARN.

SSRF × IMDSv1

A URL preview feature in the dashboard was fetching external URLs server-side — and there was no filtering to block requests to http://169.254.169.254. IMDSv1 requires no token, so SSRF access to the link-local address yields EC2 role temporary credentials directly.

Mitigations:

Enforce IMDSv2 (HttpTokens: required) on all EC2 instances.
URL-fetching features should use an allowlist, and must block private IP ranges and link-local addresses.

Secrets Persisted in Docker Image Layers

A Dockerfile pattern like COPY secret.txt . → RUN python setup.py → RUN rm secret.txt produces a final image where secret.txt is not visible at runtime. However, downloading the image layers directly from the ECR API reveals secret.txt intact in a prior layer's tarball.

Mitigations:

Use multi-stage builds; never copy secrets into build contexts.
Retrieve secrets from Secrets Manager at runtime instead.

Broken Multi-Tenant Permission Design

The S3 Vectors resource policy was set to Resource: "*", allowing a role scoped to one tenant to query another tenant's vector data. Tenant isolation in SaaS demands rigorous permission separation.

Mitigations:

Constrain the Resource and Condition in resource policies to tenant-specific identifiers.
If sharing a vector bucket, scope queries and metadata access by tenant at the API level.

Cognito Authorization Design Flaw

Passing custom:role=admin in --user-attributes during aws cognito-idp sign-up was enough to self-declare administrator status, which the application then trusted for authorization decisions.

Mitigations:

Control role assignment server-side (e.g., in a Pre Sign-up Lambda Trigger).
Never use attributes that external parties can set as the basis for authorization.

AI Agent Over-Privilege

The Bedrock agent's IAM role had access to S3 buckets that the DataAnalystRole itself could not read. By asking the agent a natural-language question, data from otherwise-inaccessible buckets was pulled out indirectly.

Mitigations:

Apply the principle of least privilege to AI agent roles as well.
Explicitly enumerate and restrict the resources an agent is permitted to access.

CloudTrail for Evidence Preservation

With CloudTrail logs in place, "what happened, when, and by whom" can be reconstructed almost completely. A handful of jq filters were enough to trace the attacker's full activity.

Mitigations:

Enable CloudTrail in all regions and ship logs to S3.
Pair with GuardDuty for real-time detection.
Apply Object Lock to the log bucket to prevent tampering.

Closing Thoughts

All 27 challenges together gave me a visceral sense of how AWS misconfigurations cascade. Each individual problem represented a realistic "seen-in-the-wild" vulnerability — but what made this CTF special was that they were all woven into a single coherent story.

AWS certifications don't teach you why something is dangerous. Solving these challenges hands-on — making mistakes, working around missing tools, figuring out the low-level APIs when Docker wasn't available — built an intuition that studying documentation alone never could.

Highly recommend participating if a similar opportunity comes around. And if you're building on AWS, I hope this report serves as a useful checklist of things worth double-checking in your own environment.

AWS CDK 100 Drill Exercises #007: SQS-Lambda-Firehose —— Building Event-Driven Data Pipelines

TOMOAKI ishihara — Mon, 26 Jan 2026 15:23:32 +0000

Introduction

This is the 8th installment of "AWS CDK 100 Drill Exercises". For the series overview, see here.

In this exercise, we'll build an event-driven data pipeline combining SQS, Lambda, and Firehose.

Why SQS-Lambda-Firehose?

Feature	Benefit
Event-Driven	Loosely coupled and scalable
Reliability	Robust error handling with DLQ and batch failure reporting
Cost Efficiency	Serverless with pay-per-use pricing
Reduced Ops	Minimize infrastructure management with managed services

What You'll Learn

SQS + Dead Letter Queue design
Lambda ReportBatchItemFailures for batch processing
Firehose → S3 streaming delivery
Production monitoring with CloudWatch Alarms

📁 Code Repository: GitHub

Architecture Overview

Data Flow

Producer → SQS Queue → Lambda → Firehose → S3
                ↓ (on failure)
           Dead Letter Queue

Key Components and Design Points

Component	Design Points
SQS Queue	Long Polling (20s), Visibility Timeout (30s), SSL enforcement
Dead Letter Queue	Move to DLQ after 3 failures, 14-day retention
Lambda	Batch size 5, ReportBatchItemFailures enabled, X-Ray enabled
SQS Queue (for Failure Lambda)	Long Polling (20s), Visibility Timeout (30s), SSL enforcement
Lambda (Failure)	Lambda without Firehose permissions for DLQ behavior validation
Firehose	1min/1MB buffering, partitioned prefix
S3	Lifecycle management (60d→IA, 90d→Glacier, 365d→Delete)
CloudWatch	8 alarms + SNS notifications

Implementation Highlights

1. SQS + Dead Letter Queue

DLQ configuration is critical for isolating failed messages and enabling investigation and reprocessing.

See: stacks/sqs-lambda-firehose-stack.ts#L35

// Dead Letter Queue
const deadLetterQueue = new sqs.Queue(this, 'DeadLetterQueue', {
  retentionPeriod: cdk.Duration.days(14),
  enforceSSL: true,
});

// Main Queue with DLQ
const queue = new sqs.Queue(this, 'MainQueue', {
  visibilityTimeout: cdk.Duration.seconds(30),
  receiveMessageWaitTime: cdk.Duration.seconds(20), // Long Polling
  deadLetterQueue: {
    maxReceiveCount: 3, // Move to DLQ after 3 failures
    queue: deadLetterQueue,
  },
  enforceSSL: true,
});

Best Practice: Set Visibility Timeout to at least 6 times the Lambda timeout (see documentation)

2. Lambda - ReportBatchItemFailures

When only some messages in a batch fail, you can reprocess only the failed ones.

lambdaFunction.addEventSource(
  new lambdaEventSources.SqsEventSource(queue, {
    batchSize: 5,
    reportBatchItemFailures: true, // Support partial failures
  })
);

Without Powertools vs With Powertools

❌ Without Powertools (Manual Implementation)

def lambda_handler(event, context):
    records = event.get("Records", [])
    batch_item_failures = []

    for record in records:
        message_id = record.get("messageId")
        message_body = record.get("body", "")

        try:
            process_message(message_body)
        except Exception as e:
            # Manually add failed record ID
            batch_item_failures.append({"itemIdentifier": message_id})

    # Manually construct response format
    return {"batchItemFailures": batch_item_failures}

Problems:

Need to manually add itemIdentifier to batch_item_failures
Must construct response format accurately
Lots of error handling boilerplate
Testing becomes complex

✅ With Powertools (Recommended)

from aws_lambda_powertools.utilities.batch import (
    BatchProcessor, EventType, process_partial_response
)

processor = BatchProcessor(event_type=EventType.SQS)

def record_handler(record):
    """Process each record - just raise exception on failure"""
    payload = record.json_body
    send_to_firehose(payload)

def lambda_handler(event, context):
    return process_partial_response(
        event=event,
        record_handler=record_handler,
        processor=processor,
        context=context
    )

Benefits:

itemIdentifier extraction and setting are automated
No need to worry about response format
record_handler can focus solely on processing logic
Just raise an exception on failure

Item	Manual Implementation	Powertools
Code Volume	Large	Small
Bug Risk	High (itemIdentifier mistakes, etc.)	Low
Test Ease	Complex	Simple
Metrics	Manual addition	Auto-collection available

📝 Complete Lambda Function Code (Python)

See: sqs-firehose-powertools/index.py

import json
import os
import boto3
from aws_lambda_powertools import Logger, Tracer, Metrics
from aws_lambda_powertools.utilities.batch import (
    BatchProcessor, EventType, process_partial_response
)
from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord

logger = Logger()
tracer = Tracer()
metrics = Metrics()

processor = BatchProcessor(event_type=EventType.SQS)
firehose_client = boto3.client('firehose')
delivery_stream_name = os.environ['FIREHOSE_DELIVERY_STREAM_NAME']

@tracer.capture_method
def record_handler(record: SQSRecord):
    """Process individual record"""
    payload = record.json_body
    logger.info("Processing message", extra={"message_id": record.message_id})

    response = firehose_client.put_record(
        DeliveryStreamName=delivery_stream_name,
        Record={'Data': json.dumps(payload) + '\n'}
    )

    logger.info("Sent to Firehose", extra={"record_id": response['RecordId']})
    metrics.add_metric(name="ProcessedMessages", unit="Count", value=1)

@logger.inject_lambda_context
@tracer.capture_lambda_handler
@metrics.log_metrics(capture_cold_start_metric=True)
def lambda_handler(event, context):
    return process_partial_response(
        event=event,
        record_handler=record_handler,
        processor=processor,
        context=context
    )

3. Firehose - Partitioned Delivery

Improves S3 query performance and analysis efficiency with Athena and similar tools.

See: stacks/sqs-lambda-firehose-stack.ts#L168

See: dev-params.ts#L69

const deliveryStream = new firehose.DeliveryStream(this, 'DeliveryStream', {
  destination: new firehose.S3Bucket(bucket, {
    dataOutputPrefix: '!{timestamp:yyyy/MM/dd}/',      // Partition by date
    errorOutputPrefix: '!{firehose:error-output-type}/!{timestamp:yyyy/MM/dd}/',
    bufferingInterval: cdk.Duration.minutes(1),
    bufferingSize: cdk.Size.mebibytes(1),
  }),
});

4. S3 Lifecycle Management

See: stacks/sqs-lambda-firehose-stack.ts#L88

bucket.addLifecycleRule({
  transitions: [
    { storageClass: s3.StorageClass.INFREQUENT_ACCESS, transitionAfter: cdk.Duration.days(60) },
    { storageClass: s3.StorageClass.GLACIER, transitionAfter: cdk.Duration.days(90) },
  ],
  expiration: cdk.Duration.days(365),
});

📝 Complete Stack Implementation Code

import * as cdk from 'aws-cdk-lib/core';
import { Construct } from 'constructs';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as pythonLambda from '@aws-cdk/aws-lambda-python-alpha';
import * as lambdaEventSources from 'aws-cdk-lib/aws-lambda-event-sources';
import * as firehose from 'aws-cdk-lib/aws-kinesisfirehose';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as logs from 'aws-cdk-lib/aws-logs';

export class SqsLambdaFirehoseStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: SqsLambdaFirehoseStackProps) {
    super(scope, id, props);

    // 1. Dead Letter Queue
    const deadLetterQueue = new sqs.Queue(this, 'DeadLetterQueue', {
      retentionPeriod: cdk.Duration.days(14),
      enforceSSL: true,
      encryption: sqs.QueueEncryption.SQS_MANAGED,
    });

    // 2. Main Queue
    const queue = new sqs.Queue(this, 'MainQueue', {
      visibilityTimeout: cdk.Duration.seconds(30),
      retentionPeriod: cdk.Duration.days(4),
      receiveMessageWaitTime: cdk.Duration.seconds(20),
      deadLetterQueue: { maxReceiveCount: 3, queue: deadLetterQueue },
      enforceSSL: true,
    });

    // 3. S3 Bucket
    const bucket = new s3.Bucket(this, 'DataBucket', {
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      autoDeleteObjects: true,
      enforceSSL: true,
      versioned: true,
      encryption: s3.BucketEncryption.S3_MANAGED,
    });

    bucket.addLifecycleRule({
      transitions: [
        { storageClass: s3.StorageClass.INFREQUENT_ACCESS, transitionAfter: cdk.Duration.days(60) },
        { storageClass: s3.StorageClass.GLACIER, transitionAfter: cdk.Duration.days(90) },
      ],
      expiration: cdk.Duration.days(365),
    });

    // 4. Firehose
    const deliveryStream = new firehose.DeliveryStream(this, 'DeliveryStream', {
      destination: new firehose.S3Bucket(bucket, {
        dataOutputPrefix: '!{timestamp:yyyy/MM/dd}/',
        errorOutputPrefix: '!{firehose:error-output-type}/!{timestamp:yyyy/MM/dd}/',
        bufferingInterval: cdk.Duration.minutes(1),
        bufferingSize: cdk.Size.mebibytes(1),
      }),
    });

    // 5. Lambda Function
    const lambdaFunction = new pythonLambda.PythonFunction(this, 'ProcessorFunction', {
      runtime: lambda.Runtime.PYTHON_3_14,
      handler: 'lambda_handler',
      entry: '../../common/src/python-lambda/sqs-firehose',
      timeout: cdk.Duration.seconds(5),
      memorySize: 256,
      environment: {
        FIREHOSE_DELIVERY_STREAM_NAME: deliveryStream.deliveryStreamName,
      },
      tracing: lambda.Tracing.ACTIVE,
      loggingFormat: lambda.LoggingFormat.JSON,
    });

    // 6. Permissions &amp; Event Source
    queue.grantConsumeMessages(lambdaFunction);
    deliveryStream.grantPutRecords(lambdaFunction);

    lambdaFunction.addEventSource(
      new lambdaEventSources.SqsEventSource(queue, {
        batchSize: 5,
        reportBatchItemFailures: true,
      })
    );
  }
}

CloudWatch Monitoring

In production, we configure 8 alarms with SNS notifications.

Alarm List

Category	Alarm	Threshold	Purpose
SQS	ApproximateAgeOfOldestMessage	300s	Detect message backlog
SQS	NumberOfEmptyReceives	100	Detect Long Polling issues
DLQ	ApproximateNumberOfMessagesVisible	1	Immediate failure detection
Firehose	DeliveryToS3.DataFreshness	900s	Detect delivery delays
Firehose	ThrottledRecords	1	Detect throttling
Firehose	IncomingBytes Rate	80%	Quota utilization
Firehose	IncomingRecords Rate	80%	Quota utilization
Firehose	IncomingPutRequests Rate	80%	Quota utilization

DLQ Alarm (Most Critical)

See: stacks/sqs-lambda-firehose-stack.ts#L369

const dlqAlarm = deadLetterQueue
  .metricApproximateNumberOfMessagesVisible()
  .createAlarm(this, 'DlqAlarm', {
    threshold: 1,  // Alert even with 1 message
    evaluationPeriods: 1,
  });

Firehose Quota Monitoring (Math Expression)

See: stacks/sqs-lambda-firehose-stack.ts#L390

const incomingBytesRateAlarm = new cw.Alarm(this, 'IncomingBytesRateAlarm', {
  threshold: 80, // Alert at 80% usage
  metric: new cw.MathExpression({
    expression: '100*(m1/300/m2)',  // Calculate usage as %
    usingMetrics: {
      m1: firehose.metric('IncomingBytes', { statistic: 'Sum' }),
      m2: firehose.metric('BytesPerSecondLimit', { statistic: 'Minimum' }),
    },
  }),
});

📝 SNS Notification Integration Code

const topic = new sns.Topic(this, 'AlertTopic', {
  displayName: 'SQS-Firehose Alerts',
});

[dlqAlarm, sqsAgeAlarm, firehoseFreshnessAlarm, ...otherAlarms].forEach(alarm =&gt; {
  alarm.addAlarmAction(new cw_actions.SnsAction(topic));
});

Deployment & Verification

npm run stage:deploy:all -w workspaces/sqs-lambda-firehose --project=myproject --env=dev

See: test-scripts/send-messages.sh

# Send test messages
./test-scripts/send-messages.sh --env dev --project myproject

# Verify data is saved in S3
./test-scripts/check-s3.sh --env dev --project myproject

Best Practices Summary

Component	Recommended	Avoid
SQS	Long Polling, DLQ configuration, SSL enforcement	Short Polling, no DLQ
Lambda	ReportBatchItemFailures, appropriate batch size (5-10)	Too large batches, no error handling
Firehose	Partitioning, 1-5MB buffer	No partitioning, excessively long buffer time
S3	Lifecycle management, encryption	No lifecycle, public access

Cost Estimation

💰 Monthly Estimate (Tokyo Region, Low-to-Medium Usage)

Service	Usage	Monthly Cost
SQS	1M requests	$0.40
Lambda	1M requests, 256MB	$0.83
Firehose	1GB delivery	$0.03
S3	10GB~60GB	$0.50~1.00
CloudWatch	5GB Logs	$0.27

Total: ~$7-10/month

Summary

What we learned from this pattern:

SQS + DLQ: Reliable message processing
ReportBatchItemFailures: Efficient handling of partial failures
Firehose Partitioning: Analytics-friendly data storage
CloudWatch Alarms: Essential monitoring for production

References

Let's continue learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in My GitHub Repository.

AWS CDK 100 Drill Exercises #006: VPC Peering - Cross-Account Network Integration and DNS Resolution Automation

TOMOAKI ishihara — Wed, 31 Dec 2025 11:23:15 +0000

Introduction

This is the 6th installment of "AWS CDK 100 Drill Exercises."

For more information about AWS CDK 100 Drill Exercises, please refer to this article.

In this article, we will explore cross-account network integration using VPC Peering, a key feature that enables private communication between multiple VPCs and is a core technology in a multi-account strategy.

Why VPC Peering?

Multi-account strategy: Organizations are recommended to manage development, staging, and production environments in separate accounts
Secure communication: Private communication between VPCs without going through the internet
Cost efficiency: Simpler network configuration at lower cost compared to Transit Gateway
CDK complexity: Learn CDK limitations and workarounds for cross-account resource references
Practical necessity: Essential network configuration pattern in enterprise environments

What You'll Learn

VPC Peering connection implementation (same account & cross-account)
CDK cross-account stack reference limitations and solutions
Value sharing using AWS Systems Manager Parameter Store
Cross-account parameter reading via Custom Resources
IAM Cross-Account Role design and implementation
Automatic VPC Peering connection acceptance and route configuration
Cross-account deployment best practices

📁 Code Repository: All code examples for this exercise are available on GitHub.

Architecture Overview

What we will build:

Configuration Pattern

Account A (111111111111):
├─ VPC A (10.0.0.0/16)
│  └─ Test Instance (Private Subnet)
├─ VPC B (10.1.0.0/16)
│  └─ Test Instance (Private Subnet)
└─ VPC A ↔ VPC B Peering (same account)

Account B (222222222222):
└─ VPC C (10.2.0.0/16)
   └─ Test Instance (Private Isolated Subnet)

Cross-account Peering:
VPC B (Account A) ↔ VPC C (Account B)

Key Components

VpcAStack (Account A)
- Create VPC A and VPC B
- VPC Peering connection within same account
- Save VPC B information to Parameter Store (readable by Account B)
VpcCStack (Account B)
- Create VPC C
- Save VPC C information to Parameter Store (readable by Account A)
- Create ParameterStoreReadRole (for reading from Account A)
CrossAccountPeeringStack (Account A)
- Retrieve VPC C information via Custom Resource
- Create Peering connection between VPC B and VPC C
- Save Peering connection ID to Parameter Store
- Create PeeringIdReadRole (for reading from Account B)
VpcCRoutesStack (Account B)
- Retrieve Peering connection ID via Custom Resource
- Add routes to VPC B in VPC C's route tables

Prerequisites

AWS CLI v2 installed and configured
Node.js 20+
AWS CDK CLI (npm install -g aws-cdk)
Basic TypeScript knowledge
Two AWS accounts (for development, production, etc.)
AWS CLI profile configuration for each account
Understanding of VPC basic concepts (refer to VPC Basics)

Project Directory Structure

vpc-peering/
├── bin/
│   └── vpc-peering.ts                     # Application entry point
├── lib/
│   ├── stacks/
│   │   ├── vpc-a-stack.ts                 # Step1. VPC A/B + same account peering
│   │   ├── vpc-c-stack.ts                 # Step2. VPC C (Account B)
│   │   ├── cross-account-peering-stack.ts # Step3. Cross-account peering
│   │   └── vpc-c-routes-stack.ts          # Step4. VPC C route configuration
│   ├── stages/
│   │   └── vpc-peering-stage.ts           # Deployment orchestration
│   └── types/
│       └── vpc-peering-params.ts          # Parameter type definitions
├── parameters/
│   └── environments.ts                   # Environment-specific parameters
└── test/
    ├── compliance/
    │   └── cdk-nag.test.ts               # CDK Nag compliance test
    ├── snapshot/
    │   └── snapshot.test.ts              # Snapshot test
    └── unit/
        ├── vpc-a.test.ts                 # VPC Peering stack test
        ├── vpc-c.test.ts                 # VPC C stack test
        ├── vpc-c-routes-stack.test.ts    # VPC C route configuration stack test
        └── cross-account-peering.test.ts # Cross-account peering test

Cross-Account Reference Challenges in CDK

This implementation requires cross-account value references. In this case, there are the following issues:

Problem: CloudFormation Runtime Limitations

When referencing cross-account resources in CDK, it cannot execute due to the following limitations:

// ❌ This does not work
const vpcCStack = new VpcCStack(this, 'VpcC', {
  env: { account: accountB }
});

const peeringStack = new CrossAccountPeeringStack(this, 'Peering', {
  env: { account: accountA },
  peerVpc: vpcCStack.vpc  // ❌ Cross-account reference not allowed
});

Error message:

Stack "CrossAccountPeering" cannot consume a cross reference from stack "VpcC".
Cross stack references are only supported for stacks deployed to the same environment

Why doesn't it work?

CloudFormation Export/Import limitations
- CloudFormation Export/Import only works within the same account and same region
- Cannot directly reference stack outputs between different accounts
How CDK stack references work
- CDK internally uses CloudFormation Export/Import
- This mechanism cannot be used for cross-account
Evaluation timing during deployment
- vpcCStack.vpc.vpcId is evaluated at deployment time
- Cannot retrieve values from resources deployed in Account B during Account A deployment

Solutions

To solve cross-account references, the following methods can be considered.
Ultimately, we adopted the second method.

1. Static parameter passing via CDK Context or environment variable files
2. Passing via Parameter Store and AWS Custom Resources

1. Manual Parameter Passing via CDK Context or Environment Variable Files

Implementation:

// Get parameter for VPC C ID created in Account B
const vpcCId = this.node.tryGetContext('vpcCId');

if (!vpcCId) {
  console.warn('vpcCId not provided. Run: cdk deploy --context vpcCId=vpc-xxx');
  return;
}

// Create VPC Peering
const peering = new ec2.CfnVPCPeeringConnection(this, 'Peering', {
  vpcId: localVpc.vpcId,
  peerVpcId: vpcCId,  // ✅ Works
  peerOwnerId: accountB,
});

Deployment procedure:

# 1. Create VPC C in Account B
cdk deploy --profile account-b VpcC

# 2. Manually retrieve VPC C ID
VPC_C_ID=$(aws ec2 describe-vpcs --profile account-b \
  --filters "Name=tag:Name,Values=VpcC" \
  --query 'Vpcs[0].VpcId' --output text)

# 3. Create Peering in Account A (pass VPC ID)
cdk deploy --profile account-a --context vpcCId=$VPC_C_ID CrossAccountPeering

Issues:

⚠️ Manual parameter retrieval required
Difficult to automate in CI/CD pipelines
Human error risk (copy & paste mistakes, etc.)
Lack of scalability (management becomes complex as parameters increase)

2. Parameter Store and AWS Custom Resources

Architecture:

Account B:
├─ VPC C
├─ SSM Parameter (/project/env/vpc-c/id)
└─ ParameterStoreReadRole <- Can be assumed by Account A

Account A:
├─ Custom Resource
│  ├─ assumeRole -> Account B's ParameterStoreReadRole
│  └─ getParameter -> Retrieve VPC C ID
└─ VPC Peering Connection (using retrieved VPC C ID)

Implementation:

Account B: Create Parameter Store + Read Role

// VpcCStack (Account B)
const vpcCIdParam = new ssm.StringParameter(this, 'VpcIdParam', {
  stringValue: this.vpcC.vpc.vpcId,
  parameterName: `/project/env/vpc-c/id`,
});

// IAM role for reading from Account A
const readRole = new iam.Role(this, 'ParameterStoreReadRole', {
  assumedBy: new iam.AccountPrincipal(accountA),
  roleName: 'project-env-ParameterStoreReadRole',
});

vpcCIdParam.grantRead(readRole);

Account A: Read parameter with Custom Resource

// CrossAccountPeeringStack (Account A)
import * as cr from 'aws-cdk-lib/custom-resources';

const readRoleArn = `arn:aws:iam::${accountB}:role/project-env-ParameterStoreReadRole`;

// Read Parameter Store from other account with Custom Resource
const getVpcCId = new cr.AwsCustomResource(this, 'GetVpcCId', {
  onUpdate: {
    service: 'SSM',
    action: 'getParameter',
    parameters: {
      Name: '/project/env/vpc-c/id',
    },
    region: 'ap-northeast-1',
    physicalResourceId: cr.PhysicalResourceId.of('VpcCIdLookup'),
    assumedRoleArn: readRoleArn,  // ✅ Assume Account B's Role
  },
  policy: cr.AwsCustomResourcePolicy.fromStatements([
    new iam.PolicyStatement({
      actions: ['sts:AssumeRole'],
      resources: [readRoleArn],
    }),
  ]),
});

const vpcCId = getVpcCId.getResponseField('Parameter.Value');

// Create VPC Peering
const peering = new ec2.CfnVPCPeeringConnection(this, 'Peering', {
  vpcId: localVpc.vpcId,
  peerVpcId: vpcCId,  // ✅ Use automatically retrieved value
  peerOwnerId: accountB,
});

Benefits:

Fully automated: No manual parameter retrieval required
CI/CD ready: Can be integrated into pipelines
Secure: Strict access control with IAM roles
Scalable: Supports multiple parameters
Error detection: Errors are automatically detected at deployment time

Drawbacks:

Lambda function is automatically created (slight cost increase)
Implementation is somewhat complex
Custom Resource lifecycle management required

VPC Peering Within Same Account

The implementation of VPC Peering within the same account is as follows.

VpcAStack Implementation

// lib/stacks/vpc-peering-stack.ts

export class VpcAStack extends cdk.Stack {
  public readonly vpcA: VpcConstruct;
  public readonly vpcB: VpcConstruct;

  constructor(scope: Construct, id: string, props: VpcAStackProps) {
    super(scope, id, props);

    // Create VPC A
    this.vpcA = new VpcConstruct(this, 'VpcA', {
        // Specify parameters
    });

    // Create VPC B
    this.vpcB = new VpcConstruct(this, 'VpcB', {
        // Specify parameters
    });

    // VPC Peering connection
    const peering = new VpcPeering(this, 'VpcABPeering', {
      vpc: this.vpcA.vpc,
      peerVpc: this.vpcB.vpc,
    });
  }
}

VPC Peering Construct

// common/constructs/vpc/vpc-peering.ts
export class VpcPeering extends Construct {
  public readonly vpcPeeringConnection: ec2.CfnVPCPeeringConnection;
  public readonly localSecurityGroup: ec2.ISecurityGroup;
  public readonly peeringSecurityGroup: ec2.ISecurityGroup;

  constructor(scope: Construct, id: string, props: VpcPeeringProps) {
    super(scope, id);

    // CIDR overlap check
    if (props.vpc.vpcCidrBlock === props.peerVpc.vpcCidrBlock) {
      throw new Error(`VPC CIDR blocks overlap`);
    }

    // Create VPC Peering connection
    this.vpcPeeringConnection = new ec2.CfnVPCPeeringConnection(this, 'Connection', {
      vpcId: props.vpc.vpcId,
      peerVpcId: props.peerVpc.vpcId,
    });

    // Add routes (Local VPC -> Peer VPC)
    props.vpc.privateSubnets.forEach((subnet, index) => {
      new ec2.CfnRoute(this, `RouteToPeer${index}`, {
        routeTableId: subnet.routeTable.routeTableId,
        destinationCidrBlock: props.peerVpc.vpcCidrBlock,
        vpcPeeringConnectionId: this.vpcPeeringConnection.ref,
      });
    });

    // Add routes (Peer VPC -> Local VPC)
    props.peerVpc.privateSubnets.forEach((subnet, index) => {
      new ec2.CfnRoute(this, `RouteToLocal${index}`, {
        routeTableId: subnet.routeTable.routeTableId,
        destinationCidrBlock: props.vpc.vpcCidrBlock,
        vpcPeeringConnectionId: this.vpcPeeringConnection.ref,
      });
    });

    // Security groups (allow mutual communication)
    :

  }
}

Cross-Account VPC Peering

Implementation

Step 3. Deploy to Account A: CrossAccountPeeringStack
   ├─ Retrieve VPC C information via Custom Resource
   ├─ Create VPC B <-> VPC C Peering
   ├─ Save Peering ID to Parameter Store
   └─ Create PeeringIdReadRole

Step 4. Deploy to Account B: VpcCRoutesStack
   ├─ Retrieve Peering ID via Custom Resource
   └─ Update VPC C route tables

VpcCStack (Account B)

// lib/stacks/vpc-c-stack.ts

export class VpcCStack extends cdk.Stack {
  public readonly vpcC: VpcConstruct;

  constructor(scope: Construct, id: string, props: VpcCStackProps) {
    super(scope, id, props);

    // Create VPC C
    this.vpcC = new VpcConstruct(this, 'VpcC', {
        // Specify parameters
    });

    if (props.params.accountAId) {
      // Save VPC C information to Parameter Store
      const vpcCIdParam = new ssm.StringParameter(this, 'VpcIdParam', {
        stringValue: this.vpcC.vpc.vpcId,
        description: 'VPC C ID in Account B',
        parameterName: `/${props.project}/${props.environment}/vpc-c/id`,
      });

      const parameterReadRole = new iam.Role(this, 'ParameterStoreReadRole', {
        assumedBy: new iam.AccountPrincipal(props.params.accountAId),
        roleName: `${props.project}-${props.environment}-ParameterStoreReadRole`,
        description: `Role to allow Account ${props.params.accountAId} to read VPC C parameters from Parameter Store`,
      });
      // Grant read access to VPC C parameters
      vpcCIdParam.grantRead(parameterReadRole);

      // Peering acceptance role
      const peeringRole = new iam.Role(this, 'VpcPeeringRole', {
        assumedBy: new iam.AccountPrincipal(props.params.accountAId),
        roleName: props.peeringRoleName,
      });

      peeringRole.addToPolicy(new iam.PolicyStatement({
        actions: ['ec2:AcceptVpcPeeringConnection'],
        resources: ['*'],
      }));
    }
  }
}

CrossAccountPeeringStack (Account A)

// lib/stacks/cross-account-peering-stack.ts

export class CrossAccountPeeringStack extends cdk.Stack {
  public readonly peeringConnection: ec2.CfnVPCPeeringConnection;

  constructor(scope: Construct, id: string, props: CrossAccountPeeringStackProps) {
    super(scope, id, props);

    // Retrieve VPC C ID via Custom Resource
    const getVpcCId = new cr.AwsCustomResource(this, 'GetVpcCId', {
      onUpdate: {
        service: 'SSM',
        action: 'getParameter',
        parameters: {
          Name: parameterName,
        },
        region: region,
        physicalResourceId: cr.PhysicalResourceId.of('VpcCIdLookup'),
        assumedRoleArn: parameterReadRoleArn,
      },
      policy: cr.AwsCustomResourcePolicy.fromStatements([
        new iam.PolicyStatement({
          actions: ['sts:AssumeRole'],
          resources: [parameterReadRoleArn],
        }),
      ]),
    });
    const vpcCId = getVpcCId.getResponseField('Parameter.Value');

    // Create VPC Peering
    this.peeringConnection = new ec2.CfnVPCPeeringConnection(this, 'VpcBCPeering', {
      vpcId: props.requestorVpc.vpcId,
      peerVpcId: vpcCId,
      peerOwnerId: props.params.accountBId,
      peerRegion: props.params.regionB,
      peerRoleArn: `arn:aws:iam::${props.params.accountBId}:role/${props.peeringRoleName}`,
    });

    // Save Peering ID to Parameter Store
    const peeringIdParam = new ssm.StringParameter(this, 'PeeringConnectionIdParam', {
      stringValue: this.peeringConnectionId,
      description: 'VPC Peering Connection ID for VPC B <-> VPC C',
      parameterName: `/${props.project}/${props.environment}/peering/vpc-b-vpc-c/id`,
    });
    const peeringIdReadRole = new iam.Role(this, 'PeeringIdReadRole', {
      assumedBy: new iam.AccountPrincipal(props.params.accountBId),
      roleName: `${props.project}-${props.environment}-PeeringIdReadRole`,
      description: `Role to allow Account ${props.params.accountBId} to read Peering Connection ID from Parameter Store`,
    });
    peeringIdParam.grantRead(peeringIdReadRole);
  }
}

VpcCRoutesStack (Account B)

// lib/stacks/vpc-c-routes-stack.ts

export class VpcCRoutesStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: VpcCRoutesStackProps) {
    super(scope, id, props);

    // Account A role
    const parameterReadRoleArn = `arn:aws:iam::${props.params.accountAId}:role/${props.project}-${props.environment}-PeeringIdReadRole`;

    // Retrieve Peering ID via Custom Resource
    const getPeeringConnectionId = new cr.AwsCustomResource(this, 'GetPeeringConnectionId', {
      onUpdate: {
        service: 'SSM',
        action: 'getParameter',
        parameters: {
          Name: props.peeringIdParamName, // /${props.project}/${props.environment}/peering/vpc-b-vpc-c/id,
        },
        region: regionA,
        physicalResourceId: cr.PhysicalResourceId.of('PeeringConnectionIdLookup'),
        assumedRoleArn: parameterReadRoleArn,
      },
      policy: cr.AwsCustomResourcePolicy.fromStatements([
        new iam.PolicyStatement({
          actions: ['sts:AssumeRole'],
          resources: [parameterReadRoleArn],
        }),
      ]),
    });
    const peeringConnectionId = getPeeringConnectionId.getResponseField('Parameter.Value');

    // target Subnet
    // private or isolated subnets
    const targetSubnets =
      (props.vpc.privateSubnets && props.vpc.privateSubnets.length > 0)
        ? props.vpc.privateSubnets
        : props.vpc.isolatedSubnets;
    // Add routes to VPC C private subnets pointing to VPC B
    targetSubnets.forEach((subnet, index) => {
      new ec2.CfnRoute(this, `VpcCToVpcBRoute${index + 1}`, {
        routeTableId: subnet.routeTable.routeTableId,
        destinationCidrBlock: props.vpcBCidr,
        vpcPeeringConnectionId: peeringConnectionId,
      });
    });
  }
}

VPC Peering DNS Resolution Options Automation (Custom Resource Implementation Example)

Why DNS Resolution Options are Needed

Just creating a VPC Peering connection doesn't enable resolution of private DNS names in the peer VPC. For example, if you want to access an EC2 instance in VPC B from VPC A using a private DNS name like ip-10-1-0-10.ap-northeast-1.compute.internal, you need to enable DNS resolution options.

Normally, this option is configured manually from the console or set using AWS CLI as follows (documentation is here):

aws ec2 modify-vpc-peering-connection-options \
  --vpc-peering-connection-id pcx-xxxxx \
  --requester-peering-connection-options AllowDnsResolutionFromRemoteVpc=true \
  --accepter-peering-connection-options AllowDnsResolutionFromRemoteVpc=true

In this implementation, we automate this configuration using AWS CDK's AwsCustomResource.

For Same Account

For same account, specify both RequesterPeeringConnectionOptions and AccepterPeeringConnectionOptions.

    // Enable DNS resolution over VPC Peering
    const onCreate: cr.AwsSdkCall = {
        service: 'EC2',
        action: 'modifyVpcPeeringConnectionOptions',
        parameters: {
            VpcPeeringConnectionId: this.peeringConnection.ref,
                RequesterPeeringConnectionOptions: {
                    AllowDnsResolutionFromRemoteVpc: true
                },
                AccepterPeeringConnectionOptions: {
                    AllowDnsResolutionFromRemoteVpc: true,
                }
        },
        region: props.env?.region,
        physicalResourceId: cr.PhysicalResourceId.of(`EnableVpcPeeringDnsResolution:${this.peeringConnection.ref}`),
    }
    const onUpdate = onCreate;
    const onDelete: cr.AwsSdkCall = {
        service: "EC2",
        action: "modifyVpcPeeringConnectionOptions",
        parameters: {
            VpcPeeringConnectionId: this.peeringConnection.ref,
            RequesterPeeringConnectionOptions: {
                AllowDnsResolutionFromRemoteVpc: false
            },
            AccepterPeeringConnectionOptions: {
                AllowDnsResolutionFromRemoteVpc: false,
            }
        },
    }
    new cr.AwsCustomResource(this, 'EnableVpcPeeringDnsResolution', {
        onUpdate,
        onCreate,
        onDelete,
        policy: cr.AwsCustomResourcePolicy.fromSdkCalls({resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE}),
    });

For Cross-Account

For cross-account VPC Peering, configure separately from Requester and Accepter side stacks.

※Duplicate parts from the same account case are omitted.

Requester side (Account A): CrossAccountPeeringStack

        parameters: {
            VpcPeeringConnectionId: this.peeringConnection.ref,
            RequesterPeeringConnectionOptions: {
                AllowDnsResolutionFromRemoteVpc: true
            },
        },

Accepter side (Account B): VpcCRoutesStack

        parameters: {
            VpcPeeringConnectionId: peeringConnectionId,
            AccepterPeeringConnectionOptions: {
                AllowDnsResolutionFromRemoteVpc: true,
            }
        },

Deployment and Cleanup

1. Environment Parameter Configuration

// parameters/environments.ts

export const params: Record<Environment, EnvParams> = {
  [Environment.DEVELOPMENT]: {
    accountAId: '111111111111',  // Account A
    accountBId: '222222222222',  // Account B
    regionA: 'ap-northeast-1',
    regionB: 'ap-northeast-1',
    vpcAConfig: {
      createConfig: {
        cidr: '10.0.0.0/16',
        maxAzs: 2,
        natGateways: 1,
      },
    },
    vpcBConfig: {
      createConfig: {
        cidr: '10.1.0.0/16',
        maxAzs: 2,
        natGateways: 1,
      },
    },
    vpcCConfig: {
      createConfig: {
        cidr: '10.2.0.0/16',
        maxAzs: 2,
        natGateways: 0,
      },
    },
  },
};

2. Bootstrap

1. Bootstrap Account A (pipeline account)

# Basic bootstrap to host pipeline in Account A
npx cdk bootstrap \
    --profile account-a-admin \
    --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess \
    aws://111111111111/ap-northeast-1

2. Bootstrap Account B (target account)

# Bootstrap Account B and trust Account A
npx cdk bootstrap \
    --profile account-b-admin \
    --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess \
    --trust 111111111111 \
    aws://222222222222/ap-northeast-1

Important points:

--trust 111111111111: Allows Account A to deploy to Account B
This configuration allows Account A's bootstrap role to AssumeRole to Account B's resources

How `--trust` option works

1. IAM Role Creation

When you bootstrap with the --trust option, the following IAM roles are created:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

This trust policy allows Account A's principals to:

CDKDeployRole: Deploy CloudFormation stacks
CDKFilePublishingRole: Upload file assets to S3
CDKImagePublishingRole: Push Docker images to ECR

3. Deployment

CDK_CROSS_ACCOUNT_ID=222222222222 npm run stage:deploy:all -w workspaces/vpc-peering --project=myproject --env=dev

4. Connection Verification

Example connection verification from VPC A. In the example below, we connect to EC2, but you can also use CloudShell.

# Ping from VPC A instance to VPC B instance
aws ssm start-session --profile account-a \
  --target i-0xxxxxxxxxxxxx  # VPC A Instance ID

# Connectivity check to VPC B
ping 10.1.x.x  # VPC B Private IP

# Ping to VPC C (cross-account) -> Cannot connect from VPC A
ping 10.2.x.x  # VPC C Private IP

5. Cleanup

If you connected to VPC using CloudShell for connection verification, delete the CloudShell environment before cleanup. Stack deletion will fail.

# Delete all resources
CDK_CROSS_ACCOUNT_ID=222222222222 npm run stage:destroy:all -w workspaces/vpc-peering --project=myproject --env=dev

Summary

This article explained the implementation of VPC Peering using AWS CDK, focusing on cross-account challenges and solutions.

What We Learned

CDK cross-account limitations
- CloudFormation Export/Import only works within the same account
- Stack references do not work cross-account
Implementation patterns
- Parameter Store: Store and share values
- IAM Cross-Account Role: Secure access control
- Custom Resource: Dynamic value retrieval at deployment time

References

Let's continue learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in My GitHub Repository.

DEV Community: TOMOAKI ishihara

AWS CDK 100 Drill Exercises #013: CloudFront S3 Static Website — Serving a Private Bucket over HTTPS with OAC and Cross-Region WAF

Introduction

What you'll learn in this article

Architecture Overview

Data Flow

Implementation Highlights

1. Private bucket + OAC — the polar opposite of "who can read this"

2. Bulk security headers via ResponseHeadersPolicy

3. Gentle with the SPA, honest about real failures

4. CLOUDFRONT-scoped WAFv2 is pinned to us-east-1 — wire it with crossRegionReferences

5. Allow-listing before or after the managed rules

6. WAF logs go directly to S3, with sensitive headers masked

Deploy & Verify

Cost Estimate

Summary

References

AWS CDK 100 Drill Exercises #012: S3 Static Web Site — Getting a "Public" Bucket Policy Past Block Public Access

Introduction

What you'll learn in this article

Architecture Overview

Data Flow

Implementation Highlights

1. websiteIndexDocument alone doesn't finish the job — the tug-of-war with Block Public Access

2. enforceSSL: false — a setting that looks wrong at first glance

3. Auto-detecting the deploy operator's IP so the site is instantly viewable

Deploy & Verify

Cost Estimate

Summary

References

AWS CDK 100 Drill Exercises #011: A CodePipeline for CloudFront & S3 — and a self-reference bug

Introduction

What you'll learn in this article

Architecture Overview

Data Flow

Implementation Highlights

1. Let CDK grant IAM permissions per action — don't hand-roll a pipeline role

2. Split "upload" from "cleanup" instead of one aws s3 sync --delete

3. The continuation-token pattern for CloudFront invalidation

4. The bug: a pipeline that referenced itself

5. Two more bugs that only a real Template.fromStack() run would catch

Deploy & Verify

Cost Estimate

Summary

References

AWS CDK 100 Drill Exercises #009: CloudWatch Logs S3 Archive — Comparing 3 Log Archive Patterns in One Repository

Introduction

What you'll learn in this article

Architecture Overview

Why three patterns

Data Flow

Key Components

Implementation Highlights

1. Pattern A — Kinesis Data Firehose subscription

1a. Per-log-group dynamic partitioning (Stack 1 – Basic)

2. Tiered S3 lifecycle (Stack 2 – Lifecycle)

3. Attaching to an existing log group (Stack 3 – Existing)

4. Pattern B — Scheduled Export Task

5. Pattern C — Subscription filter → Lambda → S3

Deploy & Verify

Cost Estimate

Summary

References

AWS CDK 100 Drill Exercises #010: Budget Alerts & Cost Anomaly Detection — 5 FinOps Alerting Patterns in CDK

Introduction

What you'll learn in this article

Architecture Overview

Why five patterns

Data Flow

Key Components

Implementation Highlights

1. Data-driven budget notification rules

2. SNS topic policies aren't optional — and each service does it differently

3. Cost Anomaly Detection's thresholdExpression

4. Stack 3 doesn't create its own anomaly monitor — AWS's "one per account" limit

5. AWS Chatbot's guardrail policy default is AdministratorAccess (for both Slack and Teams)

6. Reusable cost constructs (common/constructs/cost/)

7. The cost digest's JSONata state machine (Stack 5)

Deploy & Verify

Cost Estimate

2. Bulk security headers via `ResponseHeadersPolicy`

4. CLOUDFRONT-scoped WAFv2 is pinned to us-east-1 — wire it with `crossRegionReferences`

1. `websiteIndexDocument` alone doesn't finish the job — the tug-of-war with Block Public Access

2. `enforceSSL: false` — a setting that looks wrong at first glance

2. Split "upload" from "cleanup" instead of one `aws s3 sync --delete`

5. Two more bugs that only a real `Template.fromStack()` run would catch

3. Cost Anomaly Detection's `thresholdExpression`

5. AWS Chatbot's guardrail policy default is `AdministratorAccess` (for both Slack and Teams)

6. Reusable cost constructs (`common/constructs/cost/`)

5. The incident-response escape hatch (`publicAlbFailover`)