The CDK has been released in two major versions, v1 and v2. AWS CDK v1 entered maintenance on June 1, 2022. As part of some recent talks and demos I was doing, I wanted to update the CDK stacks for Managed Workflows for Apache Airflow (MWAA) and thought I would outline what I learned along the way.
Prepping for CDKv2
In AWS CDKv1, the different CDK constructs were imported as discrete Python libraries (for example aws-cdk.aws-s3-deployment, aws-cdk.aws-mwaa, aws_cdk.aws-kms etc) that were imported in the requirements.txt file. AWS CDKv2 has a different approach, so I needed to clean up all the CDK construct modules that I had imported for the CDKv1 stacks. I ended up using the following command to do this.
You can read more about the differences in this post, Migrating to AWS CDK v2
pip list |grep cdk | xargs pip uninstall -y
I updated my requirements.txt file as follows:
aws-cdk-lib==2.2.0
constructs>=10.0.0,<11.0.0
Finally, I had to update the cdk.json file as some of the values have changed.
{
"app": "python app.py",
"context": {
"@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": false,
"@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": false,
"@aws-cdk/aws-rds:lowercaseDbIdentifier": false,
"@aws-cdk/core:stackRelativeExports": false
}
}
Installing AWS CDKv2
To install CDKv2, I did the following:
npm install -g aws-cdk
You can check that it has updated by running:
cdk --version
2.28.1 (build d035432)
It should display 2.x - if it displays 1.x, then check your alias commands, alternatives or try and uninstall and then reinstall.
If you need to keep both CDKv1 and v2, then consider using alias commands to help you.
alias cdk1="npx aws-cdk@1.x" alias cdk="npx aws-cdk@2.x"
Re-bootstraping AWS CDK
One of the first things once I fixed/updated my CDK stacks was to re-bootstrap CDK. I did not know I had to do that, but the first time I did a "cdk deploy" I was prompted with this message first:
current credentials could not be used to assume 'arn:aws:iam::704533066374:role/cdk-hnb659fds-lookup-role-704533066374-eu-central-1', but are for the right account. Proceeding anyway.
followed quickly by an error message
mwaa-hybrid-backend failed: Error: mwaa-hybrid-backend: SSM parameter /cdk-bootstrap/hnb659fds/version not found. Has the environment been bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html)
at CloudFormationDeployments.validateBootstrapStackVersion (/Users/ricsue/.nvm/versions/node/v14.18.2/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:482:13)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at CloudFormationDeployments.publishStackAssets (/Users/ricsue/.nvm/versions/node/v14.18.2/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:457:7)
at CloudFormationDeployments.deployStack (/Users/ricsue/.nvm/versions/node/v14.18.2/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:339:7)
at CdkToolkit.deploy (/Users/ricsue/.nvm/versions/node/v14.18.2/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:209:24)
at initCommandLine (/Users/ricsue/.nvm/versions/node/v14.18.2/lib/node_modules/aws-cdk/lib/cli.ts:341:12)
mwaa-hybrid-backend: SSM parameter /cdk-bootstrap/hnb659fds/version not found. Has the environment been bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html)
I was able to fix this quickly by running the CDK Bootstrap command:
cdk bootstrap aws://704533066374/eu-central-1
!Note I was doing this with an IAM user with AdminAccess.
Using default execution policy of 'arn:aws:iam::aws:policy/AdministratorAccess'. Pass '--cloudformation-execution-policies' to customize.In the past when I have talked with builders who do not have Admin access but are using either an IAM user or a Role, they have to following this doc and modify the bootstrap template with the role and permissions they have. (--cloudformation-execution-policies) also see this issue
Changes to imports and other niggles
The first thing I needed to do was get my head around the changes to the imports and libraries within my stacks. For example, I had to change from
from aws_cdk import core
from aws_cdk.core import Tags
import aws_cdk.aws_ec2 as ec2
import aws_cdk.aws_s3 as s3
import aws_cdk.aws_s3_deployment as s3deploy
import aws_cdk.aws_mwaa as mwaa
import aws_cdk.aws_iam as iam
import aws_cdk.aws_kms as kms
to this
from aws_cdk import (
aws_iam as iam,
aws_ec2 as ec2,
aws_s3 as s3,
aws_s3_deployment as s3deploy,
aws_mwaa as mwaa,
aws_kms as kms,
Stack,
CfnOutput,
Tags
)
from constructs import Construct
Which is actually nicer and cleaner I think. I had to update references to the classes within my code, but that was pretty straight forward and just took some time to go through to make sure I had not missed anything.
Fixing changes to the VPC private subnet constructs
In some of the constructs, it looks like some of the values have changed between v1 and v2. In the VPC construct for example, creating a Private subnet used to be done via:
self.vpc = ec2.Vpc(
self,
id="MWAA-Hybrid-ApacheAirflow-VPC",
cidr="10.192.0.0/16",
max_azs=2,
nat_gateways=1,
subnet_configuration=[
ec2.SubnetConfiguration(
name="public", cidr_mask=24,
reserved=False, subnet_type=ec2.SubnetType.PUBLIC),
ec2.SubnetConfiguration(
name="private", cidr_mask=24,
reserved=False, subnet_type=ec2.SubnetType.PRIVATE)
],
enable_dns_hostnames=True,
enable_dns_support=True
)
and this had to be updated to
self.vpc = ec2.Vpc(
self,
id="MWAA-Hybrid-ApacheAirflow-VPC",
cidr="10.192.0.0/16",
max_azs=2,
nat_gateways=1,
subnet_configuration=[
ec2.SubnetConfiguration(
name="public", cidr_mask=24,
reserved=False, subnet_type=ec2.SubnetType.PUBLIC),
ec2.SubnetConfiguration(
name="private", cidr_mask=24,
reserved=False, subnet_type=ec2.SubnetType.PRIVATE_WITH_NAT)
],
enable_dns_hostnames=True,
enable_dns_support=True
)
This was the only one (PRIVATE to PRIVATE_WITH_NAT) that I needed to adjust, but you may have others.
Fixing SSM Errors
Once the CDK stacks were validated and clear for deployment, I encountered a "feature" when generating IAM policies and roles. It was whilst fixing a different CDK stack (Orchestrating Hybrid Workflows with Apache Airflow, GitHub repo blogpost-airflow-hybrid that I found that even though the stack had synthesised ok, during the deployment, I got the following error:
Invalid principal in policy: "SERVICE":"ssm.eu-west-1.amazonaws.com" (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 5763050d-c789-49ef-b7d7-9e80a62f0e9e; Proxy: null)
This error appeared in both the command prompt where I ran the "cdk deploy", but also appeared in the CloudFormation console which is a good place to go when you are trying to figure out what is going wrong.
Upon looking at the synthesised CloudFormation template, you could see that IAM Principal was regionalising the 'ssm.amazonaws.com' (so it was creating this as 'ssm.eu-west-1.amazonaws.com' in my case) which was a problem. In my stack I defined the following role:
external_task_def_policy_document_role = iam.Role(
self,
"ExternalECSAnywhereRole",
role_name=f"{props['ecsclustername']}-ExternalECSAnywhereRole",
assumed_by=iam.ServicePrincipal("ssm.amazonaws.com")
)
and this was failing (it worked fine in CDKv1). In the end, after reading GitHub issues 17646 and 16188, I was able to fix this issue within my stack by adding this code:
ssmfix = external_task_def_policy_document_role.node.default_child
ssmfix.add_property_override(
"AssumeRolePolicyDocument.Statement.0.Principal.Service", "ssm.amazonaws.com"
)
Deploying the updated stacks
Once I had made these changes, I was then able to deploy the MWAA stack, first the backend (VPC and Networking) and then the environment.
You can find the code for this in the GitHub repository, blogpost-cdk-mwaa
Deployment of the MWAA network (backend)
Here is the output that you will see (more or less)
✨ Synthesis time: 7.09s
mwaa-hybrid-backend: deploying...
[0%] start: Publishing 2695cb7a9f601cf94a4151c65c9069787d9ec312084346f2f4359e3f55ff2310:704533066374-eu-central-1
[100%] success: Published 2695cb7a9f601cf94a4151c65c9069787d9ec312084346f2f4359e3f55ff2310:704533066374-eu-central-1
mwaa-hybrid-backend: creating CloudFormation changeset...
✅ mwaa-hybrid-backend
✨ Deployment time: 172.13s
Outputs:
mwaa-hybrid-backend.ExportsOutputRefMWAAHybridApacheAirflowVPC677B092EF6F2F587 = vpc-0bbdeee3652ef21ff
mwaa-hybrid-backend.ExportsOutputRefMWAAHybridApacheAirflowVPCprivateSubnet1Subnet2A6995DF7F8D3134 = subnet-01e48db64381efc7f
mwaa-hybrid-backend.ExportsOutputRefMWAAHybridApacheAirflowVPCprivateSubnet2SubnetA28659530C36370A = subnet-0321530b8154f9bd2
mwaa-hybrid-backend.VPCId = vpc-0bbdeee3652ef21ff
Stack ARN:
arn:aws:cloudformation:eu-central-1:704533066374:stack/mwaa-hybrid-backend/b05897d0-f087-11ec-b5f3-02db3f47a5ca
✨ Total time: 179.22s
Deployment of the MWAA environment
Here is the output that you will see (more or less)
Including dependency stacks: mwaa-hybrid-backend
[Warning at /mwaa-hybrid-environment/mwaa-sg] Ignoring Egress rule since 'allowAllOutbound' is set to true; To add customize rules, set allowAllOutbound=false on the SecurityGroup
✨ Synthesis time: 12.37s
mwaa-hybrid-backend
mwaa-hybrid-backend: deploying...
[0%] start: Publishing 2695cb7a9f601cf94a4151c65c9069787d9ec312084346f2f4359e3f55ff2310:704533066374-eu-central-1
[100%] success: Published 2695cb7a9f601cf94a4151c65c9069787d9ec312084346f2f4359e3f55ff2310:704533066374-eu-central-1
✅ mwaa-hybrid-backend (no changes)
✨ Deployment time: 1.97s
Outputs:
mwaa-hybrid-backend.ExportsOutputRefMWAAHybridApacheAirflowVPC677B092EF6F2F587 = vpc-0bbdeee3652ef21ff
mwaa-hybrid-backend.ExportsOutputRefMWAAHybridApacheAirflowVPCprivateSubnet1Subnet2A6995DF7F8D3134 = subnet-01e48db64381efc7f
mwaa-hybrid-backend.ExportsOutputRefMWAAHybridApacheAirflowVPCprivateSubnet2SubnetA28659530C36370A = subnet-0321530b8154f9bd2
mwaa-hybrid-backend.VPCId = vpc-0bbdeee3652ef21ff
Stack ARN:
arn:aws:cloudformation:eu-central-1:704533066374:stack/mwaa-hybrid-backend/b05897d0-f087-11ec-b5f3-02db3f47a5ca
✨ Total time: 14.35s
mwaa-hybrid-environment
This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).
Please confirm you intend to make the following modifications:
IAM Statement Changes
┌───┬──────────────────────────────┬────────┬──────────────────────────────┬──────────────────────────────┬─────────────────────────────────┐
│ │ Resource │ Effect │ Action │ Principal │ Condition │
├───┼──────────────────────────────┼────────┼──────────────────────────────┼──────────────────────────────┼─────────────────────────────────┤
│ + │ ${Custom::CDKBucketDeploymen │ Allow │ sts:AssumeRole │ Service:lambda.amazonaws.com │ │
│ │ t8693BB64968944B69AAFB0CC9EB │ │ │ │ │
│ │ 8756C/ServiceRole.Arn} │ │ │ │ │
├───┼──────────────────────────────┼────────┼──────────────────────────────┼──────────────────────────────┼─────────────────────────────────┤
│ + │ ${mwaa-dags.Arn} │ Deny │ s3:ListAllMyBuckets │ AWS:${mwaa-service-role} │ │
│ │ ${mwaa-dags.Arn}/* │ │ │ │ │
│ + │ ${mwaa-dags.Arn} │ Allow │ s3:* │ AWS:${mwaa-service-role} │ │
│ │ ${mwaa-dags.Arn}/* │ │ │ │ │
│ + │ ${mwaa-dags.Arn} │ Allow │ s3:Abort* │ AWS:${Custom::CDKBucketDeplo │ │
│ │ ${mwaa-dags.Arn}/* │ │ s3:DeleteObject* │ yment8693BB64968944B69AAFB0C │ │
│ │ │ │ s3:GetBucket* │ C9EB8756C/ServiceRole} │ │
│ │ │ │ s3:GetObject* │ │ │
│ │ │ │ s3:List* │ │ │
│ │ │ │ s3:PutObject │ │ │
├───┼──────────────────────────────┼────────┼──────────────────────────────┼──────────────────────────────┼─────────────────────────────────┤
│ + │ ${mwaa-hybrid-demoKey.Arn} │ Allow │ kms:CancelKeyDeletion │ AWS:arn:${AWS::Partition}:ia │ │
│ │ │ │ kms:Create* │ m::704533066374:root │ │
│ │ │ │ kms:Decrypt* │ │ │
│ │ │ │ kms:Delete* │ │ │
│ │ │ │ kms:Describe* │ │ │
│ │ │ │ kms:Disable* │ │ │
│ │ │ │ kms:Enable* │ │ │
│ │ │ │ kms:GenerateDataKey* │ │ │
│ │ │ │ kms:Get* │ │ │
│ │ │ │ kms:List* │ │ │
│ │ │ │ kms:Put* │ │ │
│ │ │ │ kms:Revoke* │ │ │
│ │ │ │ kms:ScheduleKeyDeletion │ │ │
│ │ │ │ kms:Update* │ │ │
│ + │ ${mwaa-hybrid-demoKey.Arn} │ Allow │ kms:Decrypt* │ Service:logs.eu-central-1.am │ "ArnLike": { │
│ │ │ │ kms:Describe* │ azonaws.com │ "kms:EncryptionContext:aws:lo │
│ │ │ │ kms:Encrypt* │ │ gs:arn": "arn:aws:logs:eu-centr │
│ │ │ │ kms:GenerateDataKey* │ │ al-1:704533066374:*" │
│ │ │ │ kms:PutKeyPolicy │ │ } │
│ │ │ │ kms:ReEncrypt* │ │ │
├───┼──────────────────────────────┼────────┼──────────────────────────────┼──────────────────────────────┼─────────────────────────────────┤
│ + │ ${mwaa-service-role.Arn} │ Allow │ sts:AssumeRole │ Service:airflow-env.amazonaw │ │
│ │ │ │ │ s.com │ │
│ │ │ │ │ Service:airflow.amazonaws.co │ │
│ │ │ │ │ m │ │
│ │ │ │ │ Service:ecs-tasks.amazonaws. │ │
│ │ │ │ │ com │ │
├───┼──────────────────────────────┼────────┼──────────────────────────────┼──────────────────────────────┼─────────────────────────────────┤
│ + │ * │ Allow │ logs:DescribeLogGroups │ AWS:${mwaa-service-role} │ │
│ + │ * │ Allow │ ecs:DescribeTaskDefinition │ AWS:${mwaa-service-role} │ │
│ │ │ │ ecs:DescribeTasks │ │ │
│ │ │ │ ecs:ListTasks │ │ │
│ │ │ │ ecs:RegisterTaskDefinition │ │ │
│ │ │ │ ecs:RunTask │ │ │
│ + │ * │ Allow │ iam:PassRole │ AWS:${mwaa-service-role} │ "StringLike": { │
│ │ │ │ │ │ "iam:PassedToService": "ecs-t │
│ │ │ │ │ │ asks.amazonaws.com" │
│ │ │ │ │ │ } │
│ + │ * │ Allow │ kms:Decrypt │ AWS:${mwaa-service-role} │ "StringEquals": { │
│ │ │ │ kms:DescribeKey │ │ "kms:ViaService": [ │
│ │ │ │ kms:Encrypt │ │ "sqs.eu-central-1.amazonaws │
│ │ │ │ kms:GenerateDataKey* │ │ .com", │
│ │ │ │ kms:PutKeyPolicy │ │ "s3.eu-central-1.amazonaws. │
│ │ │ │ │ │ com" │
│ │ │ │ │ │ ] │
│ │ │ │ │ │ } │
├───┼──────────────────────────────┼────────┼──────────────────────────────┼──────────────────────────────┼─────────────────────────────────┤
│ + │ arn:${AWS::Partition}:s3:::c │ Allow │ s3:GetBucket* │ AWS:${Custom::CDKBucketDeplo │ │
│ │ dk-hnb659fds-assets-70453306 │ │ s3:GetObject* │ yment8693BB64968944B69AAFB0C │ │
│ │ 6374-eu-central-1 │ │ s3:List* │ C9EB8756C/ServiceRole} │ │
│ │ arn:${AWS::Partition}:s3:::c │ │ │ │ │
│ │ dk-hnb659fds-assets-70453306 │ │ │ │ │
│ │ 6374-eu-central-1/* │ │ │ │ │
├───┼──────────────────────────────┼────────┼──────────────────────────────┼──────────────────────────────┼─────────────────────────────────┤
│ + │ arn:aws:airflow:eu-central-1 │ Allow │ airflow:PublishMetrics │ AWS:${mwaa-service-role} │ │
│ │ :704533066374:environment/mw │ │ │ │ │
│ │ aa-hybrid-demo │ │ │ │ │
├───┼──────────────────────────────┼────────┼──────────────────────────────┼──────────────────────────────┼─────────────────────────────────┤
│ + │ arn:aws:logs:eu-central-1:70 │ Allow │ logs:CreateLogGroup │ AWS:${mwaa-service-role} │ │
│ │ 4533066374:log-group:airflow │ │ logs:CreateLogStream │ │ │
│ │ -mwaa-hybrid-demo-* │ │ logs:DescribeLogGroups │ │ │
│ │ │ │ logs:GetLogEvents │ │ │
│ │ │ │ logs:GetLogGroupFields │ │ │
│ │ │ │ logs:GetLogRecord │ │ │
│ │ │ │ logs:GetQueryResults │ │ │
│ │ │ │ logs:PutLogEvents │ │ │
├───┼──────────────────────────────┼────────┼──────────────────────────────┼──────────────────────────────┼─────────────────────────────────┤
│ + │ arn:aws:sqs:eu-central-1:*:a │ Allow │ sqs:ChangeMessageVisibility │ AWS:${mwaa-service-role} │ │
│ │ irflow-celery-* │ │ sqs:DeleteMessage │ │ │
│ │ │ │ sqs:GetQueueAttributes │ │ │
│ │ │ │ sqs:GetQueueUrl │ │ │
│ │ │ │ sqs:ReceiveMessage │ │ │
│ │ │ │ sqs:SendMessage │ │ │
└───┴──────────────────────────────┴────────┴──────────────────────────────┴──────────────────────────────┴─────────────────────────────────┘
IAM Policy Changes
┌───┬───────────────────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────┐
│ │ Resource │ Managed Policy ARN │
├───┼───────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────┤
│ + │ ${Custom::CDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C/Ser │ arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasic │
│ │ viceRole} │ ExecutionRole │
└───┴───────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────┘
Security Group Changes
┌───┬────────────────────┬─────┬────────────┬────────────────────┐
│ │ Group │ Dir │ Protocol │ Peer │
├───┼────────────────────┼─────┼────────────┼────────────────────┤
│ + │ ${mwaa-sg.GroupId} │ In │ Everything │ ${mwaa-sg.GroupId} │
│ + │ ${mwaa-sg.GroupId} │ Out │ Everything │ Everyone (IPv4) │
└───┴────────────────────┴─────┴────────────┴────────────────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)
Do you wish to deploy these changes (y/n)? y
mwaa-hybrid-environment: deploying...
[0%] start: Publishing e9882ab123687399f934da0d45effe675ecc8ce13b40cb946f3e1d6141fe8d68:704533066374-eu-central-1
[0%] start: Publishing 983c442a2fe823a8b4ebb18d241a5150ae15103dacbf3f038c7c6343e565aa4c:704533066374-eu-central-1
[0%] start: Publishing 91ab667f7c88c3b87cf958b7ef4158ef85fb9ba8bd198e5e0e901bb7f904d560:704533066374-eu-central-1
[0%] start: Publishing f2a926ee3d8ca4bd02b0cf073eb2bbb682e94c021925bf971a9730045ef4fb02:704533066374-eu-central-1
[25%] success: Published 983c442a2fe823a8b4ebb18d241a5150ae15103dacbf3f038c7c6343e565aa4c:704533066374-eu-central-1
[50%] success: Published 91ab667f7c88c3b87cf958b7ef4158ef85fb9ba8bd198e5e0e901bb7f904d560:704533066374-eu-central-1
[75%] success: Published f2a926ee3d8ca4bd02b0cf073eb2bbb682e94c021925bf971a9730045ef4fb02:704533066374-eu-central-1
[100%] success: Published e9882ab123687399f934da0d45effe675ecc8ce13b40cb946f3e1d6141fe8d68:704533066374-eu-central-1
mwaa-hybrid-environment: creating CloudFormation changeset...
✅ mwaa-hybrid-environment
✨ Deployment time: 1412.35s
Outputs:
mwaa-hybrid-environment.MWAASecurityGroup = sg-0ea83e01caded2bb3
Stack ARN:
arn:aws:cloudformation:eu-central-1:704533066374:stack/mwaa-hybrid-environment/450337a0-f088-11ec-a169-06ba63bfdfb2
✨ Total time: 1424.72s
Conclusion
It was pretty straight forward in the end to update my stacks, but it did take me longer thanks to this being the first time and also encountering the SSM issue along the way. As I have updated subsequent stacks, it has definitely become much easier and quicker, so I recommend that you begin on a simple stack to gain confidence and then apply that to your other stacks.
If you encounter any issues running my stacks on CDKv2, please raise an issue on the GitHub rep and I will take a look and fix.
Top comments (3)
There is another special case if you are using API Gateway v2 (HTTP API).
I know this might not be relevant to your stacks in particular, but it was the main thing that made trying to update mine difficult until I found it.
They require the following Python packages:
aws-cdk.aws-apigatewayv2-alpha
aws-cdk.aws-apigatewayv2-integrations-alpha
Yeah, I suspect as I start to update other stacks that use more esoteric construct libraries I will bump into these. Thanks for sharing.
Love CDK, but I wish they did a better job with the documentation site migration.
Even though CDK 2 has been out for months now, google/ddg search results pretty much always take me to CDK 1 site and clicking on the link to new version takes me to the home page (for some of the sub-projects I think it retains the context but for all others I have to find what I needed from the huge sidebar menu afresh).