Running AWS CDK from a Lambda function

This is a step by step guide to running AWS CDK inside an AWS Lambda Function using Lambda layers.

We will create an example CDK code, which will create an S3 bucket (can be any AWS resource, this is just an example) and then create another CDK code to deploy a Lambda function which will run the first one. Do you remember the Russian toy called "matryoshka"?

Why do we need this? There are two main reasons:

• For fun, to play with Lambda layers and AWS CDK.
• It can be sometimes useful and we actually use it in DevOpsBox.

All the source code is available here: https://github.com/devopsbox-io/example-cdk-from-lambda

Prerequisites

You will need several tools:

• AWS CDK (tested with 1.71.0)
• Java JDK (tested with OpenJDK 11.0.9)
• NodeJS (required by AWS CDK, tested with v12.18.3)
• Docker (tested with 18.09.5)
• Maven (tested with 3.6.3)
• AWS CLI (only for testing, tested with 1.18.57)

AWS account with proper credentials is also required. The code will probably work with other versions too.

The solution

First of all, we will create two CDK projects, one inside another (matryoshka):

mkdir example-cdk-from-lambda
cd example-cdk-from-lambda
cdk init app --language=java

mkdir run-cdk-lambda
cd run-cdk-lambda
cdk init app --language=java

Yes, we are using Java here... You can achieve the same results in any other language supported by AWS CDK.

I will not change any package name just to allow you to easily do git diff HEAD~1 and to see all my changes, however, I have removed all the tests and test dependencies - they are not important here.

The smallest matryoshka

Here we just create an S3 bucket but it could be any CDK code.

We need the CDK S3 dependency run-cdk-lambda/pom.xml

<dependency>
    <groupId>software.amazon.awscdk</groupId>
    <artifactId>s3</artifactId>
    <version>${cdk.version}</version>
</dependency>

Next, we have to write some CDK code run-cdk-lambda/src/main/java/com/myorg/RunCdkLambdaStack.java:

new Bucket(this, "created-by-cdk-from-lambda", BucketProps.builder()
    .removalPolicy(RemovalPolicy.DESTROY)
    .build());

The medium matryoshka

If you want to run the AWS CDK, you will have to execute the cdk binary because this is how it works. That is why we need to create a Lambda handler, which will be just a wrapper and will run the cdk.

We have to add the aws-lambda-java-core dependency to our run-cdk-lambda/pom.xml:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-lambda-java-core</artifactId>
    <version>1.2.1</version>
</dependency>

Then, create the CdkWrapper class run-cdk-lambda/src/main/java/com/myorg/CdkWrapper.java:

public class CdkWrapper implements RequestHandler<Map<String, String>, String> {

    public static final String CDK_OUT_DIR = "/tmp/cdk.out";
    private static final String CDK_COMMAND = "java -cp . ";

    @Override
    public String handleRequest(Map<String, String> input, Context context) {

        runCdk(RunCdkLambdaApp.class);

        return "";
    }

    public static void runCdk(Class<?> cdkClass) {
        ProcessBuilder processBuilder = new ProcessBuilder(
                "cdk",
                "deploy",
                "--verbose",
                "--output", CDK_OUT_DIR,
                "--app", CDK_COMMAND + cdkClass.getName(),
                "--require-approval", "\"never\""
        );
        processBuilder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
        processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT);

        Process process;
        try {
            process = processBuilder.start();
        } catch (IOException e) {
            throw new RuntimeException("Cannot start cdk process!", e);
        }

        try {
            process.waitFor();
        } catch (InterruptedException e) {
            throw new RuntimeException("Cdk process interrupted!", e);
        }

        int exitValue = process.exitValue();

        if (exitValue != 0) {
            throw new RuntimeException("Exception while executing CDK!");
        }
    }
}

We are doing this in Java using the ProcessBuilder class because our CDK code is also written in Java and we will use the Java 11 runtime in Lambda.

We also have to pack our Java code into an "uber" jar (jar with all the dependencies). We use maven-shade-plugin to do that run-cdk-lambda/pom.xml:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.2.4</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
        </execution>
    </executions>
</plugin>

The largest matryoshka

The Java 11 Lambda runtime does not have NodeJS and AWS CDK binaries. We will use Lambda layers to put them inside. We will copy binaries from a docker image created from docker/cdk/Dockerfile:

FROM node:lts

RUN apt-get update && \
    apt-get install -y zip && \
    rm -rf /var/lib/apt/lists/*

ENV AWS_CDK_VERSION=1.71.0
RUN mkdir -p /nodejs && \
    npm config set prefix /nodejs/bin && \
    npm install -g aws-cdk@${AWS_CDK_VERSION}

RUN cd /nodejs/bin && \
    zip -r --symlinks /opt/aws-cdk.zip *
RUN cd /usr/local && \
    zip -r /opt/node.zip bin/node

We need also the code to create the "uber" jar, the docker image with NodeJS and CDK binaries, and copy files from it (actually creating a temporary docker container). This will be a bash script scripts/prepare-bin:

#!/bin/bash
set -e

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
PROJECT_DIR=$(realpath "${SCRIPT_DIR}/..")
DOCKER_IMAGE_TAG=aws-cdk-bin:latest
TMP_BIN_DIR="${PROJECT_DIR}/tmp"

mkdir -p ${TMP_BIN_DIR}

mvn package -f "${PROJECT_DIR}/run-cdk-lambda/pom.xml"
cp ${PROJECT_DIR}/run-cdk-lambda/target/run-cdk-lambda-*.jar ${TMP_BIN_DIR}/run-cdk-lambda.jar

docker build -t ${DOCKER_IMAGE_TAG} ${PROJECT_DIR}/docker/cdk

rm -f ${TMP_BIN_DIR}/docker-cid
docker create --cidfile ${TMP_BIN_DIR}/docker-cid ${DOCKER_IMAGE_TAG}
CID=$(cat ${TMP_BIN_DIR}/docker-cid)

trap "docker rm ${CID}" EXIT

docker cp ${CID}:/opt/node.zip ${TMP_BIN_DIR}/node.zip
docker cp ${CID}:/opt/aws-cdk.zip ${TMP_BIN_DIR}/aws-cdk.zip

Now we have to create the outer CDK code to create the lambda with layers. We need the CDK Lambda dependency pom.xml:

<dependency>
    <groupId>software.amazon.awscdk</groupId>
    <artifactId>lambda</artifactId>
    <version>${cdk.version}</version>
</dependency>

and the CDK code src/main/java/com/myorg/ExampleCdkFromLambdaStack.java:

public class ExampleCdkFromLambdaStack extends Stack {
    public ExampleCdkFromLambdaStack(final Construct scope, final String id) {
        this(scope, id, null);
    }

    public ExampleCdkFromLambdaStack(final Construct scope, final String id, final StackProps props) {
        super(scope, id, props);

        createRunCdkLambda();
    }

    private void createRunCdkLambda() {
        String tmpBinDir = getTmpBinDir();

        PolicyStatement cloudformationPolicy = new PolicyStatement(PolicyStatementProps.builder()
                .resources(Arrays.asList(
                        "*"
                ))
                .actions(Arrays.asList(
                        "cloudformation:DescribeStacks",
                        "cloudformation:CreateChangeSet",
                        "cloudformation:DescribeChangeSet",
                        "cloudformation:GetTemplate",
                        "cloudformation:GetTemplateSummary",
                        "cloudformation:DescribeStackEvents",
                        "cloudformation:ExecuteChangeSet",
                        "cloudformation:DeleteChangeSet"
                ))
                .build());

        // we can restrict this policy to certain buckets only
        PolicyStatement s3Policy = new PolicyStatement(PolicyStatementProps.builder()
                .resources(Arrays.asList(
                        "*"
                ))
                .actions(Arrays.asList(
                        "s3:*"
                ))
                .build());

        LayerVersion nodeLayer = LayerVersion.Builder.create(this, "node-layer")
                .description("Layer containing node binary")
                .code(
                        Code.fromAsset(tmpBinDir + "/node.zip")
                )
                .build();

        LayerVersion cdkLayer = LayerVersion.Builder.create(this, "aws-cdk-layer")
                .description("Layer containing AWS CDK")
                .code(
                        Code.fromAsset(tmpBinDir + "/aws-cdk.zip")
                )
                .build();

        Function lambda = new Function(this, "RunCdk", FunctionProps.builder()
                .runtime(Runtime.JAVA_11)
                .handler("com.myorg.CdkWrapper")
                .code(Code.fromAsset(tmpBinDir + "/run-cdk-lambda.jar"))
                .layers(Arrays.asList(
                        nodeLayer,
                        cdkLayer
                ))
                .timeout(Duration.seconds(300))
                .memorySize(512)
                .initialPolicy(Arrays.asList(
                        cloudformationPolicy,
                        s3Policy
                ))
                .functionName("run-cdk")
                .build());
    }

    private String getTmpBinDir() {
        Path tmpPath = Paths.get("tmp");
        return tmpPath.toAbsolutePath().toString();
    }
}

This is our third, outermost "matryoshka" layer. Here we are creating the Lambda policies (CloudFormation execution and S3 full access), layers (NodeJS, AWS CDK binaries), and the actual Lambda resource with the com.myorg.CdkWrapper handler and the code from run-cdk-lambda.jar.

We will also change the cdk.json file to run the prepare-bin script before the CDK execution cdk.json:

{
  "app": "./scripts/prepare-bin && mvn -e -q compile exec:java",
  "context": {
    "@aws-cdk/core:enableStackNameDuplicates": "true",
    "aws-cdk:enableDiffNoFail": "true",
    "@aws-cdk/core:stackRelativeExports": "true"
  }
}

Deploying

You have to set your AWS credentials first. You can do this in the ~/.aws/credentials file or using environment variables. Then, you have to bootstrap the CDK:

cdk bootstrap

Now you can create the Lambda function:

cdk deploy --require-approval never

Testing

Check your S3 buckets using:

aws s3 ls

This should not return any bucket with the "runcdklambdastack-createdbycdkfromlambda" prefix.

Run the Lambda function:

aws lambda invoke --function-name run-cdk tmp/lambda-out

It will run the AWS CDK and create the bucket. The first run can take about 1 minute to complete. Check again your S3 buckets then:

aws s3 ls

It should return a bucket created by CDK run inside a Lambda function (with the "runcdklambdastack-createdbycdkfromlambda" prefix).

Conclusion

Running AWS CDK inside a Lambda function is not an easy task. However, sometimes it is useful and I hope that this article will help you to do it, or maybe will only show you how to use lambda Layers to execute almost any executable binary inside an AWS Lamda. We do use it in DevOpsBox to create appropriate IAM roles and policies based on the roles read from Keycloak.

For more details about the DevOpsBox platform please visit https://www.devopsbox.io/

Comments

[desawsume]

This is a AWESOME post. I am running into an issue when upgrading to CDKv2, the size is over the limit, working through a way to see if anything can help. Now is a DONT KNOW

[Maciej Raszplewicz]

Thanks @desawsume!

I've updated the repository some time ago (thanks to github.com/YotillaJonas for his PR). I've tested that and it works but didn't try anything bigger.

I would try Lambda container images now. It didn't exist when I was writing the article.

[desawsume]

yes, another way to be worry free is using ECR with LAMBDA, 10GB, more than enough for our use case, i might also create a post to share. currently working through the docker stuff

[Thomas]

I'm on the same ship - did you end up writing up ideas?