Rust + Lambda using CDK & Github Actions (Part 2)

#rust #aws #lambda #serverless

In continuing where things left off in part 1, here we will be using GitHub Actions to automatically deploy a CDK stack to provision all the cloud resources.

There are a few prerequisites that going to be assumed here:

You have an AWS account and know you way around the console
You have a Github Repository
The CDK CLI is installed (npm i -g cdk)

CDK Stack

To have things work as intended, an IAM User will be required that has an Access Key Id and Secret Access Key that can be passed into the deployment pipeline in Github. The permissions that this role has will depend on the resources that is required to provision, but for our purposes setting the following AWS-managed policies should be sufficient:

IAMFullAccess
AmazonS3FullAccess
AmazonDynamoDBFullAccess
AmazonAPIGatewayAdministrator
AmazonSSMFullAccess
AWSCloudFormationFullAccess
AWSLambda_FullAccess

Note: Caution should be used when creating IAM Users with too much access, but as a general rule User's should be creating with the minimum needed permissions. The User defined here is overly simplified for demo purposes.

With a User created, you will need to create an Access Key and download the credentials excel file. These will be the secrets that need to be injected into the Github repo. To add them, go to your repository settings on Github, then from the Security section create two secrets named AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with their respective values set:

Next we can create everything required for the CDK stack. Create a new directory in the project root folder called cdk. In this folder a new cdk project should be initialized via cdk init app --language typescript, which will generate all the required files needed to deploy a stack. Some important ones to note:

/cdk/package.json: Where stack dependencies are defined
/cdk/lib/cdk-stack.ts: Where the stack definition will go
/cdk/bin/cdk.ts: File that instantiates an instance of the stack defined in cdk-stack.ts and sets tags, environment config, etc
/cdk/cdk.json: Invokes the code in cdk.ts when cli commands are ran to initiate deployment

Some other things will generate such as a tsconfig.json file and some Jest tests, but those can be ignored for now.

First we can define what dependencies will be needed by updating the package.json file:

{
  "name": "cdk",
  "version": "0.1.0",
  "bin": {
    "cdk": "bin/cdk.js"
  },
  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "test": "jest",
    "cdk": "cdk"
  },
  "devDependencies": {
    "@aws-cdk/core": "1.143.0",
    "@aws-cdk/aws-apigatewayv2": "1.143.0",
    "@aws-cdk/aws-apigatewayv2-integrations": "1.143.0",
    "@aws-cdk/aws-dynamodb": "1.143.0",
    "@aws-cdk/aws-iam": "1.143.0",
    "@aws-cdk/aws-lambda": "1.143.0",
    "@types/jest": "^26.0.10",
    "@types/node": "17.0.15",
    "jest": "^26.4.2",
    "ts-jest": "^26.2.0",
    "aws-cdk": "2.10.0",
    "ts-node": "10.4.0",
    "typescript": "4.5.5"
  },
  "dependencies": {
    "aws-cdk-lib": "2.15.0",
    "constructs": "^10.0.0",
    "source-map-support": "^0.5.16"
  }
}

@aws-cdk/core provides basic Stack definitions, while all the other @aws-cdk/ prefixed dependencies provide constructs to provision resources related to that service. Everything else is boilerplate dependencies required to compile the project, though with updated versions.

There is a section in cdk.json that can uncommented to set the account and region that will be used for deployment. The name of stack class will be generated based upon the project name but can renamed if desired (renamed to CdkStack here). Additionally, we are going to use cdk from @aws-cdk/core instead of aws-cdk-lib:

import cdk = require('@aws-cdk/core');

new CdkStack(app, 'CdkStack', {
  env: { account: accountNumber, region: 'us-east-1' },
});

This will create a new CDK app using a CdkStack class defined in cdk-stack.ts:

import cdk = require('@aws-cdk/core');

export class CdkStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const appName = "rust-lambda"

As you can see, we are just extending the provided base stack that AWS provides. The appName variable will be referenced while naming all of our resources. to begin with, we know we need a DynamoDB table to save records to so let's start with that.

import { AttributeType, BillingMode, Table } from "@aws-cdk/aws-dynamodb"

export class CdkStack extends cdk.Stack {
--snip--
const dynamoTable = new Table(this, `DynamoTable`, {
      tableName: `${appName}-table`,
      partitionKey: { name: "userId", type: AttributeType.STRING },
      sortKey: { name: "modelTypeAndId", type: AttributeType.STRING },
      billingMode: BillingMode.PAY_PER_REQUEST
    })
--snip--
}

Lambda's will have an attached IAM Role that is used to control what services the Lambda is able to interact with. Creating a new role is just as simple:

--snip--
const lambdaRole = new Role(this, `LambdaRole`, {
            roleName: `${appName}-role`,
            assumedBy: new ServicePrincipal("lambda.amazonaws.com"),
            managedPolicies: [
ManagedPolicy.fromAwsManagedPolicyName("service-role/AWSLambdaVPCAccessExecutionRole")
            ]
})

dynamoTable.grantReadWriteData(lambdaRole)
        lambdaRole.addToPolicy(new PolicyStatement({
            effect: Effect.ALLOW,
            actions: ["dynamodb:Query", "dynamodb:Scan", "dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:DeleteItem"],
            resources: [
                `arn:aws:dynamodb:${Aws.REGION}:${Aws.ACCOUNT_ID}:table/${dynamoTable.tableName}/index/*`
            ]
        }))
--snip--

Here, a Role is being provisioned and we are granting it permission to use a specific set of DynamoDB actions. If we omitted dynamodb:PutItem for instance, the lambda would not be able to perform that operation. This gives very fine grain control on what a particular Lambda should be able to do.

Next, we can provision the actual Lambda, assign it the Role that we just created, and create what is called a Lambda Integration:

--snip
const lambdaFunction = new Function(this, `LambdaFunction`, {
      functionName: `${appName}-lambda`,
      runtime: Runtime.PROVIDED_AL2,
      role: lambdaRole,
      code: Code.fromAsset("../target/x86_64-unknown-linux-musl/release/lambda.zip"),
      handler: "main",
      environment: {
        RUST_BACKTRACE: '1'
      }
    })
    lambdaFunction.grantInvoke(new ServicePrincipal("apigateway.amazonaws.com"))

    const lambdaIntegration = new HttpLambdaIntegration("HttpLambdaIntegration", lambdaFunction)
--snip--

Since Rust does not currently have an official Rust runtime, we need to deploy it as a custom runtime using Amazon Linux 2. The code asset that is required needs to point to zip file that has the executable in a file called Bootstrap (read more on custom runtimes here). We will define the actual build steps to support this requirement last.

Since we want to invoke the Lambda via an http request, we need to state that the API Gateway service can invoke it. We also create a HttpLambdaIntegration that will be passed to the individual api routes. This pattern enables each different route to invoke different Lambdas.

There are a couple different ways to set up an APIGateway, but here we can use an HttpApi. Refer to this page to see the differences between ApiGateway's HttpApi and RestApi.

const api = new HttpApi(this, `RestAPIGateway`, {
    apiName: "rust-lambda-api",
    corsPreflight: {
        allowHeaders: ['Authorization', 'Access-Control-Allow-Origin','Access-Control-Allow-Headers','Content-Type'],
        allowMethods: [
            CorsHttpMethod.ANY
        ],
        allowOrigins: ['*'],
    },
})

api.addRoutes({
    path: "/users",
    methods: [HttpMethod.GET, HttpMethod.POST],
    integration: lambdaIntegration
})

api.addRoutes({
    path: "/users/{proxy+}",
    methods: [HttpMethod.GET, HttpMethod.POST],
    integration: lambdaIntegration
})

For this example we will just need a /users endpoint that will accept only POST and GET requests. If desired, you can allow all methods will HttpMethod.ANY. Having the additional /users/{proxy+} endpoint will enable any endpoint that extends from /users/. Now we have a complete stack that we're ready to deploy!

Note: If your IDE is complaining about about argument of type 'this' is not assignable to parameter of type 'Construct', you can add an //@ts-ignore to the line right above the resource provisioning.

GitHub Actions

In order to automatically deploy the stack to your AWS account, let's set up an Actions Workflow. Github will automatically scan your project for a .github/workflows directory, and execute files it finds.

A number of things need to be accounted for in order for the build and deployment to be successful:

AWS credentials are injected
Rust is installed, set for a Linux target like x86_64-unknown-linux-musl
Executable is named bootstrap and zipped into a file

name: Deploy CDK Stack

on: 
  push:
    branchs: [main]


jobs:
  aws_cdk:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        target:
          - x86_64-unknown-linux-musl
    steps:
      - name: Checkout Code
        uses: actions/checkout@v2

      - name: Set AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{secrets.AWS_ACCESS_KEY_ID}}
          aws-secret-access-key: ${{secrets.AWS_SECRET_ACCESS_KEY}}
          aws-region: us-east-1

      - name: Install Node
        uses: actions/setup-node@v1
        with:
          node-version: 14

      - name: Install Rust
        uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
          target: ${{ matrix.target }}
          override: true

      - name: Install Stack Dependencies
        run: 'sudo npm ci'
        working-directory: ./cdk

      - name: Install NPM
        run: 'sudo apt update -y && sudo apt install nodejs npm -y'

      - name: Install CDK CLI
        run: 'sudo npm install -g aws-cdk'

      - name: Run Build
        uses: actions-rs/cargo@v1
        with:
          use-cross: true
          command: build
          args: --release --all-features --target=${{ matrix.target }}

      - name: Rename binary to bootstrap
        run: 'mv ./rust_lambda ./bootstrap'
        working-directory: ./target/x86_64-unknown-linux-musl/release

      - name: Zip Code for Deployment
        run: 'sudo zip -j lambda.zip bootstrap'
        working-directory: ./target/x86_64-unknown-linux-musl/release

      - name: CDK Synth
        run: 'cdk synth'
        working-directory: ./cdk

      - name: Run CDK Bootstrap
        run: 'cdk bootstrap aws://415023725722/us-east-1'
        working-directory: ./cdk

      - name: CDK Deploy
        run: 'cdk deploy --require-approval never'
        working-directory: ./cdk

As configured, this Action will execute anytime something to the main branch gets pushed. Let's give that a go now! Rust takes a little while to compile, especially on the Github VMs, so you might want to grab a cup of coffee 🙂

Invoking

With everything deployed, we are finally ready to try to invoke our lambda! To get the endpoint that is generated, go to your AWS console and navigate to your Lambda's page, then go to the Configuration tab, then Triggers. You should see an API Gateway listed with a given URL: